search

icon-project / icon-bridge

The centralized bridge of ICON
Apache License 2.0
21 stars 15 forks source link

fix(bmr): submit transaction to pool failed #760

Closed manishbista28 closed 1 year ago

manishbista28 commented 1 year ago

Overview

Relay submitted the following transaction to snow chain but did not get a receipt even after waiting for it until the retry counter reached the max value (30) which is 3 minutes. As relay then tried to resubmit the transaction, the error send tx error=submit transaction to pool failed: Pool(TemporarilyBanned) was thrown. Relay could not get past the error and became stuck.

Submitted Transaction Log
D|15:03:03.294471|0xb6|snow|i2s|tx_sender.go:270 handleRelayMessage: tx sent tx={"type":"0x0","nonce":"0x98","gasPrice":"0x8","maxPriorityFeePerGas":null,"maxFeePerGas":null,"gas":"0x16e3600","value":"0x0","input":"0x21b1e9bb000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000396274703a2f2f3078312e69636f6e2f63783233613931656533646432393034383661393131336136613432343239383235643831336465353300000000000000000000000000000000000000000000000000000000000000000000000000015af90157f90154b90151f9014e01b90145f90142f9013fb83b6274703a2f2f30783232382e736e6f772f3078663830384662623535434644446133374430353142393632623739313861384433393831666544358200afb8fdf8fbb8396274703a2f2f3078312e69636f6e2f637832336139316565336464323930343836613931313361366134323432393832356438313364653533b83b6274703a2f2f30783232382e736e6f772f3078663830384662623535434644446133374430353142393632623739313861384433393831666544358362747382009cb87af87800b875f873aa687864663263633163663839303463666233363836646563346239666436306436396539353535633765aa307831303930663436394631353835376146314261376434423136306438366241333961363638323866dcdb906274702d3078312e69636f6e2d4943588905301ab2f5ace2000084036af969000000000000","v":"0x474","r":"0x2d4fbf34097bc9a459f4f371193b33ec88fae5a7bb505784f5ae43618bd11608","s":"0x60f8154cae9a9bd3353ff68df6faf763ce4e1a6e2764570948b61fc8f92d43ac","to":"0xf808fbb55cfdda37d051b962b7918a8d3981fed5","hash":"0xed20267b28ae16e6122a412c589b1755eb4cb5d3f0b43bf6e8871df15db6e2dc"}

When the exact transaction is retried on substrate chain, the latter transaction is blocked and an error is returned. The error Pool(TemporarilyBanned) is especially problematic because any further transaction from the source address will be blocked from submission to the blockchain for tens of minutes. With the relay account blocked plus the relay stuck trying to submit the transaction (that has been rejected), relay now has two reasons to become stuck.

Relay's logic of resubmitting a transaction seems to be at odds with substrate chain's behavior when it encounters the re-submittted transaction.

Error Generated after resubmitting transaction
D|15:06:10.607476|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)

We'd like to know details surrounding the initial transaction (was it submitted to blockchain, if so was it processed or were there problems processing it and such) Identifying problem with relay's initial transaction will help us handle such cases appropriately. Else we could be fixing the wrong parameter and such issues can repeat. Secondly, we should consider if relay's retry logic needs to be adjusted to ensure substrate chain interactions can be handled.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected Behavior

Relay should not get stuck.

Additional Context

Add any other context about the problem here.

manishbista28 commented 1 year ago

Links to similar problems on other projects. https://stackoverflow.com/questions/57270007/in-substrate-what-does-code-1012-transaction-is-temporarily-banned-mean#:~:text=A%20transaction%20is%20only%20temporarily,4%2C000%20transactions%20on%20the%20blacklist

https://docs.swapdex.network/troubleshooting/01-substrate-error-codes/#common-errors

https://github.com/PureStake/moonbeam/pull/175

manishbista28 commented 1 year ago

Following is the first transaction that got processes after relay was restarted.

D|16:58:54.368487|0xb6|relay|i2s|relay.go:113 relaySignal
D|16:58:54.371470|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|16:58:54.377333|0xb6|snow|i2s|tx_sender.go:270 handleRelayMessage: tx sent tx={"type":"0x0","nonce":"0x98","gasPrice":"0x10","maxPriorityFeePerGas":null,"maxFeePerGas":null,"gas":"0x16e3600","value":"0x0","input":"0x21b1e9bb000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000396274703a2f2f3078312e69636f6e2f6378323361393165653364643239303438366139313133613661343234323938323564383133646535330000000000000000000000000000000000000000000000000000000000000000000000000002aef902abf902a8b90151f9014e01b90145f90142f9013fb83b6274703a2f2f30783232382e736e6f772f3078663830384662623535434644446133374430353142393632623739313861384433393831666544358200afb8fdf8fbb8396274703a2f2f3078312e69636f6e2f637832336139316565336464323930343836613931313361366134323432393832356438313364653533b83b6274703a2f2f30783232382e736e6f772f3078663830384662623535434644446133374430353142393632623739313861384433393831666544358362747382009cb87af87800b875f873aa687864663263633163663839303463666233363836646563346239666436306436396539353535633765aa307831303930663436394631353835376146314261376434423136306438366241333961363638323866dcdb906274702d3078312e69636f6e2d4943588905301ab2f5ace2000084036af969b90151f9014e01b90145f90142f9013fb83b6274703a2f2f30783232382e736e6f772f3078663830384662623535434644446133374430353142393632623739313861384433393831666544358200b0b8fdf8fbb8396274703a2f2f3078312e69636f6e2f637832336139316565336464323930343836613931313361366134323432393832356438313364653533b83b6274703a2f2f30783232382e736e6f772f3078663830384662623535434644446133374430353142393632623739313861384433393831666544358362747382009db87af87800b875f873aa687835636137626630323034666535613037346565396663386565663566313337396634626132636135aa307837453038333143623745626334313135663234353034383234343444303562306141646233453335dcdb906274702d3078312e69636f6e2d4943588907e5fe620c5e6a000084036af9f7000000000000000000000000000000000000","v":"0x474","r":"0x528e40dffe1f62b3b15949cf8e8d541d52f33dbbcda33f470f4fa2451d300493","s":"0x1ddc55fb6681cdb4d8867a63540579b136526c71f1f694d6f18fcc254a135840","to":"0xf808fbb55cfdda37d051b962b7918a8d3981fed5","hash":"0xdfcfc0c89a593ee5d8868407590ec37eed9e1b65719d0aa959b1e88eefd86294"}
manishbista28 commented 1 year ago

Following is the error log from the point the initial transaction was intended to be submitted to the point it was resubmitted.

D|15:02:55.213052|----|-|icon|receiver.go:637 block notification height=57342311
D|15:02:57.625311|----|-|icon|receiver.go:637 block notification height=57342312
D|15:02:58.282160|0xb6|relay|i2s|relay.go:113 relaySignal
D|15:02:59.213420|----|-|icon|receiver.go:637 block notification height=57342313
D|15:02:59.234401|0xb6|relay|i2s|relay.go:162 srcMsg added seq=[175 175]
D|15:03:00.134257|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:01.637194|----|-|icon|receiver.go:637 block notification height=57342314
D|15:03:03.116739|----|-|icon|receiver.go:637 block notification height=57342315
D|15:03:03.289570|0xb6|relay|i2s|relay.go:113 relaySignal
D|15:03:03.292519|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:03:03.294471|0xb6|snow|i2s|tx_sender.go:270 handleRelayMessage: tx sent tx={"type":"0x0","nonce":"0x98","gasPrice":"0x8","maxPriorityFeePerGas":null,"maxFeePerGas":null,"gas":"0x16e3600","value":"0x0","input":"0x21b1e9bb000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000396274703a2f2f3078312e69636f6e2f63783233613931656533646432393034383661393131336136613432343239383235643831336465353300000000000000000000000000000000000000000000000000000000000000000000000000015af90157f90154b90151f9014e01b90145f90142f9013fb83b6274703a2f2f30783232382e736e6f772f3078663830384662623535434644446133374430353142393632623739313861384433393831666544358200afb8fdf8fbb8396274703a2f2f3078312e69636f6e2f637832336139316565336464323930343836613931313361366134323432393832356438313364653533b83b6274703a2f2f30783232382e736e6f772f3078663830384662623535434644446133374430353142393632623739313861384433393831666544358362747382009cb87af87800b875f873aa687864663263633163663839303463666233363836646563346239666436306436396539353535633765aa307831303930663436394631353835376146314261376434423136306438366241333961363638323866dcdb906274702d3078312e69636f6e2d4943588905301ab2f5ace2000084036af969000000000000","v":"0x474","r":"0x2d4fbf34097bc9a459f4f371193b33ec88fae5a7bb505784f5ae43618bd11608","s":"0x60f8154cae9a9bd3353ff68df6faf763ce4e1a6e2764570948b61fc8f92d43ac","to":"0xf808fbb55cfdda37d051b962b7918a8d3981fed5","hash":"0xed20267b28ae16e6122a412c589b1755eb4cb5d3f0b43bf6e8871df15db6e2dc"}
D|15:03:04.437403|hx20|snow|s2i|rx_receiver.go:464 block notification height=251992
D|15:03:05.021607|----|-|icon|receiver.go:637 block notification height=57342316
D|15:03:05.147450|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:07.011523|----|-|icon|receiver.go:637 block notification height=57342317
D|15:03:09.026222|----|-|icon|receiver.go:637 block notification height=57342318
D|15:03:09.417205|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=1
D|15:03:10.155606|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:11.479758|----|-|icon|receiver.go:637 block notification height=57342319
D|15:03:13.024913|----|-|icon|receiver.go:637 block notification height=57342320
D|15:03:15.157364|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:15.228254|----|-|icon|receiver.go:637 block notification height=57342321
D|15:03:15.585993|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=2
D|15:03:15.890785|hx20|snow|s2i|rx_receiver.go:272 poll height latest=251994 next=251994
D|15:03:17.566467|----|-|icon|receiver.go:637 block notification height=57342322
D|15:03:19.168390|----|-|icon|receiver.go:637 block notification height=57342323
D|15:03:20.162118|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:21.012670|----|-|icon|receiver.go:637 block notification height=57342324
D|15:03:21.670117|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=3
D|15:03:23.124662|----|-|icon|receiver.go:637 block notification height=57342325
D|15:03:25.125155|----|-|icon|receiver.go:637 block notification height=57342326
D|15:03:25.162408|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:27.160233|----|-|icon|receiver.go:637 block notification height=57342327
D|15:03:27.808039|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=4
D|15:03:28.400910|hx20|snow|s2i|rx_receiver.go:464 block notification height=251993
D|15:03:29.110548|----|-|icon|receiver.go:637 block notification height=57342328
D|15:03:30.177044|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:31.004341|----|-|icon|receiver.go:637 block notification height=57342329
D|15:03:33.171900|----|-|icon|receiver.go:637 block notification height=57342330
D|15:03:33.911370|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry retry=5 error=not found
D|15:03:34.980995|----|-|icon|receiver.go:637 block notification height=57342331
D|15:03:35.178159|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:37.012554|----|-|icon|receiver.go:637 block notification height=57342332
D|15:03:39.650161|----|-|icon|receiver.go:637 block notification height=57342333
D|15:03:39.955091|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=6
D|15:03:40.179178|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:40.425282|hx20|snow|s2i|rx_receiver.go:464 block notification height=251994
D|15:03:41.134285|----|-|icon|receiver.go:637 block notification height=57342334
D|15:03:43.159538|----|-|icon|receiver.go:637 block notification height=57342335
D|15:03:45.195549|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:45.758832|----|-|icon|receiver.go:637 block notification height=57342336
D|15:03:46.048914|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=7
D|15:03:47.656598|----|-|icon|receiver.go:637 block notification height=57342337
D|15:03:49.199270|----|-|icon|receiver.go:637 block notification height=57342338
D|15:03:50.201930|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:51.660056|----|-|icon|receiver.go:637 block notification height=57342339
D|15:03:51.880384|hx20|snow|s2i|rx_receiver.go:272 poll height latest=251997 next=251996
D|15:03:52.086948|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=8
D|15:03:52.407205|hx20|snow|s2i|rx_receiver.go:464 block notification height=251995
D|15:03:52.410520|hx20|snow|s2i|rx_receiver.go:464 block notification height=251996
D|15:03:53.108409|----|-|icon|receiver.go:637 block notification height=57342340
D|15:03:55.059662|----|-|icon|receiver.go:637 block notification height=57342341
D|15:03:55.212092|hx20|relay|s2i|relay.go:113 relaySignal
D|15:03:57.019974|----|-|icon|receiver.go:637 block notification height=57342342
D|15:03:58.277755|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=9
D|15:03:59.065724|----|-|icon|receiver.go:637 block notification height=57342343
D|15:04:00.222661|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:01.032127|----|-|icon|receiver.go:637 block notification height=57342344
D|15:04:03.057449|----|-|icon|receiver.go:637 block notification height=57342345
D|15:04:04.324874|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry retry=10 error=not found
D|15:04:04.404623|hx20|snow|s2i|rx_receiver.go:464 block notification height=251997
D|15:04:05.223498|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:05.252439|----|-|icon|receiver.go:637 block notification height=57342346
D|15:04:07.636655|----|-|icon|receiver.go:637 block notification height=57342347
D|15:04:09.170712|----|-|icon|receiver.go:637 block notification height=57342348
D|15:04:10.227883|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:10.430611|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=11
D|15:04:11.026150|----|-|icon|receiver.go:637 block notification height=57342349
D|15:04:13.652064|----|-|icon|receiver.go:637 block notification height=57342350
D|15:04:15.228247|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:15.682294|----|-|icon|receiver.go:637 block notification height=57342351
D|15:04:16.391681|hx20|snow|s2i|rx_receiver.go:464 block notification height=251998
D|15:04:16.646508|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=12
D|15:04:17.148877|----|-|icon|receiver.go:637 block notification height=57342352
W|15:04:18.425849|----|-|stat|stat.go:102 System Alert Message=HeartBeat Message Level=info Time=2022-10-31T14:49:18.409Z LoadAverage=map[LoadAvg5:3.71]
I|15:04:18.442582|----|-|stat|stat.go:111 HeartBeat Message Message=HeartBeat Message Level=info Time=2022-10-31T14:49:18.409Z
I|15:04:18.445568|----|-|stat|stat.go:115 System Info LoadAverage=map[LoadAvg5:3.71] MemoryUsage=map[UsedPercent:19.85134852394664] DiskUsage=[map[MountedOn:/ UsedPercent:30] map[MountedOn:/bmr UsedPercent:30]] Message=HeartBeat Message Level=info Time=2022-10-31T14:49:18.409Z
D|15:04:19.104865|----|-|icon|receiver.go:637 block notification height=57342353
D|15:04:20.236594|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:21.011531|----|-|icon|receiver.go:637 block notification height=57342354
D|15:04:22.690846|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=13
D|15:04:23.156915|----|-|icon|receiver.go:637 block notification height=57342355
D|15:04:25.021066|----|-|icon|receiver.go:637 block notification height=57342356
D|15:04:25.237103|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:27.044151|----|-|icon|receiver.go:637 block notification height=57342357
D|15:04:27.883894|hx20|snow|s2i|rx_receiver.go:272 poll height latest=252000 next=252000
D|15:04:28.762437|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=14
D|15:04:29.166935|----|-|icon|receiver.go:637 block notification height=57342358
D|15:04:30.248115|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:31.135260|----|-|icon|receiver.go:637 block notification height=57342359
D|15:04:33.176162|----|-|icon|receiver.go:637 block notification height=57342360
D|15:04:34.853743|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=15
D|15:04:35.248136|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:35.248448|----|-|icon|receiver.go:637 block notification height=57342361
D|15:04:37.150662|----|-|icon|receiver.go:637 block notification height=57342362
D|15:04:39.702284|----|-|icon|receiver.go:637 block notification height=57342363
D|15:04:40.250266|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:40.395547|hx20|snow|s2i|rx_receiver.go:464 block notification height=251999
D|15:04:40.943701|0xb6|relay|i2s|relay.go:244 tx.Receipt: retry error=not found retry=16
D|15:04:41.170035|----|-|icon|receiver.go:637 block notification height=57342364
D|15:04:43.088934|----|-|icon|receiver.go:637 block notification height=57342365
D|15:04:45.061783|----|-|icon|receiver.go:637 block notification height=57342366
D|15:04:45.260826|hx20|relay|s2i|relay.go:113 relaySignal
W|15:04:46.990875|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=17
D|15:04:47.064358|----|-|icon|receiver.go:637 block notification height=57342367
D|15:04:49.065420|----|-|icon|receiver.go:637 block notification height=57342368
D|15:04:50.271538|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:51.017393|----|-|icon|receiver.go:637 block notification height=57342369
D|15:04:52.424139|hx20|snow|s2i|rx_receiver.go:464 block notification height=252000
D|15:04:53.067230|----|-|icon|receiver.go:637 block notification height=57342370
W|15:04:53.096584|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=18
D|15:04:55.285261|hx20|relay|s2i|relay.go:113 relaySignal
D|15:04:55.753675|----|-|icon|receiver.go:637 block notification height=57342371
D|15:04:57.134160|----|-|icon|receiver.go:637 block notification height=57342372
W|15:04:59.157653|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=19
D|15:04:59.707172|----|-|icon|receiver.go:637 block notification height=57342373
D|15:05:00.288593|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:01.031458|----|-|icon|receiver.go:637 block notification height=57342374
D|15:05:03.689938|----|-|icon|receiver.go:637 block notification height=57342375
D|15:05:03.903856|hx20|snow|s2i|rx_receiver.go:272 poll height latest=252003 next=252002
D|15:05:04.468890|hx20|snow|s2i|rx_receiver.go:464 block notification height=252001
W|15:05:05.261445|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=20
D|15:05:05.289099|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:05.675517|----|-|icon|receiver.go:637 block notification height=57342376
D|15:05:07.696687|----|-|icon|receiver.go:637 block notification height=57342377
D|15:05:09.125016|----|-|icon|receiver.go:637 block notification height=57342378
D|15:05:10.289979|hx20|relay|s2i|relay.go:113 relaySignal
W|15:05:11.276496|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=21
D|15:05:11.507351|----|-|icon|receiver.go:637 block notification height=57342379
D|15:05:13.651708|----|-|icon|receiver.go:637 block notification height=57342380
D|15:05:15.290780|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:15.516190|----|-|icon|receiver.go:637 block notification height=57342381
D|15:05:16.372300|hx20|snow|s2i|rx_receiver.go:464 block notification height=252002
D|15:05:17.013173|----|-|icon|receiver.go:637 block notification height=57342382
W|15:05:17.284664|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=22
D|15:05:19.653649|----|-|icon|receiver.go:637 block notification height=57342383
D|15:05:20.291811|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:21.654301|----|-|icon|receiver.go:637 block notification height=57342384
W|15:05:23.293464|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=23
D|15:05:23.643877|----|-|icon|receiver.go:637 block notification height=57342385
D|15:05:25.253325|----|-|icon|receiver.go:637 block notification height=57342386
D|15:05:25.292672|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:27.645897|----|-|icon|receiver.go:637 block notification height=57342387
D|15:05:28.370403|hx20|snow|s2i|rx_receiver.go:464 block notification height=252003
W|15:05:29.302081|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=24
D|15:05:29.703974|----|-|icon|receiver.go:637 block notification height=57342388
D|15:05:30.293001|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:31.651535|----|-|icon|receiver.go:637 block notification height=57342389
D|15:05:33.080574|----|-|icon|receiver.go:637 block notification height=57342390
D|15:05:35.077380|----|-|icon|receiver.go:637 block notification height=57342391
D|15:05:35.294086|hx20|relay|s2i|relay.go:113 relaySignal
W|15:05:35.310180|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=25
D|15:05:37.043930|----|-|icon|receiver.go:637 block notification height=57342392
D|15:05:39.025093|----|-|icon|receiver.go:637 block notification height=57342393
D|15:05:39.865030|hx20|snow|s2i|rx_receiver.go:272 poll height latest=252005 next=252005
D|15:05:40.303542|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:41.019690|----|-|icon|receiver.go:637 block notification height=57342394
W|15:05:41.319896|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry retry=26 error=not found
D|15:05:43.047003|----|-|icon|receiver.go:637 block notification height=57342395
D|15:05:45.304073|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:45.760463|----|-|icon|receiver.go:637 block notification height=57342396
D|15:05:47.120951|----|-|icon|receiver.go:637 block notification height=57342397
W|15:05:47.328945|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=27
D|15:05:49.699864|----|-|icon|receiver.go:637 block notification height=57342398
D|15:05:50.304770|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:51.059282|----|-|icon|receiver.go:637 block notification height=57342399
D|15:05:52.370494|hx20|snow|s2i|rx_receiver.go:464 block notification height=252004
W|15:05:53.337311|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=28
D|15:05:53.660759|----|-|icon|receiver.go:637 block notification height=57342400
D|15:05:55.305494|hx20|relay|s2i|relay.go:113 relaySignal
D|15:05:55.666649|----|-|icon|receiver.go:637 block notification height=57342401
D|15:05:57.152676|----|-|icon|receiver.go:637 block notification height=57342402
W|15:05:59.382068|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=29
D|15:05:59.640667|----|-|icon|receiver.go:637 block notification height=57342403
D|15:06:00.306430|hx20|relay|s2i|relay.go:113 relaySignal
D|15:06:01.025844|----|-|icon|receiver.go:637 block notification height=57342404
D|15:06:03.695148|----|-|icon|receiver.go:637 block notification height=57342405
D|15:06:04.394089|hx20|snow|s2i|rx_receiver.go:464 block notification height=252005
D|15:06:05.065536|----|-|icon|receiver.go:637 block notification height=57342406
D|15:06:05.314188|hx20|relay|s2i|relay.go:113 relaySignal
W|15:06:05.456147|0xb6|relay|i2s|relay.go:242 tx.Receipt: retry error=not found retry=30
D|15:06:07.049014|----|-|icon|receiver.go:637 block notification height=57342407
D|15:06:09.724851|----|-|icon|receiver.go:637 block notification height=57342408
D|15:06:10.317394|hx20|relay|s2i|relay.go:113 relaySignal
D|15:06:10.586036|0xb6|relay|i2s|relay.go:113 relaySignal
W|15:06:10.587993|0xb6|relay|i2s|relay.go:138 relay wallet balance below threshold balance=19979905499347188080 threshold=200000000000000000000
D|15:06:10.588578|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:10.607476|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:11.110223|0xb6|relay|i2s|relay.go:215 tx.Send: retry=1 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:11.110368|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:11.144875|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:11.645555|0xb6|relay|i2s|relay.go:215 tx.Send: retry=2 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:11.645639|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:11.647913|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:11.681813|----|-|icon|receiver.go:637 block notification height=57342409
D|15:06:12.154481|0xb6|relay|i2s|relay.go:215 tx.Send: retry=3 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:12.154571|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:12.156360|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:12.669149|0xb6|relay|i2s|relay.go:215 tx.Send: retry=4 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:12.669299|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:12.671923|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:13.173140|0xb6|relay|i2s|relay.go:215 tx.Send: retry=5 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:13.173221|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:13.175061|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:13.227103|----|-|icon|receiver.go:637 block notification height=57342410
D|15:06:13.675579|0xb6|relay|i2s|relay.go:215 tx.Send: retry=6 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:13.675868|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:13.684807|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:14.190208|0xb6|relay|i2s|relay.go:215 tx.Send: retry=7 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:14.190433|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:14.192630|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:14.705473|0xb6|relay|i2s|relay.go:215 tx.Send: retry=8 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:14.705717|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:14.708209|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:15.208576|0xb6|relay|i2s|relay.go:215 tx.Send: retry=9 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:15.208655|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:15.210525|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:15.278606|----|-|icon|receiver.go:637 block notification height=57342411
D|15:06:15.333779|hx20|relay|s2i|relay.go:113 relaySignal
D|15:06:15.723014|0xb6|relay|i2s|relay.go:215 tx.Send: retry=10 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:15.723264|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:15.725203|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:15.885688|hx20|snow|s2i|rx_receiver.go:272 poll height next=252007 latest=252008
D|15:06:16.229850|0xb6|relay|i2s|relay.go:215 tx.Send: retry=11 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:16.230088|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:16.232724|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:16.386799|hx20|snow|s2i|rx_receiver.go:464 block notification height=252006
D|15:06:16.733871|0xb6|relay|i2s|relay.go:215 tx.Send: retry=12 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:16.733971|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:16.735830|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:17.236808|0xb6|relay|i2s|relay.go:215 tx.Send: retry=13 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:17.237027|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:17.238775|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:17.676613|----|-|icon|receiver.go:637 block notification height=57342412
D|15:06:17.778018|0xb6|relay|i2s|relay.go:215 tx.Send: retry=14 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:17.818582|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:17.911883|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:18.426540|0xb6|relay|i2s|relay.go:215 tx.Send: retry=15 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:18.426795|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:18.428665|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
W|15:06:18.929719|0xb6|relay|i2s|relay.go:213 tx.Send: retry=16 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:18.929810|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:18.931604|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:19.225711|----|-|icon|receiver.go:637 block notification height=57342413
W|15:06:19.444113|0xb6|relay|i2s|relay.go:213 tx.Send: retry=17 error=submit transaction to pool failed: Pool(TemporarilyBanned)
D|15:06:19.444311|0xb6|snow|i2s|tx_sender.go:255 handleRelayMessage: send tx prev=btp://0x1.icon/cx23a91ee3dd290486a9113a6a42429825d813de53
D|15:06:19.445954|0xb6|snow|i2s|tx_sender.go:276 handleRelayMessage: send tx error=submit transaction to pool failed: Pool(TemporarilyBanned)
W|15:06:19.950500|0xb6|relay|i2s|relay.go:213 tx.Send: retry=18 error=submit transaction to pool failed: Pool(TemporarilyBanned)
manishbista28 commented 1 year ago

Are we certain gas price was the reason why initial transaction was not processed ?

Relay waited for transaction receipt between 251992 and 252010, it used a gas price of 8 units for the transaction. As can be seen from the following list of transactions that had happened during the period, it is evident that transactions with gas price of 8 were getting their turn. So a higher gas price could have helped better but it is not safe to assume that gas price alone is the sole factor that lead to the chain from discarding (not prioritizing) the transaction

Also notice that there are blocks in between which do not have any transactions in them, so the chain being unable to process because of network activity may not make perfect sense.

BlockNumber, TransactionID, TransactionHash, GasPrice
251991 0 0xd4e32b6e8f93708a271d020ec556d1d45a4361411342ebf2a88d51ba99e5ee30 8
------------------------------------------
------------------------------------------
251993 0 0x7c622f7a0487c6abfdcc1740d5424d61643027259046ce46171c353b5188a320 8
------------------------------------------
------------------------------------------
251995 0 0x8fd3f9477b3afbcb633c757d8239efada87ee970e8f7c4f5b20bf1ff94c0f383 1500000009
------------------------------------------
251996 0 0x07e1f9b308dbdf12065b1b9935e004786afbe36e5462ca9d19eb7b59b4d75521 1500000009
251996 1 0xe1614766783f45adcbb2005e1dbfbc60afa9ab4a7140569ae85ab5d103188700 8
251996 2 0xf54368694a36d000ebffc65b24b02af990e94ef214bb89ba52c1b475ae5415b2 8
------------------------------------------
251997 0 0xf9f72b553da19d2d68e7627c3fb05766f40fba0998d35239c60885322f237a91 8
251997 1 0xc108cd149cb059dfc297a8c02710afae26ecf3ddf37861ce5a3fa45fd6f0f760 8
251997 2 0x623953c5dbcce4aef4374a35f74c522f71356b382384c3fc750f2e7c88583753 8
------------------------------------------
251998 0 0x195975ecf6803e32432c750e95c9b79649e07bdae99f2c8bc1c191fdfe26c93d 8
------------------------------------------
251999 0 0x1c381600b9294b7a0e2c8778f504373fe5ac6ee1c85362f061906723e25b4bd5 8
------------------------------------------
------------------------------------------
------------------------------------------
252002 0 0x1d5c6269e1b8ec1bcae72f37ba7364cf5f9d9874000d35976dc7a5f73f0cec72 9
------------------------------------------
252003 0 0x2577960e775184e8963c9db772478231ac5bac53cb9acfd07b3891d31eee9dbf 1500000009
252003 1 0x04ab6d659951c9afb74567a57d2565e51a6839722345052c668b56f503901dd4 1500000009
252003 2 0xc036485a9d7efd2e9845ce1dc85902eb9543bbfd2a345ad21a8e3bfefb91d6a9 8
------------------------------------------
------------------------------------------
252005 0 0xf7e81923110e053e90b85bed65bbb1b971b398a276155c2a8db0a8a340e44494 8
------------------------------------------
252006 0 0xfdc2fc3088061594a41c78ba9fcbeeae146a0ba464d80ce5cda78515c1e7022f 1500000009
252006 1 0x71747550d7992fc0c09e75c99ca1a7480566e98a744de7da2dab4119c2e8281c 8
------------------------------------------
------------------------------------------
------------------------------------------
252009 0 0x99ef1943c6212a5a8ee4a6cb2b7c91c7b06c4a3e29835c9d1de3f4f894908a7f 8
252009 1 0x773a657925239d73b50cfcd5beb2a820cb2d3dccffaf6416f44c11159226605a 8
------------------------------------------
252010 0 0x991795bea65ee418a7b0c5726ad1ba71bf4b5476f21c3022221c9e40f6576e06 1500000009
------------------------------------------
andrii-kl commented 1 year ago

The reason can be in use Legacy transactions. I think it make sense to migrate to a new transactions.

Screen Shot 2022-11-01 at 12 23 02 PM
andrii-kl commented 1 year ago

In additionally it would be good to research if it possible to check tx pool on transaction existence, to avoid submission transactions several times and blacklisting