search

informalsystems / hermes

IBC Relayer in Rust
https://hermes.informal.systems
Apache License 2.0
430 stars 317 forks source link

Clearing packets apparently blocks Hermes from properly working #4071

Open freak12techno opened 2 days ago

freak12techno commented 2 days ago

Summary of Bug

There might be multiple issues at once here, not sure whether it would be better to report these as a separate ones, but here we go. So, after something happened (not sure what, but still), Hermes started to think there's more than 20k packets pending:

monitoring@monitoring-logging ~ ❯ hermes query packet pending --chain osmosis-1 --channel channel-0 --port transfer
2024-06-30T20:15:58.243203Z  INFO ThreadId(01) using default configuration from '/home/monitoring/.hermes/config.toml'
2024-06-30T20:15:58.244169Z  INFO ThreadId(01) running Hermes v1.10.0+fc998de4
SUCCESS Summary of pending packets:
Packets pending on source chain (osmosis-1):
  Unreceived packets:
    3395732..=3396057
    3396059..=3396063
    3396065..=3396109
    3396111..=3396210
    3396212..=3396312
    3396315..=3396458
    3396460..=3396662
    3396664..=3396712
    3396715..=3396863
    3396865..=3396902
    3396904..=3397231
    3397233..=3397251
    3397255..=3397675
    3397677..=3397709
    3397711..=3398118
    3398120..=3398932
    3398934..=3399103
    3399343..=3399367
    3399369..=3399419
    3399421..=3399425
    3399427..=3399521
    3399523..=3399680
    3399682..=3399746
    3399748..=3399833
    3399836..=3399885
    3399888..=3399937
    3399939..=3400017
    3400019..=3400039
    3400041..=3400140
    3400142..=3400191
    3400193..=3400243
    3400246..=3400402
    3400405..=3400498
    3400500..=3400531
    3400533..=3400642
    3400644..=3400670
    3400672..=3400719
    3400721..=3400881
    3400883..=3401022
    3401024..=3401503
    3401505..=3401826
    3401828..=3402110
    3402112..=3402546
    3402548..=3402568
    3402571..=3402721
    3402724..=3402774
    3402776..=3402837
    3402839..=3402876
    3402878..=3403077
    3403081..=3403283
    3403287..=3403386
    3403388..=3403438
    3403440..=3403674
    3403676..=3403681
    3403683..=3403696
    3403698..=3403749
    3403751..=3403952
    3403954..=3403974
    3403976..=3404131
    3404133..=3404147
    3404149..=3404151
    3404153..=3404156
    3404158..=3404179
    3404181..=3404296
    3404298..=3404333
    3404335..=3404360
    3404362..=3404405
    3404407..=3404418
    3404420..=3404435
    3404437..=3404448
    3404450..=3404711
    3404713..=3404737
    3404739..=3404816
    3404818..=3404820
    3404822..=3404879
    3404881..=3404928
    3404930..=3404968
    3404970..=3404998
    3405000..=3405129
    3405131..=3405195
    3405197..=3405256
    3405258..=3405347
    3405349..=3405389
    3405391..=3405397
    3405399..=3405438
    3405440..=3405577
    3405579..=3405884
    3405886..=3405904
    3405906..=3405961
    3405963..=3405966
    3405968..=3406042
    3406044..=3406183
    3406185..=3406189
    3406191..=3406280
    3406282..=3406288
    3406290..=3406299
    3406301..=3406340
    3406342..=3406351
    3406353..=3406397
    3406399..=3406402
    3406404..=3406409
    3406411..=3406493
    3406495..=3406502
    3406504..=3406507
    3406509..=3406541
    3406543..=3406630
    3406632..=3406733
    3406735..=3406791
    3406793..=3406811
    3406813..=3406933
    3406935..=3406955
    3406957..=3406958
    3406960..=3407189
    3407193..=3407241
    3407243..=3407276
    3407278..=3407293
    3407295..=3407345
    3407347..=3407396
    3407399..=3407437
    3407439..=3407549
    3407551..=3407555
    3407557..=3407601
    3407603..=3407618
    3407620..=3407662
    3407664..=3407877
    3407879..=3407934
    3407936..=3408006
    3408008..=3408057
    3408059..=3408105
    3408107..=3408136
    3408138..=3408155
    3408157..=3408159
    3408161..=3408210
    3408214..=3408263
    3408265..=3408265
    3408267..=3408280
    3408282..=3408414
    3408418..=3408466
    3408468..=3408507
    3408509..=3408517
    3408519..=3408567
    3408569..=3408638
    3408640..=3408668
    3408670..=3408769
    3408771..=3408872
    3408874..=3409026
    3409028..=3409069
    3409071..=3409153
    3409155..=3409335
    3409337..=3409385
    3409387..=3409407
    3409409..=3409427
    3409429..=3409429
    3409431..=3409575
    3409577..=3409682
    3409684..=3409692
    3409694..=3409785
    3409787..=3409826
    3409828..=3410147
    3410149..=3410160
    3410162..=3410268
    3410270..=3410283
    3410285..=3410301
    3410304..=3410565
    3410567..=3410709
    3410711..=3410717
    3410719..=3410812
    3410814..=3411214
    3411216..=3411222
    3411224..=3411431
    3411433..=3411574
    3411576..=3411630
    3411633..=3411924
    3411926..=3412294
    3412296..=3412341
    3412343..=3412599
    3412601..=3412727
    3412729..=3412837
    3412839..=3413090
    3413092..=3413154
    3413156..=3413182
    3413184..=3413206
    3413208..=3413309
    3413314..=3413367
    3413369..=3413401
    3413403..=3413633
    3413635..=3413668
    3413670..=3413918
    3413920..=3414459
    3414461..=3414473
    3414475..=3414577
    3414580..=3414628
    3414630..=3414777
    3414779..=3415100
    3415102..=3415387
    3415389..=3415458
    3415460..=3415527
    3415529..=3415685
    3415687..=3415967
    3415969..=3416011
    3416013..=3416276
    3416278..=3416377
    3416380..=3416453
    3416455..=3416711
    3416713..=3416883
    3416885..=3416910
    3416912..=3416933
    3416935..=3416985
    3416987..=3417016
    3417018..=3417087
    3417089..=3417139
    3417141..=3417181
    3417183..=3417248
    3417250..=3417293
    3417295..=3417342
    3417345..=3417390
    3417392..=3417445
    3417449..=3417497
    3417499..=3417549
    3417551..=3417582
    3417584..=3417601
    3417603..=3417613
    3417615..=3417702
    3417704..=3417754
    3417756..=3417802
    3417804..=3417804
    3417808..=3417908
    3417910..=3418338
    3418340..=3418462
    3418464..=3418515
    3418517..=3418566
    3418568..=3418944
    3418946..=3419268
    3419270..=3419340
    3419342..=3419365
    3419367..=3419576
    3419578..=3419595
    3419597..=3419616
    3419618..=3419623
    3419625..=3419675
    3423615..=3423617
    3423619..=3423620
    3423622..=3423623
    3423625..=3423627
    3423629..=3423629
    3423631..=3423634
    3423636..=3423644
    3423646..=3423646
    3423648..=3423657
    3423660..=3423664
    3423666..=3423668
    3423670..=3423692
    3423695..=3423698
    3423700..=3423708
    3423712..=3423712
    3423715..=3423719
    3423721..=3423723
    3423725..=3423729
    3423731..=3423731
    3423733..=3423741
    3423743..=3423751
    3423753..=3423759
    3423761..=3423762
    3423766..=3423771
    3423773..=3423780
    3423782..=3423787
    3423789..=3423789
    3423791..=3423794
    3423796..=3423799
    3423801..=3423811
    3423813..=3423814
    3423816..=3423820
    3423822..=3423825
    3423827..=3423829
    3423831..=3423831
    3423833..=3423834
    3423836..=3423839
    3423841..=3423843
    3423845..=3423854
    3423856..=3423867
    3423869..=3423869
    3423871..=3423884
    3423886..=3423888
    3423890..=3423893
    3423896..=3423900
    3423902..=3423906
    3423908..=3423910
    3423912..=3423917
    3423919..=3423927
    3423931..=3423931
    3423933..=3423946
    3423949..=3423950
    3423952..=3423954
    3423956..=3423968
    3423970..=3423992
    3423994..=3423995
    3423998..=3424005
    3462042..=3462043
  Unreceived acks:
Packets pending on destination chain (cosmoshub-4):
  Unreceived packets:
  Unreceived acks:
    3655048..=3655048

and it tries to clear these. Looking at the logs, this might take really long time (like more than 24h for sure), and during all this time, Hermes somehow isn't broadcasting any transactions (I can see it both from logs and from metrics). And apparently trying to clear the packets (both manually via hermes clear packets --chain osmosis-1 --channel channel-0 --port transfer and by waiting for Hermes main process packet clear routine to finish) doesn't work, as once it's done, Hermes runs the clean packets routine again, and everything is blocked again.

Basically there are 3 issues here: 1) Hermes somehow thinks there are 20k packets pending from osmosis-1 -> cosmoshub-4 2) When clearing packets, nothing happens, and the next time packets clearing routing is executed, it still shows the same 20k packets 3) (the main one) Once the packet clear routine is being executed, Hermes isn't doing anything else.

I am not sure what can cause it and what was changed so it started behaving that way, but it seems wrong. I fixed it by disabling the check routine, but that seems like a workaround.

Here are Hermes logs:

Started Relayer.
2024-06-30T20:19:14.665887Z  INFO ThreadId(01) using default configuration from '/home/monitoring/.hermes/config.toml'
2024-06-30T20:19:14.666410Z  INFO ThreadId(01) running Hermes v1.10.0+fc998de4
2024-06-30T20:19:14.667780Z  INFO ThreadId(01) rest: REST server disabled
2024-06-30T20:19:14.667832Z  INFO ThreadId(19) telemetry service running, exposing metrics at http://127.0.0.1:3001/metrics
2024-06-30T20:19:14.688425Z  WARN ThreadId(21) health_check{chain=cosmoshub-4}: found potential misconfiguration for chain 'cosmoshub-4'
2024-06-30T20:19:14.688441Z  WARN ThreadId(21) health_check{chain=cosmoshub-4}: reason: semantic config validation failed for option `max_tx_size` for chain 'cosmoshub-4', reason: `max_tx_size` = 2000000 is greater than 90% of the consensus parameter `max_size` = 2000000
2024-06-30T20:19:14.688444Z  WARN ThreadId(21) health_check{chain=cosmoshub-4}: some Hermes features may not work in this mode!
2024-06-30T20:19:14.688465Z  WARN ThreadId(01) health_check{chain=cosmoshub-4}: chain is not healthy: semantic config validation failed for option `max_tx_size` for chain 'cosmoshub-4', reason: `max_tx_size` = 2000000 is greater than 90% of the consensus parameter `max_size` = 2000000
2024-06-30T20:19:15.768792Z  WARN ThreadId(22) health_check{chain=osmosis-1}: chain 'osmosis-1' has no minimum gas price value configured for denomination 'uosmo'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
2024-06-30T20:19:16.971506Z  WARN ThreadId(22) health_check{chain=osmosis-1}: Will use fallback value for max_block_time: `30s`. Error: response error: Internal error: genesis response is large, please use the genesis_chunked API instead (code: -32603)
2024-06-30T20:19:16.971580Z  INFO ThreadId(01) health_check{chain=osmosis-1}: chain is healthy
2024-06-30T20:19:16.972112Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: scanning chain...
2024-06-30T20:19:16.972162Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-06-30T20:19:16.972167Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: allowed ports/channels: transfer/channel-229, transfer/channel-141, transfer/channel-186
2024-06-30T20:19:16.972175Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: querying allowed channels...
2024-06-30T20:19:16.972182Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: querying channel...
2024-06-30T20:19:16.982881Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: found connection and client connection=connection-395 client=07-tendermint-481
2024-06-30T20:19:16.982903Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: querying client... client=07-tendermint-481
2024-06-30T20:19:16.984495Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: found counterparty chain for client client=07-tendermint-481 counterparty_chain=bitsong-2b
2024-06-30T20:19:16.984524Z ERROR ThreadId(01) scan.chain{chain=cosmoshub-4}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-229
2024-06-30T20:19:16.984536Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: querying channel...
2024-06-30T20:19:16.991021Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found connection and client connection=connection-257 client=07-tendermint-259
2024-06-30T20:19:16.991039Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: querying client... client=07-tendermint-259
2024-06-30T20:19:16.992196Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty chain for client client=07-tendermint-259 counterparty_chain=osmosis-1
2024-06-30T20:19:17.349801Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty channel counterparty_channel=channel-0
2024-06-30T20:19:17.597171Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty connection state counterparty_connection_state=OPEN
2024-06-30T20:19:17.597262Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: querying channel...
2024-06-30T20:19:17.607225Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: found connection and client connection=connection-341 client=07-tendermint-388
2024-06-30T20:19:17.607248Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: querying client... client=07-tendermint-388
2024-06-30T20:19:17.608879Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: found counterparty chain for client client=07-tendermint-388 counterparty_chain=sentinelhub-2
2024-06-30T20:19:17.608905Z ERROR ThreadId(01) scan.chain{chain=cosmoshub-4}: failed to scan channel, reason: spawn: missing chain config for 'sentinelhub-2' in configuration file channel=channel-186
2024-06-30T20:19:17.608924Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}: scanning chain...
2024-06-30T20:19:17.608976Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-06-30T20:19:17.608980Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}: allowed ports/channels: transfer/channel-73, transfer/channel-0, transfer/channel-412, transfer/channel-2, wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn/channel-78439
2024-06-30T20:19:17.608989Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}: querying allowed channels...
2024-06-30T20:19:17.608994Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: querying channel...
2024-06-30T20:19:18.215471Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: found connection and client connection=connection-1216 client=07-tendermint-1551
2024-06-30T20:19:18.215497Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: querying client... client=07-tendermint-1551
2024-06-30T20:19:18.334420Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: found counterparty chain for client client=07-tendermint-1551 counterparty_chain=bitsong-2b
2024-06-30T20:19:18.334460Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-73
2024-06-30T20:19:18.334472Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: querying channel...
2024-06-30T20:19:18.691059Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found connection and client connection=connection-1 client=07-tendermint-1
2024-06-30T20:19:18.691085Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: querying client... client=07-tendermint-1
2024-06-30T20:19:18.809703Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty chain for client client=07-tendermint-1 counterparty_chain=cosmoshub-4
2024-06-30T20:19:18.816134Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty channel counterparty_channel=channel-141
2024-06-30T20:19:18.816177Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty connection state counterparty_connection_state=OPEN
2024-06-30T20:19:18.816235Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: querying channel...
2024-06-30T20:19:19.418229Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: found connection and client connection=connection-1762 client=07-tendermint-2273
2024-06-30T20:19:19.418258Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: querying client... client=07-tendermint-2273
2024-06-30T20:19:19.536807Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: found counterparty chain for client client=07-tendermint-2273 counterparty_chain=jackal-1
2024-06-30T20:19:19.536851Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'jackal-1' in configuration file channel=channel-412
2024-06-30T20:19:19.536864Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: querying channel...
2024-06-30T20:19:20.138088Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: found connection and client connection=connection-3 client=07-tendermint-2
2024-06-30T20:19:20.138116Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: querying client... client=07-tendermint-2
2024-06-30T20:19:20.257000Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: found counterparty chain for client client=07-tendermint-2 counterparty_chain=sentinelhub-2
2024-06-30T20:19:20.257038Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'sentinelhub-2' in configuration file channel=channel-2
2024-06-30T20:19:20.257053Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: querying channel...
2024-06-30T20:19:20.615793Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: found connection and client connection=connection-1216 client=07-tendermint-1551
2024-06-30T20:19:20.615821Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: querying client... client=07-tendermint-1551
2024-06-30T20:19:20.843420Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: found counterparty chain for client client=07-tendermint-1551 counterparty_chain=bitsong-2b
2024-06-30T20:19:20.843458Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-78439
2024-06-30T20:19:20.843476Z  INFO ThreadId(01) scanned chains:
2024-06-30T20:19:20.843480Z  INFO ThreadId(01) # Chain: cosmoshub-4
  - Client: 07-tendermint-259
    * Connection: connection-257
      | State: OPEN
      | Counterparty state: OPEN
      + Channel: channel-141
        | Port: transfer
        | State: OPEN
        | Counterparty: channel-0
# Chain: osmosis-1
  - Client: 07-tendermint-1
    * Connection: connection-1
      | State: OPEN
      | Counterparty state: OPEN
      + Channel: channel-0
        | Port: transfer
        | State: OPEN
        | Counterparty: channel-141
2024-06-30T20:19:20.843516Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: connection is OPEN, state on destination chain is OPEN chain=cosmoshub-4 connection=connection-257 counterparty_chain=osmosis-1
2024-06-30T20:19:20.843523Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: connection is already open, not spawning Connection worker chain=cosmoshub-4 connection=connection-257
2024-06-30T20:19:20.843531Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: no connection workers were spawn chain=cosmoshub-4 connection=connection-257
2024-06-30T20:19:20.843539Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}: channel is OPEN, state on destination chain is OPEN chain=cosmoshub-4 counterparty_chain=osmosis-1 channel=channel-141
2024-06-30T20:19:21.116508Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}: spawned client worker: client::osmosis-1->cosmoshub-4:07-tendermint-259
2024-06-30T20:19:24.629223Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: done spawning channel workers chain=cosmoshub-4 channel=channel-141
2024-06-30T20:19:24.629334Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}: spawning Wallet worker: wallet::cosmoshub-4
2024-06-30T20:19:24.629371Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: connection is OPEN, state on destination chain is OPEN chain=osmosis-1 connection=connection-1 counterparty_chain=cosmoshub-4
2024-06-30T20:19:24.629379Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: connection is already open, not spawning Connection worker chain=osmosis-1 connection=connection-1
2024-06-30T20:19:24.629386Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: no connection workers were spawn chain=osmosis-1 connection=connection-1
2024-06-30T20:19:24.629393Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=osmosis-1 counterparty_chain=cosmoshub-4 channel=channel-0
2024-06-30T20:19:24.871166Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: spawned client worker: client::cosmoshub-4->osmosis-1:07-tendermint-1
2024-06-30T20:19:28.461923Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: spawned packet worker: packet::channel-0/transfer:osmosis-1->cosmoshub-4
2024-06-30T20:19:28.461950Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: done spawning channel workers chain=osmosis-1 channel=channel-0
2024-06-30T20:19:28.461998Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}: spawning Wallet worker: wallet::osmosis-1
2024-06-30T20:19:29.779724Z  INFO ThreadId(01) Hermes has started
2024-06-30T20:19:32.664232Z  INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}: clearing packets
2024-06-30T20:20:06.201132Z  INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395732..=3395781; events.total=23757 events.left=23707
2024-06-30T20:20:37.426075Z  INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395782..=3395831; events.total=23757 events.left=23657
2024-06-30T20:21:08.440887Z  INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395832..=3395881; events.total=23757 events.left=23607
2024-06-30T20:21:39.215104Z  INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395882..=3395931; events.total=23757 events.left=23557
2024-06-30T20:22:10.734854Z  INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17369387 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17369386 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17369386}: pulled packet data for 0 events out of 50 sequences: 3395932..=3395981; events.total=23757 events.left=23507
# and many many like the previous one

And here's the minimal config I was able to reproduce it with:

[global]
log_level = "info"

[mode.clients]
enabled = true
refresh = true
misbehaviour = false

[mode.connections]
enabled = false

[mode.channels]
enabled = false

[mode.packets]
enabled = true
# Commenting out these 2 lines and uncommenting the next 2 lines after that one fixes it.
clear_interval = 100
clear_on_start = true
# clear_interval = 0
# clear_on_start = false
tx_confirmation = true
auto_register_counterparty_payee = false
ics20_max_memo_size = { enabled = true, size = "32KiB" }
ics20_max_receiver_size = { enabled = true, size = "2KiB" }

[rest]
enabled = false
host = "127.0.0.1"
port = 3000

[telemetry]
enabled = true
host = "127.0.0.1"
port = 3001

[telemetry.buckets.latency_submitted]
start = 500
end = 20000
buckets = 10

[telemetry.buckets.latency_confirmed]
start = 1000
end = 30000
buckets = 10

[[chains]]
id = "cosmoshub-4"
type = "CosmosSdk"
rpc_addr = "https://rpc.cosmos.quokkastake.io"
grpc_addr = "http://rpc.cosmos.quokkastake.io:9090"
rpc_timeout = "20s"
trusted_node = false
account_prefix = "cosmos"
key_name = "relayer"
key_store_type = "Test"
store_prefix = "ibc"
default_gas = 800000
max_gas = 3500000
gas_multiplier = 1.5
max_msg_num = 30
max_tx_size = 2000000
max_grpc_decoding_size = 33554432
clock_drift = "5s"
max_block_time = "30s"
ccv_consumer_chain = false
memo_prefix = "Relayed with ❤️  by 🐹 Quokka Stake"
sequential_batch_tx = false
event_source = { mode = "push", url = "wss://rpc.cosmos.quokkastake.io/websocket", batch_delay = "1000ms" }
trust_threshold = { numerator = "1", denominator = "3" }
gas_price = { price = 0.005, denom = "uatom" }
address_type = { derivation = "cosmos" }

[chains.packet_filter]
policy = "allow"
list = [
    ["transfer", "channel-229"],
    ["transfer", "channel-141"],
    ["transfer", "channel-186"],
]

[[chains]]
id = "osmosis-1"
type = "CosmosSdk"
rpc_addr = "https://rpc-osmosis.sentinelgrowthdao.com"
grpc_addr = "http://rpc-osmosis.sentinelgrowthdao.com:9090"
rpc_timeout = "600s"
trusted_node = false
account_prefix = "osmo"
key_name = "relayer"
key_store_type = "Test"
store_prefix = "ibc"
default_gas = 1000000
max_gas = 120000000
gas_multiplier = 1.4
max_msg_num = 30
max_tx_size = 1800000
max_grpc_decoding_size = 33554432
clock_drift = "5s"
max_block_time = "30s"
ccv_consumer_chain = false
memo_prefix = "Relayed with ❤️  by 🐹 Quokka Stake"
sequential_batch_tx = false
trust_threshold = { numerator = "1", denominator = "3" }
gas_price = { price = 0.01, denom = 'uosmo' }
event_source = { mode = "push", url = "wss://rpc-osmosis.sentinelgrowthdao.com/websocket", batch_delay = "500ms" }
address_type = { derivation = "cosmos" }

[chains.packet_filter]
policy = "allow"
list = [
    ["transfer", "channel-73"],
    ["transfer", "channel-0"],
    ["transfer", "channel-412"],
    ["transfer", "channel-2"],
    ["wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn", "channel-78439"] # polytone from bitsong-2b
]

[tracing_server]
enabled = false
port = 5555

Version

hermes 1.10.0+fc998de4

Steps to Reproduce

See above for the minimal config example and for logs.

Acceptance Criteria

For Admin Use

freak12techno commented 2 days ago

Might be related: https://github.com/informalsystems/hermes/issues/2612

ljoss17 commented 2 days ago

A first issue I have noticed with the RPC endpoint is that it doesn't seem to persist ABCI responses. For example, if I try to query https://rpc-osmosis.sentinelgrowthdao.com/block_results?height=17198597 I will get:

{"jsonrpc":"2.0","id":-1,"error":{"code":-32603,"message":"Internal error","data":"node is not persisting abci responses"}}

This is usually due to the CometBFT configuration discard_abci_responses being set to true. Could you verify this configuration?

And have you tried using a Node with more historical data? The one being used doesn't seem to have the packet data anymore: pulled packet data for 0 events out of 50 sequences.

freak12techno commented 2 days ago

@ljoss17 most likely, will check. I do not have Osmosis archive node unfortunately, so I cannot check it with older heights, but I will check the discard_abci_responses a bit later. The pulled packet data for 0 events out of 50 sequences part also a bit counter-unintuitive; IMO it should be a warning (like, reporting something like could not pull X/50 events if X > 0) and it ideally should report the reason what was the issue when pulling events, do you think it's possible?

ljoss17 commented 2 days ago

It is a valid point, this log message can be confusing, thanks for the feedback! I will look into this and see what information I can extract from the query and improve the log message.

freak12techno commented 2 days ago

@ljoss17 did you also check the point about clear packets routine thing blocking the rest? This seems to be the biggest issue here, for me at least.

ljoss17 commented 2 days ago

I will look into that right now. Would you be able to share logs at trace level with the instance running for a few minutes while the clearing is triggered? Just a heads-up, this will generate a lot of logs, so you might need to store the output to a file.

freak12techno commented 2 days ago

@ljoss17 here you go, let me know if you need more.

2024-07-01T11:54:03.805896Z  INFO ThreadId(01) using default configuration from '/home/monitoring/.hermes/config.toml'
2024-07-01T11:54:03.806405Z  INFO ThreadId(01) running Hermes v1.10.0+fc998de4
2024-07-01T11:54:03.807674Z  INFO ThreadId(01) rest: REST server disabled
2024-07-01T11:54:03.807789Z  INFO ThreadId(19) telemetry service running, exposing metrics at http://127.0.0.1:3001/metrics
2024-07-01T11:54:03.815481Z TRACE ThreadId(01) health_check{chain=cosmoshub-4}: spawned chain runtime chain=cosmoshub-4
2024-07-01T11:54:03.829073Z TRACE ThreadId(21) health_check{chain=cosmoshub-4}: parsed version specification application=gaiad version=v17.2.0 git_commit=17f2ba0b90d1d2884f7b67518ec08dfd37f001a7 sdk_version=Some(Version { major: 0, minor: 47, patch: 11 }) ibc_go_status=Some(Version { major: 7, minor: 4, patch: 0 }) consensus_version=Some(Comet(Version { major: 0, minor: 37, patch: 6 }))
2024-07-01T11:54:03.829103Z DEBUG ThreadId(21) health_check{chain=cosmoshub-4}: running diagnostic on version specs: Cosmos SDK 0.47.11, IBC-Go 7.4.0, CometBFT 0.37.6
2024-07-01T11:54:03.836498Z  WARN ThreadId(21) health_check{chain=cosmoshub-4}: found potential misconfiguration for chain 'cosmoshub-4'
2024-07-01T11:54:03.836513Z  WARN ThreadId(21) health_check{chain=cosmoshub-4}: reason: semantic config validation failed for option `max_tx_size` for chain 'cosmoshub-4', reason: `max_tx_size` = 2000000 is greater than 90% of the consensus parameter `max_size` = 2000000
2024-07-01T11:54:03.836516Z  WARN ThreadId(21) health_check{chain=cosmoshub-4}: some Hermes features may not work in this mode!
2024-07-01T11:54:03.836541Z  WARN ThreadId(01) health_check{chain=cosmoshub-4}: chain is not healthy: semantic config validation failed for option `max_tx_size` for chain 'cosmoshub-4', reason: `max_tx_size` = 2000000 is greater than 90% of the consensus parameter `max_size` = 2000000
2024-07-01T11:54:04.207422Z TRACE ThreadId(01) health_check{chain=osmosis-1}: spawned chain runtime chain=osmosis-1
2024-07-01T11:54:04.948556Z  WARN ThreadId(22) health_check{chain=osmosis-1}: chain 'osmosis-1' has no minimum gas price value configured for denomination 'uosmo'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
2024-07-01T11:54:05.311784Z TRACE ThreadId(22) health_check{chain=osmosis-1}: parsed version specification application=osmosisd version=25.1.3 git_commit=dd51aa80ce509c7577238580feda8246bfbe7b82 sdk_version=Some(Version { major: 0, minor: 47, patch: 8 }) ibc_go_status=Some(Version { major: 7, minor: 4, patch: 1 }) consensus_version=Some(Comet(Version { major: 0, minor: 37, patch: 4 }))
2024-07-01T11:54:05.311828Z DEBUG ThreadId(22) health_check{chain=osmosis-1}: running diagnostic on version specs: Cosmos SDK 0.47.8, IBC-Go 7.4.1, CometBFT 0.37.4
2024-07-01T11:54:06.174247Z  WARN ThreadId(22) health_check{chain=osmosis-1}: Will use fallback value for max_block_time: `30s`. Error: response error: Internal error: genesis response is large, please use the genesis_chunked API instead (code: -32603)
2024-07-01T11:54:06.174286Z  INFO ThreadId(01) health_check{chain=osmosis-1}: chain is healthy
2024-07-01T11:54:06.174743Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: scanning chain...
2024-07-01T11:54:06.174796Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-07-01T11:54:06.174800Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: allowed ports/channels: transfer/channel-229, transfer/channel-141, transfer/channel-186
2024-07-01T11:54:06.174815Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}: querying allowed channels...
2024-07-01T11:54:06.174822Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: querying channel...
2024-07-01T11:54:06.184413Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: found connection and client connection=connection-395 client=07-tendermint-481
2024-07-01T11:54:06.184440Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: querying client... client=07-tendermint-481
2024-07-01T11:54:06.186133Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-229}: found counterparty chain for client client=07-tendermint-481 counterparty_chain=bitsong-2b
2024-07-01T11:54:06.186165Z ERROR ThreadId(01) scan.chain{chain=cosmoshub-4}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-229
2024-07-01T11:54:06.186178Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: querying channel...
2024-07-01T11:54:06.193753Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found connection and client connection=connection-257 client=07-tendermint-259
2024-07-01T11:54:06.193775Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: querying client... client=07-tendermint-259
2024-07-01T11:54:06.195314Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty chain for client client=07-tendermint-259 counterparty_chain=osmosis-1
2024-07-01T11:54:06.567450Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty channel counterparty_channel=channel-0
2024-07-01T11:54:06.815949Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-141}: found counterparty connection state counterparty_connection_state=OPEN
2024-07-01T11:54:06.816054Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: querying channel...
2024-07-01T11:54:06.825456Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: found connection and client connection=connection-341 client=07-tendermint-388
2024-07-01T11:54:06.825480Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: querying client... client=07-tendermint-388
2024-07-01T11:54:06.827001Z  INFO ThreadId(01) scan.chain{chain=cosmoshub-4}:scan.channel{port=transfer channel=channel-186}: found counterparty chain for client client=07-tendermint-388 counterparty_chain=sentinelhub-2
2024-07-01T11:54:06.827029Z ERROR ThreadId(01) scan.chain{chain=cosmoshub-4}: failed to scan channel, reason: spawn: missing chain config for 'sentinelhub-2' in configuration file channel=channel-186
2024-07-01T11:54:06.827045Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}: scanning chain...
2024-07-01T11:54:06.827086Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}: chain uses an allow list (without wildcards), skipping scan for fast startup
2024-07-01T11:54:06.827088Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}: allowed ports/channels: transfer/channel-73, transfer/channel-0, transfer/channel-412, transfer/channel-2, wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn/channel-78439
2024-07-01T11:54:06.827095Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}: querying allowed channels...
2024-07-01T11:54:06.827098Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: querying channel...
2024-07-01T11:54:07.445781Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: found connection and client connection=connection-1216 client=07-tendermint-1551
2024-07-01T11:54:07.445816Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: querying client... client=07-tendermint-1551
2024-07-01T11:54:07.568820Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-73}: found counterparty chain for client client=07-tendermint-1551 counterparty_chain=bitsong-2b
2024-07-01T11:54:07.568861Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-73
2024-07-01T11:54:07.568873Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: querying channel...
2024-07-01T11:54:07.939226Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found connection and client connection=connection-1 client=07-tendermint-1
2024-07-01T11:54:07.939255Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: querying client... client=07-tendermint-1
2024-07-01T11:54:08.062658Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty chain for client client=07-tendermint-1 counterparty_chain=cosmoshub-4
2024-07-01T11:54:08.070205Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty channel counterparty_channel=channel-141
2024-07-01T11:54:08.070256Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-0}: found counterparty connection state counterparty_connection_state=OPEN
2024-07-01T11:54:08.070315Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: querying channel...
2024-07-01T11:54:08.705140Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: found connection and client connection=connection-1762 client=07-tendermint-2273
2024-07-01T11:54:08.705169Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: querying client... client=07-tendermint-2273
2024-07-01T11:54:08.829093Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-412}: found counterparty chain for client client=07-tendermint-2273 counterparty_chain=jackal-1
2024-07-01T11:54:08.829132Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'jackal-1' in configuration file channel=channel-412
2024-07-01T11:54:08.829143Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: querying channel...
2024-07-01T11:54:09.453813Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: found connection and client connection=connection-3 client=07-tendermint-2
2024-07-01T11:54:09.453834Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: querying client... client=07-tendermint-2
2024-07-01T11:54:09.576891Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=transfer channel=channel-2}: found counterparty chain for client client=07-tendermint-2 counterparty_chain=sentinelhub-2
2024-07-01T11:54:09.576935Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'sentinelhub-2' in configuration file channel=channel-2
2024-07-01T11:54:09.576947Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: querying channel...
2024-07-01T11:54:09.948716Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: found connection and client connection=connection-1216 client=07-tendermint-1551
2024-07-01T11:54:09.948745Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: querying client... client=07-tendermint-1551
2024-07-01T11:54:10.071991Z  INFO ThreadId(01) scan.chain{chain=osmosis-1}:scan.channel{port=wasm.osmo1ew5q937st3vemnk3cjk85rkceapvtmmk673pt0yc5mqt4ffhfa9qa4xfjn channel=channel-78439}: found counterparty chain for client client=07-tendermint-1551 counterparty_chain=bitsong-2b
2024-07-01T11:54:10.072032Z ERROR ThreadId(01) scan.chain{chain=osmosis-1}: failed to scan channel, reason: spawn: missing chain config for 'bitsong-2b' in configuration file channel=channel-78439
2024-07-01T11:54:10.072050Z  INFO ThreadId(01) scanned chains:
2024-07-01T11:54:10.072052Z  INFO ThreadId(01) # Chain: cosmoshub-4
  - Client: 07-tendermint-259
    * Connection: connection-257
      | State: OPEN
      | Counterparty state: OPEN
      + Channel: channel-141
        | Port: transfer
        | State: OPEN
        | Counterparty: channel-0
# Chain: osmosis-1
  - Client: 07-tendermint-1
    * Connection: connection-1
      | State: OPEN
      | Counterparty state: OPEN
      + Channel: channel-0
        | Port: transfer
        | State: OPEN
        | Counterparty: channel-141
2024-07-01T11:54:10.072100Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: connection is OPEN, state on destination chain is OPEN chain=cosmoshub-4 connection=connection-257 counterparty_chain=osmosis-1
2024-07-01T11:54:10.072105Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: connection is already open, not spawning Connection worker chain=cosmoshub-4 connection=connection-257
2024-07-01T11:54:10.072112Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: no connection workers were spawn chain=cosmoshub-4 connection=connection-257
2024-07-01T11:54:10.072118Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}: channel is OPEN, state on destination chain is OPEN chain=cosmoshub-4 counterparty_chain=osmosis-1 channel=channel-141
2024-07-01T11:54:10.321022Z DEBUG ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.client.refresh{client=07-tendermint-259 src_chain=osmosis-1 dst_chain=cosmoshub-4}: spawning task
2024-07-01T11:54:10.321130Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}: spawned client worker: client::osmosis-1->cosmoshub-4:07-tendermint-259
2024-07-01T11:54:12.695528Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}: done spawning channel workers chain=cosmoshub-4 channel=channel-141
2024-07-01T11:54:12.695587Z DEBUG ThreadId(01) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: spawning task
2024-07-01T11:54:12.695651Z  INFO ThreadId(01) spawn:chain{chain=cosmoshub-4}: spawning Wallet worker: wallet::cosmoshub-4
2024-07-01T11:54:12.695683Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: connection is OPEN, state on destination chain is OPEN chain=osmosis-1 connection=connection-1 counterparty_chain=cosmoshub-4
2024-07-01T11:54:12.695688Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: connection is already open, not spawning Connection worker chain=osmosis-1 connection=connection-1
2024-07-01T11:54:12.695692Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: no connection workers were spawn chain=osmosis-1 connection=connection-1
2024-07-01T11:54:12.695698Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=osmosis-1 counterparty_chain=cosmoshub-4 channel=channel-0
2024-07-01T11:54:12.698097Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:12.944423Z DEBUG ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-1 src_chain=cosmoshub-4 dst_chain=osmosis-1}: spawning task
2024-07-01T11:54:12.944513Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: spawned client worker: client::cosmoshub-4->osmosis-1:07-tendermint-1
2024-07-01T11:54:15.154861Z DEBUG ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}: spawning task
2024-07-01T11:54:15.154951Z DEBUG ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}: spawning task
2024-07-01T11:54:15.155002Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}: spawned packet worker: packet::channel-0/transfer:osmosis-1->cosmoshub-4
2024-07-01T11:54:15.155015Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}: done spawning channel workers chain=osmosis-1 channel=channel-0
2024-07-01T11:54:15.155032Z DEBUG ThreadId(01) spawn:chain{chain=osmosis-1}:wallet{chain=osmosis-1}: spawning task
2024-07-01T11:54:15.155065Z  INFO ThreadId(01) spawn:chain{chain=osmosis-1}: spawning Wallet worker: wallet::osmosis-1
2024-07-01T11:54:15.160407Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: tm.event = 'NewBlock'
2024-07-01T11:54:15.160820Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: message.module = 'ibc_client'
2024-07-01T11:54:15.161208Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: message.module = 'ibc_connection'
2024-07-01T11:54:15.161460Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: message.module = 'ibc_channel'
2024-07-01T11:54:15.161700Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribing to query: message.module = 'interchainquery'
2024-07-01T11:54:15.161931Z TRACE ThreadId(21) supervisor.init_subscriptions:event_source.init_subscriptions{chain=cosmoshub-4}: subscribed to all queries
2024-07-01T11:54:15.162066Z DEBUG ThreadId(30) event_source.websocket{chain=cosmoshub-4}: collecting events
2024-07-01T11:54:15.402123Z TRACE ThreadId(28) spawn:chain{chain=osmosis-1}:wallet{chain=osmosis-1}: wallet balance amount=47405742 denom=uosmo account=osmo17ndx5qfku28ymxgmq6zq4a6d02dvpfjjul0hyh
2024-07-01T11:54:15.893597Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: tm.event = 'NewBlock'
2024-07-01T11:54:16.015357Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: message.module = 'ibc_client'
2024-07-01T11:54:16.137118Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: message.module = 'ibc_connection'
2024-07-01T11:54:16.258815Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: message.module = 'ibc_channel'
2024-07-01T11:54:16.380717Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribing to query: message.module = 'interchainquery'
2024-07-01T11:54:16.502562Z TRACE ThreadId(22) supervisor.init_subscriptions:event_source.init_subscriptions{chain=osmosis-1}: subscribed to all queries
2024-07-01T11:54:16.502714Z DEBUG ThreadId(31) event_source.websocket{chain=osmosis-1}: collecting events
2024-07-01T11:54:16.502834Z DEBUG ThreadId(01) worker.batch{chain=cosmoshub-4}: spawning task
2024-07-01T11:54:16.502926Z DEBUG ThreadId(01) worker.batch{chain=osmosis-1}: spawning task
2024-07-01T11:54:16.502952Z DEBUG ThreadId(01) worker.cmd: spawning task
2024-07-01T11:54:16.502994Z DEBUG ThreadId(01) cleanup_worker: spawning task
2024-07-01T11:54:16.503094Z  INFO ThreadId(01) Hermes has started
2024-07-01T11:54:16.889123Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:17.389731Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:17.389773Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395791
2024-07-01T11:54:17.556227Z  INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}: clearing packets
2024-07-01T11:54:17.702882Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:19.398574Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:19.799829Z DEBUG ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}: sequence numbers of unreceived packets to send to the destination chain out of the ones with commitments on the source chain dst_chain=cosmoshub-4 src_chain=osmosis-1 total=23757 sequences=3395732..=3396057, 3396059..=3396063, 3396065..=3396109, 3396111..=3396210, 3396212..=3396312, 3396315..=3396458, 3396460..=3396662, 3396664..=3396712, 3396715..=3396863, 3396865..=3396902, 3396904..=3397231, 3397233..=3397251, 3397255..=3397675, 3397677..=3397709, 3397711..=3398118, 3398120..=3398932, 3398934..=3399103, 3399343..=3399367, 3399369..=3399419, 3399421..=3399425, 3399427..=3399521, 3399523..=3399680, 3399682..=3399746, 3399748..=3399833, 3399836..=3399885, 3399888..=3399937, 3399939..=3400017, 3400019..=3400039, 3400041..=3400140, 3400142..=3400191, 3400193..=3400243, 3400246..=3400402, 3400405..=3400498, 3400500..=3400531, 3400533..=3400642, 3400644..=3400670, 3400672..=3400719, 3400721..=3400881, 3400883..=3401022, 3401024..=3401503, 3401505..=3401826, 3401828..=3402110, 3402112..=3402546, 3402548..=3402568, 3402571..=3402721, 3402724..=3402774, 3402776..=3402837, 3402839..=3402876, 3402878..=3403077, 3403081..=3403283, 3403287..=3403386, 3403388..=3403438, 3403440..=3403674, 3403676..=3403681, 3403683..=3403696, 3403698..=3403749, 3403751..=3403952, 3403954..=3403974, 3403976..=3404131, 3404133..=3404147, 3404149..=3404151, 3404153..=3404156, 3404158..=3404179, 3404181..=3404296, 3404298..=3404333, 3404335..=3404360, 3404362..=3404405, 3404407..=3404418, 3404420..=3404435, 3404437..=3404448, 3404450..=3404711, 3404713..=3404737, 3404739..=3404816, 3404818..=3404820, 3404822..=3404879, 3404881..=3404928, 3404930..=3404968, 3404970..=3404998, 3405000..=3405129, 3405131..=3405195, 3405197..=3405256, 3405258..=3405347, 3405349..=3405389, 3405391..=3405397, 3405399..=3405438, 3405440..=3405577, 3405579..=3405884, 3405886..=3405904, 3405906..=3405961, 3405963..=3405966, 3405968..=3406042, 3406044..=3406183, 3406185..=3406189, 3406191..=3406280, 3406282..=3406288, 3406290..=3406299, 3406301..=3406340, 3406342..=3406351, 3406353..=3406397, 3406399..=3406402, 3406404..=3406409, 3406411..=3406493, 3406495..=3406502, 3406504..=3406507, 3406509..=3406541, 3406543..=3406630, 3406632..=3406733, 3406735..=3406791, 3406793..=3406811, 3406813..=3406933, 3406935..=3406955, 3406957..=3406958, 3406960..=3407189, 3407193..=3407241, 3407243..=3407276, 3407278..=3407293, 3407295..=3407345, 3407347..=3407396, 3407399..=3407437, 3407439..=3407549, 3407551..=3407555, 3407557..=3407601, 3407603..=3407618, 3407620..=3407662, 3407664..=3407877, 3407879..=3407934, 3407936..=3408006, 3408008..=3408057, 3408059..=3408105, 3408107..=3408136, 3408138..=3408155, 3408157..=3408159, 3408161..=3408210, 3408214..=3408263, 3408265..=3408265, 3408267..=3408280, 3408282..=3408414, 3408418..=3408466, 3408468..=3408507, 3408509..=3408517, 3408519..=3408567, 3408569..=3408638, 3408640..=3408668, 3408670..=3408769, 3408771..=3408872, 3408874..=3409026, 3409028..=3409069, 3409071..=3409153, 3409155..=3409335, 3409337..=3409385, 3409387..=3409407, 3409409..=3409427, 3409429..=3409429, 3409431..=3409575, 3409577..=3409682, 3409684..=3409692, 3409694..=3409785, 3409787..=3409826, 3409828..=3410147, 3410149..=3410160, 3410162..=3410268, 3410270..=3410283, 3410285..=3410301, 3410304..=3410565, 3410567..=3410709, 3410711..=3410717, 3410719..=3410812, 3410814..=3411214, 3411216..=3411222, 3411224..=3411431, 3411433..=3411574, 3411576..=3411630, 3411633..=3411924, 3411926..=3412294, 3412296..=3412341, 3412343..=3412599, 3412601..=3412727, 3412729..=3412837, 3412839..=3413090, 3413092..=3413154, 3413156..=3413182, 3413184..=3413206, 3413208..=3413309, 3413314..=3413367, 3413369..=3413401, 3413403..=3413633, 3413635..=3413668, 3413670..=3413918, 3413920..=3414459, 3414461..=3414473, 3414475..=3414577, 3414580..=3414628, 3414630..=3414777, 3414779..=3415100, 3415102..=3415387, 3415389..=3415458, 3415460..=3415527, 3415529..=3415685, 3415687..=3415967, 3415969..=3416011, 3416013..=3416276, 3416278..=3416377, 3416380..=3416453, 3416455..=3416711, 3416713..=3416883, 3416885..=3416910, 3416912..=3416933, 3416935..=3416985, 3416987..=3417016, 3417018..=3417087, 3417089..=3417139, 3417141..=3417181, 3417183..=3417248, 3417250..=3417293, 3417295..=3417342, 3417345..=3417390, 3417392..=3417445, 3417449..=3417497, 3417499..=3417549, 3417551..=3417582, 3417584..=3417601, 3417603..=3417613, 3417615..=3417702, 3417704..=3417754, 3417756..=3417802, 3417804..=3417804, 3417808..=3417908, 3417910..=3418338, 3418340..=3418462, 3418464..=3418515, 3418517..=3418566, 3418568..=3418944, 3418946..=3419268, 3419270..=3419340, 3419342..=3419365, 3419367..=3419576, 3419578..=3419595, 3419597..=3419616, 3419618..=3419623, 3419625..=3419675, 3423615..=3423617, 3423619..=3423620, 3423622..=3423623, 3423625..=3423627, 3423629..=3423629, 3423631..=3423634, 3423636..=3423644, 3423646..=3423646, 3423648..=3423657, 3423660..=3423664, 3423666..=3423668, 3423670..=3423692, 3423695..=3423698, 3423700..=3423708, 3423712..=3423712, 3423715..=3423719, 3423721..=3423723, 3423725..=3423729, 3423731..=3423731, 3423733..=3423741, 3423743..=3423751, 3423753..=3423759, 3423761..=3423762, 3423766..=3423771, 3423773..=3423780, 3423782..=3423787, 3423789..=3423789, 3423791..=3423794, 3423796..=3423799, 3423801..=3423811, 3423813..=3423814, 3423816..=3423820, 3423822..=3423825, 3423827..=3423829, 3423831..=3423831, 3423833..=3423834, 3423836..=3423839, 3423841..=3423843, 3423845..=3423854, 3423856..=3423867, 3423869..=3423869, 3423871..=3423884, 3423886..=3423888, 3423890..=3423893, 3423896..=3423900, 3423902..=3423906, 3423908..=3423910, 3423912..=3423917, 3423919..=3423927, 3423931..=3423931, 3423933..=3423946, 3423949..=3423950, 3423952..=3423954, 3423956..=3423968, 3423970..=3423992, 3423994..=3423995, 3423998..=3424005
2024-07-01T11:54:19.971781Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:20.091893Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:20.092127Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:1006445, path:channel-6994/transfer->channel-2/transfer, toh:0-1794340, tos:2024-07-01T12:04:17.128890343Z) })
2024-07-01T11:54:20.092143Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: AF2A93CA2D779B1BA1668E429C30E340626AE0F09CBFFFB8CA3F64172DBAF2FF event="SendPacket"
2024-07-01T11:54:20.093191Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:20.093700Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1829, client_type: ClientType(07-tendermint), consensus_height: 1-12798982 } })
2024-07-01T11:54:20.214064Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:20.214607Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:174631, path:channel-1/transfer->channel-199/transfer, toh:1-17395935, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:20.400009Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:54:20.400055Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102090
2024-07-01T11:54:20.716422Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=4
2024-07-01T11:54:20.716462Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 4 events chain=osmosis-1 count=4 height=1-17395792
2024-07-01T11:54:22.374852Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:22.706440Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:22.875697Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:22.875741Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395793
2024-07-01T11:54:23.695084Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:23.937195Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:23.937853Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:397246, path:channel-326/transfer->channel-5/transfer, toh:1-9718212, tos:NoTimestamp)})
2024-07-01T11:54:23.938388Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:23.938941Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-2119, client_type: ClientType(07-tendermint), consensus_height: 1-9718068 } })
2024-07-01T11:54:24.440376Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:54:24.440416Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395794
2024-07-01T11:54:25.707634Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:25.707967Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3695541, path:channel-141/transfer->channel-0/transfer, toh:no timeout, tos:2024-07-01T11:59:15.747931392Z) })
2024-07-01T11:54:25.707983Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: tx hash: 8E336A019F65AE465E67BA1B1AFCE19135F82F1E86881999813FE277DB55B9C3 event="SendPacket"
2024-07-01T11:54:25.711726Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_client' chain=cosmoshub-4
2024-07-01T11:54:25.712230Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-769, client_type: ClientType(07-tendermint), consensus_height: 1-20018536 } })
2024-07-01T11:54:25.716790Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:25.717310Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:110217, path:channel-343/transfer->channel-0/transfer, toh:no timeout, tos:2078-12-30T23:57:56.700908103Z)})
2024-07-01T11:54:25.717339Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463359, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T12:03:50.068Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:25.719194Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:26.279973Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:26.401827Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:26.402504Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3086, client_type: ClientType(07-tendermint), consensus_height: 1-2210434 } })
2024-07-01T11:54:26.402632Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:26.402992Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3091, client_type: ClientType(07-tendermint), consensus_height: 1-5958504 } })
2024-07-01T11:54:26.523328Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:26.524031Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463360, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:13.1479427Z) })
2024-07-01T11:54:26.524046Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: F9DF32F67FF5E2D4E7FB0FE7D90A30C594552ACEDA2802EF1607E7FA36C67C55 event="SendPacket"
2024-07-01T11:54:26.524073Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:318670, path:channel-2/transfer->channel-19774/transfer, toh:1-17395940, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:26.524137Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:26.524510Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted TimeoutPacket(TimeoutPacket { packet: seq:4773, path:channel-20082/transfer->channel-4/transfer, toh:1-5958502, tos:NoTimestamp)})
2024-07-01T11:54:26.525132Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:26.525607Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3012, client_type: ClientType(07-tendermint), consensus_height: 0-1793341 } })
2024-07-01T11:54:26.643824Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:26.644492Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463361, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:18.8916283Z) })
2024-07-01T11:54:26.644509Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 8CE0DEFDE942706429C15CE461610533CDAA9501586C0DDA2D215EAB0B06C3B1 event="SendPacket"
2024-07-01T11:54:26.644535Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881267, path:channel-2/transfer->channel-6994/transfer, toh:1-17395914, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:26.720101Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=5
2024-07-01T11:54:26.720151Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 5 events chain=cosmoshub-4 count=5 height=4-21102091
2024-07-01T11:54:27.146505Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=9
2024-07-01T11:54:27.146550Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 9 events chain=osmosis-1 count=9 height=1-17395795
2024-07-01T11:54:27.710245Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:31.720571Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:31.901267Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:32.143341Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:32.144010Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-2119, client_type: ClientType(07-tendermint), consensus_height: 1-9718069 } })
2024-07-01T11:54:32.144682Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:32.145313Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:511086, path:channel-5/transfer->channel-326/transfer, toh:1-17395939, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:32.645693Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:54:32.645736Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395796
2024-07-01T11:54:32.713482Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:32.722606Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:54:32.722652Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102092
2024-07-01T11:54:33.521836Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:33.644836Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:33.645565Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1, client_type: ClientType(07-tendermint), consensus_height: 4-21102092 } })
2024-07-01T11:54:33.765331Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:33.766062Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3695541, path:channel-141/transfer->channel-0/transfer, toh:no timeout, tos:2024-07-01T11:59:15.747931392Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:33.766089Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463359, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T12:03:50.068Z)})
2024-07-01T11:54:33.766317Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:33.766562Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3009, client_type: ClientType(07-tendermint), consensus_height: 1-19389198 } })
2024-07-01T11:54:33.884566Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:33.884951Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463362, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:26.0441795Z) })
2024-07-01T11:54:33.884962Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 54DEE69F6F4F6DB83ADC305634E247B879938C2680A3D04FD31A57D8D890F4BC event="SendPacket"
2024-07-01T11:54:33.884982Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:43562, path:channel-3/transfer->channel-6787/transfer, toh:1-17395945, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:34.386168Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=7
2024-07-01T11:54:34.386203Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 7 events chain=osmosis-1 count=7 height=1-17395797
2024-07-01T11:54:34.924276Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:35.045451Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:35.046012Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-0, client_type: ClientType(07-tendermint), consensus_height: 2-16980827 } })
2024-07-01T11:54:35.048543Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:35.049145Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:286454, path:channel-9/transfer->channel-1/transfer, toh:1-17395940, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:35.550155Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:54:35.550204Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395798
2024-07-01T11:54:37.020740Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:37.522057Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:37.522113Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395799
2024-07-01T11:54:37.717023Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:37.937172Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:37.937864Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463360, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:13.1479427Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:37.937900Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463361, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:18.8916283Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:37.943833Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_client' chain=cosmoshub-4
2024-07-01T11:54:37.944493Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-259, client_type: ClientType(07-tendermint), consensus_height: 1-17395796 } })
2024-07-01T11:54:37.947579Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:37.948259Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3695541, path:channel-141/transfer->channel-0/transfer, toh:no timeout, tos:2024-07-01T11:59:15.747931392Z)})
2024-07-01T11:54:37.948300Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463362, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:26.0441795Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:37.951240Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_client' chain=cosmoshub-4
2024-07-01T11:54:37.951837Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-259, client_type: ClientType(07-tendermint), consensus_height: 1-17395798 } })
2024-07-01T11:54:37.954562Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:38.787607Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:38.788290Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1562, client_type: ClientType(07-tendermint), consensus_height: 1-14294667 } })
2024-07-01T11:54:38.908659Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:38.909341Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:695056, path:channel-75/transfer->channel-0/transfer, toh:no timeout, tos:2024-07-01T11:59:16.5936522Z)})
2024-07-01T11:54:38.955765Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=7
2024-07-01T11:54:38.955801Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 7 events chain=cosmoshub-4 count=7 height=4-21102093
2024-07-01T11:54:39.030626Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:39.032008Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:39.032554Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3012, client_type: ClientType(07-tendermint), consensus_height: 0-1793342 } })
2024-07-01T11:54:39.152014Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:39.152693Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463363, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:34.7399004Z) })
2024-07-01T11:54:39.152714Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 54F1BAB866492C143F5644C5C0931B022220FEA052BCE85414CEC0031ED000DC event="SendPacket"
2024-07-01T11:54:39.152743Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881268, path:channel-2/transfer->channel-6994/transfer, toh:no timeout, tos:2024-07-01T11:58:34.739882496Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:39.152789Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463364, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:57.311676Z) })
2024-07-01T11:54:39.152795Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 54F1BAB866492C143F5644C5C0931B022220FEA052BCE85414CEC0031ED000DC event="SendPacket"
2024-07-01T11:54:39.152814Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881269, path:channel-2/transfer->channel-6994/transfer, toh:no timeout, tos:2024-07-01T11:58:57.311660544Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:39.152835Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881270, path:channel-2/transfer->channel-6994/transfer, toh:1-17395936, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:39.152849Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:1006444, path:channel-6994/transfer->channel-2/transfer, toh:no timeout, tos:2024-07-01T11:59:01.6933353Z)})
2024-07-01T11:54:39.654329Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=10
2024-07-01T11:54:39.654367Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 10 events chain=osmosis-1 count=10 height=1-17395800
2024-07-01T11:54:40.610769Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:41.112279Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:41.112323Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395801
2024-07-01T11:54:42.721940Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:43.413374Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:43.535268Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:43.535854Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3012, client_type: ClientType(07-tendermint), consensus_height: 0-1793343 } })
2024-07-01T11:54:43.655875Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:43.656503Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:1006445, path:channel-6994/transfer->channel-2/transfer, toh:0-1794340, tos:2024-07-01T12:04:17.128890343Z)})
2024-07-01T11:54:43.871890Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:44.158277Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:54:44.158321Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395802
2024-07-01T11:54:44.873473Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:54:44.873515Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102094
2024-07-01T11:54:45.817717Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:46.060133Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:46.060905Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1, client_type: ClientType(07-tendermint), consensus_height: 4-21102094 } })
2024-07-01T11:54:46.181104Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:46.181919Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463360, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:13.1479427Z)})
2024-07-01T11:54:46.181958Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463361, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:18.8916283Z)})
2024-07-01T11:54:46.181973Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463362, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:26.0441795Z)})
2024-07-01T11:54:46.682686Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=5
2024-07-01T11:54:46.682734Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 5 events chain=osmosis-1 count=5 height=1-17395803
2024-07-01T11:54:47.289092Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:47.289316Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:47.289479Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:272019, path:channel-19774/transfer->channel-2/transfer, toh:1-2210587, tos:NoTimestamp) })
2024-07-01T11:54:47.289491Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 46AD07326AD57EE7CF57C89E26A364B7BC8CB87458B33BD1EEFDC65B4F6A09D7 event="SendPacket"
2024-07-01T11:54:47.726305Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:47.790507Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=2
2024-07-01T11:54:47.790561Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 2 events chain=osmosis-1 count=2 height=1-17395804
2024-07-01T11:54:50.270143Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:50.770888Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:50.770931Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395805
2024-07-01T11:54:50.805191Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_client' chain=cosmoshub-4
2024-07-01T11:54:50.805985Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-259, client_type: ClientType(07-tendermint), consensus_height: 1-17395801 } })
2024-07-01T11:54:50.811260Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: message.module = 'ibc_channel' chain=cosmoshub-4
2024-07-01T11:54:50.811780Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463363, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:34.7399004Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:50.811808Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:3463364, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:57.311676Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 65, 81, 61, 61, 34, 125 ] })
2024-07-01T11:54:50.814157Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:51.815020Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=4
2024-07-01T11:54:51.815070Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 4 events chain=cosmoshub-4 count=4 height=4-21102095
2024-07-01T11:54:52.547690Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:52.668038Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:52.668276Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:1006446, path:channel-6994/transfer->channel-2/transfer, toh:0-1793493, tos:NoTimestamp) })
2024-07-01T11:54:52.668299Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 621949C5E5E3E3979153E8CCB47E0678B392711410C426545BD133CEE8FDF84D event="SendPacket"
2024-07-01T11:54:52.730567Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:53.169163Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=2
2024-07-01T11:54:53.169208Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 2 events chain=osmosis-1 count=2 height=1-17395806
2024-07-01T11:54:53.571382Z TRACE ThreadId(22) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}:query_send_packet_events{chain=osmosis-1 height=<=1-17395790 sequences=[3395732, 3395733, 3395734, 3395735, 3395736, 3395737, 3395738, 3395739, 3395740, 3395741, 3395742, 3395743, 3395744, 3395745, 3395746, 3395747, 3395748, 3395749, 3395750, 3395751, 3395752, 3395753, 3395754, 3395755, 3395756, 3395757, 3395758, 3395759, 3395760, 3395761, 3395762, 3395763, 3395764, 3395765, 3395766, 3395767, 3395768, 3395769, 3395770, 3395771, 3395772, 3395773, 3395774, 3395775, 3395776, 3395777, 3395778, 3395779, 3395780, 3395781]}: start_block_events []
2024-07-01T11:54:53.571424Z TRACE ThreadId(22) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}:query_send_packet_events{chain=osmosis-1 height=<=1-17395790 sequences=[3395732, 3395733, 3395734, 3395735, 3395736, 3395737, 3395738, 3395739, 3395740, 3395741, 3395742, 3395743, 3395744, 3395745, 3395746, 3395747, 3395748, 3395749, 3395750, 3395751, 3395752, 3395753, 3395754, 3395755, 3395756, 3395757, 3395758, 3395759, 3395760, 3395761, 3395762, 3395763, 3395764, 3395765, 3395766, 3395767, 3395768, 3395769, 3395770, 3395771, 3395772, 3395773, 3395774, 3395775, 3395776, 3395777, 3395778, 3395779, 3395780, 3395781]}: tx_events []
2024-07-01T11:54:53.571432Z TRACE ThreadId(22) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}:query_send_packet_events{chain=osmosis-1 height=<=1-17395790 sequences=[3395732, 3395733, 3395734, 3395735, 3395736, 3395737, 3395738, 3395739, 3395740, 3395741, 3395742, 3395743, 3395744, 3395745, 3395746, 3395747, 3395748, 3395749, 3395750, 3395751, 3395752, 3395753, 3395754, 3395755, 3395756, 3395757, 3395758, 3395759, 3395760, 3395761, 3395762, 3395763, 3395764, 3395765, 3395766, 3395767, 3395768, 3395769, 3395770, 3395771, 3395772, 3395773, 3395774, 3395775, 3395776, 3395777, 3395778, 3395779, 3395780, 3395781]}: end_block_events []
2024-07-01T11:54:53.571511Z  INFO ThreadId(26) spawn:chain{chain=osmosis-1}:client{client=07-tendermint-1}:connection{connection=connection-1}:channel{channel=channel-0}:worker.packet.cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 17395791 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 17395790 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-17395790}: pulled packet data for 0 events out of 50 sequences: 3395732..=3395781; events.total=23757 events.left=23707
2024-07-01T11:54:54.244497Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:54.746271Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:54.746306Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395807
2024-07-01T11:54:56.793982Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:54:57.393693Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:57.733578Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:54:57.794935Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:54:57.794975Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102096
2024-07-01T11:54:57.894908Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:54:57.894980Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395808
2024-07-01T11:54:59.155443Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:54:59.398303Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:54:59.399067Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-1, client_type: ClientType(07-tendermint), consensus_height: 4-21102096 } })
2024-07-01T11:54:59.519024Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:54:59.519828Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463363, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:34.7399004Z)})
2024-07-01T11:54:59.519862Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:3463364, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:58:57.311676Z)})
2024-07-01T11:55:00.020810Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=4
2024-07-01T11:55:00.020851Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 4 events chain=osmosis-1 count=4 height=1-17395809
2024-07-01T11:55:01.936597Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:02.178261Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:55:02.178936Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted AcknowledgePacket(AcknowledgePacket { packet: seq:272019, path:channel-19774/transfer->channel-2/transfer, toh:1-2210587, tos:NoTimestamp)})
2024-07-01T11:55:02.179565Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:55:02.180152Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3086, client_type: ClientType(07-tendermint), consensus_height: 1-2210440 } })
2024-07-01T11:55:02.681642Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=3
2024-07-01T11:55:02.681679Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 3 events chain=osmosis-1 count=3 height=1-17395810
2024-07-01T11:55:02.737250Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:55:02.881140Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:55:03.325513Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:03.826986Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:03.827028Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395811
2024-07-01T11:55:03.882122Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:55:03.882165Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102097
2024-07-01T11:55:05.150059Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:05.651584Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:05.651654Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395812
2024-07-01T11:55:07.742155Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:55:08.249223Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:08.249821Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:55:08.250176Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-2704, client_type: ClientType(07-tendermint), consensus_height: 1-7409828 } })
2024-07-01T11:55:08.371065Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:55:08.371562Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:69947, path:channel-38946/transfer->channel-1/transfer, toh:no timeout, tos:2024-07-01T12:00:00.208658Z) })
2024-07-01T11:55:08.371580Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 7930F302C61795FB709D7F2D92BC20FB6A99DC4F5C4DC40BA7567FA36B7064E9 event="SendPacket"
2024-07-01T11:55:08.371606Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:220305, path:channel-1/transfer->channel-750/transfer, toh:1-17395959, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:55:08.492184Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:55:08.492856Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3086, client_type: ClientType(07-tendermint), consensus_height: 1-2210441 } })
2024-07-01T11:55:08.494246Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:55:08.494882Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:1006447, path:channel-6994/transfer->channel-2/transfer, toh:no timeout, tos:2024-07-01T11:59:53.5405522Z) })
2024-07-01T11:55:08.494902Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: 093B3CE7D6F773A00B5FFB48E0804BFD900FB165E8928E41AED88DC1DE5F0C95 event="SendPacket"
2024-07-01T11:55:08.494929Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:318671, path:channel-2/transfer->channel-19774/transfer, toh:1-17395957, tos:NoTimestamp), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:55:08.613252Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_client' chain=osmosis-1
2024-07-01T11:55:08.613846Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: extracted ibc_client event UpdateClient(UpdateClient { Attributes { client_id: 07-tendermint-3012, client_type: ClientType(07-tendermint), consensus_height: 0-1793345 } })
2024-07-01T11:55:08.615150Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: message.module = 'ibc_channel' chain=osmosis-1
2024-07-01T11:55:08.615724Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:3463365, path:channel-0/transfer->channel-141/transfer, toh:no timeout, tos:2024-07-01T11:59:29.158915Z) })
2024-07-01T11:55:08.615740Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: A1263FA3B6B3F69BBB13C4DA90122CBEB96D89F06D414F7F7B2A521F58A1F265 event="SendPacket"
2024-07-01T11:55:08.615766Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881271, path:channel-2/transfer->channel-6994/transfer, toh:no timeout, tos:2024-07-01T11:59:29.158901248Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:55:08.615798Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted SendPacket(SendPacket { packet: seq:1006448, path:channel-6994/transfer->channel-2/transfer, toh:no timeout, tos:2024-07-01T12:11:44.661126659Z) })
2024-07-01T11:55:08.615803Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: tx hash: A1263FA3B6B3F69BBB13C4DA90122CBEB96D89F06D414F7F7B2A521F58A1F265 event="SendPacket"
2024-07-01T11:55:08.615817Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}:ibc_channel event: extracted WriteAcknowledgement(WriteAcknowledgement { packet: seq:881272, path:channel-2/transfer->channel-6994/transfer, toh:no timeout, tos:2024-07-01T12:06:37.589Z), ack: [ 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 34, 101, 121, 74, 106, 98, 50, 53, 48, 99, 109, 70, 106, 100, 70, 57, 121, 90, 88, 78, 49, 98, 72, 81, 105, 79, 109, 53, 49, 98, 71, 119, 115, 73, 109, 108, 105, 89, 49, 57, 104, 89, 50, 115, 105, 79, 105, 74, 108, 101, 85, 112, 53, 87, 108, 104, 79, 77, 87, 74, 73, 85, 87, 108, 80, 97, 85, 112, 67, 86, 86, 81, 119, 79, 85, 108, 117, 77, 68, 48, 105, 102, 81, 61, 61, 34, 125 ] })
2024-07-01T11:55:09.116935Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=12
2024-07-01T11:55:09.116978Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 12 events chain=osmosis-1 count=12 height=1-17395813
2024-07-01T11:55:09.133164Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: received an RPC event: tm.event = 'NewBlock' chain=cosmoshub-4
2024-07-01T11:55:09.281800Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:09.783435Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:09.783483Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395814
2024-07-01T11:55:10.134346Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: assembled batch chain=cosmoshub-4 len=1
2024-07-01T11:55:10.134387Z TRACE ThreadId(30) event_source.websocket{chain=cosmoshub-4}: broadcasting batch of 1 events chain=cosmoshub-4 count=1 height=4-21102098
2024-07-01T11:55:11.786030Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:12.286728Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:12.286772Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395815
2024-07-01T11:55:12.746255Z TRACE ThreadId(24) spawn:chain{chain=cosmoshub-4}:wallet{chain=cosmoshub-4}: wallet balance amount=6742630 denom=uatom account=cosmos17ndx5qfku28ymxgmq6zq4a6d02dvpfjj5yu8j9
2024-07-01T11:55:13.097484Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: received an RPC event: tm.event = 'NewBlock' chain=osmosis-1
2024-07-01T11:55:13.599145Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: assembled batch chain=osmosis-1 len=1
2024-07-01T11:55:13.599183Z TRACE ThreadId(31) event_source.websocket{chain=osmosis-1}: broadcasting batch of 1 events chain=osmosis-1 count=1 height=1-17395816
ljoss17 commented 2 days ago

Ok so it seems the clear_on_start does indeed block the relaying. This is because this part should clear any pending packets having the normal relaying run concurrently will likely generate redundant packet relaying. I was under the impression that the clear_interval shouldn't block the relaying, but I will look into this as well.

freak12techno commented 2 days ago

@ljoss17 can packets clearing interfere with Hermes functioning properly? (my thought was they might submit transaction and mess with account sequence, but not sure if if it's the case or if there's more caveats)

ljoss17 commented 2 days ago

Clear on start will first finish trying to clear all packets, due to the high number of packets in your case the instance is "stuck" at the clearing phase and that is why it isn't relaying. Could you try adding the pending packets to the excluded_sequences and see if the relaying works even with the clearing enabled?

For interval clearing, if it happens concurrently to the relaying, one of the two tentatives to relay might result in redundant packet error (either the relaying is faster or the clearing) if there is a new packet when clearing is triggered.

freak12techno commented 2 days ago

@ljoss17 one more thing I just discovered: so I used the same config as above, with trace logging, but without clear_on_start (but with clear_interval), and it seems to also not relay anything since the interval clearing was executed. It did some transactions when starting up, but then it stopped, I usually see way more broadcast errors when the app is working correctly, here's what I see now:

изображение

Here's the same metric for the last 6 hours:

изображение

Basically it has 3 periods:

Here are the trace logs for the period since 16:30, hope it would be of any help: https://gist.github.com/freak12techno/d6688f271f34ec6dc70053d938375062

For excluded_sequences, gonna check it out a bit later.

ljoss17 commented 2 days ago

Interesting, thank you very much for the additional information!

freak12techno commented 2 days ago

@ljoss17 just to clarify: my biggest concern here is not the packets being not relayed correctly (this is likely either the node being misconfigured, or blocks being pruned), but rather Hermes not doing anything once the packets are being cleared. I think that might be disturbing if let's say there are indeed some packets being pending, a lot of them, so clearing would take some time, and during that time Hermes won't function properly. This specific case is likely something I can resolve; problem is once this happens again, it'll (apparently) block Hermes from doing anything else.

ljoss17 commented 2 days ago

I see, for the clear on start I will look into it to see if there is a way to cleanly split the clearing part and relaying part so that both can run concurrently without interference. For the clear interval it might be a bit more tricky but it should also be possible.

If the clearing and standard relaying have a clear separation it will be easier to have them run concurrently.

freak12techno commented 1 day ago

@ljoss17 okay so, the excluded_sequences config part did the trick apparently, therefore I confirm it's working with it.

изображение