"Account sequence mismatch" with the same sequence multiple times

freak12techno commented 5 months ago

Summary of Bug

Getting a lot of "Account sequence mismatch" errors on osmosis-1 <=> cosmoshub-4 channel with the same sequence:

something is submitted
the logs are showing the error with account sequence mismatch
something else is happening (not sure what can influence it, but in my case it was a tx that was sent but its confirmation has timed out apparently)
something else is submitted
the logs are showing the same error with the same sequence (although it should've been updated once it failed?)

I guess when timing out it assumes that the tx went through and increases sequence while in fact the tx failed, but I may be wrong here.

Version

v1.8.0, although I've faced it a lot on 1.7.4 as well.

Steps to Reproduce

Run Hermes on cosmoshub-4 <=> osmosis-1 channel
not sure what else can be relevant here

Logs illustrating the issue (I omitted some not relevant to this issue, let me know if you need all of them):

2024-01-25T19:49:34.232408Z  WARN ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}:foreign_client.wait_and_build_update_client_with_trusted{client=cosmoshub-4->osmosis-1:07-tendermint-1 target_height=4-18877181}:foreign_client.build_update_client_with_trusted{client=cosmoshub-4->osmosis-1:07-tendermint-1 target_height=4-18877181}:foreign_client.solve_trusted_height{client=cosmoshub-4->osmosis-1:07-tendermint-1 target_height=4-18877181}: resolving trusted height from the full list of consensus state heights for target height 4-18877181; this may take a while
2024-01-25T19:49:40.968061Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}: assembled batch of 4 message(s)
2024-01-25T19:49:41.553307Z  WARN ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=ca44850d}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}:estimate_gas: failed to simulate tx, falling back on default gas because the error is potentially recoverable: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 163, got 164: incorrect account sequence [cosmos/cosmos-sdk@v0.47.5/x/auth/ante/sigverify.go:269] With gas wanted: '300000000' and gas used: '1040081' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "13449501"} }
2024-01-25T19:49:41.802723Z  WARN ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=ca44850d}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}: failed to broadcast tx because of a mismatched account sequence number, refreshing account sequence number and retrying once response=Response { code: Err(32), data: b"", log: "account sequence mismatch, expected 163, got 164: incorrect account sequence", hash: Hash::Sha256(D73369273F639559318AE388F6877F8CE03BA027B042CFC974DB3F3B7983E554) }
2024-01-25T19:49:42.036170Z  INFO ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=ca44850d}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}: refreshed account sequence number old=164 new=163
2024-01-25T19:49:43.432560Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}: response(s): 1; Error with code 22:D3BE52D57BD5C52AA5DB741E4B5D33F22A47548CAF64FC822BE9FB68D0CACF85 target_chain=osmosis-1

2024-01-25T19:49:43.432595Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}: submitted
2024-01-25T19:49:56.327062Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=e1aed966 ->Destination @4-18877185; len=1}: assembled batch of 2 message(s)
2024-01-25T19:49:57.751350Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=e1aed966 ->Destination @4-18877185; len=1}: response(s): 1; Ok:822A1D5CE4780665562EC62F4D27B3304582ED0034057A2F331FF5FE09226FF9 target_chain=osmosis-1

2024-01-25T19:49:57.751383Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=e1aed966 ->Destination @4-18877185; len=1}: submitted
2024-01-25T19:50:03.251855Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=c764403c ->Destination @4-18877186; len=2}: assembled batch of 3 message(s)
2024-01-25T19:50:04.664632Z  WARN ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=c764403c ->Destination @4-18877186; len=2}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=c764403c}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}: failed to broadcast tx because of a mismatched account sequence number, refreshing account sequence number and retrying once response=Response { code: Err(32), data: b"", log: "account sequence mismatch, expected 163, got 164: incorrect account sequence", hash: Hash::Sha256(8CCADFE27612EF2FF6859D35FC241146630B4E5C8DE2B5A649885EB286E68324) }
2024-01-25T19:50:04.902131Z  INFO ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=c764403c ->Destination @4-18877186; len=2}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=c764403c}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}: refreshed account sequence number old=164 new=163

Acceptance Criteria

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate milestone (priority) applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

freak12techno commented 5 months ago

Might be the same issue: https://github.com/informalsystems/hermes/issues/3547

adizere commented 3 months ago

Hey @freak12techno is this still happening regularly?

freak12techno commented 3 months ago

@adizere I guess

Apr 02 23:46:59 monitoring-logging hermes[634298]: 2024-04-02T21:46:59.002990Z  WARN ThreadId(25) worker.batch{chain=cosmoshub-4}:supervisor.handle_batch{chain=cosmoshub-4}:supervisor.process_batch{chain=cosmoshub-4}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=35f843ed ->Destination @4-19830002; len=1}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=35f843ed}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=6071}:estimate_gas: failed to simulate tx, falling back on default gas because the error is potentially recoverable: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 6070, got 6071: incorrect account sequence [cosmos/cosmos-sdk@v0.47.8/x/auth/ante/sigverify.go:269] With gas wanted: '300000000' and gas used: '831344' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "14653501"} }

informalsystems / hermes