search

cosmos / relayer

An IBC relayer for ibc-go
Apache License 2.0
390 stars 1.71k forks source link

Flush not complete #1464

Open avolon42x opened 6 months ago

avolon42x commented 6 months ago

I saw this error in my relayer and it keeps coming back:

Flush not complete {"error": "failed to enqueue pending messages for flush: no ibc messages found for write_acknowledgement query: write_acknowledgement.packet_dst_channel='channel-13' AND write_acknowledgement.packet_sequence='58128'"}

rly tx flush does not seems to resolve the problem. Any ideas?

Reecepbcups commented 5 months ago

Hey @avolon42x did you get this resolved? It's possible that packet was already relayed or the packet was never created properly. Can you confirm if this was the case?

(it's also possible you have a relay max size limit set, and thus packets of select size are not relayed)

avolon42x commented 5 months ago

Hey @Reecepbcups - no haven't solved it yet. I think it must have been something like not being created properly. Any other way to clear/flush this channel?

Reecepbcups commented 5 months ago

@avolon42x There is a new rly tx flush --stuck-packet-height-start flag which helps to get previous heights and push stuck packets. Mess with this and see if it can be pushed through

avolon42x commented 5 months ago

I tried several heights and ranges with the start and end-flag but it's still not going through.. What else is possible?

agouin commented 2 months ago

I believe this is due to the relayer trying to flush out old packets that were previously filtered (e.g. the IBC spam packets). Once the height that contains the packet commitment is pruned from IAVL on the node that rly is connected to, rly cannot query it anymore and the flush will get stuck like this.

This is both a relayer level issue and a chain level issue, because if all relayers are configured to ignore certain packets, then the chain should probably also remove them from state with an upgrade migration.

TheMarstonConnell commented 1 month ago

Dealing with this same thing now, any fix?