omni-network / omni

Monorepo for Omni node, contracts and other related tools
https://omni.network
GNU General Public License v3.0
86 stars 51 forks source link

add retries to avoid relatively benign WARN logs #1415

Open arajasek opened 3 months ago

arajasek commented 3 months ago

Problem: there are 2 aspects of the system that can generate WARN logs a little too zealously. These are the Xchain Provider's FetchBatch method, which may fail due to being very slightly behind another node, and the relayer's send method, which might have submissions be reverted on very dynamic chains due to out of gas (eg. here). In both cases, the components upstream will retry anyway, and likely succeed, but still emit a failure log / metric.

Solution: given that this is relatively benign, let's retry a few times within these methods before erroring.

arajasek commented 2 months ago

I'm not sure I like doing this for the relayer's Send method. We should probably capture that failure immediately.