Come up with better agent FallbackProvider liveness heuristic

[daniel-savu]

Problem

In the fallback provider, the current rule for deciding if the current provider should be deprioritized is: "have there been no blocks seen by this provider in the last 2 mins"? This assumption is imperfect because it assumes that chains always produce blocks. This turns out not to be the case for Arbitrum Orbit chains, so we should come up with a better rule. From the Arbitrum docs:

block creation depends entirely on chain usage, meaning that blocks are only produced when there are transactions to sequence

Unless this heuristic is improved, we can't use a fallback provider for Arbitrum Orbit chains.

Solution

Describe the solution you'd like One idea we had was to query all providers at once to see what the max seen block is, and only deprioritize if the current provider is behind that. There could be better solutions though.

[tkporter]

Just wanna confirm that fallback provider is still usable but just has ugly logs?

[tkporter]

If so do you think https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3226 would cover this?

[tkporter]

note we also had this for inevm for similar reasons https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/3235, I closed this as 3226 would fix

[daniel-savu]

If so do you think #3226 would cover this?

@tkporter It would, assuming we have more than one working RPC url. How realistic would it be to assume this for new rollups?

[tkporter]

hahah good point probably not likely

[avious00]

hey @daniel-savu is this other issue https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2570 a dupe of this issue?

[tkporter]

@avious00 see my comment https://github.com/hyperlane-xyz/hyperlane-monorepo/issues/2570#issuecomment-1976416859, I vote closing that one in favor of this, they're slightly different