[pool] Improve chain tracking on inconsistent RPC providers

[dancoombs]

Describe the feature Inconsistent RPC provider: an RPC provider whose "consistency level" block height/block hash may be different for simultaneous requests. Consistency level: On L1: "latest", "safe", "finalized". Typically tags that can be added to RPC requests.

Inconsistent RPCs cause issues for Rundler, but mostly in chain tracking. When the chain tracker sees a new block, it turns around and issues requests to get info about those logs, and then publishes that block to the pool and the builder. The pool and the builder then process the new state and my issue new RPC queries tagged with the new block hash that the chain tracker published.

Many of these subsequent requests fail with "unknown block" or a multitude of other RPC errors that are returned when the request is routed to a node that doesn't yet know about a block, or has processed a different fork of the chain.

There isn't much we can do about removing inconsistent RPCs. Many chains we support only have inconsistent RPC providers.

There are, however, things we can do to improve this.

1. In some sort of alloy middleware, we can catch errors that are related to inconsistent RPC providers, and issue retries. This will help hide inconsistency to the higher level logic, at the expense of higher latency.
2. Audit the codebase to find the places where we are tagging requests with a block number or a block hash. These are the areas that are going to be the most susceptible to issues here. We should ensure we're using the right logic in these places to get a consistent view of the chain.

[andysim3d]

it make sense to define 3 categories:

• retry-able error(like resource exhausted),
• inconsistent error(like unknown blocks)
• non-retry-able error(like invalid argument),

and config different retry policies, e.g. interval, max retry count, etc.