Improving RPC health check robustness

Keyrxng commented 4 months ago

The handler is now a proxy which routes on-chain read/writes through it handling retry and reselection logic. The original issue was that eth_call is not supported was being thrown by some endpoints due to network sync issues on the provider's part. While the proxy approach resolves this indirectly it could be beneficial to deal with the underlying issue.

While we can do our best to serve a healthy provider from the beginning it's possible a user call may still fail although with an improved health check this is far less likely and with our proxy the other endpoints should be also be perfectly healthy

Option 1

Current: fetch block number and ensure it's valid data Proposed: block number + permit2 read op

The health check can be improved by a number of methods but one that makes sense to me is repeating the eth_call which highlighted the issue originally

The downsides with this is:

That it has been proven that the same eth_call can and will fail intermittently even with some of the best RPCs for that network, so would require best-of 3 calls for example
If checks are 'fragile' like this then it's entirely possible we encounter a 'no providers found' scenario in error
The more checks added the higher the chance the above happens
It's still possible for a call to fail due to provider error
The more calls we add the higher the network demand on load
^ the slower we are to returning a provider
Our data/provider is static after load unless re-rendered/recalled

The upsides with this is:

We are guaranteed healthy endpoints
Less chance retry logic will be executed

Option 2

Adopt a polling-based approach, first gather our healthy endpoints and then poll them according to config settings ensuring that they remain healthy and pop them if they fault. We could collect the first pool using just the block number check and then any that make it to round two are polled with eth_call read ops.

Downsides:

Perpetual read ops called
runtime network demand depending on settings and amount of providers to poll
We may hit call limits with our check and serve it for the user to get rate limited (low chance)

Upsides:

Dynamic health status
Potentially makes retry logic obsolete
network load demand is on par with option 1

Option two would require a pretty big overhaul of the current system as opposed to option one

Any other suggestions are welcome

0x4007 commented 3 months ago

2 seems more robust so that seems like the better choice.

surafeldev commented 1 month ago

/start

surafeldev commented 1 month ago

/help