indexsupply / shovel

An Ethereum to Postgres indexer
https://indexsupply.com/shovel
MIT License
172 stars 22 forks source link

multiple rpc urls for eth source #257

Closed ryandotsmith closed 4 months ago

ryandotsmith commented 4 months ago

If an RPC is failing for whatever reason it would be nice for Shovel to use a backup RPC URL.

I am thinking we can do something like this:

"eth_sources": [
  {
    "name": "main",
    "urls": ["https://foo", "https://bar"],
    "timeout": "1s"
  }
]

As Shovel downloads data, it will evenly balance requests across the URLs. If a request fails from non-200 or a timeout, the request will be retried with the next url in the list.

ryandotsmith commented 4 months ago

Another idea is to explicitly provide a "backup" url. I like this approach less because it is common for backup resources to be flawed since they are rarely used.

ryandotsmith commented 4 months ago

Shovel should expose metrics and logs with data about QoS for each url

ryandotsmith commented 4 months ago

Added in #259