search

paradigmxyz / cryo

cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes
Apache License 2.0
1.22k stars 116 forks source link

Handle several RPCs provider at once #132

Open peyha opened 1 year ago

peyha commented 1 year ago

Is your feature request related to a problem? Please describe. Using an Alchemy RPC is pretty difficult with Cryo even after fine-tuning the concurrency parameters. Half of the requests exceed the compute units per second capacity.

Describe the solution you'd like One solution could be to allow users to enter several RPCs in the command (or in an env variable) and to make cryo dynamically change the RPC when it runs

Describe alternatives you've considered I don't have my own node yet

Additional context cryo is cool

aramalipoor commented 3 weeks ago

Having a local instance of erpc (maybe via dockerfile) can handle this with many required evm-focused logic for failover etc. The auto-tuner feature will decrease the max gradually if it still sees 429s (or increase if no errors), so after a while it'll settle at the right capacity.

One feature I was thinking to add is "credit unit mapping" to further optimize for this use-case 🤔

https://github.com/erpc/erpc

I'm happy to create a PR within cryo as well if maintainers think it could be valuable.

yenicelik commented 1 week ago

was looking for something like this, would be super valuable. managing a bunch of RPC keys manually here.