chifra cmd - should have backup rpcEndpoints in case of downtime

[tjayrush]

See https://github.com/TrueBlocks/trueblocks-core/issues/3424

~Not a nice thing to do, but we could, conceivably use this list~

~https://chainlist.org/chain/1~

~and 100 go routines to speed up the processing of remote data access by splitting the queries across 100 remote RPCs~

[dszlachta]

Interesting idea. It'd actually work like http://pokt.network - maybe we should investigate it? We could start small with just a tutorial on how to use Pokt with TrueBlocks, so the users could set their own Pokt accounts (250,000 relays per day are free) as RPC.

I did some research:

For not-so-active account (100 transactions)

chifra list takes 0m6.27s chifra list --count takes 0m0.034s chifra export —fmt json takes 0m4.208s 14 relays total

For active account (4000 transactions) chifra list --count takes 0m7.274s chifra export --fmt json takes 23m44.503s 4,261 relays total

Calling chifra export --fmt json again has similar results: takes 20m10.950s and 5,927 relays

The client is Erigon, and I've asked on Pokt Discord if it's always Erigon. Waiting for answer.

[tjayrush]

Can you do this?

chifra monitors --decache <address>

To clear the cache, then repeat the experiment but add --cache to the chifra export command. Then re-run a second time.

By the way -- you've almost already written the article...

[dszlachta]

Adding this comment after our discussion about supporting multiple parallel and/or backup RPCs.

I think it will be easier to start with backup RPCs and then allow parallel connections. What I think is needed:

1. Change configuration so that RPC section of each chain can be an array
2. Make Query and FromRpc and similar functions methods of Connection struct
3. Add thread-safe flag to Connection so it knows which RPC endpoint it should return (primary or backup)
4. Add Provider() getter to Connection, which simply returns primary or backup RPC endpoint
5. For long running processes (scrape) we might add a retry mechanism:

   if using backup endpoint, after N requests try primary one again

For parallel connections, we would need to add scheduler to Connection and call it in Connection.Provider() getter

[tjayrush]

Do you think you can try to complete this one? Make a PR?

On Tue, Nov 7, 2023 at 3:50 AM Dawid Szlachta @.***> wrote:

Adding this comment after our discussion about supporting multiple parallel and/or backup RPCs.

I think it will be easier to start with backup RPCs and then allow parallel connections. What I think is needed:

1. Change configuration so that RPC section of each chain can be an array
2. Make Query and FromRpc and similar functions methods of Connection struct
3. Add thread-safe flag to Connection so it knows which RPC endpoint it should return (primary or backup)
4. Add Provider() getter to Connection, which simply returns primary or backup RPC endpoint

5. For long running processes (scrape) we might add a retry mechanism:

   if using backup endpoint, after N requests try primary one again

For parallel connections, we would need to add scheduler to Connection and call it in Connection.Provider() getter

— Reply to this email directly, view it on GitHub https://github.com/TrueBlocks/trueblocks-core/issues/2857#issuecomment-1798059874, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJKXPXSZ3IVNCOFN2SANN3YDHY3ZAVCNFSM6AAAAAAW67SQUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJYGA2TSOBXGQ . You are receiving this because you authored the thread.Message ID: @.***>

-- Thomas Jay Rush http://t http://quickblocks.iorueblocks.io @.***

[dszlachta]

I think I should wait until your changes are merged?

[tjayrush]

Totally agree. So, you're saying never? Just kidding.

On Fri, Nov 10, 2023 at 9:26 AM Dawid Szlachta @.***> wrote:

I think I should wait until your changes are merged?

— Reply to this email directly, view it on GitHub https://github.com/TrueBlocks/trueblocks-core/issues/2857#issuecomment-1805835266, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJKXPROIV37IJ76I5XB2SLYDY2R7AVCNFSM6AAAAAAW67SQUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBVHAZTKMRWGY . You are receiving this because you authored the thread.Message ID: @.***>

-- Thomas Jay Rush http://t http://quickblocks.iorueblocks.io @.***

[tjayrush]

Closing for now. Will re-open.

[tjayrush]

List of chains supported by EtherScan. We can use these in chifra slurp.

https://etherscan.io/eaas

[tjayrush]

Closing one more time. Will re-open.