reece/indexer (SQL, Async)

[Reecepbcups]

Almost production ready

flow

• query endpoint with 50-100 requests at the same time. Wait for all to return a response
• get all responses (block data JSON) async
• decode all Txs (async, up to CPU core threads)
• iterate over each blocks data, and format it the way we need
• save to SQL synchronously since we already calculated everything

[Reecepbcups]

There are likely still areas of improvement, especially since I moved the amino_to_json converted from its own file into the running process.

Things to try

• async SQL
• run multiple batches at the same time (ex: thread 1: first 500 blocks, threads 2 second 500 blocks, etc)
• batchHTTP RPC queries for each of the above ^

For 10 threads = 5k requests per time