Experimental Parallel ETH Scanner

[pdet]

This Scanner is based on the JSON-RPC protocol. There are two primary ways to use it. One is fully local, where you must start an Ethereum client locally (e.g., GETH or nethermind. The other option is to use a provider (e.g.,infura, if you don't have the necessary disk space to run a local node.

Currently, only the eth_getLogs method is supported, but otherswill come soon.

Set Ethereum provider

To set the Ethereum provider one must set the following:

set eth_node_url= 'https://mainnet.infura.io/v3/{key}';

This is set to http://127.0.0.1:8545 by default, assuming a local node.

Scanner

The scanner function is called read_eth. It takes the following five parameters:

1. The address to be tracked. This must be either a hex or a known coin (e.g., USDT)
2. The topic. This must be either a hex or a known topic (e.g., Transfer)
3. From Block. The start block id the scanner starts from
4. To Block. The end block id the scanners ends.
5. blocks_per_thread (Optional), the number of blocks to be processed per thread, this not only affects performance but might also be important if you are using a provider that limits the number of blocks you can query per request.

Example Usage:

set eth_node_url= 'https://mainnet.infura.io/v3/{key}';

FROM read_eth(
    'USDT',
    'Transfer',
    20034078,
    20034100, 
    blocks_per_thread=1
)
order by all
limit 1
----
"0xdac17f958d2ee523a2206206994597c13d831ec7"    Transfer    "0x0208a65db173bee464fe73c39156a25aa63b8e7f21b658dce7b933b563fe31f6"    20034081    0   121 false   [0x000000000000000000000000d0e4ccee1bbaf4bb701dd2a659c88ea849972095, 0x000000000000000000000000076aaf6914030df63fed94e4bcf22bdcfe05648e]    "0xe58cdbcc396c7bfcca433315c45192b81b9d0f2431b3fe0dcc3f4994242fd7b0"    19

Future Work

• More Quality of Life adjustments. For example, accept lists for the address and the topics instead of just a single value. Implement statistics, progress bars and whatnot for the scanners.
• Expend the scanners to other client json rpc functionality.
• Better error handling testing. More unhappy testing.
• Implement custom types uint256 and int256 instead of hoping things fit nicely in (U)BIGINTs or (U)HugeInts
• Drop JSON RPC for local scans. JSON RPC is obviously super slow, both producing, consuming, and transferring it. Underneath all that, these clients just use key-value stores like LevelDB and RocksDB. Their storage is not that crazy, so it should be possible to directly scan the file formats the client metadata.

[ngotchac]

I would strongly advise anyone wanting to try this out, to run it against an Erigon or a Reth node, which have way faster log querying than any other nodes. There are some free Reth RPC node, eg. here https://freerpc.merkle.io/