search

xiaoyao1991 / presto-ethereum

Presto Ethereum Connector -- SQL on Ethereum
Apache License 2.0
471 stars 56 forks source link

Optimize ERC20 retrieval by using Filters #41

Open medvedev1088 opened 6 years ago

medvedev1088 commented 6 years ago

Using log filters instead of retrieving transaction receipts significantly improves performance of ERC20 retrieval https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_getfilterlogs

My tests show 100x improvement for some blocks. Here is an example implementation in python https://github.com/medvedev1088/ethereum-etl/blob/88fd6792707fa8c0fdfd5ff199bc3ff44b9a1aae/ethereumetl/jobs.py#L155

xiaoyao1991 commented 6 years ago

Good point. I didn't choose to implement that way because Infura does not support this API call yet.

medvedev1088 commented 6 years ago

What do you think about ERC20RetrievalStrategy with 2 implementations ReceiptRetrievalStrategy, LogFilterRetrievalStrategy? It can be configured with a property, erc20.retrieval-strategy=receipt|log-filter|auto.

BTW, do you know if Infura have plans to add support for filter APIs?

xiaoyao1991 commented 6 years ago

I like that.

I don't know if Infura will support filter APIs. It's kinda hard cuz having filters makes their API stateful.