search

blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
https://t.me/BlockchainETL
MIT License
2.94k stars 841 forks source link

Streaming to postgresql is Extremely slow. #457

Open B1Q opened 1 year ago

B1Q commented 1 year ago

I noticed that streaming is very very slow.

I'm running an Erigon Archive node locally

with this command ethereumetl stream -e block,transaction,token_transfer,token,log --provider-uri http://127.0.0.1:8545 --output=postgresql+pg8000://user:pw@127.0.0.1:5432/db--period-seconds 0 --max-workers 1000

It took 2 hours to index 15963 blocks. (recorded the time) this is indexing old blocks starting from block 10,000,000 I am also running another one that's indexing the blockchain from the last block onwards (to keep database up to date)

Any advice on how to speed it up?

medvedev1088 commented 1 year ago

This could be useful https://github.com/blockchain-etl/ethereum-etl-postgres

B1Q commented 1 year ago

This could be useful https://github.com/blockchain-etl/ethereum-etl-postgres

Thank you, I have another question:

I am trying to download the data locally, is that a possibility? I do not wish to use Cloud SQL