Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Using geth.ipc for streaming data can not locate blocks to sync (but works if syncing using uri as MainNet):
# Does not work:
sudo ethereumetl stream --provider-uri file:///var/database/raw_data/geth/geth.ipc --output=postgresql+pg8000://ethereum:ethereum@127.0.0.1:5432/ethereum --last-synced-block-file /var/database/raw_data/geth/last_synced_block.txt
Output: 2020-12-02 14:42:26,129 - root [INFO] - Current block 0, target block 0, last synced block 11371542, blocks to sync 0
# The following all works:# stream using Mainnet:
sudo ethereumetl stream --provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c --output=postgresql+pg8000://ethereum:ethereum@127.0.0.1:5432/ethereum --last-synced-block-file /var/database/raw_data/geth/last_synced_block.txt
# Extracting data to .csv using .ipc creates the respective files and fills with data:
sudo ethereumetl export_blocks_and_transactions --start-block 0 --end-block 10 --blocks-output blocks.csv --transactions-output transactions.csv --provider-uri file:///var/database/raw_data/geth/geth.ipc
Using geth.ipc for streaming data can not locate blocks to sync (but works if syncing using uri as MainNet):
# Does not work: sudo ethereumetl stream --provider-uri file:///var/database/raw_data/geth/geth.ipc --output=postgresql+pg8000://ethereum:ethereum@127.0.0.1:5432/ethereum --last-synced-block-file /var/database/raw_data/geth/last_synced_block.txt
Output: 2020-12-02 14:42:26,129 - root [INFO] - Current block 0, target block 0, last synced block 11371542, blocks to sync 0
# The following all works: # stream using Mainnet: sudo ethereumetl stream --provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c --output=postgresql+pg8000://ethereum:ethereum@127.0.0.1:5432/ethereum --last-synced-block-file /var/database/raw_data/geth/last_synced_block.txt
# Extracting data to .csv using .ipc creates the respective files and fills with data: sudo ethereumetl export_blocks_and_transactions --start-block 0 --end-block 10 --blocks-output blocks.csv --transactions-output transactions.csv --provider-uri file:///var/database/raw_data/geth/geth.ipc
# Start geth: nohup geth --datadir /var/database/raw_data/geth/ --cache=1024 &
#Launch geth console: sudo geth --datadir /var/database/raw_data/geth/ attach ipc://var/database/raw_data/geth/geth.ipc