blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
https://t.me/BlockchainETL
MIT License
2.95k stars 847 forks source link

stream to Postgres - blocks to sync 0 - using uri: geth.ipc - not working #228

Open DrIanGregory opened 3 years ago

DrIanGregory commented 3 years ago

Using geth.ipc for streaming data can not locate blocks to sync (but works if syncing using uri as MainNet):

# Does not work: sudo ethereumetl stream --provider-uri file:///var/database/raw_data/geth/geth.ipc --output=postgresql+pg8000://ethereum:ethereum@127.0.0.1:5432/ethereum --last-synced-block-file /var/database/raw_data/geth/last_synced_block.txt

Output: 2020-12-02 14:42:26,129 - root [INFO] - Current block 0, target block 0, last synced block 11371542, blocks to sync 0

# The following all works: # stream using Mainnet: sudo ethereumetl stream --provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c --output=postgresql+pg8000://ethereum:ethereum@127.0.0.1:5432/ethereum --last-synced-block-file /var/database/raw_data/geth/last_synced_block.txt

# Extracting data to .csv using .ipc creates the respective files and fills with data: sudo ethereumetl export_blocks_and_transactions --start-block 0 --end-block 10 --blocks-output blocks.csv --transactions-output transactions.csv --provider-uri file:///var/database/raw_data/geth/geth.ipc

# Start geth: nohup geth --datadir /var/database/raw_data/geth/ --cache=1024 &

#Launch geth console: sudo geth --datadir /var/database/raw_data/geth/ attach ipc://var/database/raw_data/geth/geth.ipc

eth.syncing { currentBlock: 11373547, highestBlock: 11373617, knownStates: 222634446, pulledStates: 222633464, startingBlock: 11371541 }

SirFrankieCrisp commented 3 years ago

Not 100% sure about this, but I think geth is not supported in the stream command, only parity. See the answer of #223