blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
https://t.me/BlockchainETL
MIT License
2.94k stars 845 forks source link

use ethereumetl stream #265

Closed sznariOsmosis closed 2 years ago

sznariOsmosis commented 3 years ago

ethereumetl stream --start-block 500000 -e block,transaction --log-file log.txt \ --provider-uri file:///datalake/blockdata/geth.ipc \ --output=postgresql+pg8000://postgres:admin@127.0.0.1:5432/ethereum

why? root [INFO] - Current block 0, target block 0, last synced block 499999, blocks to sync 0 oot [INFO] - Nothing to sync. Sleeping for 10 seconds...

medvedev1088 commented 3 years ago

It looks like the node is not synced.

getBlock('latest') must be returning 0 https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/streaming/eth_streamer_adapter.py#L40

sznariOsmosis commented 3 years ago

看起来节点没有同步。

getBlock('latest') 必须返回 0 https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/streaming/eth_streamer_adapter.py#L40

The above error was solved. I synced directly to block 46146 and reported the following error: "Make sure Ethereum node is synced", but my Ethereum node is syncing

eth.syncing { currentBlock: 12961805, highestBlock: 12962575, knownStates: 0, pulledStates: 0, startingBlock: 12962264 }

michael-stajer commented 2 years ago

Bumping this issue... Here is what I am facing...

  1. My eth.syncing Screen Shot 2021-11-15 at 10 31 22 AM

  2. but, getBlock('latest') is show the block number==0: Screen Shot 2021-11-15 at 10 32 05 AM

  3. When I run the streaming, it shows 0 blocks and waits... Screen Shot 2021-11-15 at 10 32 58 AM

My geth node has been syncing for about 24 hours. According to this stack exchange, I have to wait for the node to fully sync before the getBlock('latest') will update properly: https://ethereum.stackexchange.com/questions/16537/why-is-getblock-latest-returning-0

michael-stajer commented 2 years ago

For future people researching this issue... the answer seems to be to wait, even several hours after full sync:

https://github.com/ethereum/go-ethereum/issues/14338:

Geth 1.6.0 uses fast sync by default. The block states are unavailable for intermediate blocks and are downloaded only for the recent state. As such, up to the point that sync completes, the latest complete block is zero, since all newer blocks are incomplete during sync. When fast sync finishes, it downloads the associated state for the head, so the head block becomes complete and the latest block can be bumped to the current head.

and...

Issue resolved, Solution is to wait till all state entries complete download after all blocks sync, There are around 300 million state entries which could take anything between 3 to 4 hour or more depending on net speed.

michael-stajer commented 2 years ago

Final update... 24 hours after eth.syncing was showing currentBlock as up to date, the states are finally downloaded and GETH is fully synced. Now getBlock('latest').number is showing the currentBlock.

medvedev1088 commented 2 years ago

Thanks, updated the docs https://ethereum-etl.readthedocs.io/en/latest/exporting-the-blockchain/

limingyang2020 commented 2 years ago

Final update... 24 hours after eth.syncing was showing currentBlock as up to date, the states are finally downloaded and GETH is fully synced. Now getBlock('latest').number is showing the currentBlock.

so geth is fully synced when getBlock('latest').number is what eth.syncing .currentBlock showing,right?

michael-stajer commented 2 years ago

It is when the STATES are fully downloaded. At the same time, getBlock('latest') == eth.currentBlock

so geth is fully synced when getBlock('latest').number is what eth.syncing .currentBlock showing,right?