search

blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
https://t.me/BlockchainETL
MIT License
2.94k stars 845 forks source link

Streaming transactions, logs, token_transfer, trace to PostgreSQL doesn't work #231

Open SirFrankieCrisp opened 3 years ago

SirFrankieCrisp commented 3 years ago

I succeeded in streaming block data into a postgresql database from a local openethereum node. When I try streaming transactions, traces, or any other of ther possible entities I get the below listed output. If I do a select statement in the database, I don't see any entries (except for the blocks). Does it maybe have to do with openethereum (since as I saw form another issue, the devs use parity 2.5) or is it a bug in the script? Is it also correct that contracts are not yet supported for the streaming into postgresql?

Command ethereumetl stream --provider-uri file://$HOME/.local/share/openethereum/jsonrpc.ipc --start-block 0 --output postgresql+pg8000://postgres:password@127.0.0.1:5432/openeth_etl -e transaction,log,token_transfer,trace

Output The below listed output is just a snippet, it goes on like this for quite some time. 

2020-12-17 10:08:18,245 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,246 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,247 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.001733.
2020-12-17 10:08:18,247 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,247 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000099.
2020-12-17 10:08:18,247 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,247 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000091.
2020-12-17 10:08:18,247 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,251 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,251 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.003582.
2020-12-17 10:08:18,251 - root [INFO] - Exporting with PostgresItemExporter
2020-12-17 10:08:18,251 - root [INFO] - Writing last synced block 1838
2020-12-17 10:08:18,253 - root [INFO] - Current block 1740260, target block 1839, last synced block 1838, blocks to sync 1
2020-12-17 10:08:18,254 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,255 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,256 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.001944.
2020-12-17 10:08:18,256 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,256 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000101.
2020-12-17 10:08:18,256 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,256 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000084.
2020-12-17 10:08:18,256 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,260 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,261 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.004245.
2020-12-17 10:08:18,261 - root [INFO] - Exporting with PostgresItemExporter
2020-12-17 10:08:18,261 - root [INFO] - Writing last synced block 1839
2020-12-17 10:08:18,263 - root [INFO] - Current block 1740260, target block 1840, last synced block 1839, blocks to sync 1
2020-12-17 10:08:18,263 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,265 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,266 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.002211.
2020-12-17 10:08:18,266 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,266 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000096.
2020-12-17 10:08:18,266 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,266 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000079.
2020-12-17 10:08:18,266 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,270 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,270 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.003434.
2020-12-17 10:08:18,270 - root [INFO] - Exporting with PostgresItemExporter
2020-12-17 10:08:18,270 - root [INFO] - Writing last synced block 1840
2020-12-17 10:08:18,273 - root [INFO] - Current block 1740260, target block 1841, last synced block 1840, blocks to sync 1
2020-12-17 10:08:18,273 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,275 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,275 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.001923.
2020-12-17 10:08:18,275 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,275 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000077.
2020-12-17 10:08:18,276 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,276 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000118.
2020-12-17 10:08:18,276 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,279 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,279 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.003494.
2020-12-17 10:08:18,280 - root [INFO] - Exporting with PostgresItemExporter
2020-12-17 10:08:18,280 - root [INFO] - Writing last synced block 1841
2020-12-17 10:08:18,283 - root [INFO] - Current block 1740260, target block 1842, last synced block 1841, blocks to sync 1
2020-12-17 10:08:18,283 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,285 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,285 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.001863.
2020-12-17 10:08:18,285 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,285 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000081.
2020-12-17 10:08:18,285 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,285 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000064.
2020-12-17 10:08:18,285 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,288 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,289 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.003132.
2020-12-17 10:08:18,289 - root [INFO] - Exporting with PostgresItemExporter
2020-12-17 10:08:18,289 - root [INFO] - Writing last synced block 1842
2020-12-17 10:08:18,292 - root [INFO] - Current block 1740260, target block 1843, last synced block 1842, blocks to sync 1
2020-12-17 10:08:18,292 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,294 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,294 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.001705.
2020-12-17 10:08:18,294 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,294 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000087.
2020-12-17 10:08:18,294 - ProgressLogger [INFO] - Started work.
2020-12-17 10:08:18,294 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000072.
2020-12-17 10:08:18,295 - ProgressLogger [INFO] - Started work. Items to process: 1.
2020-12-17 10:08:18,298 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2020-12-17 10:08:18,298 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.003387.
2020-12-17 10:08:18,298 - root [INFO] - Exporting with PostgresItemExporter
2020-12-17 10:08:18,298 - root [INFO] - Writing last synced block 1843
2020-12-17 10:08:18,302 - root [INFO] - Current block 1740260, target block 1844, last synced block 1843, blocks to sync 1
medvedev1088 commented 3 years ago

Hello, the first few thousand blocks don't have any transactions and logs / token transfers. You should see traces though

SirFrankieCrisp commented 3 years ago

@medvedev1088 Thanks for your reponse. I now tried streaming blocks, transactions, traces, token_transfers and logs all at the same time, but besides blocks there is nothing in the database. There are now at least 15'000 blocks in the db but still no transactions or anything. Also no traces :/ No error messages are displayed, it looks like it is really just importing blocks or so

SirFrankieCrisp commented 3 years ago

You where right. The first transaction happens on block 46167. Now transactions show up. Still no logs, traces, receipts or token_transfers though:/ Let's see if anything changes a few tens of thousand of blocks down the line.