blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
https://t.me/BlockchainETL
MIT License
2.87k stars 812 forks source link

Cant stream contract and/or tokens #397

Open academe-01 opened 1 year ago

academe-01 commented 1 year ago

Anyone know how to fix this?

root@b:/mnt/md7/eth/export# ethereumetl stream -s 15625286 -p http://127.0.0.1:8545 -e block,transaction,log,token_transfer,contract
2022-09-27 14:42:47,320 - root [INFO] - Using http://127.0.0.1:8545
2022-09-27 14:42:47,336 - root [INFO] - Current block 15625355, target block 15625286, last synced block 15625285, blocks to sync 1
2022-09-27 14:42:47,336 - ProgressLogger [INFO] - Started work. Items to process: 1.
2022-09-27 14:42:47,367 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2022-09-27 14:42:47,367 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.030639.
2022-09-27 14:42:47,367 - ProgressLogger [INFO] - Started work.
2022-09-27 14:42:47,452 - ProgressLogger [INFO] - Finished work. Total items processed: 236. Took 0:00:00.084809.
2022-09-27 14:42:47,453 - ProgressLogger [INFO] - Started work.
2022-09-27 14:42:47,460 - ProgressLogger [INFO] - Finished work. Total items processed: 554. Took 0:00:00.007202.
2022-09-27 14:42:47,460 - ProgressLogger [INFO] - Started work. Items to process: 1.
2022-09-27 14:42:47,464 - root [ERROR] - An exception occurred while syncing block data.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/blockchainetl/streaming/streamer.py", line 77, in _do_stream
    synced_blocks = self._sync_cycle()
  File "/usr/local/lib/python3.10/site-packages/blockchainetl/streaming/streamer.py", line 98, in _sync_cycle
    self.blockchain_streamer_adapter.export_all(self.last_synced_block + 1, target_block)
  File "/usr/local/lib/python3.10/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 62, in export_all
    traces = self._export_traces(start_block, end_block)
  File "/usr/local/lib/python3.10/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 159, in _export_traces
    job.run()
  File "/usr/local/lib/python3.10/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
    self._end()
  File "/usr/local/lib/python3.10/site-packages/ethereumetl/jobs/export_traces_job.py", line 106, in _end
    self.batch_work_executor.shutdown()
  File "/usr/local/lib/python3.10/site-packages/ethereumetl/executors/batch_work_executor.py", line 97, in shutdown
    self.executor.shutdown()
  File "/usr/local/lib/python3.10/site-packages/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown
    self._check_completed_futures()
  File "/usr/local/lib/python3.10/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/ethereumetl/executors/batch_work_executor.py", line 63, in _fail_safe_execute
    work_handler(batch)
  File "/usr/local/lib/python3.10/site-packages/ethereumetl/jobs/export_traces_job.py", line 90, in _export_batch
    json_traces = self.web3.parity.traceBlock(block_number)
  File "/usr/local/lib/python3.10/site-packages/web3/module.py", line 57, in caller
    result = w3.manager.request_blocking(method_str,
  File "/usr/local/lib/python3.10/site-packages/web3/manager.py", line 198, in request_blocking
    return self.formatted_response(response,
  File "/usr/local/lib/python3.10/site-packages/web3/manager.py", line 171, in formatted_response
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'the method trace_block does not exist/is not available'}
2022-09-27 14:42:47,465 - root [INFO] - Nothing to sync. Sleeping for 10 seconds...
^C
Aborted!

instance: Geth/v1.10.25-stable-69568c55/linux-amd64/go1.18.5

everything breaks if I use entity token and/or contract

sfsf9797 commented 1 year ago

I guess Geth doesn't support the trace_block method.

academe-01 commented 1 year ago

I guess Geth doesn't support the trace_block method.

Have any idea which client does ? I need to stream on The Merge.

sfsf9797 commented 1 year ago

you may try quicknode

ddl-hust commented 1 year ago

I guess Geth doesn't support the trace_block method.

Have any idea which client does ? I need to stream on The Merge.

erigon can do it ;but when use erigon client not support optimism's trace method, you may handle it by yourself