search

blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
https://t.me/BlockchainETL
MIT License
2.87k stars 812 forks source link

EthereumETL Traces not working after merge #394

Closed abhishektvz closed 1 year ago

abhishektvz commented 1 year ago

On running the following command, we are getting an error like so:

ethereumetl export_traces --start-block 15537406 --end-block 15544330  --provider-uri  <provider> --batch-size 100 --output traces.csv

Error:

ValueError: {'code': -32000, 'message': 'first run for txIndex 0 error: nonce too high: address 0x4DF96C744748b3Ae07D2A43364b01256e4E366E6, tx: 9 state: 8'}
abhishektvz commented 1 year ago

We are running this on an erigon node(pos_sync branch) with a lighthouse beacon node doing a checkpoint-sync using one of the public beacon nodes, and I can confirm that the node is currently at the head of the blockchain so the data is synced

medvedev1088 commented 1 year ago

Try with this fix https://github.com/blockchain-etl/ethereum-etl/pull/392

abhishektvz commented 1 year ago

I am already running ethereumetl 2.0.5

abhishektvz commented 1 year ago 
ethereumetl --version                                                                                                                                                    
ethereumetl, version 2.0.5

Just saw that I am still getting insufficient funds error for some blocks:

 ethereumetl export_traces --start-block 15540015 --end-block 15544330 --batch-size 100 --output traces.csv

2022-09-16 14:04:02,655 - ProgressLogger [INFO] - Started work. Items to process: 4316.
2022-09-16 14:04:03,261 - root [ERROR] - An "insufficient funds" error occurred while tracing block 15540018. See here for more details: https://github.com/ledgerwatch/erigon/issues/5284. The block will be skipped.
Traceback (most recent call last):
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/jobs/export_traces_job.py", line 91, in _export_batch
    json_traces = self.web3.parity.traceBlock(block_number)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/web3/module.py", line 57, in caller
    result = w3.manager.request_blocking(method_str,
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/web3/manager.py", line 198, in request_blocking
    return self.formatted_response(response,
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/web3/manager.py", line 171, in formatted_response
    raise ValueError(response["error"])
ValueError: {'code': -32000, 'message': 'first run for txIndex 154 error: insufficient funds for gas * price + value: address 0x9D8e2dc5615c674F329d18786D52AF10a65Af08b have 810165122227742022 want 1319055670879248455'}
Traceback (most recent call last):
  File "/home/achopra/.asdf/installs/python/3.10.6/bin/ethereumetl", line 33, in <module>
    sys.exit(load_entry_point('ethereum-etl==2.0.5', 'console_scripts', 'ethereumetl')())
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/cli/export_traces.py", line 66, in export_traces
    job.run()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
    self._end()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/jobs/export_traces_job.py", line 115, in _end
    self.batch_work_executor.shutdown()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/executors/batch_work_executor.py", line 97, in shutdown
    self.executor.shutdown()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown
    self._check_completed_futures()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/blockchainetl/jobs/base_job.py", line 28, in run
    self._export()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/jobs/export_traces_job.py", line 66, in _export
    self.batch_work_executor.execute(
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/executors/batch_work_executor.py", line 59, in execute
    self.executor.submit(self._fail_safe_execute, work_handler, batch)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit
    self._check_completed_futures()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/executors/batch_work_executor.py", line 63, in _fail_safe_execute
    work_handler(batch)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/jobs/export_traces_job.py", line 99, in _export_batch
    raise e
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/ethereumetl/jobs/export_traces_job.py", line 91, in _export_batch
    json_traces = self.web3.parity.traceBlock(block_number)
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/web3/module.py", line 57, in caller
    result = w3.manager.request_blocking(method_str,
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/web3/manager.py", line 198, in request_blocking
    return self.formatted_response(response,
  File "/home/achopra/.asdf/installs/python/3.10.6/lib/python3.10/site-packages/web3/manager.py", line 171, in formatted_response
    raise ValueError(response["error"])
medvedev1088 commented 1 year ago

Got it, for the "insufficient funds" error the block is simply skipped for now as can be seen in the log. Another error "nonce too high" has to be handled as well

abhishektvz commented 1 year ago

Also, started getting an error while exporting receipts: unknown

abhishektvz commented 1 year ago 
 ethereumetl export_receipts_and_logs --transaction-hashes transaction_hashes.txt --receipts-output receipts.csv --logs-output logs.csv

same block range

abhishektvz commented 1 year ago

@medvedev1088 I worked with the erigon team to get these fixed, you can feel free to undo the previous temp fix as well, things seem to be working fine now

abhishektvz commented 1 year ago

Already using ethereumetl 2.0.5

On Fri, 16 Sep 2022 at 1:06 PM, Evgeny Medvedev @.***> wrote:

Try with this fix #392 https://github.com/blockchain-etl/ethereum-etl/pull/392

— Reply to this email directly, view it on GitHub https://github.com/blockchain-etl/ethereum-etl/issues/394#issuecomment-1249024417, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWOIHERPHXQ4NG7ZEK7LIV3V6QPOPANCNFSM6AAAAAAQOB3R5Y . You are receiving this because you authored the thread.Message ID: @.***>