analyseether / ether_sql

A python library to push ethereum blockchain data into an sql database.
http://ether-sql.readthedocs.io
Apache License 2.0
69 stars 22 forks source link

Scrape fails after 46,147 blocks - receipt_data is NoneType #62

Open whydna opened 3 years ago

whydna commented 3 years ago

Describe the bug Setup went smoothly using a local Geth node. Ran ether_sql scrape_block_range and everything seemed to be working smoothly. It now seems to be stuck on block 46,147 with the following error every time I try to run it:

(venv) user1@full-node-1:~/ether_sql$ ether_sql scrape_block_range
[2021-06-25 13:45:37,600][INFO][ether_sql.session] Connected to the db ether_sql
[2021-06-25 13:45:37,604][INFO][ether_sql.session] Connected to Geth node
[2021-06-25 13:45:37,604][INFO][ether_sql.globals] Pushing the session DefaultSettings in local thread
[2021-06-25 13:45:39,128][INFO][ether_sql.cli] 0 missing blocks detected
Traceback (most recent call last):
  File "/home/user1/venv/bin/ether_sql", line 33, in <module>
    sys.exit(load_entry_point('ether-sql', 'console_scripts', 'ether_sql')())
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/user1/venv/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user1/ether_sql/ether_sql/cli/__init__.py", line 91, in scrape_block_range
    mode=mode)
  File "/home/user1/ether_sql/ether_sql/tasks/scrapper.py", line 66, in scrape_blocks
    add_block_number(block_number)
  File "/home/user1/venv/lib/python3.6/site-packages/celery/local.py", line 193, in __call__
    return self._get_current_object()(*a, **kw)
  File "/home/user1/venv/lib/python3.6/site-packages/celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  File "/home/user1/ether_sql/ether_sql/tasks/scrapper.py", line 143, in add_block_number
    timestamp=iso_timestamp)
  File "/home/user1/ether_sql/ether_sql/models/receipts.py", line 68, in add_receipt
    receipt = cls(transaction_hash=to_hex(receipt_data['transactionHash']),
TypeError: 'NoneType' object is not subscriptable

If I skip that block by running: scrape_block_range --start_block_number 46148

It will scrape a few more blocks and then fail on the next one without receipt data:

[2021-06-25 13:49:25,923][INFO][ether_sql.tasks.scrapper] Commiting block: 46162 to sql
[2021-06-25 13:49:26,020][INFO][ether_sql.tasks.scrapper] Commiting block: 46163 to sql
[2021-06-25 13:49:26,094][INFO][ether_sql.tasks.scrapper] Commiting block: 46164 to sql
[2021-06-25 13:49:26,267][INFO][ether_sql.tasks.scrapper] Commiting block: 46165 to sql
[2021-06-25 13:49:26,334][INFO][ether_sql.tasks.scrapper] Commiting block: 46166 to sql
[2021-06-25 13:49:26,414][INFO][ether_sql.tasks.scrapper] Commiting block: 46167 to sql
[2021-06-25 13:49:26,465][INFO][ether_sql.tasks.scrapper] Commiting block: 46168 to sql
Traceback (most recent call last):
  File "/home/user1/venv/bin/ether_sql", line 33, in <module>
    sys.exit(load_entry_point('ether-sql', 'console_scripts', 'ether_sql')())
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user1/venv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/user1/venv/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user1/ether_sql/ether_sql/cli/__init__.py", line 91, in scrape_block_range
    mode=mode)
  File "/home/user1/ether_sql/ether_sql/tasks/scrapper.py", line 66, in scrape_blocks
    add_block_number(block_number)
  File "/home/user1/venv/lib/python3.6/site-packages/celery/local.py", line 193, in __call__
    return self._get_current_object()(*a, **kw)
  File "/home/user1/venv/lib/python3.6/site-packages/celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  File "/home/user1/ether_sql/ether_sql/tasks/scrapper.py", line 143, in add_block_number
    timestamp=iso_timestamp)
  File "/home/user1/ether_sql/ether_sql/models/receipts.py", line 68, in add_receipt
    receipt = cls(transaction_hash=to_hex(receipt_data['transactionHash']),
TypeError: 'NoneType' object is not subscriptable

To Reproduce Steps to reproduce the behavior: Just run the the scraper.

Expected behavior It should handle the case where receipt_data is missing?

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context Add any other context about the problem here.