1.2.9: AssertionError: [6112] Failed to execute script 'start_full_node' due to unhandled exception!

[PSLLSP]

chia-blockchain_1.2.9_amd64.deb Ubuntu 20.04.3

I think this is new issue in 1.2.9. Version 1.2.7 was running without this issue. But it is not 100% repeatable, other machine with chia 1.2.9 runs ok... I am not sure but it could be created by a server "crash" (unexpected power outage).

$ # chia processes are not running
$ ps aux | grep chia
ubuntu         5978  0.0  0.0   8160   672 pts/1    S+   11:59   0:00 grep --color=auto chia

$ python3 -V
Python 3.8.10

$ du -sh ~/.chia/
33G /home/ubuntu/.chia/

$ ln -sf /usr/lib/chia-blockchain/resources/app.asar.unpacked/daemon/chia chia

$ ./chia version
1.2.9

$ ./chia start node
Daemon not started yet
Starting daemon
chia_full_node: started

# after few seconds, exception is dumped to the console:
$ Traceback (most recent call last):
  File "chia/server/start_full_node.py", line 65, in <module>
  File "chia/server/start_full_node.py", line 60, in main
  File "chia/server/start_service.py", line 255, in run_service
  File "asyncio/runners.py", line 44, in run
  File "asyncio/base_events.py", line 616, in run_until_complete
  File "chia/server/start_service.py", line 249, in async_run_service
  File "chia/server/start_service.py", line 171, in run
  File "chia/server/start_service.py", line 135, in start
  File "chia/full_node/full_node.py", line 178, in _start
  File "chia/full_node/full_node.py", line 1105, in peak_post_processing
  File "chia/consensus/blockchain.py", line 503, in get_sp_and_ip_sub_slots
AssertionError
[6112] Failed to execute script 'start_full_node' due to unhandled exception!
Exception in thread Thread-1:
Traceback (most recent call last):
  File "aiosqlite/core.py", line 109, in run
  File "asyncio/base_events.py", line 764, in call_soon_threadsafe
  File "asyncio/base_events.py", line 508, in _check_closed
RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 932, in _bootstrap_inner
  File "aiosqlite/core.py", line 117, in run
  File "asyncio/base_events.py", line 764, in call_soon_threadsafe
  File "asyncio/base_events.py", line 508, in _check_closed
RuntimeError: Event loop is closed

$ ps aux | grep chia
ubuntu         5987  0.0  0.0   4952   660 pts/1    S    12:01   0:00 ./chia run_daemon --wait-for-unlock
ubuntu         5988  2.0  1.3 286944 55336 pts/1    Sl   12:01   0:01 chia_daemon
ubuntu         6443  0.0  0.0   8160   724 pts/1    S+   13:42   0:00 grep --color=auto chia

This exception is repeatable on this computer, I just cannot start node...

[PSLLSP]

It is possible that this exception was triggered because there was not enough free space on the disk. There was only about 2GB of free space (of 50GB disk), I missed that. I stopped virtual computer, extended disk with another 50 GB and it seems I can start the chia node now. Chia block-chain grows fast, I assume I will have to extend the disk again in less than a year.

UPDATE: wrong observation. Disk was extended but issue was not fixed, I still cannot start node...

[emlowe]

Highly likely this is a blockchain DB corruption error. While getting the block from the DB, it returned None back which is causing the assert. You almost certainly need to resync the blockchain DB.

[paulhainsworth-chia]

@emlowe can we find out if this is an issue created in 1.2.9, created by low free disk space, or something else?

[github-actions[bot]]

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

[github-actions[bot]]

This issue was automatically closed because it has been flagged as stale, and subsequently passed 7 days with no further activity from the submitter or watchers.