search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

The Status is the Syncing but there is no progress at a block height #10392

Closed TomaszWM closed 2 years ago

TomaszWM commented 2 years ago

What happened?

Peak Height: 1,307,652

Peak Time: 12/21/2021 03:22:09

debug.log : 2022-02-23T12:24:31.281 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed

This is the 5th time. I can't upload the database. It was similar at blocks 1,567,637, 1,047,860, 997,940, and 130,959. No philosophy please.

Version

1,2,11

What platform are you using?

Windows

What ui mode are you using?

GUI

Relevant log output

2022-02-23T12:24:31.281 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
TomaszWM commented 2 years ago

The last message from debug.log;

2022-02-23T12:22:31.727 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:24:31.281 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:29:49.528 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:34:12.953 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:35:57.519 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:39:21.796 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:42:21.130 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:42:32.699 full_node chia.full_node.full_node: WARNING Received unsolicited/late block from peer {'host': '112.85.209.39', 'port': 8444} 2022-02-23T12:44:48.860 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:47:28.431 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:51:16.314 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:54:39.751 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:57:43.800 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T12:59:46.961 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:01:05.724 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:03:16.210 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:04:34.763 full_node full_node_server : WARNING Banning 181.31.73.18 for 600 seconds 2022-02-23T13:04:34.766 full_node chia.full_node.full_node: ERROR Error with syncing: <class 'ValueError'>Traceback (most recent call last): File "chia\full_node\full_node.py", line 823, in _sync File "chia\full_node\weight_proof.py", line 606, in validate_weight_proof concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "chia\full_node\full_node.py", line 826, in _sync ValueError: Weight proof validation threw an error A process in the process pool was terminated abruptly while the future was running or pending.

2022-02-23T13:22:16.125 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:24:58.730 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:25:22.418 full_node chia.full_node.full_node: WARNING Received unsolicited/late block from peer {'host': '221.211.155.39', 'port': 8444} 2022-02-23T13:27:17.294 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed

TomaszWM commented 2 years ago

and the last:

2022-02-23T13:30:32.271 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:32:56.536 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:35:36.719 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:37:00.529 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:39:06.411 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:41:30.573 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:42:16.176 full_node chia.full_node.full_node: WARNING Received unsolicited/late block from peer {'host': '95.54.169.242', 'port': 8444} 2022-02-23T13:44:08.419 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:46:30.179 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:48:18.594 full_node chia.full_node.full_node: ERROR Error with syncing: <class 'RuntimeError'>Traceback (most recent call last): File "chia\full_node\full_node.py", line 807, in _sync RuntimeError: Weight proof did not arrive in time from peer: 95.54.169.242

2022-02-23T13:51:05.408 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:51:30.557 full_node chia.full_node.full_node: WARNING Received unsolicited/late block from peer {'host': '114.85.58.164', 'port': 8444} 2022-02-23T13:52:32.767 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:52:47.588 full_node chia.full_node.full_node: WARNING Received unsolicited/late block from peer {'host': '112.102.212.27', 'port': 8444} 2022-02-23T13:54:47.685 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:55:03.855 full_node chia.full_node.full_node: WARNING Received unsolicited/late block from peer {'host': '111.16.229.47', 'port': 8444} 2022-02-23T13:55:11.691 full_node chia.full_node.full_node: WARNING Received unsolicited/late block from peer {'host': '120.42.239.56', 'port': 8444} 2022-02-23T13:56:50.104 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:59:32.598 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T13:59:55.684 full_node chia.rpc.rpc_server : WARNING Error while handling message: Traceback (most recent call last): File "chia\rpc\rpc_server.py", line 217, in safe_handle File "chia\rpc\rpc_server.py", line 205, in ws_api File "chia\rpc\rpc_server.py", line 175, in close_connection ValueError: Connection with node_id 898209a4d7ba71b8e7efdf882bc4653b74812932bfa9257dfcabb742780e7089 does not exist

2022-02-23T14:00:37.798 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T14:00:43.818 full_node full_node_server : WARNING Banning 178.131.102.35 for 10 seconds 2022-02-23T14:00:43.820 full_node full_node_server : WARNING Invalid handshake with peer. Maybe the peer is running old software. 2022-02-23T14:01:53.250 full_node full_node_server : ERROR Exception: database disk image is malformed, {'host': '121.182.0.77', 'port': 8444}. Traceback (most recent call last): File "chia\server\server.py", line 566, in wrapped_coroutine File "chia\full_node\full_node_api.py", line 124, in new_peak File "chia\full_node\full_node.py", line 522, in new_peak File "chia\full_node\full_node.py", line 392, in short_sync_batch File "chia\full_node\full_node.py", line 366, in short_sync_batch File "chia\full_node\full_node.py", line 1037, in receive_block_batch File "chia\consensus\blockchain.py", line 274, in receive_block File "chia\consensus\blockchain.py", line 414, in _reconsider_peak File "chia\full_node\coin_store.py", line 115, in new_block File "chia\full_node\coin_store.py", line 415, in _add_coin_records File "aiosqlite\core.py", line 210, in executemany File "aiosqlite\core.py", line 129, in _execute File "aiosqlite\core.py", line 102, in run sqlite3.DatabaseError: database disk image is malformed

2022-02-23T14:01:53.253 full_node full_node_server : ERROR Exception: database disk image is malformed <class 'sqlite3.DatabaseError'>, closing connection {'host': '121.182.0.77', 'port': 8444}. Traceback (most recent call last): File "chia\server\server.py", line 576, in api_call File "asyncio\tasks.py", line 442, in wait_for File "chia\server\server.py", line 573, in wrapped_coroutine File "chia\server\server.py", line 566, in wrapped_coroutine File "chia\full_node\full_node_api.py", line 124, in new_peak File "chia\full_node\full_node.py", line 522, in new_peak File "chia\full_node\full_node.py", line 392, in short_sync_batch File "chia\full_node\full_node.py", line 366, in short_sync_batch File "chia\full_node\full_node.py", line 1037, in receive_block_batch File "chia\consensus\blockchain.py", line 274, in receive_block File "chia\consensus\blockchain.py", line 414, in _reconsider_peak File "chia\full_node\coin_store.py", line 115, in new_block File "chia\full_node\coin_store.py", line 415, in _add_coin_records File "aiosqlite\core.py", line 210, in executemany File "aiosqlite\core.py", line 129, in _execute File "aiosqlite\core.py", line 102, in run sqlite3.DatabaseError: database disk image is malformed

2022-02-23T14:01:53.255 full_node full_node_server : WARNING Banning 121.182.0.77 for 10 seconds 2022-02-23T14:03:41.139 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-02-23T14:05:26.228 full_node chia.full_node.full_node: ERROR Error with syncing: <class 'RuntimeError'>Traceback (most recent call last): File "chia\full_node\full_node.py", line 807, in _sync RuntimeError: Weight proof did not arrive in time from peer: 222.189.178.241

TomaszWM commented 2 years ago

I don't see any advices so I've started download database for the sixth time.

lcgogo commented 2 years ago

@TomaszWM They said will release a 1.3 beta but not publish now. https://github.com/Chia-Network/chia-blockchain/issues/9677#issuecomment-1035103956

emlowe commented 2 years ago

database disk image is malformed is somewhat self-explanatory - but your DB is corrupt. The only recourse here is to delete the node db and resync it. However, getting this same error after several syncing attempts is very atypical. I recommend you download the latest 1.3 beta (which has version 1.2.12.dev355) and attempt to sync with that version.

TomaszWM commented 2 years ago

The Current progress of downloading the blocks: Peak Height: 741,532 Peak Time: 08/20/2021 11:11:29

lcgogo commented 2 years ago

database disk image is malformed is somewhat self-explanatory - but your DB is corrupt. The only recourse here is to delete the node db and resync it. However, getting this same error after several syncing attempts is very atypical. I recommend you download the latest 1.3 beta (which has version 1.2.12.dev355) and attempt to sync with that version.

How to checkout 1.2.12.dev355? not find this tag yet.

TomaszWM commented 2 years ago

The blocks is uploading; Peak Height: 1,127,479 Peak Time: 11/11/2021 22:34:30

TomaszWM commented 2 years ago

For now it's OK;

Full Node Overview Status: Synced Connection Status: Connected Network Name: mainnet Peak Height: 1,620,301 Peak Time: 02/26/2022 18:53:56 Difficulty: 2,576

TomaszWM commented 2 years ago

The Chia has stuck on the 1,633,514 block. Again.

Full Node Overview Status: Syncing 1,633,514/1,637,715 Connection Status: Connected Network Name: mainnet Peak Height: 1,633,514 Peak Time: 03/01/2022 15:26:47 Difficulty: 2,512 VDF Sub Slot Iterations: 135,266,304 Total Iterations: 6,369,773,969,282 Estimated Network Space: 30.406 EiB

Blocks Header Hash: d5077edfc379cb12c6a063065d6650fcd9749d1cffda12498d88a8f71fb28435 Height: 1,633,514 State: Finished

Debug.log 2022-03-02T13:38:23.032 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T13:40:41.396 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T13:43:32.282 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T13:45:25.743 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T13:47:53.062 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T13:49:18.046 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T13:51:37.020 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T13:54:11.808 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T13:56:54.248 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed

Any suggestions ?

TomaszWM commented 2 years ago

Traditionally, I don't see any hints. I have started downloading the base 7th times from the end of February . It is absurd.

Debug.log 2022-03-02T19:47:18.505 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T19:47:46.819 full_node asyncio : ERROR Task was destroyed but it is pending! task: <Task pending name='Task-1726' coro=<FullNode.sync_from_fork_point..fetch_block_batches() running at chia\full_node\full_node.py:893> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x00000210DDC1D1C0>()]> cb=[gather.._done_callback() at asyncio\tasks.py:766]> 2022-03-02T19:49:44.075 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T19:50:16.141 full_node chia.full_node.full_node: WARNING Received unsolicited/late block from peer {'host': '211.21.154.175', 'port': 8444} 2022-03-02T19:51:06.439 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T19:53:46.174 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T19:55:21.059 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed 2022-03-02T19:56:34.391 full_node chia.full_node.full_node: ERROR sync from fork point failed err: database disk image is malformed

TomaszWM commented 2 years ago

The block database has been loaded How long will it last ?

Full Node Overview;

Status - Synced Connection Status - Connected Network Name - mainnet Peak Height - 1,652,970 Peak Time - 03/05/2022 20:46:49 Difficulty - 2,464 VDF Sub Slot Iterations - 132,120,576 Total Iterations - 6,451,643,031,621 Estimated Network Space - 29.693 EiB

SecurID commented 2 years ago

Have the same problem #10432

emlowe commented 2 years ago

database disk image is malformed - you need to delete your DB and resync. please attempt with v1.3

If you get this error each time you resync - I believe it suggests some type of problem with the disk. Right now I don't have any other explanation why your DB would get corrupted in such a fashion on every re-resync

github-actions[bot] commented 2 years ago

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

github-actions[bot] commented 2 years ago

This issue was automatically closed because it has been flagged as stale, and subsequently passed 7 days with no further activity from the submitter or watchers.