search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[Bug] Syncing - malformed database on the same block #14413

Closed jakub-polak closed 1 year ago

jakub-polak commented 1 year ago

What happened?

I was trying to sync my database multiple times and it always get malformed around blocks 728k - Added blocks 728007 to 728039. Multiple times it was exactly this block, some times it goes up to 731k.

There were no crashes while syncing. Removal of db and wallet directories didn't help.

Please let me know what details could be helpful for solving that.

Version

1.6.2

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

'0x8ecad8ab2a4dda82f4586ad6c57051b036b75788df299377ef394719c13e42d8f7983a3d133739778e316d72d2405334', 'payout_instructions': 'bc7c29e6f9271023c65705480b8e5625ab3a46a88d3642949c6d0ca034628036', 'current_difficulty': 6, 'current_points': 33}
2023-01-27T01:38:55.966 farmer chia.farmer.farmer         : INFO     PUT /farmer response: {'authentication_public_key': False, 'payout_instructions': False, 'suggested_difficulty': False}
2023-01-27T01:38:56.169 farmer chia.farmer.farmer         : INFO     GET /farmer response: {'authentication_public_key': '0x8ecad8ab2a4dda82f4586ad6c57051b036b75788df299377ef394719c13e42d8f7983a3d133739778e316d72d2405334', 'payout_instructions': 'a189c0f017dde757df4e89d1aad632f1db1fcce464a8b77f63996fafb0356e0a', 'current_difficulty': 1, 'current_points': 0}
2023-01-27T01:38:56.269 farmer chia.farmer.farmer         : INFO     PUT /farmer response: {'authentication_public_key': False, 'payout_instructions': False, 'suggested_difficulty': False}
2023-01-27T01:38:57.421 full_node chia.full_node.full_node: INFO     Added blocks 726567 to 726599
2023-01-27T01:39:00.019 full_node chia.full_node.full_node: INFO     Added blocks 726599 to 726631
2023-01-27T01:39:02.861 full_node chia.full_node.full_node: INFO     Added blocks 726631 to 726663
2023-01-27T01:39:05.026 full_node chia.full_node.full_node: INFO     Added blocks 726663 to 726695
2023-01-27T01:39:07.004 full_node chia.full_node.full_node: INFO     Added blocks 726695 to 726727
2023-01-27T01:39:09.036 full_node full_node_server        : INFO     Cannot connect to host 194.150.142.100:8444 ssl:<ssl.SSLContext object at 0x7f8d0c9c7bc0> [Connect call failed ('194.150.142.100', 8444)]
2023-01-27T01:39:09.177 full_node chia.full_node.full_node: INFO     Added blocks 726727 to 726759
2023-01-27T01:39:11.148 full_node chia.full_node.full_node: INFO     Added blocks 726759 to 726791
2023-01-27T01:39:13.178 full_node chia.full_node.full_node: INFO     Added blocks 726791 to 726823
2023-01-27T01:39:15.140 full_node chia.full_node.full_node: INFO     Added blocks 726823 to 726855
2023-01-27T01:39:17.545 full_node chia.full_node.full_node: INFO     Added blocks 726855 to 726887
2023-01-27T01:39:20.450 full_node chia.full_node.full_node: INFO     Added blocks 726887 to 726919
2023-01-27T01:39:22.488 full_node chia.full_node.full_node: INFO     Added blocks 726919 to 726951
2023-01-27T01:39:25.553 full_node chia.full_node.full_node: INFO     Added blocks 726951 to 726983
2023-01-27T01:39:27.373 full_node chia.full_node.full_node: INFO     Added blocks 726983 to 727015
2023-01-27T01:39:29.162 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162973 timestamp: 1674779958
2023-01-27T01:39:29.162 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162973
2023-01-27T01:39:29.796 full_node chia.full_node.full_node: INFO     Added blocks 727015 to 727047
2023-01-27T01:39:31.674 full_node chia.full_node.full_node: INFO     Added blocks 727047 to 727079
2023-01-27T01:39:34.469 full_node chia.full_node.full_node: INFO     Added blocks 727079 to 727111
2023-01-27T01:39:37.186 full_node chia.full_node.full_node: INFO     Added blocks 727111 to 727143
2023-01-27T01:39:39.258 full_node chia.full_node.full_node: INFO     Added blocks 727143 to 727175
2023-01-27T01:39:41.856 full_node chia.full_node.full_node: INFO     Added blocks 727175 to 727207
2023-01-27T01:39:44.663 full_node chia.full_node.full_node: INFO     Added blocks 727207 to 727239
2023-01-27T01:39:47.713 full_node chia.full_node.full_node: INFO     Added blocks 727239 to 727271
2023-01-27T01:39:49.401 full_node chia.full_node.full_node: INFO     Added blocks 727271 to 727303
2023-01-27T01:39:51.576 full_node chia.full_node.full_node: INFO     Added blocks 727303 to 727335
2023-01-27T01:39:51.609 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event started, loaded 0, removed 0, processed 0, remaining 0, duration: 0.00 seconds, total plots: 0
2023-01-27T01:39:51.610 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event done, loaded 0, removed 0, processed 0, remaining 0, duration: 0.00 seconds, total plots: 0
2023-01-27T01:39:52.106 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162974 timestamp: 1674779958
2023-01-27T01:39:52.107 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162974
2023-01-27T01:39:54.232 full_node chia.full_node.full_node: INFO     Added blocks 727335 to 727367
2023-01-27T01:39:56.398 full_node chia.full_node.full_node: INFO     Added blocks 727367 to 727399
2023-01-27T01:39:58.560 full_node chia.full_node.full_node: INFO     Added blocks 727399 to 727431
2023-01-27T01:40:00.434 full_node chia.full_node.full_node: INFO     Added blocks 727431 to 727463
2023-01-27T01:40:03.500 full_node chia.full_node.full_node: INFO     Added blocks 727463 to 727495
2023-01-27T01:40:06.345 full_node chia.full_node.full_node: INFO     Added blocks 727495 to 727527
2023-01-27T01:40:09.423 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162975 timestamp: 1674779979
2023-01-27T01:40:09.424 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162975
2023-01-27T01:40:09.963 full_node chia.full_node.full_node: INFO     Added blocks 727527 to 727559
2023-01-27T01:40:12.312 full_node chia.full_node.full_node: INFO     Added blocks 727559 to 727591
2023-01-27T01:40:13.681 full_node chia.full_node.full_node: INFO     Added blocks 727591 to 727623
2023-01-27T01:40:15.928 full_node chia.full_node.full_node: INFO     Added blocks 727623 to 727655
2023-01-27T01:40:16.816 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162976 timestamp: 1674779979
2023-01-27T01:40:16.817 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162976
2023-01-27T01:40:17.785 full_node chia.full_node.full_node: INFO     Added blocks 727655 to 727687
2023-01-27T01:40:19.645 full_node chia.full_node.full_node: INFO     Added blocks 727687 to 727719
2023-01-27T01:40:20.061 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162977 timestamp: 1674779979
2023-01-27T01:40:20.061 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162977
2023-01-27T01:40:21.628 full_node chia.full_node.full_node: INFO     Added blocks 727719 to 727751
2023-01-27T01:40:23.060 full_node chia.full_node.full_node: INFO     Added blocks 727751 to 727783
2023-01-27T01:40:24.741 full_node chia.full_node.full_node: INFO     Added blocks 727783 to 727815
2023-01-27T01:40:26.413 full_node chia.full_node.full_node: INFO     Added blocks 727815 to 727847
2023-01-27T01:40:28.480 full_node chia.full_node.full_node: INFO     Added blocks 727847 to 727879
2023-01-27T01:40:30.041 full_node chia.full_node.full_node: INFO     Added blocks 727879 to 727911
2023-01-27T01:40:31.899 full_node chia.full_node.full_node: INFO     Added blocks 727911 to 727943
2023-01-27T01:40:33.853 full_node chia.full_node.full_node: INFO     Added blocks 727943 to 727975
2023-01-27T01:40:35.277 full_node chia.full_node.full_node: INFO     Added blocks 727975 to 728007
2023-01-27T01:40:36.995 full_node chia.full_node.full_node: INFO     Added blocks 728007 to 728039
2023-01-27T01:40:38.285 full_node chia.consensus.blockchain: ERROR    Error while adding block fa70a7e7f0940f4385218f5b0c027613c1d966d4728b5cb9c842b3fe5eb24368 height 728058, rolling back: Traceback (most recent call last):
  File "chia/consensus/blockchain.py", line 270, in receive_block
  File "chia/consensus/blockchain.py", line 406, in _reconsider_peak
  File "chia/full_node/coin_store.py", line 156, in new_block
  File "chia/full_node/coin_store.py", line 522, in _add_coin_records
  File "aiosqlite/core.py", line 210, in executemany
  File "aiosqlite/core.py", line 129, in _execute
  File "aiosqlite/core.py", line 102, in run
sqlite3.DatabaseError: database disk image is malformed
 database disk image is malformed
2023-01-27T01:40:38.286 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2023-01-27T01:40:38.286 full_node chia.full_node.full_node: INFO     long sync done
2023-01-27T01:40:38.287 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 728057, weight 853845120, hh 52bcefd53e9e816139d814a8dcf771882ccf40266ecda36c68993369fa601150, forked at 728056, rh: 1325a4ad2896e8aa3d64af1f5af260cf2d5a1e5d896ea59013000ea4ceecbdbb, total iters: 2559974519605, overflow: False, deficit: 0, difficulty: 2656, sub slot iters: 130547712, Generator size: No tx, Generator ref list size: No tx
2023-01-27T01:40:38.293 full_node chia.full_node.weight_proof: INFO     create weight proof peak 52bcefd53e9e816139d814a8dcf771882ccf40266ecda36c68993369fa601150 728057
2023-01-27T01:40:38.881 full_node asyncio                 : ERROR    Task was destroyed but it is pending!
task: <Task pending name='Task-1092' coro=<FullNode.sync_from_fork_point.<locals>.fetch_block_batches() running at chia/full_node/full_node.py:1124> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f8cd59dedf0>()]> cb=[gather.<locals>._done_callback() at asyncio/tasks.py:769]>
2023-01-27T01:40:38.915 full_node chia.full_node.weight_proof: INFO     recent chain, start: 727335 end:  728057 
2023-01-27T01:40:59.767 full_node chia.full_node.full_node: INFO     Starting to perform sync.
2023-01-27T01:40:59.767 full_node chia.full_node.full_node: INFO     Waiting to receive peaks from peers.
2023-01-27T01:41:00.573 full_node chia.full_node.full_node: INFO     Collected a total of 3 peaks.
2023-01-27T01:41:00.574 full_node chia.full_node.full_node: INFO     Selected peak Peak(header_hash=<bytes32: b5bab17b61d7c42fe3e282ce0734623d1da51a7ec7deb6494c80b3af566587f2>, height=3162978, weight=6482320144)
2023-01-27T01:41:01.450 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162978 timestamp: 1674780027
2023-01-27T01:41:01.451 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162978
2023-01-27T01:41:02.055 full_node chia.full_node.full_node: INFO     Total of 7 peers with peak 3162978
2023-01-27T01:41:02.055 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 201.231.187.151 up to height 3162978
2023-01-27T01:41:12.613 full_node chia.full_node.weight_proof: INFO     validating 8236 sub epochs
2023-01-27T01:41:12.647 full_node chia.full_node.weight_proof: INFO     validate weight proof peak height 3162978
2023-01-27T01:41:13.096 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162979 timestamp: 1674780027
2023-01-27T01:41:13.096 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162979
2023-01-27T01:41:17.409 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162980 timestamp: 1674780027
2023-01-27T01:41:17.410 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162980
2023-01-27T01:41:27.446 full_node chia.full_node.full_node: INFO     Re-checked peers: total of 7 peers with peak 3162978
2023-01-27T01:41:27.989 full_node chia.full_node.full_node: INFO     Start syncing from fork point at 726943 up to 3162978
2023-01-27T01:41:30.314 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 3162981 timestamp: 1674780027
2023-01-27T01:41:30.314 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 3162981
2023-01-27T01:41:33.709 full_node chia.consensus.blockchain: ERROR    Error while adding block fa70a7e7f0940f4385218f5b0c027613c1d966d4728b5cb9c842b3fe5eb24368 height 728058, rolling back: Traceback (most recent call last):
  File "chia/consensus/blockchain.py", line 270, in receive_block
  File "chia/consensus/blockchain.py", line 406, in _reconsider_peak
  File "chia/full_node/coin_store.py", line 156, in new_block
  File "chia/full_node/coin_store.py", line 522, in _add_coin_records
  File "aiosqlite/core.py", line 210, in executemany
  File "aiosqlite/core.py", line 129, in _execute
  File "aiosqlite/core.py", line 102, in run
sqlite3.DatabaseError: database disk image is malformed
 database disk image is malformed
MumfMeisterT commented 1 year ago

The most obvious explanation is that your disk is going bad. Can you try syncing to a different SSD?

rones-dev commented 1 year ago

The most obvious explanation is that your disk is going bad. Can you try syncing to a different SSD?

This is not entirely true. I had several times "database disk image is malformed" then exchanged SSD to new one then got again "database disk image is malformed" then I exchanged (again) to NEW SSD NVME M2 and the second time in the middle ( on the second SSD) I have database disk image is malformed during synchronization.

It would be good if CHIA TEAM could focus on such CONSTANT & REPEATING issues not only bullshit like the last NFT updates !!!!

MumfMeisterT commented 1 year ago

Please check the setting of db_sync in config.yaml for both wallet and node. Please make sure it is set to either "auto" or "full".

jakub-polak commented 1 year ago

The most obvious explanation is that your disk is going bad. Can you try syncing to a different SSD?

This is not entirely true. I had several times "database disk image is malformed" then exchanged SSD to new one then got again "database disk image is malformed" then I exchanged (again) to NEW SSD NVME M2 and the second time in the middle ( on the second SSD) I have database disk image is malformed during synchronization.

It would be good if CHIA TEAM could focus on such CONSTANT & REPEATING issues not only bullshit like the last NFT updates !!!!

I can confirm this. I have tried different disks and the issue persists.

rones-dev commented 1 year ago

Please check the setting of db_sync in config.yaml for both wallet and node. Please make sure it is set to either "auto" or "full".

OK - I had "auto" switched to "full" nad continue sync as for now ...

rones-dev commented 1 year ago

And again problem with sync - log says:

2023-03-04T22:42:44.022 full_node asyncio : ERROR Task was destroyed but it is pending! 2023-03-04T22:42:44.332 full_node asyncio : ERROR Task was destroyed but it is pending! 2023-03-04T22:43:05.069 full_node chia.full_node.full_node: ERROR sync from fork point failed err: b'\x8b\xce\xac\xeb\xa4\xc5\x0e\xf6\x1f\x16\xa0vA\x84-\xd8\x8d\xbe,+R\x8e\x01_Is#\x9ba\x84\xeb\xd1' 2023-03-04T22:43:45.569 full_node chia.full_node.full_node: ERROR sync from fork point failed err: b'\x8b\xce\xac\xeb\xa4\xc5\x0e\xf6\x1f\x16\xa0vA\x84-\xd8\x8d\xbe,+R\x8e\x01_Is#\x9ba\x84\xeb\xd1' 2023-03-04T22:44:29.613 full_node chia.full_node.full_node: ERROR sync from fork point failed err: b'\x8b\xce\xac\xeb\xa4\xc5\x0e\xf6\x1f\x16\xa0vA\x84-\xd8\x8d\xbe,+R\x8e\x01_Is#\x9ba\x84\xeb\xd1' 2023-03-04T22:44:49.270 full_node asyncio : ERROR Task was destroyed but it is pending! 2023-03-04T22:44:49.861 full_node asyncio : ERROR Task was destroyed but it is pending! 2023-03-04T22:44:59.161 full_node chia.full_node.full_node: WARNING Syncing failed, CancelledError 2023-03-04T22:46:16.534 full_node chia.full_node.full_node: ERROR sync from fork point failed err: b'\x8b\xce\xac\xeb\xa4\xc5\x0e\xf6\x1f\x16\xa0vA\x84-\xd8\x8d\xbe,+R\x8e\x01_Is#\x9ba\x84\xeb\xd1' 2023-03-04T22:47:21.999 full_node chia.full_node.full_node: ERROR sync from fork point failed err: b'\x8b\xce\xac\xeb\xa4\xc5\x0e\xf6\x1f\x16\xa0vA\x84-\xd8\x8d\xbe,+R\x8e\x01_Is#\x9ba\x84\xeb\xd1' 2023-03-04T22:48:58.617 full_node chia.full_node.full_node: ERROR sync from fork point failed err: b'\x8b\xce\xac\xeb\xa4\xc5\x0e\xf6\x1f\x16\xa0vA\x84-\xd8\x8d\xbe,+R\x8e\x01_Is#\x9ba\x84\xeb\xd1' 2023-03-04T22:49:48.622 full_node chia.full_node.full_node: ERROR sync from fork point failed err: b'\x8b\xce\xac\xeb\xa4\xc5\x0e\xf6\x1f\x16\xa0vA\x84-\xd8\x8d\xbe,+R\x8e\x01_Is#\x9ba\x84\xeb\xd1'

What the hell does it mean ???!!!

MumfMeisterT commented 1 year ago

@rones-dev Could you please make further responses within your issue #14558?

github-actions[bot] commented 1 year ago

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

github-actions[bot] commented 1 year ago

This issue was automatically closed because it has been flagged as stale, and subsequently passed 7 days with no further activity from the submitter or watchers.