search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[Bug] database disk image malformed, despite repeated re-syncing #10685

Closed erikjber closed 2 years ago

erikjber commented 2 years ago

What happened?

I have been unable to sync for several weeks. After a few days of syncing, the syncing stops with the message:
sync from fork point failed err: database disk image is malformed

So far I have tried:

I'm lost for what to do here.

I'm running on a Raspberry Pi 4 with 4 gigs of ram.

Version

1.3

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

These are the last successfully added blocks:  

2022-03-12T02:57:07.159 full_node chia.full_node.full_node: INFO     Added blocks 1310112 to 1310144
2022-03-12T02:57:16.594 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event done, loaded 0, removed 0, processed 539, remaining 0, duration: 0.10 seconds, total plots: 539
2022-03-12T02:57:18.298 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '222.90.12.228', 'port': 8444}.Inbound limit reached.
2022-03-12T02:57:18.592 full_node full_node_server        : INFO     Connection closed: 222.90.12.228, node id: <redacted>
2022-03-12T02:57:18.594 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '222.90.12.228', 'port': 8444}
2022-03-12T02:57:22.708 farmer chia.farmer.farmer         : INFO     GET /farmer response: {'authentication_public_key': '<redacted>', 'payout_instructions': '<redacted>', 'current_difficulty': 6, 'current_points': 30}
2022-03-12T02:57:25.365 farmer chia.farmer.farmer         : INFO     PUT /farmer response: {'authentication_public_key': False, 'payout_instructions': False, 'suggested_difficulty': False}
2022-03-12T02:57:27.841 full_node chia.full_node.full_node: INFO     Added blocks 1310144 to 1310176
2022-03-12T02:57:41.752 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '140.246.250.117', 'port': 8444}.Inbound limit reached.
2022-03-12T02:57:42.030 full_node full_node_server        : INFO     Connection closed: 140.246.250.117, node id: <redacted>
2022-03-12T02:57:42.031 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '140.246.250.117', 'port': 8444}
2022-03-12T02:57:46.077 full_node chia.full_node.full_node: WARNING  Block pre-validation time: 18.21 seconds (32 blocks, start height: 1310177)
2022-03-12T02:57:51.049 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '104.208.83.236', 'port': 8444}.Inbound limit reached.
2022-03-12T02:57:51.284 full_node full_node_server        : INFO     Connection closed: 104.208.83.236, node id: <redacted>
2022-03-12T02:57:51.286 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '104.208.83.236', 'port': 8444}
2022-03-12T02:57:52.998 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-03-12T02:57:52.999 full_node chia.full_node.full_node: INFO     long sync done
2022-03-12T02:57:53.007 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 1310192, weight 2556023248, hh <redacted>, forked at 1310191, rh: <redacted>, total iters: 5006315216972, overflow: False, deficit: 0, difficulty: 2400, sub slot iters: 131596288, Generator size: No tx, Generator ref list size: No tx
2022-03-12T02:57:53.033 full_node chia.full_node.weight_proof: INFO     create weight proof peak <redacted> 1310192
2022-03-12T02:57:55.173 full_node chia.full_node.weight_proof: INFO     recent chain, start: 1309451 end:  1310192 
2022-03-12T02:58:00.407 full_node asyncio                 : ERROR    Task was destroyed but it is pending!
task: <Task pending name='Task-1518' coro=<FullNode.sync_from_fork_point.<locals>.fetch_block_batches() running at chia/full_node/full_node.py:948> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0xffff8a21c2e0>()]> cb=[gather.<locals>._done_callback() at asyncio/tasks.py:769]>
2022-03-12T02:58:01.314 full_node chia.full_node.full_node: INFO     Starting to perform sync.
2022-03-12T02:58:01.316 full_node chia.full_node.full_node: INFO     Waiting to receive peaks from peers.
2022-03-12T02:58:04.752 full_node full_node_server        : INFO     Connection closed: 86.12.43.84, node id: <redacted>
2022-03-12T02:58:04.753 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '86.12.43.84', 'port': 8444}

This is what the log displays now, pretty much over and over:  

2022-03-12T08:39:08.485 full_node chia.full_node.full_node: INFO     Starting to perform sync.
2022-03-12T08:39:08.487 full_node chia.full_node.full_node: INFO     Waiting to receive peaks from peers.
2022-03-12T08:39:08.964 full_node full_node_server        : INFO     Connection closed: 60.125.250.77, node id: <redacted>
2022-03-12T08:39:08.966 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '60.125.250.77', 'port': 8444}
2022-03-12T08:39:23.918 wallet chia.server.address_manager_store: INFO     Serializing peer data
2022-03-12T08:39:40.042 full_node chia.full_node.full_node: INFO     Collected a total of 11 peaks.
2022-03-12T08:39:40.044 full_node chia.full_node.full_node: INFO     Selected peak 1683169, <redacted>
2022-03-12T08:39:50.061 full_node chia.full_node.full_node: INFO     Total of 11 peers with peak 1683169
2022-03-12T08:39:50.063 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 31.160.151.98 up to height 1683169
2022-03-12T08:39:59.540 full_node asyncio                 : ERROR    Task was destroyed but it is pending!
task: <Task pending name='Task-21549' coro=<FullNode.sync_from_fork_point.<locals>.fetch_block_batches() running at chia/full_node/full_node.py:948> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0xffff879190a0>()]> cb=[gather.<locals>._done_callback() at asyncio/tasks.py:769]>
2022-03-12T08:40:00.275 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683170 timestamp: 1647074370
2022-03-12T08:40:00.278 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683170
2022-03-12T08:40:01.786 full_node full_node_server        : INFO     Connection closed: 121.205.99.233, node id: <redacted>
2022-03-12T08:40:01.787 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '121.205.99.233', 'port': 8444}
2022-03-12T08:40:03.711 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683171 timestamp: 1647074370
2022-03-12T08:40:03.714 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683171
2022-03-12T08:40:05.084 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event done, loaded 0, removed 0, processed 539, remaining 0, duration: 0.09 seconds, total plots: 539
2022-03-12T08:40:09.286 full_node chia.full_node.weight_proof: INFO     validate weight proof peak height 1683169
2022-03-12T08:40:09.669 full_node chia.full_node.weight_proof: INFO     validating 4383 sub epochs
2022-03-12T08:40:10.710 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683172 timestamp: 1647074370
2022-03-12T08:40:10.712 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683172
2022-03-12T08:40:13.095 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683173 timestamp: 1647074370
2022-03-12T08:40:13.102 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683173
2022-03-12T08:40:25.304 full_node full_node_server        : INFO     Connection closed: 51.154.74.190, node id: <redacted>
2022-03-12T08:40:25.306 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '51.154.74.190', 'port': 8444}
2022-03-12T08:40:26.300 full_node full_node_server        : INFO     Connection closed: 79.143.47.213, node id: <redacted>
2022-03-12T08:40:26.302 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '79.143.47.213', 'port': 8444}
2022-03-12T08:40:27.211 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683174 timestamp: 1647074370
2022-03-12T08:40:27.213 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683174
2022-03-12T08:40:36.977 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '104.162.75.53', 'port': 8444}.Inbound limit reached.
2022-03-12T08:40:37.085 full_node full_node_server        : INFO     Connection closed: 104.162.75.53, node id: <redacted>
2022-03-12T08:40:37.086 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '104.162.75.53', 'port': 8444}
2022-03-12T08:40:53.790 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683175 timestamp: 1647074424
2022-03-12T08:40:53.792 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683175
2022-03-12T08:41:03.728 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '92.117.213.91', 'port': 8444}.Inbound limit reached.
2022-03-12T08:41:03.763 full_node full_node_server        : INFO     Connection closed: 92.117.213.91, node id: <redacted>
2022-03-12T08:41:03.764 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '92.117.213.91', 'port': 8444}
2022-03-12T08:41:08.640 full_node chia.full_node.full_node: INFO     Re-checked peers: total of 11 peers with peak 1683169
2022-03-12T08:41:09.991 full_node chia.full_node.full_node: INFO     Start syncing from fork point at 1309057 up to 1683169
2022-03-12T08:41:10.244 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '24.134.209.185', 'port': 8444}.Inbound limit reached.
2022-03-12T08:41:10.284 full_node full_node_server        : INFO     Connection closed: 24.134.209.185, node id: <redacted>
2022-03-12T08:41:10.287 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '24.134.209.185', 'port': 8444}
2022-03-12T08:41:10.649 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '36.63.241.219', 'port': 8444}.Inbound limit reached.
2022-03-12T08:41:10.958 full_node full_node_server        : INFO     Connection closed: 36.63.241.219, node id: <redacted>
2022-03-12T08:41:10.960 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '36.63.241.219', 'port': 8444}
2022-03-12T08:41:14.974 full_node full_node_server        : INFO     Connection closed: 74.111.45.155, node id: <redacted>
2022-03-12T08:41:14.976 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '74.111.45.155', 'port': 8449}
2022-03-12T08:41:15.233 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '182.240.144.163', 'port': 8444}.Inbound limit reached.
2022-03-12T08:41:15.554 full_node full_node_server        : INFO     Connection closed: 182.240.144.163, node id: <redacted>
2022-03-12T08:41:15.556 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '182.240.144.163', 'port': 8444}
2022-03-12T08:41:17.748 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683176 timestamp: 1647074424
2022-03-12T08:41:17.752 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683176
2022-03-12T08:41:20.128 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683177 timestamp: 1647074424
2022-03-12T08:41:20.130 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683177
2022-03-12T08:41:26.264 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '36.4.201.205', 'port': 8444}.Inbound limit reached.
2022-03-12T08:41:26.559 full_node full_node_server        : INFO     Connection closed: 36.4.201.205, node id: <redacted>
2022-03-12T08:41:26.560 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '36.4.201.205', 'port': 8444}
2022-03-12T08:41:27.072 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683178 timestamp: 1647074424
2022-03-12T08:41:27.074 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683178
2022-03-12T08:41:33.168 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 1683179 timestamp: 1647074463
2022-03-12T08:41:33.171 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 1683179
2022-03-12T08:41:37.675 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '86.123.37.222', 'port': 8444}.Inbound limit reached.
2022-03-12T08:41:38.067 full_node full_node_server        : INFO     Connection closed: 86.123.37.222, node id: <redacted>
2022-03-12T08:41:38.068 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '86.123.37.222', 'port': 8444}
2022-03-12T08:41:38.939 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '92.86.13.174', 'port': 8444}.Inbound limit reached.
2022-03-12T08:41:39.379 full_node full_node_server        : INFO     Connection closed: 92.86.13.174, node id: <redacted>
2022-03-12T08:41:39.380 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '92.86.13.174', 'port': 8444}
2022-03-12T08:41:44.726 full_node chia.full_node.full_node: WARNING  Block pre-validation time: 25.80 seconds (32 blocks, start height: 1310193)
2022-03-12T08:41:44.908 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-03-12T08:41:44.909 full_node chia.full_node.full_node: INFO     long sync done
2022-03-12T08:41:44.914 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 1310192, weight 2556023248, hh <redacted>, forked at 1310191, rh: <redacted>, total iters: 5006315216972, overflow: False, deficit: 0, difficulty: 2400, sub slot iters: 131596288, Generator size: No tx, Generator ref list size: No tx
2022-03-12T08:42:03.175 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '120.31.136.55', 'port': 8444}.Inbound limit reached.
erikjber commented 2 years ago

I also sometimes see this error on the command line, although it does not appear in the debug log:

Exception ignored in: <coroutine object FullNode.sync_from_fork_point..fetch_block_batches at 0xffff86f03440> Traceback (most recent call last): File "chia/util/streamable.py", line 339, in RuntimeError: coroutine ignored GeneratorExit

emlowe commented 2 years ago

SD cards are unable to handle the I/O required to run the node, we recommend you use an SSD drive. It's possible your SD card is itself getting corrupted.

github-actions[bot] commented 2 years ago

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

github-actions[bot] commented 2 years ago

This issue was automatically closed because it has been flagged as stale, and subsequently passed 7 days with no further activity from the submitter or watchers.

blackmennewstyle commented 1 year ago

I'm facing the same exact issue since a couple of days it started since i upgraded to 1.6.1, i'm running GNU/Linux Debian 11 with a NVME SSD.

I deleted multiple times the ~/.chia/mainnet/db and start full resync but nothing changes. At some point i always get the same error:

  File "chia/full_node/coin_store.py", line 157, in new_block
  File "chia/full_node/coin_store.py", line 523, in _add_coin_records
  File "aiosqlite/core.py", line 210, in executemany
  File "aiosqlite/core.py", line 129, in _execute
  File "aiosqlite/core.py", line 102, in run
sqlite3.DatabaseError: database disk image is malformed
 database disk image is malformed
2022-11-13T00:42:11.415 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-11-13T00:43:32.389 full_node chia.consensus.blockchain: ERROR    Error while adding block 6871643598e9e48cc1862bcfef5c9603343585e2e35128923a39586c9905106e height 1762191, rolling back: Traceback (most recent call last):
  File "chia/consensus/blockchain.py", line 270, in receive_block
  File "chia/consensus/blockchain.py", line 406, in _reconsider_peak
  File "chia/full_node/coin_store.py", line 157, in new_block
  File "chia/full_node/coin_store.py", line 523, in _add_coin_records
  File "aiosqlite/core.py", line 210, in executemany
  File "aiosqlite/core.py", line 129, in _execute
  File "aiosqlite/core.py", line 102, in run
sqlite3.DatabaseError: database disk image is malformed
 database disk image is malformed
2022-11-13T00:43:32.391 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed