search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[Bug] sync from fork point failed: DatabaseError: database disk image is malformed #17456

Closed nbauernfeind closed 8 months ago

nbauernfeind commented 8 months ago

What happened?

I recently updated to v2.1.4. Somehow the database got malformed, so I deleted it and have been trying to sync from other nodes on the network. Again I'm getting errors that the database is malformed.

The SSD that hosts the db is showing no data or integrity errors; so I don't believe it is a hardware failure. There is also 1.7TB of free space on the drive; so it's not an out-of-memory issue either.

Version

2.1.4

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

2024-01-31T21:13:30.280 full_node chia.full_node.full_node: INFO     Start syncing from fork point at 984198 up to 4880022
2024-01-31T21:13:30.764 full_node chia.full_node.full_node: ERROR    sync from fork point failed: DatabaseError: database disk image is malformed
Traceback (most recent call last):
  File "/home/nate/code/chia-blockchain/chia/util/log_exceptions.py", line 20, in log_exceptions
    yield
  File "/home/nate/code/chia-blockchain/chia/full_node/full_node.py", line 1173, in sync_from_fork_point
    await asyncio.gather(fetch_task, validate_task)
  File "/home/nate/code/chia-blockchain/chia/full_node/full_node.py", line 1135, in validate_block_batches
    success, state_change_summary, err = await self.add_block_batch(
  File "/home/nate/code/chia-blockchain/chia/full_node/full_node.py", line 1307, in add_block_batch
    result, error, state_change_summary = await self.blockchain.add_block(
  File "/home/nate/code/chia-blockchain/chia/consensus/blockchain.py", line 415, in add_block
    error_code, _ = await validate_block_body(
  File "/home/nate/code/chia-blockchain/chia/consensus/block_body_validation.py", line 369, in validate_block_body
    unspent_records = await coin_store.get_coin_records(removals_from_db)
  File "/home/nate/code/chia-blockchain/chia/full_node/coin_store.py", line 169, in get_coin_records
    for row in await cursor.fetchall():
  File "/home/nate/code/chia-blockchain/venv/lib/python3.10/site-packages/aiosqlite/cursor.py", line 76, in fetchall
    return await self._execute(self._cursor.fetchall)
  File "/home/nate/code/chia-blockchain/venv/lib/python3.10/site-packages/aiosqlite/cursor.py", line 40, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
  File "/home/nate/code/chia-blockchain/venv/lib/python3.10/site-packages/aiosqlite/core.py", line 133, in _execute
    return await future
  File "/home/nate/code/chia-blockchain/venv/lib/python3.10/site-packages/aiosqlite/core.py", line 106, in run
    result = function()
sqlite3.DatabaseError: database disk image is malformed

2024-01-31T21:13:30.764 full_node chia.full_node.full_node: INFO     long sync done
2024-01-31T21:13:30.764 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 985340, weight 1606716448, hh 1c833a972d32dd424e59daf07e3dc3d82d83c35a5713d68256cc9c20b4f89023, forked at 985339, rh: 53f22385f1243d7c05890fffa9df0e42b40956f1bed5adaaf80d2cd56e69520a, total iters: 3630169982569, overflow: True, deficit: 16, difficulty: 3104, sub slot iters: 135266304, Generator size: No tx, Generator ref list size: No tx
 'header_hash': '0x1c833a972d32dd424e59daf07e3dc3d82d83c35a5713d68256cc9c20b4f89023',
 'height': 985340,
 'weight': 1606716448}

2024-01-31T21:16:27.961 full_node chia.full_node.full_node: INFO     Selected peak Peak(header_hash=<bytes32: af135bc352883e33a3d552a9a7a1d377313ac460211c2a7f06c31bf3a10887cb>, height=4880031, weight=14276741584)
 'header_hash': '0xaf135bc352883e33a3d552a9a7a1d377313ac460211c2a7f06c31bf3a10887cb',
 'height': 4880031,
 'weight': 14276741584}
2024-01-31T21:16:28.139 full_node chia.full_node.full_node: INFO     Total of 1 peers with peak 4880031
2024-01-31T21:16:28.140 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 107.182.204.7 up to height 4880031
2024-01-31T21:16:29.466 wallet chia.wallet.wallet_blockchain: INFO     Peak set to: 4880031 timestamp: 1706735764
2024-01-31T21:16:29.467 wallet chia.wallet.wallet_node    : INFO     Finished processing new peak of 4880031
nbauernfeind commented 8 months ago

It must have been one of the other files in the db directory. Even after copying the sql database from another host I couldn't get that node to sync. Then I blew away the entire db directory, but kept the copy of the database, and the farmer was able to restart.

No idea what was actually corrupt. Leaving this note in hopes to help the next poor soul who thinks they could fix it with a resync from the network.