Ion 5.0.00 error: Corruption: block checksum mismatch

LordDarkHelmet commented 3 years ago

Got an error that I had not seen before Fatal Error: "Corruption: block checksum mismatch".

Describe the issue

From the log file:

2021-05-05 20:01:21 Writing info to mncache.dat...
2021-05-05 20:01:21 Written info to mncache.dat  4ms
2021-05-05 20:01:21      Masternodes: meta infos object count: 0, nDsqCount: 0
2021-05-05 20:01:21 mncache.dat dump finished  5ms
2021-05-05 20:01:21 Verifying governance.dat format...
2021-05-05 20:01:21 Loaded info from governance.dat  59ms
2021-05-05 20:01:21      Governance Objects: 0 (Proposals: 0, Triggers: 0, Other: 0; Erased: 0), Votes: 0
2021-05-05 20:01:21 Writing info to governance.dat...
2021-05-05 20:01:21 Written info to governance.dat  5ms
2021-05-05 20:01:21      Governance Objects: 0 (Proposals: 0, Triggers: 0, Other: 0; Erased: 0), Votes: 0
2021-05-05 20:01:21 governance.dat dump finished  65ms
2021-05-05 20:01:21 Verifying netfulfilled.dat format...
2021-05-05 20:01:21 Loaded info from netfulfilled.dat  0ms
2021-05-05 20:01:21      Nodes with fulfilled requests: 0
2021-05-05 20:01:21 Writing info to netfulfilled.dat...
2021-05-05 20:01:21 Written info to netfulfilled.dat  3ms
2021-05-05 20:01:21      Nodes with fulfilled requests: 0
2021-05-05 20:01:21 netfulfilled.dat dump finished  3ms
2021-05-05 20:01:21 Verifying sporks.dat format...
2021-05-05 20:01:21 Loaded info from sporks.dat  1ms
2021-05-05 20:01:21      Sporks: 8
2021-05-05 20:01:21 Writing info to sporks.dat...
2021-05-05 20:01:21 Written info to sporks.dat  3ms
2021-05-05 20:01:21      Sporks: 8
2021-05-05 20:01:21 sporks.dat dump finished  4ms
2021-05-05 20:01:21 Dumped mempool: 0s to copy, 0.025031s to dump
2021-05-05 20:01:21 Corruption: block checksum mismatch: C:\Users\x\AppData\Roaming\ioncoin\evodb\001339.ldb
2021-05-05 20:01:21 *** System error while flushing: Database corrupted
2021-05-05 20:01:40 Corruption: block checksum mismatch: C:\Users\x\AppData\Roaming\ioncoin\evodb\001339.ldb
2021-05-05 20:01:40 *** System error while flushing: Database corrupted
2021-05-05 20:01:42 Shutdown: done

v 5.0.00 When reloading I get:

Log file from restart:

2021-05-05 20:09:12 Ion Core version v5.0.00
2021-05-05 20:09:12 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2021-05-05 20:09:13 Assuming ancestors of block 1599d484cfd57e3dc0ef8a2e8bc428c6b12ed8f8f73d6f5afd6c405c100f9a15 have valid signatures.
2021-05-05 20:09:13 Setting nMinimumChainWork=0000000000000000000000000000000000000000000000ab5cc326ba5cc01516
2021-05-05 20:09:13 Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2021-05-05 20:09:13 Using RdRand as an additional entropy source
2021-05-05 20:09:18 GUI: "registerShutdownBlockReason: Successfully registered: Ion Core didn't yet exit safely..."
2021-05-05 20:09:18 Default data directory C:\Users\x\AppData\Roaming\ioncoin
2021-05-05 20:09:18 Using data directory C:\Users\x\AppData\Roaming\ioncoin
2021-05-05 20:09:18 Using config file C:\Users\x\AppData\Roaming\ioncoin\ioncoin.conf
2021-05-05 20:09:18 Using at most 125 automatic connections (2048 file descriptors available)
2021-05-05 20:09:18 Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2021-05-05 20:09:18 Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2021-05-05 20:09:18 Using 2 threads for script verification
2021-05-05 20:09:18 RenameThread: thread new name ion-scriptch
2021-05-05 20:09:18 RenameThread: thread new name ion-scheduler
2021-05-05 20:09:18 scheduler thread start
2021-05-05 20:09:18 HTTP: creating work queue of depth 16
2021-05-05 20:09:18 Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.
2021-05-05 20:09:18 HTTP: starting 4 worker threads
2021-05-05 20:09:18 RenameThread: thread new name ion-httpworker
2021-05-05 20:09:18 RenameThread: thread new name ion-httpworker
2021-05-05 20:09:18 RenameThread: thread new name ion-httpworker
2021-05-05 20:09:18 RenameThread: thread new name ion-httpworker
2021-05-05 20:09:18 RenameThread: thread new name ion-http
2021-05-05 20:09:18 Creating backup of C:\Users\x\AppData\Roaming\ioncoin\wallet.dat -> C:\Users\x\AppData\Roaming\ioncoin\backups\wallet.dat.2021-05-05-20-09
2021-05-05 20:09:18 Old backup deleted: "C:\Users\x\AppData\Roaming\ioncoin\backups\wallet.dat.2021-01-10-15-44"
2021-05-05 20:09:18 init message: Verifying wallet(s)...
2021-05-05 20:09:18 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2021-05-05 20:09:18 Using wallet wallet.dat
2021-05-05 20:09:18 CDBEnv::Open: LogDir=C:\Users\x\AppData\Roaming\ioncoin\database ErrorFile=C:\Users\x\AppData\Roaming\ioncoin\db.log
2021-05-05 20:09:18 fLiteMode 0
2021-05-05 20:09:18 init message: Loading sporks cache...
2021-05-05 20:09:18 Reading info from sporks.dat...
2021-05-05 20:09:18 Loaded info from sporks.dat  0ms
2021-05-05 20:09:18      Sporks: 8
2021-05-05 20:09:18 Read: Cleaning....
2021-05-05 20:09:18      Sporks: 8
2021-05-05 20:09:18 Cache configuration:
2021-05-05 20:09:18 * Using 37.5MiB for block index database
2021-05-05 20:09:18 * Using 8.0MiB for chain state database
2021-05-05 20:09:18 * Using 254.5MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
2021-05-05 20:09:18 init message: Loading block index...
2021-05-05 20:09:18 Opening LevelDB in C:\Users\x\AppData\Roaming\ioncoin\evodb
2021-05-05 20:09:18 Opened LevelDB successfully
2021-05-05 20:09:18 Using obfuscation key for C:\Users\x\AppData\Roaming\ioncoin\evodb: 0000000000000000
2021-05-05 20:09:18 Opening LevelDB in C:\Users\x\AppData\Roaming\ioncoin\blocks\index
2021-05-05 20:09:19 Opened LevelDB successfully
2021-05-05 20:09:19 Using obfuscation key for C:\Users\x\AppData\Roaming\ioncoin\blocks\index: 0000000000000000
2021-05-05 20:09:19 Opening LevelDB in C:\Users\x\AppData\Roaming\ioncoin\llmq
2021-05-05 20:09:19 Opened LevelDB successfully
2021-05-05 20:09:19 Using obfuscation key for C:\Users\x\AppData\Roaming\ioncoin\llmq: 0000000000000000
2021-05-05 20:09:19 Opening LevelDB in C:\Users\x\AppData\Roaming\ioncoin\zerocoin
2021-05-05 20:09:19 Opened LevelDB successfully
2021-05-05 20:09:19 Using obfuscation key for C:\Users\x\AppData\Roaming\ioncoin\zerocoin: 0000000000000000
2021-05-05 20:09:19 Opening LevelDB in C:\Users\x\AppData\Roaming\ioncoin\tokens
2021-05-05 20:09:19 Opened LevelDB successfully
2021-05-05 20:09:19 Using obfuscation key for C:\Users\x\AppData\Roaming\ioncoin\tokens: 0000000000000000
2021-05-05 20:10:13 LoadBlockIndexDB: last block file = 13
2021-05-05 20:10:13 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=70270, size=58533477, heights=2080619...2150888, time=2021-03-13...2021-05-05)
2021-05-05 20:10:13 Checking all blk files are present...
2021-05-05 20:10:14 LoadBlockIndexDB: transaction index enabled
2021-05-05 20:10:14 LoadBlockIndexDB: address index disabled
2021-05-05 20:10:14 LoadBlockIndexDB: timestamp index disabled
2021-05-05 20:10:14 LoadBlockIndexDB: spent index disabled
2021-05-05 20:10:14 init message: Loading token data...
2021-05-05 20:10:14 Opening LevelDB in C:\Users\x\AppData\Roaming\ioncoin\chainstate
2021-05-05 20:10:14 Opened LevelDB successfully
2021-05-05 20:10:14 Using obfuscation key for C:\Users\x\AppData\Roaming\ioncoin\chainstate: e7e46b5bdb215c43
2021-05-05 20:10:15 Loaded best chain: hashBestChain=8b381dcf8fede2021fe181c985c1e14b5686a6e4ff2e396d7e092376b68a345a height=2150888 date=2021-05-05 20:01:24 progress=0.999896
2021-05-05 20:10:15 init message: Verifying tokens...
2021-05-05 20:10:15 init message: Verifying blocks...
2021-05-05 20:10:15 Verifying last 6 blocks at level 3
2021-05-05 20:10:15 [0%]...

Log file for the previous ~2 hours: https://pastebin.com/51mUWPu3

FYI, launching with --reindex seems to fix the issue. I will let you know if there are any issues with the repair. Strange error. I have never seen it before.

Wallet has fully synced after --reindex and seems to be working without any issues.

Can you reliably reproduce the issue?

Unable to reproduce

What version of Ion Core are you using?

5.0.00

Machine specs:

OS: Windows 10 Pro (20H2, 19042.928)
CPU: Intel(R) Core(TM) i3-4160 CPU @ 3.60GHz 3.60 GHz
RAM: 16 GB
Disk size: 1 TB
Disk Type (HD/SDD): SSD

LordDarkHelmet commented 3 years ago

The original error occurred when I was attempting to shutdown the wallet. The wallet was running for a while before the attempted shutdown. It most likely had plenty of connections. The wallet was not mining or minting. The wallet was locked.

LordDarkHelmet commented 3 years ago

From discord discussion: The first error happened when I was attempting to close the wallet. It was a fatal error. The wallet caught it gracefully and an error was presented to the user. While this error should not have happened, at least the wallet caught it. The error was a checksum mismatch and a corruption of the database. Hence the need for a --reindex. Even if the wallet was on a fork, the database should not have been corrupted. Hence the reason I posted it to GitHub as an issue.

For the second error that happened on restart, was while reading the corrupted database. In this case there was no graceful exit. The program completely crashed out. While this is an unexpected issue (the database should not have been corrupted) there should have been a check in the code that caught unexpected errors while reading the DB. There is an assertion() check in the code (https://github.com/ioncoincore/ion/blob/ba48de3f1da907218d4336f2b8e8343efca4c2f9/src/evo/evodb.cpp#L71). Most of the time assertions are used in test cases. In this case the assertion is unhanded and the program crashes out. Basically the check is valid, but handling the case where the check fails does not happen. The try-catch-finally method is the reason why assertions are not common in modern non-test code. In an ideal world the assertion would happen and either the program would throw up the issue to the user with steps that can be taken to resolve, or the program itself should know the DB is corrupted and attempt a repair. The repair is computationally expensive, so I would recommend the notice for the QT wallet. I would recommend the auto repair for daemon based instances as there is no user to make a decision.

The --reindex fixes the DB error because it just deletes whatever database is there (in our case the corrupted one) and replaces it with a fresh one.

ioncoincore / ion