Add action to delete txindex if it gets corrupted

elvece commented 2 years ago

Environment:

raspi 8gb
dev box (its in the office, regular black embassy plugged into ethernet in wall behind my desk)
crucial drive (our default product we ship, but I do not know the specifics of the SSD, I can run a command to find out if provided)

Steps to recreate:

fresh install of Bitcoin Core
toggle txindex on
all other config settings are default (ie. full archival node)
let sync for ~1 day (was at about 4%)
logs indicated a crash loop with the following recurring message (relevant details at end):

Other notes:

Related (see comment with other links as well)


2022-03-29T11:09:06Z Bitcoin Core version v22.0.0 (release build)
2022-03-29T11:09:06Z InitParameterInteraction: parameter interaction: -externalip set -> setting -discover=0
2022-03-29T11:09:06Z Assuming ancestors of block 00000000000000000008a89e854d57e5667df88f1cdef6fde2fbca1de5b639ad have valid signatures.
2022-03-29T11:09:06Z Setting nMinimumChainWork=00000000000000000000000000000000000000001fa4663bbbe19f82de910280
2022-03-29T11:09:06Z Using the 'standard' SHA256 implementation
2022-03-29T11:09:06Z Default data directory /root/.bitcoin
2022-03-29T11:09:06Z Using data directory /root/.bitcoin
2022-03-29T11:09:06Z Config file: /root/.bitcoin/bitcoin.conf
2022-03-29T11:09:06Z Config file arg: avoidpartialspends="1"
2022-03-29T11:09:06Z Config file arg: bind="0.0.0.0:8333"
2022-03-29T11:09:06Z Config file arg: disablewallet="0"
2022-03-29T11:09:06Z Config file arg: discardfee="0.0001"
2022-03-29T11:09:06Z Config file arg: listen="1"
2022-03-29T11:09:06Z Config file arg: maxmempool="300"
2022-03-29T11:09:06Z Config file arg: mempoolexpiry="336"
2022-03-29T11:09:06Z Config file arg: persistmempool="1"
2022-03-29T11:09:06Z Config file arg: rpcallowip="0.0.0.0/0"
2022-03-29T11:09:06Z Config file arg: rpcbind=****
2022-03-29T11:09:06Z Config file arg: rpcpassword=****
2022-03-29T11:09:06Z Config file arg: rpcserialversion="1"
2022-03-29T11:09:06Z Config file arg: rpcservertimeout="30"
2022-03-29T11:09:06Z Config file arg: rpcthreads="1"
2022-03-29T11:09:06Z Config file arg: rpcuser=****
2022-03-29T11:09:06Z Config file arg: rpcworkqueue="16"
2022-03-29T11:09:06Z Config file arg: txindex="1"
2022-03-29T11:09:06Z Config file arg: whitelist="172.18.0.0/16"
2022-03-29T11:09:06Z Config file arg: zmqpubrawblock="tcp://0.0.0.0:28332"
2022-03-29T11:09:06Z Config file arg: zmqpubrawtx="tcp://0.0.0.0:28333"
2022-03-29T11:09:06Z Command-line arg: conf="/root/.bitcoin/bitcoin.conf"
2022-03-29T11:09:06Z Command-line arg: datadir="/root/.bitcoin"
2022-03-29T11:09:06Z Command-line arg: externalip="slctejn6ynv4ajs4pz3uxjfelkz7zefggvgbibtcme5hjsmf7enksrad.onion"
2022-03-29T11:09:06Z Command-line arg: onion="172.18.0.1:9050"
2022-03-29T11:09:06Z Using at most 125 automatic connections (1048576 file descriptors available)
2022-03-29T11:09:06Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2022-03-29T11:09:06Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2022-03-29T11:09:06Z Script verification uses 3 additional threads
2022-03-29T11:09:06Z scheduler thread start
2022-03-29T11:09:06Z WARNING: the RPC server is not safe to expose to untrusted networks such as the public internet
2022-03-29T11:09:06Z HTTP: creating work queue of depth 16
2022-03-29T11:09:06Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
2022-03-29T11:09:06Z HTTP: starting 1 worker threads
2022-03-29T11:09:06Z Using wallet directory /root/.bitcoin
2022-03-29T11:09:06Z init message: Verifying wallet(s)…
2022-03-29T11:09:06Z init message: Loading banlist…
2022-03-29T11:09:06Z SetNetworkActive: true
2022-03-29T11:09:06Z Failed to read fee estimates from /root/.bitcoin/fee_estimates.dat. Continue anyway.
2022-03-29T11:09:06Z AddLocal(slctejn6ynv4ajs4pz3uxjfelkz7zefggvgbibtcme5hjsmf7enksrad.onion:8333,4)
2022-03-29T11:09:06Z Using /16 prefix for IP bucketing
2022-03-29T11:09:06Z Cache configuration:
2022-03-29T11:09:06Z * Using 2.0 MiB for block index database
2022-03-29T11:09:06Z * Using 56.0 MiB for transaction index database
2022-03-29T11:09:06Z * Using 8.0 MiB for chain state database
2022-03-29T11:09:06Z * Using 384.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2022-03-29T11:09:06Z init message: Loading block index…
2022-03-29T11:09:06Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2022-03-29T11:09:06Z Opening LevelDB in /root/.bitcoin/blocks/index
2022-03-29T11:09:06Z Opened LevelDB successfully
2022-03-29T11:09:06Z Using obfuscation key for /root/.bitcoin/blocks/index: 0000000000000000
2022-03-29T11:09:17Z LoadBlockIndexDB: last block file = 421
2022-03-29T11:09:17Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=156, size=129708690, heights=393979...394143, time=2016-01-19...2016-01-20)
2022-03-29T11:09:17Z Checking all blk files are present...
2022-03-29T11:09:17Z Opening LevelDB in /root/.bitcoin/chainstate
2022-03-29T11:09:17Z Opened LevelDB successfully
2022-03-29T11:09:17Z Using obfuscation key for /root/.bitcoin/chainstate: 3d0879cfb3246fa2
2022-03-29T11:09:18Z Loaded best chain: hashBestChain=0000000000000000051a6089df758eb75106fabf9a51e46c1dd9f4bc249e2e4a height=394133 date=2016-01-20T07:08:11Z progress=0.147453
2022-03-29T11:09:18Z init message: Verifying blocks…
2022-03-29T11:09:18Z Verifying last 6 blocks at level 3
2022-03-29T11:09:18Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2022-03-29T11:09:19Z No coin database inconsistencies in last 6 blocks (6258 transactions)
2022-03-29T11:09:19Z  block index           13062ms
2022-03-29T11:09:19Z Opening LevelDB in /root/.bitcoin/indexes/txindex
2022-03-29T11:09:19Z Opened LevelDB successfully
2022-03-29T11:09:19Z Using obfuscation key for /root/.bitcoin/indexes/txindex: 0000000000000000
2022-03-29T11:09:19Z txindex thread start
2022-03-29T11:09:19Z txindex is enabled at height 394133
2022-03-29T11:09:19Z block tree size = 729500
2022-03-29T11:09:19Z nBestHeight = 394133
2022-03-29T11:09:19Z txindex thread exit
2022-03-29T11:09:19Z loadblk thread start
2022-03-29T11:09:19Z Failed to open mempool file from disk. Continuing anyway.
2022-03-29T11:09:19Z loadblk thread exit
2022-03-29T11:09:19Z Bound to 0.0.0.0:8333
2022-03-29T11:09:19Z Bound to 127.0.0.1:8334
2022-03-29T11:09:19Z init message: Loading P2P addresses…
2022-03-29T11:09:19Z torcontrol thread start
2022-03-29T11:09:19Z Loaded 16730 addresses from peers.dat  113ms
2022-03-29T11:09:19Z Missing or invalid file /root/.bitcoin/anchors.dat
2022-03-29T11:09:19Z 0 block-relay-only anchors will be tried for connections.
2022-03-29T11:09:19Z init message: Starting network threads…
2022-03-29T11:09:19Z net thread start
2022-03-29T11:09:19Z addcon thread start
2022-03-29T11:09:19Z dnsseed thread start
2022-03-29T11:09:19Z init message: Done loading
2022-03-29T11:09:19Z Waiting 300 seconds before querying DNS seeds.
2022-03-29T11:09:19Z opencon thread start
2022-03-29T11:09:19Z msghand thread start
2022-03-29T11:09:20Z New outbound peer connected: version: 70015, blocks=729542, peer=0 (outbound-full-relay)
2022-03-29T11:09:20Z Synchronizing blockheaders, height: 729542 (~100.00%)
2022-03-29T11:09:21Z UpdateTip: new best=00000000000000000837d8df4299feed2573af12778081bf041fe8f8ebd63df8 height=394134 version=0x00000004 log2_work=83.943519 tx=104619196 date='2016-01-20T07:09:42Z' progress=0.147454 cache=0.9MiB(6790txo)
2022-03-29T11:09:21Z UpdateTip: new best=000000000000000007fd5fa7c84e7f71f2fb0c2ee1625fb226daa5750ee25502 height=394135 version=0x00000004 log2_work=83.943557 tx=104620197 date='2016-01-20T07:19:19Z' progress=0.147455 cache=1.9MiB(14324txo)
2022-03-29T11:09:21Z Fatal LevelDB error: Corruption: block checksum mismatch: /root/.bitcoin/indexes/txindex/032765.ldb
2022-03-29T11:09:21Z You can use -debug=leveldb to get more complete diagnostic messages
2022-03-29T11:09:21Z

************************
EXCEPTION: 15dbwrapper_error
Fatal LevelDB error: Corruption: block checksum mismatch: /root/.bitcoin/indexes/txindex/032765.ldb
bitcoin in scheduler

ProofOfKeags commented 2 years ago

And you said that disabling the txindex directive caused it to recover itself?

elvece commented 2 years ago

And you said that disabling the txindex directive caused it to recover itself?

Yes. Did not have to reinstall bitcoin core. Toggling txindex off allowed progress to continue. Started sync on 3/28, noticed this issue morning of 3/29, afternoon of 3/30 sync is at 87.33%.

ProofOfKeags commented 2 years ago

It appears from your logs that there is a leveldb file that is corrupted in the txindex data.

Fatal LevelDB error: Corruption: block checksum mismatch: /root/.bitcoin/indexes/txindex/032765.ldb

Corruption can happen if it is aggressively killed, perhaps the solution here is to have an action wipe the txindex.

chrisguida commented 2 years ago

This is almost certainly a hardware issue, and nothing to do with bitcoind. Disk corruptions from I/O errors affect all data, not just txindex

ProofOfKeags commented 2 years ago

yeah but unfortunately we have to be able to recover from these sorts of corruptions. We could do it on an adhoc basis but as it stands right now, if @elvece was a customer, this would be a support issue and one that would require a post-init script or ssh to resolve.

chrisguida commented 2 years ago

Sounds like she just turned off txindex in the config

ProofOfKeags commented 2 years ago

Right but she can't turn it back on anymore which means it is "stuck" until the corruption is cleaned up. Realistically this feels like a bug in Core, but we have a limited ability to do anything about that.

chrisguida commented 2 years ago

Ok let's add an action to delete the txindex if it gets corrupted

chrisguida commented 2 years ago

Fixed by https://github.com/Start9Labs/bitcoind-wrapper/pull/78

Start9Labs / bitcoind-startos

Add action to delete txindex if it gets corrupted #60