Corrupted store: Block ... referenced from index not found

[viquezclaudio]

During a replication test in the private testnet, one of the non-malicious validators panicked due to:

Oct 09 21:34:28 validator-3 nimiq-client[834121]: 2023-10-09T21:34:28.293515285Z ERROR panic | thread 'tokio-runtime-worker' panicked at 'Corrupted store: Block bc7758b2a44dce7a229f1db0c538fc7191659fad2f713e81403c059f903c1cfa referenced from index not found': blockchain/src/chain_store.rs:340

The scenario is as follows:

• There are 11 nodes in the network: 8 validators, 1 seed node, 1 faucet, and 1 prover
• 3 of the 11 validators are tainted: they will respond to erroneous information when they receive a request from another peer
• The test net is started fresh, from scratch.
• 45 minutes later, all the nodes (except the seed node) are shut down
• The state of all the nodes (except the seed node) is erased.
• The nodes are started again
• The nodes sync and produce a few blocks
• The panic is observed in validator3 (a non-tainted validator)

Before the panic happened, the block in question was tried to be applied in validator 3:

Oct 09 21:34:09 validator-3 nimiq-client[834121]: 2023-10-09T21:34:09.608464937Z DEBUG accounts             | Reverting block block=#2940191:MI:ba6968c1b7 is_skip=false num_transactions=0 num_equivocation_proofs=1
Oct 09 21:34:09 validator-3 nimiq-client[834121]: 2023-10-09T21:34:09.609283056Z WARN  verify               | Rejecting block block=#2940191:MI:270dd76019 block_root=15a4073f511d0ca5b0071f4e0c6a39354320cdeebcff3c1969b37b864c2ec35f history_root=3ff7cc4585389abb5044001b2dcf6dd8dfa0c6a16c1569fc4d01e9bab199e409 reason="History root doesn't match real history root"
Oct 09 21:34:09 validator-3 nimiq-client[834121]: 2023-10-09T21:34:09.609322040Z WARN  push                 | Rejecting block block=#2940191:MI:270dd76019 reason="bad state" error=Invalid block: Invalid history root error.sources=[Invalid history root]
Oct 09 21:34:09 validator-3 nimiq-client[834121]: 2023-10-09T21:34:09.609568972Z WARN  push                 | Rejecting block block=#2940192:MI:bc7758b2a4 reason="failed to apply for block while rebranching" fork_block=#2940191:MI:270dd76019 error=Invalid block: Invalid history root error.sources=[Invalid history root]
Oct 09 21:34:09 validator-3 nimiq-client[834121]: 2023-10-09T21:34:09.611095597Z WARN  queue                | Failed to push missing block bc7758b2a44dce7a229f1db0c538fc7191659fad2f713e81403c059f903c1cfa: Invalid fork

privtestnet_validator3.log

This block was rejected because it is invalid. However, some minutes later, panic is observed in Validator3 Full validator 3 logs are attached

The complete logs are located within the CC machine at: "privtestnet-logs-20230910-1"

[jsdanielh]

Looking at the issue, from the logs it looks like a bad logic here. I think it should be:

this.chain_store.remove_chain_info(
    &mut write_txn,
    &block.0,
    block.1.head.block_number(),
)

[viquezclaudio]

After the fix in #1888 the corrupted store panic is no longer observed, however, there is still "invalid history root" issues when reverting blocks, even when there are no tainted or malicious validators