Post Mortem: Understanding reasons behind the chain halt

[byte-bandit]

We need to build a better understand of what exactly was the root cause behind the different issues encountered in main net after the upgrade to Cosmos SDK v0.50.5 that ultimately ended in a chain restart.

After careful consolidation with the devs over at Cosmos, we have arrived at the following facts:

• About 1/3 of active validators failed to perform the upgrade to Paloma v1.13.0, which introduced the upgrade to Cosmos SDK v0.50.5. Those validators were unable to join the chain again, neither by using state synching, snapshots or rewinds.
• Encountered problems consisted mostly of AppHash errors (not BlockHash), without further indications of underlaying erros in the log files, as well as missing keys from a store, see:
  • https://github.com/palomachain/paloma/issues/1106
  • https://github.com/palomachain/paloma/issues/1115
  • https://github.com/palomachain/paloma/issues/1121
• During the lifetime of Paloma v1.13.0 (approx 2 weeks), additional validators eventually encountered an AppHash error and remained unable to rejoin the network, no matter which strategy was attempted. In some instances, these validators had been part of the active valset for more than a week before encountering this issue.
• The team then made additional changes around existing upgrade handlers, IAVL version as well as CometBFT version and released Paloma v1.13.1
• During the upgrade of v1.13.1, validators failed to build consensus, and most nodes encountered one or more problems described above
• Only a small subset of validates were able to upgrade without issues
• There seems to be no noticeable difference in node hardware, or self-built vs. binary release

Without additional information on what exactly caused the observed issues, Paloma remains at risk of losing validators and ending up in a chain halt again. It's crucial we build an understanding of the underlaying issues that led to the behaviours outlined above. Therefore, we have designed the following testing and debugging steps together with the Cosmos dev team:

1. Grab a snapshot made prior to the upgrade height of v1.13.1 and test loading this snapshot into a node of Paloma v1.13.1, verify whether it's able to replay all blocks and ready to form a network.
2. Grab a snapshot made prior to the upgrade height of v1.13.1, modify a version of Paloma v1.13.0 to use IAVL v1.0.3 only, no other changes. Verify whether the node is able to replay all blocks and ready to form a network.
3. Grab a snapshot made prior to the upgrade height of v1.13.1, load into v1.13.0 node, sync until upgrade height. Stop at upgrade height, modify DB to be a single validator (instead of a full valset) and see if network continues. See https://docs.cosmos.network/main/build/building-apps/app-testnet for more information.
4. IF the network continues, test different configurations to verify whether the issue is non-deterministic
5. IF the issue persists, use the https://github.com/crypto-com/python-iavl tool to inspect the state during upgrade height and identify differences

Resources

• Snapshot: TBD

[vishal-kanna]

@byte-bandit I'll start looking into it

[taariq]

Fixed with upgrade of IAVL to DyDX Version on mainnet https://github.com/palomachain/paloma/pull/1139/commits/e4e5ec212bf016ca1110622d7e0d4329bd611fd0