search

palomachain / paloma

The fast blockchain messenger protocol
Apache License 2.0
291 stars 135 forks source link

Vitwit Support: SDK 04.7.1 to 0.50.5 ERR failed to get consensus params err="collections: not found: key 'no_key' of type tendermint.types.ConsensusParams" module=server #1138

Closed taariq closed 7 months ago

taariq commented 7 months ago

Background

@verabehr experienced this bug in running a test of exported genesis. We believe this is an outstanding issue from SDK 0.47.1 upgrade to SDK 0.50.5.

Error on Genesis 04/18/24: 5:57PM INF Found height=0 index=0 module=consensus wal=/root/.paloma/data/cs.wal/wal 5:57PM INF Catchup by replaying consensus messages height=15841320 module=consensus 5:57PM INF Replay: Done module=consensus 5:57PM INF Timed out dur=-142.861331 height=15841320 module=consensus round=0 step=RoundStepNewHeight 5:57PM INF commit is for a block we do not know about; set ProposalBlock=nil commit=F00F5205FB5EBDBA5F730187E463FB54C07C049B65EA77623F92CF1865AC42EC commit_round=0 height=15841320 module=consensus proposal= 5:57PM INF received proposal module=consensus proposal="Proposal{15841320/0 (0985B9F8B091911D9D5BB0D1E544008F93A0E21A7D30E1DECB96EFC5E210BD72:1:0CCA47416FDD, -1) 3224FBAB1862 @ 2024-04-18T14:36:57.14654553Z}" proposer=7214E5E75F03183ABD23FA4BA55074A7123BFEDA 5:57PM ERR failed to get consensus params err="collections: not found: key 'no_key' of type github.com/cosmos/gogoproto/tendermint.types.ConsensusParams" module=server 5:57PM INF received complete proposal block hash=F00F5205FB5EBDBA5F730187E463FB54C07C049B65EA77623F92CF1865AC42EC height=15841320 module=consensus 5:57PM ERR CONSENSUS FAILURE!!! err="+2/3 committed an invalid block: wrong Block.Header.ConsensusHash. Expected 5BFF5806D28442899AA8FBFB61FC8F1A2336D92FDE8053D81288293ACA5A4082, got 048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F" module=consensus stack="goroutine 545

Done when

To solve it, in the upgrade handler you need to create a new consensus params keeper using the wrong key again and move the data into the consensus params keeper with the right key.

Source: https://github.com/cosmos/cosmos-sdk/issues/18733#issuecomment-1854611049

taariq commented 7 months ago

@vishal-kanna okay. reposting.

  1. Yes, we have this error in the wild
  2. Here's the logs: https://gist.githubusercontent.com/pcheliniy/f55e1cde67eefc9e9b4ab4d74dd07a95/raw/60ff31f598a9ea367b16d297badcf3c7221907d6/paloma_logs_13_2_1
  3. Versions 
    root@sf2:/#  palomad version --long | grep commit
commit: d49942927324392c8a3db45fa35c6f3b1ee73f24
root@sf2:/# sha256sum ~/.paloma/config/genesis.json
57ae12350e9f38d84419dccabe283704b034db1fd16e0071243edaecad0ea9c3  /root/.paloma/config/genesis.json
verabehr commented 7 months ago

user also confirmed they did palomad comet unsafe-reset-all --home $HOME/.paloma after getting the new genesis file

verabehr commented 7 months ago 
Apr 18 21:09:21 mainnet-validator palomad[846116]: {"level":"info","module":"server","module":"consensus","height":15841320,"commit_round":0,"proposal":"","commit":"F00F5205FB5EBDBA5F730187E463FB54C07C049B65EA77623F92CF1865AC42EC","time":"2024-04-18T21:09:21Z","message":"commit is for a block we do not know about; set ProposalBlock=nil"}
Apr 18 21:09:22 mainnet-validator palomad[846116]: {"level":"info","module":"server","module":"consensus","height":15841320,"hash":"F00F5205FB5EBDBA5F730187E463FB54C07C049B65EA77623F92CF1865AC42EC","time":"2024-04-18T21:09:22Z","message":"received complete proposal block"}
Apr 18 21:09:22 mainnet-validator palomad[846116]: {"level":"error","module":"server","module":"consensus","err":"+2/3 committed an invalid block: wrong Block.Header.ConsensusHash.  Expected 5BFF5806D28442899AA8FBFB61FC8F1A2336D92FDE8053D81288293ACA5A4082, got 048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F","stack":"goroutine 512 [running]:\nruntime/debug.Stack()\n\t/opt/hostedtoolcache/go/1.22.2/x64/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\t/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:801 +0x46\npanic({0x3187e60?, 0xc00781d8e0?})\n\t/opt/hostedtoolcache/go/1.22.2/x64/src/runtime/panic.go:770 +0x132\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc005105508, 0xf1b828)\n\t/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:1713 +0x1045\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc005105508, 0xf1b828)\n\t/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:1682 +0x2e8\ngithub.com/cometbft/cometbft/consensus.(*State).handleCompleteProposal(0xc005105508, 0xf1b828)\n\t/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:2060 +0x372\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0xc005105508, {{0x46bd1c0, 0xc0078251b8}, {0xc0084de2a0, 0x28}})\n\t/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:910 +0x178\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc005105508, 0x0)\n\t/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:836 +0x3f1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 326\n\t/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:398 +0x10c\n","time":"2024-04-18T21:09:22Z","message":"CONSENSUS FAILURE!!!"}
taariq commented 7 months ago

@vishal-kanna the proposed fix is here: https://github.com/cosmos/cosmos-sdk/issues/18733#issuecomment-1854611049

taariq commented 7 months ago

This was fixed.