[BUG] terrad start panics after upgrade to v0.5.16

[LCyson]

We had problem restarting our terra node after sync to version 0.5.16, does anyone know how to fix this issue? Thanks in advance!

[ec2-user@ip-172-31-43-156 core]$ terrad start --x-crisis-skip-assert-invariants
12:23AM INF starting ABCI with Tendermint
12:24AM INF Starting multiAppConn service impl=multiAppConn module=proxy
12:24AM INF Starting localClient service connection=query impl=localClient module=abci-client
12:24AM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
12:24AM INF Starting localClient service connection=mempool impl=localClient module=abci-client
12:24AM INF Starting localClient service connection=consensus impl=localClient module=abci-client
12:24AM INF Starting EventBus service impl=EventBus module=events
12:24AM INF Starting PubSub service impl=PubSub module=pubsub
12:24AM INF Starting IndexerService service impl=IndexerService module=txindex
12:24AM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=0.5.16
12:24AM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=4724001
12:30AM INF created new capability module=ibc name=ports/transfer
12:30AM INF port binded module=x/ibc/port port=transfer
12:30AM INF claimed capability capability=1 module=transfer name=ports/transfer
panic: StoreBlockHeight (4724001) > StateBlockHeight + 1 (1)

goroutine 1 [running]:
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc000056110, 0x7}}, {0xc000056140, 0xa}, 0x481521, 0x0, {{0x0, ...}, ...}, ...}, ...)
        github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:382 +0xadf
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc039f75e50, {0x23bfb80, 0xc0000c0000})
        github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:268 +0x3c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc000056110, 0x7}}, {0xc000056140, 0xa}, 0x481521, 0x0, ...}, ...)
        github.com/tendermint/tendermint@v0.34.14/node/node.go:325 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc001081900, {0x236a050, 0xc0011e0b40}, 0xc0000d5220, {0x234c880, 0xc000133ea8}, 0x0, 0x0, 0xc0000d5390, {0x2391400, ...}, ...)
        github.com/tendermint/tendermint@v0.34.14/node/node.go:733 +0x578
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x23d1618, 0xc00116e060}, {0x0, 0x0}, {0x2399c10, 0xc001015cf0}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.44.5/server/start.go:269 +0x765
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc00109f900, {0xc001154250, 0x0, 0x1})
        github.com/cosmos/cosmos-sdk@v0.44.5/server/start.go:133 +0x193
github.com/spf13/cobra.(*Command).execute(0xc00109f900, {0xc001154240, 0x1, 0x1})
        github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000faaa00)
        github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.2.1/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra@v1.2.1/command.go:895
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0, {0xc000d57530, 0x15})
        github.com/cosmos/cosmos-sdk@v0.44.5/server/cmd/execute.go:36 +0x1eb
main.main()
        github.com/terra-money/core/cmd/terrad/main.go:15 +0x2c

[Jared-TFL]

Join the validator discord.

https://discord.gg/tHcmyJHrPJ

[HS-Joe]

Are you syncing from scratch from block 0? At block 4724000 there was the col-5 upgrade (see https://discord.com/channels/566086600560214026/566086708811268127/892031556590387230) that required the use of halt-height = 4724000 and a binary swap.

[BranfordSWBF]

@TS-Joe0x could you elaborate on what steps one should take for the binary swap?

[HS-Joe]

That depends if it's an hardfork or softfork. For hardforks the chain will stop at the specified height without nothing to do, then you'll need to replace the binary with the new version. For soft ones, the binary swap must be done before so at the expected height the node doesn't panic and corrupt the db. All this is usually announced in the discord, but in the case you sync from 0, you need to dig up the older posts to understand at which heights you need to upgrade. Using halt-height is useful in this case.

[HS-Joe]

That being said, I've never seen anyone trying to sync from scratch, we usually use https://quicksync.io/networks/terra.html snapshots

[LCyson]

resolved after moving data/ from ~/.terrad to ~/.terra