[Bug]: Panic after remote snapshot restart on n05

[daniel1302]

Problem encountered

We restarted n05.devnet1.vega.rocks with remote snapshot + unsafe reset all and we saw the panic

Observed behaviour

Panic after remote snapshot restart

Expected behaviour

Server should not panic

Steps to reproduce

1. Start the network with the below version
2. Wait for some snapshots and segments to be produced
3. Try to restart one of the node with remote snapshot

Software version

Vega CLI v0.72.1 (42d436fc016cf5425c826d8f8935ec66cb20199e)

Failing test

No response

Jenkins run

No response

Configuration used

No response

Relevant log output

Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: 2023-08-28T20:05:10.075Z        INFO        core.protocol.snapshot        snapshot/engine.go:320        snapshot engine successfully initialised
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: panic: runtime error: invalid memory address or nil pointer dereference
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc483a2]
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: goroutine 458 [running]:
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: code.vegaprotocol.io/vega/core/types.KeyFromPayload({0x0, 0x0})
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/work/vega/vega/core/types/snapshot_nodes.go:4272 +0x22
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: code.vegaprotocol.io/vega/core/types.(*Payload).GetTreeKey(...)
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/work/vega/vega/core/types/snapshot_nodes.go:865
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: code.vegaprotocol.io/vega/core/types.(*Snapshot).TreeFromSnapshot(0xc00725d320, 0x0?)
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/work/vega/vega/core/types/snapshot.go:200 +0x1ea
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: code.vegaprotocol.io/vega/core/snapshot.(*Engine).ApplySnapshot(0xc000e9aa80, {0x373f740?, 0x355d5ad?})
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/work/vega/vega/core/snapshot/engine.go:538 +0x57
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: code.vegaprotocol.io/vega/core/processor.(*App).ApplySnapshotChunk(0xc000d5d800, {0x3e2b338, 0xc000074060}, {0x0, {0xc007f80000, 0x161485, 0x161486}, {0xc0001293b0, 0x28}})
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/work/vega/vega/core/processor/abci.go:670 +0x788
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: code.vegaprotocol.io/vega/core/blockchain/abci.(*App).ApplySnapshotChunk(0x7f76dbf17e00?, {0x0, {0xc007f80000, 0x161485, 0x161486}, {0xc0001293b0, 0x28}})
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/work/vega/vega/core/blockchain/abci/abci.go:182 +0xc3
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: code.vegaprotocol.io/vega/cmd/vega/commands/node.(*appW).ApplySnapshotChunk(0x41170b?, {0x0, {0xc007f80000, 0x161485, 0x161486}, {0xc0001293b0, 0x28}})
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/work/vega/vega/cmd/vega/commands/node/app_wrapper.go:92 +0xa9
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: github.com/tendermint/tendermint/abci/client.(*localClient).ApplySnapshotChunkSync(0xc000f1f3e0, {0x0, {0xc007f80000, 0x161485, 0x161486}, {0xc0001293b0, 0x28}})
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/abci/client/local_client.go:322 +0x142
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: github.com/tendermint/tendermint/proxy.(*appConnSnapshot).ApplySnapshotChunkSync(0xc006e05310?, {0x0, {0xc007f80000, 0x161485, 0x161486}, {0xc0001293b0, 0x28}})
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/proxy/app_conn.go:194 +0x47
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: github.com/tendermint/tendermint/statesync.(*syncer).applyChunks(0xc0051e85a0, 0x3e2b370?)
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/statesync/syncer.go:367 +0x13a
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: github.com/tendermint/tendermint/statesync.(*syncer).Sync(_, _, _)
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/statesync/syncer.go:303 +0xc0a
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: github.com/tendermint/tendermint/statesync.(*syncer).SyncAny(_, _, _)
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/statesync/syncer.go:185 +0x4be
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: github.com/tendermint/tendermint/statesync.(*Reactor).Sync(_, {_, _}, _)
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/statesync/reactor.go:303 +0x4d7
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: github.com/tendermint/tendermint/node.startStateSync.func1()
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/node/node.go:675 +0xc5
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]: created by github.com/tendermint/tendermint/node.startStateSync
Aug 28 20:05:10 n05.devnet1.vega.rocks visor[2703497]:         /home/runner/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/node/node.go:674 +0x48c

[jeremyletang]

Closing as it was using an old version of the code