search

ComposableFi / composable-cosmos

The home of IBC
28 stars 95 forks source link

Panic on testnet 7.0.1 during slashing event #526

Open joe-bowman opened 3 months ago

joe-bowman commented 3 months ago

During a slashing event 51 blocks after the sdk50 testnet restart, the chain halted with the following stack trace:

May 30 16:16:05 composable-shared-artifacts-relayer-banksy5node-01 picad[234837]: 4:16PM INF service start impl=TimeoutTicker module=consensus msg="Starting TimeoutTicker service"
May 30 16:16:06 composable-shared-artifacts-relayer-banksy5node-01 picad[234837]: 4:16PM INF received proposal module=consensus proposal="Proposal{2137051/0 (9E6B1EBBAA6C39968D995DBB0112AB9FA74CF188A307AFC78F7AFF73A9B39968:1:275F7BEA5E7E, -1) E3FF88B2C838 @ 2024-05-30T16:10:43.52162477Z}" proposer=45A57A0D53D35464622CA36A938437AE93417767
May 30 16:16:06 composable-shared-artifacts-relayer-banksy5node-01 picad[234837]: 4:16PM INF received complete proposal block hash=9E6B1EBBAA6C39968D995DBB0112AB9FA74CF188A307AFC78F7AFF73A9B39968 height=2137051 module=consensus
May 30 16:16:06 composable-shared-artifacts-relayer-banksy5node-01 picad[234837]: 4:16PM INF finalizing commit of block hash=9E6B1EBBAA6C39968D995DBB0112AB9FA74CF188A307AFC78F7AFF73A9B39968 height=2137051 module=consensus num_txs=0 root=E5EF6AD01B8E418C59F6113970FDCDD31CBAF83539E8BBC9CA54FE34B7F4E82D
May 30 16:16:06 composable-shared-artifacts-relayer-banksy5node-01 picad[234837]: 4:16PM INF validator slashed by slash factor burned=904392225200000000000 module=x/staking slash_factor=0.010000000000000000 validator=picavaloper1l2xrj70jx539ln7ms9usn6t6vk5vpwe3zx7ena
May 30 16:16:06 composable-shared-artifacts-relayer-banksy5node-01 picad[234837]: 4:16PM ERR CONSENSUS FAILURE!!! err="runtime error: invalid memory address or nil pointer dereference" module=consensus stack="goroutine 803 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:801 +0x46\npanic({0x2c79de0?, 0x56746a0?})\n\truntime/panic.go:770 +0x132\ngithub.com/notional-labs/composable/v6/custom/staking/keeper.Keeper.SlashWithInfractionReason({{{0x3a6ff00, 0xc000bc26c0}, {0x3ac5590, 0xc0021e6130}, {0x3ab20c8, 0xc00172e8c0}, {0x3ac12a0, 0xc000ad9808}, {0x3ac5910, 0xc000a37320}, ...}, ...}, ...)\n\tgithub.com/notional-labs/composable/v6/custom/staking/keeper/keeper.go:175 +0x419\ngithub.com/cosmos/cosmos-sdk/x/slashing/keeper.Keeper.HandleValidatorSignature({{0x3a6ff00, 0xc000bc26d8}, {0x3ac5590, 0xc0021e6130}, 0xc000bc2330, {0x3ac88a0, 0xc000736a08}, {0xc0011336e0, 0x2b}}, {0x3aa96a8, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.50.5/x/slashing/keeper/infractions.go:132 +0xc23\ngithub.com/cosmos/cosmos-sdk/x/slashing.BeginBlocker({0x3aa96a8, 0xc2808da008}, {{0x3a6ff00, 0xc000bc26d8}, {0x3ac5590, 0xc0021e6130}, 0xc000bc2330, {0x3ac88a0, 0xc000736a08}, {0xc0011336e0, ...}})\n\tgithub.com/cosmos/cosmos-sdk@v0.50.5/x/slashing/abci.go:25 +0x245\ngithub.com/cosmos/cosmos-sdk/x/slashing.AppModule.BeginBlock(...)\n\tgithub.com/cosmos/cosmos-sdk@v0.50.5/x/slashing/module.go:166\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x3aa8f38, 0x58776a0}, {0x3ac60f0, 0xc280709280}, {{0x0, 0x0}, {0xc0021b6800, 0x10}, 0x209bdb, ...}, ...})\n\tgithub.com/cosmos/cosmos-sdk@v0.50.5/types/module/module.go:779 +0x188\ngithub.com/notional-labs/composable/v6/app.(*ComposableApp).BeginBlocker(_, {{0x3aa8f38, 0x58776a0}, {0x3ac60f0, 0xc280709280}, {{0x0, 0x0}, {0xc0021b6800, 0x10}, 0x209bdb, ...}, ...})\n\tgithub.com/notional-labs/composable/v6/app/app.go:610 +0x9a\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).beginBlock(0xc00169e6c8, 0xc2807155c0?)\n\tgithub.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:730 +0xb5\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc00169e6c8, {0x3aa8f38, 0x58776a0}, 0xc2807155c0)\n\tgithub.com/cosmos/cosmos-sdk@v0.50.5/baseapp/abci.go:760 +0xdfe\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc00169e6c8, 0xc2807155c0)\n\tgithub.com/cosmos/cosmos-sdk@v0.50.5/baseapp/abci.go:884 +0x16e\ngithub.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)\n\tgithub.com/cosmos/cosmos-sdk@v0.50.5/server/cmt_abci.go:44\ngithub.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0xc000eb9ae0?, {0x3aa98a0?, 0x58776a0?}, 0x7e0efcfb18b0?)\n\tgithub.com/cometbft/cometbft@v0.38.6/abci/client/local_client.go:185 +0xcd\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc001360c30, {0x3aa98a0, 0x58776a0}, 0xc2807155c0)\n\tgithub.com/cometbft/cometbft@v0.38.6/proxy/app_conn.go:104 +0x170\ngithub.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc01da6e31a, 0x6}}, {0xc01da6e330, 0x10}, 0x1, 0x209bda, {{0xc27eca2aa0, ...}, ...}, ...}, ...)\n\tgithub.com/cometbft/cometbft@v0.38.6/state/execution.go:213 +0x5c5\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc003e42a88, 0x209bdb)\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:1771 +0xb6c\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc003e42a88, 0x209bdb)\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:1682 +0x2e8\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:1617 +0x9c\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit(0xc003e42a88, 0x209bdb, 0x0)\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:1655 +0xc3b\ngithub.com/cometbft/cometbft/consensus.(*State).addVote(0xc003e42a88, 0xc2805fe9c0, {0xc003216a80, 0x28})\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:2334 +0x1c6d\ngithub.com/cometbft/cometbft/consensus.(*State).tryAddVote(0xc003e42a88, 0xc2805fe9c0, {0xc003216a80?, 0x0?})\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:2066 +0x26\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0xc003e42a88, {{0x3a79060, 0xc27b853ee0}, {0xc003216a80, 0x28}})\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:929 +0x39c\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc003e42a88, 0x0)\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:836 +0x3f1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 33\n\tgithub.com/cometbft/cometbft@v0.38.6/consensus/state.go:398 +0x10c\n"

On inspection, it appears the authKeeper field is not set on the custom staking keeper.