search

dydxprotocol / v4-chain

dydx v4 protocol
Other
204 stars 86 forks source link

Successful statesync is followed by a panic #791

Open qezz opened 7 months ago

qezz commented 7 months ago

Have been setting up a sentry on the dydx-testnet and mainnet. Statesync fetches and applies the chunks just fine, but then it's followed up with a call to PruneRateLimits() (or at least that's what the error message says).

Tested on dydx-testnet. Not tested on the mainnet.

...
10:53AM INF Applied snapshot chunk to ABCI app chunk=1 format=3 height=1654000 module=statesync total=2
10:53AM INF Verified ABCI app appHash="~��\x00Σ*r��k\x7f�UQo����lj\x1b�����\t���" height=1654000 module=statesync
10:53AM INF Snapshot restored format=3 hash="��w�\x05 \x18H�CC\f�[Q(o�$38\x00g[�pmל���" height=1654000 module=statesync
10:53AM INF service start impl=BlockPool module=blockchain msg={}
10:53AM INF Price update had length of 0 module=pricefeed-daemon submodule=price-updater
10:53AM ERR runTx panic'ed with PerformStatefulPremiumVotesValidation failed, err = PerformStatefulPremiumVotesValidation: failed to determine ClobPair sta
tus for perpetual with id 0: Perpetual ID 0 has no associated CLOB pairs: The provided perpetual ID does not have any associated CLOB pairs [dydxprotocol/v
4-chain/protocol/x/clob/keeper/clob_pair.go:244] /x/perpetuals=true module=server proposer_cons_addr=dydxvalcons1wr9fc5en9jzg3dh5t2vzahfqc8n8dd0h9jx0gt sta
ck trace="goroutine 1981 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngithub.com/dydxprotocol/v4-chain/protocol/app.New.NewRunTxPa
nicLoggingMiddleware.func8({0x242d5a0, 0xc033dcb6b0})\n\tgithub.com/dydxprotocol/v4-chain/protocol/app/middleware/middleware.go:20 +0x2f\ngithub.com/dydxpr
otocol/v4-chain/protocol/app.New.(*BaseApp).AddRunTxRecoveryHandler.newRecoveryMiddleware.func9({0x242d5a0?, 0xc033dcb6b0?})\n\tgithub.com/cosmos/cosmos-sd
k@v0.47.4/baseapp/recovery.go:39 +0x2d\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x242d5a0, 0xc033dcb6b0}, 0xc000600000?)\n\tgithub.com/cosmos
/cosmos-sdk@v0.47.4/baseapp/recovery.go:28 +0x2f\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x242d5a0, 0xc033dcb6b0}, 0x240?)\n\tgithub.com/cos
mos/cosmos-sdk@v0.47.4/baseapp/recovery.go:33 +0x53\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()\n\tgithub.com/cosmos/cosmos-sdk@v0.47.4/
baseapp/baseapp.go:783 +0xfe\npanic({0x242d5a0?, 0xc033dcb6b0?})\n\truntime/panic.go:920 +0x270\ngithub.com/dydxprotocol/v4-chain/protocol/x/perpetuals/kee
per.msgServer.AddPremiumVotes({{0x362d0f8?, 0xc0007a29a0?}}, {0x3613da8?, 0xc033fdb710?}, 0xc033d15518)\n\tgithub.com/dydxprotocol/v4-chain/protocol/x/perp
etuals/keeper/msg_server_add_premium_votes.go:20 +0x1fd\ngithub.com/dydxprotocol/v4-chain/protocol/x/perpetuals/types._Msg_AddPremiumVotes_Handler.func1({0
x3613da8, 0xc033fdb710}, {0x2850860?, 0xc033d15518})\n\tgithub.com/dydxprotocol/v4-chain/protocol/x/perpetuals/types/tx.pb.go:734 +0x72\ngithub.com/cosmos/
cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x3613f30, 0xc033fe42c0}, {0xc036bd06a0?, 0x4104c5?}, 0x2a0?, 0xc033d15590)\n\tgithub.com/c
osmos/cosmos-sdk@v0.47.4/baseapp/msg_service_router.go:118 +0x93\ngithub.com/dydxprotocol/v4-chain/protocol/x/perpetuals/types._Msg_AddPremiumVotes_Handler
({0x266bba0?, 0xc001359d20}, {0x3613f30, 0xc033fe42c0}, 0x30c0938, 0xc033dd1160)\n\tgithub.com/dydxprotocol/v4-chain/protocol/x/perpetuals/types/tx.pb.go:7
36 +0x135\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x3613ad0, 0x4db2ce0}, {0x362a7b0, 0xc033d3ae40}, {{0xb, 0x0},
{0xc010b00010, 0xe}, 0x193cf1, {0x636b96e, ...}, ...}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.47.4/baseapp/msg_service_router.go:139 +0x343\ngithub.
com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x3613ad0, 0x4db2ce0}, {0x362a7b0, 0xc033d3ae40}, {{0xb, 0x0}, {0xc010b00010, 0xe}, 0x193cf1, ...}, .
..}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:942 +0x271\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc00092f200, 0x3, {
0xc0163df6c0, 0x3c, 0x3c})\n\tgithub.com/cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:885 +0xe05\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx
ShouldLock(0xc00092f200, {{0xc0163df6c0?, 0x0?, 0x0?}}, 0x8?)\n\tgithub.com/cosmos/cosmos-sdk@v0.47.4/baseapp/abci.go:449 +0x252\ngithub.com/cosmos/cosmos-
sdk/baseapp.(*BaseApp).DeliverTx(...)\n\tgithub.com/cosmos/cosmos-sdk@v0.47.4/baseapp/abci.go:419\ngithub.com/cometbft/cometbft/abci/client.(*unsyncLocalCl
ient).DeliverTxAsync(0xc00035db90, {{0xc0163df6c0?, 0x0?, 0x0?}})\n\tgithub.com/cometbft/cometbft@v0.37.2/abci/client/unsync_local_client.go:73 +0x56\ngith
ub.com/cometbft/cometbft/proxy.(*appConnConsensus).DeliverTxAsync(0xc001019f38, {{0xc0163df6c0?, 0x20?, 0xb?}})\n\tgithub.com/cometbft/cometbft@v0.37.2/pro
xy/app_conn.go:106 +0x163\ngithub.com/cometbft/cometbft/state.execBlockOnProxyApp({0x3613ef8?, 0xc0019834c0}, {0x3627880, 0xc001019f38}, 0xc0166fe5a0, {0x3
62af70, 0xc000300540}, 0x193cf0?)\n\tgithub.com/cometbft/cometbft@v0.37.2/state/execution.go:378 +0x6b6\ngithub.com/cometbft/cometbft/state.(*BlockExecutor
).ApplyBlock(_, {{{0xb, 0x0}, {0x292fd91, 0x6}}, {0xc001196700, 0xe}, 0x1, 0x193cf0, {{0xc006f40a00, ...}, ...}, ...}, ...)\n\tgithub.com/cometbft/cometbft
@v0.37.2/state/execution.go:210 +0x329\ngithub.com/cometbft/cometbft/blocksync.(*Reactor).poolRoutine(0xc001dae700, 0x1)\n\tgithub.com/cometbft/cometbft@v0
.37.2/blocksync/reactor.go:389 +0xb5a\ncreated by github.com/cometbft/cometbft/blocksync.(*Reactor).SwitchToBlockSync in goroutine 245\n\tgithub.com/cometb
ft/cometbft@v0.37.2/blocksync/reactor.go:124 +0xde\n"
panic: Unexpected invocation of PruneRateLimits

goroutine 1981 [running]:
github.com/dydxprotocol/v4-chain/protocol/x/clob/rate_limit.panicRateLimiter[...].PruneRateLimits(...)
        github.com/dydxprotocol/v4-chain/protocol/x/clob/rate_limit/panic_rate_limiter.go:23
github.com/dydxprotocol/v4-chain/protocol/x/clob/keeper.(*Keeper).PruneRateLimits(_, {{0x3613ad0, 0x4db2ce0}, {0x362a7b0, 0xc033517b80}, {{0xb, 0x0}, {0xc0
10b00010, 0xe}, 0x193cf1, ...}, ...})
        github.com/dydxprotocol/v4-chain/protocol/x/clob/keeper/rate_limit.go:67 +0x5a
github.com/dydxprotocol/v4-chain/protocol/x/clob.EndBlocker({{0x3613ad0, 0x4db2ce0}, {0x362a7b0, 0xc033517b80}, {{0xb, 0x0}, {0xc010b00010, 0xe}, 0x193cf1,
 {0x636b96e, ...}, ...}, ...}, ...)
        github.com/dydxprotocol/v4-chain/protocol/x/clob/abci.go:108 +0x9c9
github.com/dydxprotocol/v4-chain/protocol/x/clob.AppModule.EndBlock({{{0x362aa60, 0xc000c9f1c0}}, 0xc001b02ea0, {0x35e76a0, 0xc000c62e60}, {0x7f2b7c344968,
 0xc0010e6420}, {0x3626fd0, 0xc001412900}, 0xc000c4ac80}, ...)
qezz commented 7 months ago

Tried the same thing on the mainnet, and I can see the same panic happening.