search

neutron-org / neutron

Smart Contract platform secured by Cosmos Hub
https://neutron.org
Apache License 2.0
110 stars 91 forks source link

Bug: release v4.0.1 SIGABRT: abort #619

Closed bert2002 closed 1 month ago

bert2002 commented 1 month ago

Hi, seems that release v4.0.1 on mainnet is frequently dying of this:

Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | 4:16AM INF computed stake-weighted median price for currency pair currency_pair=DYDX/USD final_price=1325867413 module=server num_validators=88 percent_submitted=0.750363354130234126 threshold=0.667000000000000000
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | 4:16AM INF computed stake-weighted median price for currency pair currency_pair=INJ/USD final_price=2051794820 module=server num_validators=88 percent_submitted=0.750363354130234126 threshold=0.667000000000000000
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | 4:16AM INF computed stake-weighted median price for currency pair currency_pair=PEPE/USD final_price=91020448455 module=server num_validators=88 percent_submitted=0.750363354130234126 threshold=0.667000000000000000
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | 4:16AM INF computed stake-weighted median price for currency pair currency_pair=DYM/USD final_price=1393860613 module=server num_validators=88 percent_submitted=0.750363354130234126 threshold=0.667000000000000000
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | 4:16AM INF smallest non opt out power updated module=x/host-ccvconsumer power=400378
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | 4:16AM INF fee deduct ante handle fee=[{"amount":"12962","denom":"untrn"}] gas limit=864071 min gas prices=0.005300000000000000untrn module=server
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | SIGABRT: abort
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | PC=0x3ae1e5f m=7 sigcode=18446744073709551610
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | signal arrived during cgo execution
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | goroutine 561 gp=0xc0011776c0 m=7 mp=0xc000500008 [syscall]:
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | runtime.cgocall(0x2fe2ee0, 0xc03cade780)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         runtime/cgocall.go:157 +0x4b fp=0xc03cade758 sp=0xc03cade720 pc=0x4624cb
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmvm/v2/internal/api._C2func_execute(0x7f3f8f881500, {0x0, 0xc061359940, 0x20}, {0x0, 0xc02240fe10, 0xc7}, {0x0, 0xc06d7a86c0, 0xb9}, ...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         _cgo_gotypes.go:307 +0x71 fp=0xc03cade780 sp=0xc03cade758 pc=0x1dad431
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmvm/v2/internal/api.Execute.func1({0x7f3f8f881500?, {0xc0097bf5c0?}}, {0x80?, 0xc061359940?, 0xc03cadea80?}, {0xe?, 0xc02240fe10?, 0x3d80bc0?}, {0x0, 0xc06d7a86c0, ...}, ...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/CosmWasm/wasmvm/v2@v2.0.1/internal/api/lib.go:273 +0x245 fp=0xc03cade908 sp=0xc03cade780 pc=0x1db6d25
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmvm/v2/internal/api.Execute({0x7f3f8f881500?, {0xc0097bf5c0?}}, {0xc061359940?, 0x3?, 0x3?}, {0xc02240fe10?, 0x0?, 0x0?}, {0xc06d7a86c0, 0xb9, ...}, ...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/CosmWasm/wasmvm/v2@v2.0.1/internal/api/lib.go:273 +0x7a8 fp=0xc03cadec80 sp=0xc03cade908 pc=0x1db66c8
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmvm/v2.(*VM).Execute(0xc006cc6a68, {0xc061359940, 0x20, 0x20}, {{0xbb1ecb, 0x17e10d85ebba25ce, {0xc00937e3b0, 0x9}}, 0xc0677bbf3c, {{0xc088fde910, ...}}}, ...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/CosmWasm/wasmvm/v2@v2.0.1/lib_libwasmvm.go:181 +0x287 fp=0xc03cadeda8 sp=0xc03cadec80 pc=0x1dc1cc7
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.execute({{0x527cbc0, 0xc006cc8218}, {0x530cc30, 0xc000c7ef20}, {0x529be40, 0xc0097c7400}, {0x527caa0, 0xc006ccab60}, {0x527c8c0, 0xc006cc6678}, ...}, ...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/CosmWasm/wasmd@v0.51.0/x/wasm/keeper/keeper.go:417 +0xc42 fp=0xc03cae1130 sp=0xc03cadeda8 pc=0x2647362
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmd/x/wasm/keeper.msgServer.ExecuteContract({0xc03cae1400?}, {0x52c1168, 0xc07bf09f20}, 0xc07bbca500)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/CosmWasm/wasmd@v0.51.0/x/wasm/keeper/msg_server.go:124 +0x1ba fp=0xc03cae14b8 sp=0xc03cae1130 pc=0x265a79a
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmd/x/wasm/keeper.(*msgServer).ExecuteContract(0x76182c0?, {0x52c1168?, 0xc07bf09f20?}, 0x3c75d00?)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         <autogenerated>:1 +0x2e fp=0xc03cae14e8 sp=0xc03cae14b8 pc=0x267bbee
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler.func1({0x52c1168?, 0xc07bf09f20?}, {0x428ca40?, 0xc07bbca500?})
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/CosmWasm/wasmd@v0.51.0/x/wasm/types/tx.pb.go:2265 +0xcb fp=0xc03cae1520 sp=0xc03cae14e8 pc=0x1e41e6b
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2.1({0x52c04f0, 0xc07bf19188}, {0xc03cae15b0?, 0x469465?}, 0x358?, 0xc01c4dbcf8)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cosmos/cosmos-sdk@v0.50.7/baseapp/msg_service_router.go:175 +0x8a fp=0xc03cae1570 sp=0xc03cae1520 pc=0x160392a
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler({0x425ce00, 0xc006cc8438}, {0x52c04f0, 0xc07bf19188}, 0x4b298b8, 0xc07be29180)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/CosmWasm/wasmd@v0.51.0/x/wasm/types/tx.pb.go:2267 +0x143 fp=0xc03cae15c0 sp=0xc03cae1570 pc=0x1e41cc3
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2({{0x52c1168, 0xc07be73c80}, {0x52e05f0, 0xc07bcdbc40}, {{0x0, 0x0}, {0xc00937e3b0, 0x9}, 0xbb1ecb, {0x1bf2cbce, ...}, ...}, ...}, ...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cosmos/cosmos-sdk@v0.50.7/baseapp/msg_service_router.go:198 +0x37f fp=0xc03cae2068 sp=0xc03cae15c0 pc=0x160361f
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x52c1168, 0xc07be73c80}, {0x52e05f0, 0xc07bcdbc40}, {{0x0, 0x0}, {0xc00937e3b0, 0x9}, 0xbb1ecb, ...}, ...}, ...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cosmos/cosmos-sdk@v0.50.7/baseapp/baseapp.go:1015 +0x1e6 fp=0xc03cae2698 sp=0xc03cae2068 pc=0x15fe986
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc000706b48, 0x7, {0xc05bdda8c0, 0x1be, 0x1be})
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cosmos/cosmos-sdk@v0.50.7/baseapp/baseapp.go:953 +0x126c fp=0xc03cae6558 sp=0xc03cae2698 pc=0x15fdacc
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).deliverTx(0xc000706b48, {0xc05bdda8c0?, 0x1be?, 0x0?})
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cosmos/cosmos-sdk@v0.50.7/baseapp/baseapp.go:762 +0xc6 fp=0xc03cae6628 sp=0xc03cae6558 pc=0x15fbf26
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc000706b48, {0x52c0368, 0x8844b80}, 0xc06dcbad80)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cosmos/cosmos-sdk@v0.50.7/baseapp/abci.go:790 +0x119c fp=0xc03caea438 sp=0xc03cae6628 pc=0x15f385c
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc000706b48, 0xc06dcbad80)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cosmos/cosmos-sdk@v0.50.7/baseapp/abci.go:884 +0x16e fp=0xc03caea4b8 sp=0xc03caea438 pc=0x15f3fee
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/neutron-org/neutron/v4/app.(*App).FinalizeBlock(0xc0?, 0xa8?)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         <autogenerated>:1 +0x1f fp=0xc03caea4d8 sp=0xc03caea4b8 pc=0x2fbc4bf
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cosmos/cosmos-sdk@v0.50.7/server/cmt_abci.go:44
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cosmos/cosmos-sdk/server.(*cometABCIWrapper).FinalizeBlock(0x7f3fd7db9878?, {0x40?, 0xc000500008?}, 0xc06e5de1c0?)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         <autogenerated>:1 +0x2c fp=0xc03caea4f8 sp=0xc03caea4d8 pc=0x17f4a8c
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0x52e36e0?, {0x52c1248?, 0x8844b80?}, 0x7f3f81b888b8?)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/abci/client/local_client.go:185 +0xcd fp=0xc03caea568 sp=0xc03caea4f8 pc=0x11b15ad
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc005bd1cb0, {0x52c1248, 0x8844b80}, 0xc06dcbad80)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/proxy/app_conn.go:104 +0x170 fp=0xc03caea600 sp=0xc03caea568 pc=0x11b2bd0
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc01ff8177a, 0x6}}, {0xc01ff81790, 0x9}, 0x1, 0xbb1eca, {{0xc0797f01a0, ...}, ...}, ...}, ...)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/state/execution.go:213 +0x5c5 fp=0xc03caeab60 sp=0xc03caea600 pc=0x13df1a5
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc021dc4008, 0xbb1ecb)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/consensus/state.go:1771 +0xb6c fp=0xc03caeb528 sp=0xc03caeab60 pc=0x145ab8c
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc021dc4008, 0xbb1ecb)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/consensus/state.go:1682 +0x2e8 fp=0xc03caeb628 sp=0xc03caeb528 pc=0x1459f28
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/consensus/state.go:1617 +0x9c fp=0xc03caeb660 sp=0xc03caeb628 pc=0x1459c1c
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/consensus.(*State).enterCommit(0xc021dc4008, 0xbb1ecb, 0x0)
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/consensus/state.go:1655 +0xc3b fp=0xc03caeb7e0 sp=0xc03caeb660 pc=0x1459b1b
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/consensus.(*State).addVote(0xc021dc4008, 0xc06aa6add0, {0xc01ccb0930, 0x28})
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/consensus/state.go:2334 +0x1c6d fp=0xc03caebb30 sp=0xc03caeb7e0 pc=0x145f16d
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  | github.com/cometbft/cometbft/consensus.(*State).tryAddVote(0xc021dc4008, 0xc06aa6add0, {0xc01ccb0930?, 0x0?})
Jul 11 04:16:43 neutron-1 neutron[445255]: neutron-cosmos-1  |         github.com/cometbft/cometbft@v0.38.7/consensus/state.go:2066 +0x26 fp=0xc03caebb98 sp=0xc0:
...

Using the official docker container.

Many thanks, bert

pr0n00gler commented 1 month ago

Looking, thanks for the report

webmaster128 commented 1 month ago

Whats the neutrond query wasm libwasmvm-version on the failing node?

There is a bit of a version mismatch between this stacktrace (github.com/CosmWasm/wasmvm/v2@v2.0.1) and the neutrond go.sum. Not sure if that is the cause but this mismatch should not be there.

webmaster128 commented 1 month ago

Using the official docker container.

Does this mean you follow this tutoral and then building with make build-docker-image?

bert2002 commented 1 month ago

@webmaster128 I am using your image from Docker Hub.

bert2002 commented 1 month ago 
# docker exec -it neutron-cosmos-1 neutrond query wasm libwasmvm-version
2.0.1

and the image is neutronorg/neutron:v4.0.1.

pr0n00gler commented 1 month ago

@bert2002 the release image should be fixed now, please check

bert2002 commented 1 month ago

Hi, updated and seems good so far.