search

CosmWasm / wasmvm

Go bindings to the CosmWasm VM
Apache License 2.0
177 stars 102 forks source link

AnalyzeCode cause panic with invalid stack pointer #265

Closed yun-yeo closed 2 years ago

yun-yeo commented 3 years ago

We got the report from the community. I think it is very rare case and hard to reproduce.

Seems copyAndDestroyUnmanangedVector made this panic. Details: https://github.com/terra-money/core/issues/595

image

webmaster128 commented 3 years ago

Very interesting. Thanks for the report. The pointer 0x01 indicates an empty vector, which happens when required features is empty. However, I do not yet understand how they can end up in this line because they should be handled by the v.len == cusize(0) case above.

Do you know what system this happens on (OS and CPU)?

webmaster128 commented 3 years ago

Could you run the same thing with this patch to understand how the UnmanagedVector looks like? https://github.com/CosmWasm/wasmvm/pull/266

yun-yeo commented 3 years ago

Very interesting. Thanks for the report. The pointer 0x01 indicates an empty vector, which happens when required features is empty. However, I do not yet understand how they can end up in this line because they should be handled by the v.len == cusize(0) case above.

Do you know what system this happens on (OS and CPU)?

alpine:3.12 was used

yun-yeo commented 3 years ago

Could you run the same thing with this patch to understand how the UnmanagedVector looks like? #266

Okay, I will ask to reporter to run with the PR

webmaster128 commented 3 years ago

There is a good chance this is fixed in 0.16.2 & 1.0.0-beta2 due to this patch: https://github.com/CosmWasm/wasmvm/commit/ceaebca68ca2ddbda8cff6bcf2b89316e90121b1

webmaster128 commented 2 years ago

I'm still getting this in wasmvm 1.0.0 as part of wasmd 0.27.0 in a CI job. I'm getting the impression that this can happen when the system runs out of memory. AnalyzeCode is a bit memory intense as it loads a Wasm file into memory and deserializes the bytecode into a Module structure. According to this reddit conversation, most APIs in Rust do not check allocation failures. Now when this line fails to allocate memory, it has no result type to let us know: https://github.com/CosmWasm/wasmvm/blob/v1.0.0/libwasmvm/src/cache.rs#L208. Maybe it just gives us an unallocated vector with the 0x1 dummy pointer but the capacity of the data it wanted to allocate.

Logs ``` 10:47AM INF executed block height=426 module=state num_invalid_txs=0 num_valid_txs=1 10:47AM INF commit synced commit=436F6D6D697449447B5B302032323220313636203130342038342031373920323332203132392032302031343820383720313436203135392031313820393420313138203930203137342031352031303320313036203735203131392031373720333220323020313533203232332035203437203132392032375D3A3141417D 10:47AM INF committed state app_hash=00DEA66854B3E881149457929F765E765AAE0F676A4B77B1201499DF052F811B height=426 module=state num_txs=1 10:47AM INF indexed block height=426 module=txindex 10:47AM INF Timed out dur=961.611594 height=427 module=consensus round=0 step=1 10:47AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"A8EF5212491C91E7679A1AA6E633C4A2D82916EA6BF2480BDEEFC95AEC996F5E","parts":{"hash":"B0E09938290C5A55EDB678DC7A747A0D102D033FF0EFFF7B20F2E44A10F185B7","total":1}},"height":427,"pol_round":-1,"round":0,"signature":"T3m6zpluZtp3XsVLaQ5Kgcl1dkZkJmONDicDf4W8VFmAU7zhBYR4ouWEFxps836sww5u+4B9uR+yYMJnEHBzCg==","timestamp":"2022-06-14T10:47:31.128133349Z"} 10:47AM INF received complete proposal block hash=A8EF5212491C91E7679A1AA6E633C4A2D82916EA6BF2480BDEEFC95AEC996F5E height=427 module=consensus 10:47AM INF finalizing commit of block hash=A8EF5212491C91E7679A1AA6E633C4A2D82916EA6BF2480BDEEFC95AEC996F5E height=427 module=consensus num_txs=1 root=00DEA66854B3E881149457929F765E765AAE0F676A4B77B1201499DF052F811B 10:47AM INF minted coins from module account amount=597ustake from=mint module=x/bank here we go 🚀 runtime: bad pointer in frame github.com/CosmWasm/wasmvm/api.AnalyzeCode at 0xc00133f900: 0x1 fatal error: invalid pointer found on stack runtime stack: runtime.throw({0x21a1a88, 0x3aeb720}) runtime/panic.go:1198 +0x71 fp=0x7f10000410f0 sp=0x7f10000410c0 pc=0x453bf1 runtime.adjustpointers(0x7f1000041510, 0xe9a5ff71b399ea9b, 0x384ec98, {0x384ec98, 0x3aeb720}) runtime/stack.go:617 +0x1d0 fp=0x7f1000041158 sp=0x7f10000410f0 pc=0x46c950 runtime.adjustframe(0x7f1000041510, 0x7f10000415f8) runtime/stack.go:659 +0xcc fp=0x7f1000041208 sp=0x7f1000041158 pc=0x46ca6c runtime.gentraceback(0x10, 0x7f10000417a8, 0xc001328000, 0x203001, 0x0, 0x0, 0x7fffffff, 0x283fa98, 0xe9a5ff71b399ea9e, 0x0) runtime/traceback.go:350 +0xac3 fp=0x7f1000041578 sp=0x7f1000041208 pc=0x479003 runtime.copystack(0xc000f5f040, 0x20000) runtime/stack.go:918 +0x293 fp=0x7f1000041728 sp=0x7f1000041578 pc=0x46d233 runtime.shrinkstack(0xc000f5f040) runtime/stack.go:1199 +0x126 fp=0x7f1000041748 sp=0x7f1000041728 pc=0x46e006 runtime.newstack() runtime/stack.go:1047 +0x39b fp=0x7f10000418f8 sp=0x7f1000041748 pc=0x46d6bb runtime.morestack() runtime/asm_amd64.s:461 +0x8b fp=0x7f1000041900 sp=0x7f10000418f8 pc=0x485e2b goroutine 102 [running]: runtime.newobject(0x1fc7d20) runtime/malloc.go:1233 +0x3b fp=0xc00133f848 sp=0xc00133f840 pc=0x42a2fb github.com/CosmWasm/wasmvm/api.AnalyzeCode({0x15}, {0xc0004e3bc0, 0x209fc20, 0x0}) github.com/CosmWasm/wasmvm@v1.0.0/api/lib.go:111 +0x1c5 fp=0xc00133f958 sp=0xc00133f848 pc=0x13ad4a5 github.com/CosmWasm/wasmvm.(*VM).AnalyzeCode(0xc005454550, {0xc0004e3bc0, 0xc004863710, 0x2b}) github.com/CosmWasm/wasmvm@v1.0.0/lib.go:102 +0x25 fp=0xc00133f988 sp=0xc00133f958 pc=0x13b8905 github.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.instantiate({{0x2a67a08, 0xc000fd2710}, {0x2acf878, 0xc000fd35f0}, {0x2a7a150, 0xc000fa8630}, {0x2a5ad20, 0xc000ff63c0}, {0x2a58240, 0xc0010f8478}, ...}, ...) github.com/CosmWasm/wasmd/x/wasm/keeper/keeper.go:304 +0xfd8 fp=0xc001341420 sp=0xc00133f988 pc=0x175d2d8 github.com/CosmWasm/wasmd/x/wasm/keeper.(*Keeper).instantiate(_, {{0x2a9bb58, 0xc00100c480}, {0x2ac9520, 0xc002feea40}, {{0xb, 0x0}, {0xc004c64e70, 0x7}, 0x1ab, ...}, ...}, ...) :1 +0x1a5 fp=0xc001341910 sp=0xc001341420 pc=0x1787505 github.com/CosmWasm/wasmd/x/wasm/keeper.PermissionedKeeper.Instantiate({{_, _}, {_, _}}, {{0x2a9bb58, 0xc00100c480}, {0x2ac9520, 0xc002feea40}, {{0xb, 0x0}, ...}, ...}, ...) github.com/CosmWasm/wasmd/x/wasm/keeper/contract_keeper.go:47 +0x14c fp=0xc001341bd8 sp=0xc001341910 pc=0x1750eac github.com/CosmWasm/wasmd/x/wasm/keeper.(*PermissionedKeeper).Instantiate(_, {{0x2a9bb58, 0xc00100c480}, {0x2ac9520, 0xc002feea40}, {{0xb, 0x0}, {0xc004c64e70, 0x7}, 0x1ab, ...}, ...}, ...) :1 +0x150 fp=0xc001341ea8 sp=0xc001341bd8 pc=0x1789690 github.com/CosmWasm/wasmd/x/wasm/keeper.msgServer.InstantiateContract({{0x2ac3c78, 0xc00127d3c0}}, {0x2a9bb58, 0xc0010d7e30}, 0xc0011ff500) github.com/CosmWasm/wasmd/x/wasm/keeper/msg_server.go:65 +0x590 fp=0xc001342b40 sp=0xc001341ea8 pc=0x1770bb0 github.com/CosmWasm/wasmd/x/wasm/keeper.(*msgServer).InstantiateContract(0xc000ff68a0, {0x2a9bb58, 0xc0010d7e30}, 0x2160960) :1 +0x4c fp=0xc001342b78 sp=0xc001342b40 pc=0x1788dcc github.com/CosmWasm/wasmd/x/wasm/types._Msg_InstantiateContract_Handler.func1({0x2a9bb58, 0xc0010d7e30}, {0x2101580, 0xc0011ff500}) github.com/CosmWasm/wasmd/x/wasm/types/tx.pb.go:831 +0x78 fp=0xc001342bb8 sp=0xc001342b78 pc=0x1401818 github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x2a9bb58, 0xc0010d7e00}, {0x546846, 0x42782b}, 0x2160960, 0xc00525fe48) github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/msg_service_router.go:113 +0xd5 fp=0xc001342e30 sp=0xc001342bb8 pc=0x102aa55 github.com/CosmWasm/wasmd/x/wasm/types._Msg_InstantiateContract_Handler({0x1fff8c0, 0xc000ff68a0}, {0x2a9bb58, 0xc0010d7e00}, 0x283c4b8, 0xc00587d420) github.com/CosmWasm/wasmd/x/wasm/types/tx.pb.go:833 +0x138 fp=0xc001342e88 sp=0xc001342e30 pc=0x14016d8 github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x2a9bb58, 0xc00100c480}, {0x2ac9520, 0xc002feea40}, {{0xb, 0x0}, {0xc004c64e70, 0x7}, 0x1ab, {0x6f2df36, ...}, ...}, ...}, ...) github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/msg_service_router.go:117 +0x34f fp=0xc0013439e0 sp=0xc001342e88 pc=0x102a84f github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x2a9bb58, 0xc00100c480}, {0x2ac9520, 0xc002feea40}, {{0xb, 0x0}, {0xc004c64e70, 0x7}, 0x1ab, ...}, ...}, ...) github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/baseapp.go:736 +0x46a fp=0xc001343f08 sp=0xc0013439e0 pc=0x102816a github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc0012291e0, 0x3, {0xc001032540, 0x1c0, 0x1c0}) github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/baseapp.go:693 +0xc6c fp=0xc001345eb0 sp=0xc001343f08 pc=0x10276ac github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc0012291e0, {{0xc001032540, 0x18, 0x0}}) github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/abci.go:276 +0x1df fp=0xc0013460a0 sp=0xc001345eb0 pc=0x101e9df github.com/CosmWasm/wasmd/app.(*WasmApp).DeliverTx(0xc000f75b00, {{0xc001032540, 0x20, 0xc005448d60}}) :1 +0x8a fp=0xc0013461c0 sp=0xc0013460a0 pc=0x18222ea github.com/tendermint/tendermint/abci/client.(*localClient).DeliverTxAsync(0xc000f53c80, {{0xc001032540, 0x0, 0xc000f53c80}}) github.com/tendermint/tendermint@v0.34.19/abci/client/local_client.go:93 +0x105 fp=0xc001346348 sp=0xc0013461c0 pc=0xc95825 github.com/tendermint/tendermint/proxy.(*appConnConsensus).DeliverTxAsync(0xc004b3ec60, {{0xc001032540, 0x20, 0xb}}) github.com/tendermint/tendermint@v0.34.19/proxy/app_conn.go:85 +0x26 fp=0xc001346378 sp=0xc001346348 pc=0xc9caa6 github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2a9cae0, 0xc000e12480}, {0x2ab23c0, 0xc001062e60}, 0xc0034885a0, {0x2ac4688, 0xc001039c90}, 0x1aa) github.com/tendermint/tendermint@v0.34.19/state/execution.go:320 +0x822 fp=0xc001346798 sp=0xc001346378 pc=0xec8062 github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0012507d0, 0x7}}, {0xc0012507d7, 0x7}, 0x1, 0x1aa, {{0xc0040d3080, ...}, ...}, ...}, ...) github.com/tendermint/tendermint@v0.34.19/state/execution.go:140 +0x171 fp=0xc001346c78 sp=0xc001346798 pc=0xec6571 github.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc000f69180, 0x1ab) github.com/tendermint/tendermint@v0.34.19/consensus/state.go:1655 +0xa1d fp=0xc0013475f0 sp=0xc001346c78 pc=0xefbd1d github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc000f69180, 0x1ab) github.com/tendermint/tendermint@v0.34.19/consensus/state.go:1564 +0x305 fp=0xc0013476f8 sp=0xc0013475f0 pc=0xefb205 github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1() github.com/tendermint/tendermint@v0.34.19/consensus/state.go:1499 +0x87 fp=0xc001347728 sp=0xc0013476f8 pc=0xefaee7 github.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc000f69180, 0x1ab, 0x0) github.com/tendermint/tendermint@v0.34.19/consensus/state.go:1537 +0xc06 fp=0xc0013478c0 sp=0xc001347728 pc=0xefade6 github.com/tendermint/tendermint/consensus.(*State).addVote(0xc000f69180, 0xc00376f220, {0x0, 0x0}) github.com/tendermint/tendermint@v0.34.19/consensus/state.go:2151 +0xb6e fp=0xc001347b18 sp=0xc0013478c0 pc=0xeff0ee github.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc000f69180, 0xc00376f220, {0x0, 0x4e58a6}) github.com/tendermint/tendermint@v0.34.19/consensus/state.go:1949 +0x2c fp=0xc001347b80 sp=0xc001347b18 pc=0xefe0ac github.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc000f69180, {{0x2a58c20, 0xc00342af20}, {0x0, 0x0}}) github.com/tendermint/tendermint@v0.34.19/consensus/state.go:856 +0x45f fp=0xc001347c68 sp=0xc001347b80 pc=0xef3eff github.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc000f69180, 0x0) github.com/tendermint/tendermint@v0.34.19/consensus/state.go:783 +0x512 fp=0xc001347fc0 sp=0xc001347c68 pc=0xef35b2 github.com/tendermint/tendermint/consensus.(*State).OnStart·dwrap·37() github.com/tendermint/tendermint@v0.34.19/consensus/state.go:379 +0x2a fp=0xc001347fe0 sp=0xc001347fc0 pc=0xef116a runtime.goexit() runtime/asm_amd64.s:1581 +0x1 fp=0xc001347fe8 sp=0xc001347fe0 pc=0x487f01 created by github.com/tendermint/tendermint/consensus.(*State).OnStart github.com/tendermint/tendermint@v0.34.19/consensus/state.go:379 +0x13e […] ```