search

palomachain / paloma

The fast blockchain messenger protocol
Apache License 2.0
291 stars 135 forks source link

BUG: Error: recovered: Block (unix) time must never be empty or negative nstack:ngoroutine 252403 on block 15488803 #1115

Closed taariq closed 6 months ago

taariq commented 7 months ago

What is happening?

Section description Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.

Unable to get past block 15488803. Error output:

mainnet-validator palomad[1999487]: {"level":"error","module":"server","err":"recovered: Block (unix) time must never be empty or negative \nstack:\ngoroutine 252403 [running]:\nruntime/debug.Stack()\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.func5({0x2d03080?, 0x3bfcee0})\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/recovery.go:74 +0x26\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.newRecoveryMiddleware.func7({0x2d03080?, 0x3bfcee0?})\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/recovery.go:42 +0x2d\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2d03080, 0x3bfcee0}, 0x0?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/recovery.go:31 +0x2f\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2d03080, 0x3bfcee0}, 0x0?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/recovery.go:36 +0x53\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).runTx.func1()\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:836 +0x136\npanic({0x2d03080?, 0x3bfcee0?})\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/runtime/panic.go:920 +0x270\ngithub.com/CosmWasm/wasmd/x/wasm/types.NewEnv({{0x3c3f5f8, 0x5aefe00}, {0x3c5b680, 0xc00c251a80}, {{0x0, 0x0}, {0xc002301700, 0x9}, 0x0, {0x0, ...}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/types/types.go:278 +0x398\ngithub.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.execute({{0x3c04ea0, 0xc000708ae0}, {0x3c7d068, 0xc002303cb0}, {0x3c1d4d0, 0xc00150be00}, {0x3c06340, 0xc002344720}, {0x3c04aa0, 0xc0009ca918}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/keeper/keeper.go:395 +0x6f0\ngithub.com/CosmWasm/wasmd/x/wasm/keeper.msgServer.ExecuteContract({0x4122da?}, {0x3c40058, 0xc00cf1a7b0}, 0xc00cedc1e0)\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/keeper/msg_server.go:124 +0x1ba\ngithub.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler.func1({0x3c40058, 0xc00cf1a7b0}, {0x3278580?, 0xc00cedc1e0})\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/types/tx.pb.go:2264 +0x72\ngithub.com/cosmos/cosmos-sdk/baseapp.(MsgServiceRouter).registerMsgServiceHandler.func2.1({0x3c3f198, 0xc00cf0b180}, {0xc00eb6b500?, 0x411da5?}, 0x358?, 0xc00e5b5170)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/msg_service_router.go:175 +0x93\ngithub.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler({0x324f3c0?, 0xc0001b11e0}, {0x3c3f198, 0xc00cf0b180}, 0x361beb0, 0xc00c276260)\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/types/tx.pb.go:2266 +0x135\ngithub.com/cosmos/cosmos-sdk/baseapp.(MsgServiceRouter).registerMsgServiceHandler.func2({{0x3c3f5f8, 0x5aefe00}, {0x3c5b680, 0xc00c251a80}, {{0x0, 0x0}, {0xc002301700, 0x9}, 0x0, {0x0, ...}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/msg_service_router.go:198 +0x363\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).runMsgs(_, {{0x3c3f5f8, 0x5aefe00}, {0x3c5b680, 0xc00c251a80}, {{0x0, 0x0}, {0xc002301700, 0x9}, 0x0, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:1009 +0x1e6\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).runTx(0xc0015fc480, 0x2, {0xc000cd1980, 0x168, 0x180})\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:947 +0x116c\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).Simulate(...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/test_helpers.go:28\ngithub.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x3c637a0, 0xc005c0aa50}, 0x0, {0xc000a33d30, 0x9}, {0x3c7d068, 0xc000ec39b0}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/x/auth/tx/service.go:98 +0x129\ngithub.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler.func1({0x3c40058, 0xc00cee2300}, {0x31ed560?, 0xc00c209580})\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/types/tx/service.pb.go:1443 +0x72\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).RegisterGRPCServer.func1({0x3c40058, 0xc00c0ed500}, {0x31ed560, 0xc00c209580}, 0xc00c2095a0?, 0xc00cf49d58)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/grpcserver.go:70 +0x458\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.func2.1({0x3c40058?, 0xc00c0ed500?}, {0x31ed560?, 0xc00c209580?})\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48 +0x45\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3c40058?, 0xc00c0ed500?}, {0x31ed560?, 0xc00c209580?}, 0xc00cf49d58?, 0x0?)\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/recovery/interceptors.go:33 +0xb0\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.func2({0x3c40058, 0xc00c0ed500}, {0x31ed560, 0xc00c209580}, 0xc00c2095a0, 0x414645?)\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53 +0x135\ngithub.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler({0x31b7ac0?, 0xc00099ad20}, {0x3c40058, 0xc00c0ed500}, 0xc00c0ca600, 0xc00c0ed530)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/types/tx/service.pb.go:1445 +0x135\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).RegisterGRPCServer.func2({0x31b7ac0, 0xc00099ad20}, {0x3c40058, 0xc00c0ed500}, 0x32c7180?, 0xc00c0898c0?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/grpcserver.go:84 +0x112\ngoogle.golang.org/grpc.(Server).processUnaryRPC(0xc00157fa00, {0x3c40058, 0xc00c0ed470}, {0x3c578c0, 0xc006737040}, 0xc00c0898c0, 0xc0066fea20, 0xc00520cee0, 0x0)\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1383 +0xe03\ngoogle.golang.org/grpc.(Server).handleStream(0xc00157fa00, {0x3c578c0, 0xc006737040}, 0xc00c0898c0)\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1794 +0x100c\ngoogle.golang.org/grpc.(Server).serveStreams.func2.1()\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1027 +0x8b\ncreated by google.golang.org/grpc.(Server).serveStreams.func2 in goroutine 1207\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1038 +0x135\n: panic","time":"2024-04-05T20:45:26Z","message":"panic recovered in runTx"}

Paloma and pigeon versions and logs

Section description Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.

{WRITE YOUR ANSWER HERE} Paloma

How to reproduce?

Section description Please write detailed steps of what you were doing for this bug to appear.

What is the expected behaviour?

Section description If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.

Cosmwasm time must be properly set in all contracts whose state is validated.

taariq commented 7 months ago

Fixed with https://github.com/palomachain/paloma/pull/1132

verabehr commented 6 months ago

oberseved again with https://github.com/orgs/VolumeFi/projects/4/views/5?pane=issue&itemId=61805054

taariq commented 6 months ago

Via the Cosmwasm Discord: https://discord.com/channels/737637324434833438/737640672680607764/1238178635463004192

For us, it manifested from consensus issues, since we had wasm executing in the AnteHandler. Consensus headers are what set that time value, and the default val is 0 If you are executing wasm in the antehandler then it's probably the same issue

taariq commented 6 months ago

@verabehr let's put a tracking ticket in Volume for this one.

verabehr commented 6 months ago

This could no longer be produced locally or on the lcd mainnet machine

I did see a Block (unix) time must never be empty or negative error at first, but it only happened when the address book was empty. After copying the mainnet address book, it started normally