cosmos / interchain-security

Interchain Security is an open sourced IBC application which allows cosmos blockchains to lease their proof-of-stake security to one another.
https://cosmos.github.io/interchain-security/
Other
156 stars 126 forks source link

Strange behavior with v2.0.0-rc1 #995

Closed AaronKutch closed 1 year ago

AaronKutch commented 1 year ago

When trying to test out v2.0.0-rc1 (e.x. 58a0b1a on https://github.com/onomyprotocol/onomy/tree/provider_v2.0.0-rc1_upgrade), I got much different behavior than from v1.2.0. First, it seems that even after configuring the chain-id with cosmovisor run config chain-id interchain-security-c, when cosmovisor run init --overwrite is executed it produces a genesis.json with a chain-id in the format test-chain-[random] instead of what I configured it to be. Is this just some property of the release candidate similar to how git tags affects the version?

After manually correcting the chain-id, when running tx provider assign-consensus-key in the preparation for querying the consumer genesis, I get

Error: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: a validator cannot assign the default key assignment unless its key on that consumer has already been assigned: cannot re-assign default key assignment [/home/admin/Documents/GitHub/onomy/vendor/cosmossdk.io/errors/errors.go:187] With gas wanted: '0' and gas used: '49752' : unknown request

Which makes no sense. The whole purpose of this command is to assign keys, why would it need to require keys to already have been assigned?

Edit: I fixed the hermes error, see below comment If I skip assign-consensus-key, then later after successfully opening the transfer channel, when attempting to open a producer-consumer channel via hermes it relays this error back to me. I'm guessing its because no consensus keys have been assigned and that the case is not being handled gracefully.

{"timestamp":"2023-06-07T21:37:26.938757Z","level":"ERROR","fields":{"message":"failed ChanOpenTry ChannelSide { chain: BaseChainHandle { chain_id: onomy }, client_id: 07-tendermint-0, connection_id: connection-0, port_id: provider, channel_id: None, version: None }: failed during a transaction submission step to chain 'onomy': gRPC call `send_tx_simulate` failed with status: status: Unknown, message: \"recovered: runtime error: invalid memory address or nil pointer dereference\\nstack:\\ngoroutine 2712 [running]:\\nruntime/debug.Stack()\\n\\t/home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/runtime/debug/stack.go:24 +0x7a\\ngithub.com/cosmos/cosmos-sdk/baseapp.newDefaultRecoveryMiddleware.func1({0x26f6f60, 0x3fc1de0})\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/recovery.go:71 +0x45\\ngithub.com/cosmos/cosmos-sdk/baseapp.newRecoveryMiddleware.func1({0x26f6f60, 0x3fc1de0})\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/recovery.go:39 +0x57\\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x26f6f60, 0x3fc1de0}, 0xc00100a138)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/recovery.go:28 +0x73\\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x26f6f60, 0x3fc1de0}, 0xc0051933b0)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/recovery.go:33 +0xf5\\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:632 +0x11e\\npanic({0x26f6f60, 0x3fc1de0})\\n\\t/home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/runtime/panic.go:890 +0x262\\ngithub.com/cosmos/interchain-security/v2/x/ccv/provider/keeper.Keeper.GetConsumerRewardsPoolAddressStr({{0x32773d0, 0xc000f66510}, {0x3296898, 0xc000fb57a0}, {{0x3296898, 0xc000fb57a0}, 0xc000259448, {0x32773d0, 0xc000f664a0}, {0x3277420, ...}, ...}, ...}, ...)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/interchain-security/v2/x/ccv/provider/keeper/distribution.go:35 +0x92\\ngithub.com/cosmos/interchain-security/v2/x/ccv/provider.AppModule.OnChanOpenTry({{}, 0xc000ccabd0, {{0x3296898, 0xc000fb57a0}, 0xc000259448, {0x32773d0, 0xc000f664a0}, {0x3277420, 0xc000f66530}, {0xc000f46228, ...}, ...}}, ...)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/interchain-security/v2/x/ccv/provider/ibc_module.go:88 +0x8d8\\ngithub.com/cosmos/ibc-go/v4/modules/core/keeper.Keeper.ChannelOpenTry({{0x0, 0x0}, {0x3296898, 0xc000fb57a0}, {{0x32773d0, 0xc000f664b0}, {0x3296898, 0xc000fb57a0}, {{0x3296898, 0xc000fb57a0}, ...}, ...}, ...}, ...)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/ibc-go/v4/modules/core/keeper/msg_server.go:231 +0x943\\ngithub.com/cosmos/ibc-go/v4/modules/core/04-channel/types._Msg_ChannelOpenTry_Handler.func1({0x328af90, 0xc0051ba930}, {0x289c0e0, 0xc004e715c0})\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/ibc-go/v4/modules/core/04-channel/types/tx.pb.go:1232 +0xfa\\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x328af90, 0xc0051ba930}, {0x0, 0xc00521b2d0}, 0x411187, 0xc005193068)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/msg_service_router.go:113 +0x12c\\ngithub.com/cosmos/ibc-go/v4/modules/core/04-channel/types._Msg_ChannelOpenTry_Handler({0x291b380, 0xc000d44280}, {0x328af90, 0xc0051ba900}, 0x315e730, 0xc0051bc2c0)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/ibc-go/v4/modules/core/04-channel/types/tx.pb.go:1234 +0x225\\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x328af20, 0xc000124000}, {0x3296740, 0xc00519c740}, {{0xb, 0x0}, {0xc004d60329, 0x5}, 0x3b, {0x23c2a3a3, ...}, ...}, ...}, ...)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/msg_service_router.go:126 +0x4a3\\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x328af20, 0xc000124000}, {0x3296740, 0xc00519c740}, {{0xb, 0x0}, {0xc004d60329, 0x5}, 0x3b, ...}, ...}, ...)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:760 +0x316\\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc000d28000, 0x2, {0xc004e7f000, 0x7a2, 0x800})\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:717 +0x1066\\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate(0xc000d28000, {0xc004e7f000, 0x7a2, 0x800})\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/test_helpers.go:23 +0xb2\\ngithub.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x32a0be0, 0xc0010b4b20}, {0xc00103b418, 0x5}, {0x328f358, 0xc000fb57a0}, {0x329aaa8, ...}, ...}, ...}, ...)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/x/auth/tx/service.go:120 +0x38d\\ngithub.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler.func1({0x328af90, 0xc005190d50}, {0x2895440, 0xc004df1140})\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/types/tx/service.pb.go:917 +0xfa\\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x328af90, 0xc005190d50}, {0x2895440, 0xc004df1140}, 0xc005104f98, 0xc004e0fe48)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/grpcserver.go:66 +0x71e\\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x328af90, 0xc004e67650}, {0x2895440, 0xc004df1140})\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0xae\\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x328af90, 0xc004e67650}, {0x2895440, 0xc004df1140}, 0xc004df1160, 0xc004df1180)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/grpc-ecosystem/go-grpc-middleware/recovery/interceptors.go:33 +0x182\\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x328af90, 0xc004e67650}, {0x2895440, 0xc004df1140})\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0xae\\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x328af90, 0xc004e67650}, {0x2895440, 0xc004df1140}, 0xc004df1160, 0xc004e0fe48)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0x177\\ngithub.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler({0x284db80, 0xc000fe9860}, {0x328af90, 0xc004e67650}, 0xc004dedec0, 0xc004e67680)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/types/tx/service.pb.go:919 +0x225\\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2({0x284db80, 0xc000fe9860}, {0x328af90, 0xc004e67650}, 0xc004dedec0, 0x0)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/github.com/cosmos/cosmos-sdk/baseapp/grpcserver.go:80 +0x167\\ngoogle.golang.org/grpc.(*Server).processUnaryRPC(0xc000d29dc0, {0x3294ae0, 0xc004866780}, 0xc004e2c300, 0xc000e12e10, 0xc000171400, 0x0)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/google.golang.org/grpc/server.go:1210 +0x14f6\\ngoogle.golang.org/grpc.(*Server).handleStream(0xc000d29dc0, {0x3294ae0, 0xc004866780}, 0xc004e2c300, 0x0)\\n\\t/home/admin/Documents/GitHub/onomy/vendor/google.golang.org/grpc/server.go:1533 +0x87c\\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2()\\n\\t/home/admin/Documents/GitHub/onomy/vendor/google.golang.org/grpc/server.go:871 +0x109\\ncreated by google.golang.org/grpc.(*Server).serveStreams.func1\\n\\t/home/admin/Documents/GitHub/onomy/vendor/google.golang.org/grpc/server.go:869 +0x366\\n: panic [cosmos/cosmos-sdk/baseapp/recovery.go:69] With gas wanted: '0' and gas used: '163649' \", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"x-cosmos-block-height\": \"59\"} }"},"threadId":"ThreadId(1)"}

The logs of both producer and consumer runners are both indicating normal block production and channel state updates, no warnings given. It shows both the transfer and producer-consumer channel capabilities being claimed, the only difference being that the third "channel state updated" message is not being reached for connection-1 since hermes is failing at that point.

AaronKutch commented 1 year ago

I just pushed two more commits that added a README in case you want to run my custom test harness for yourself

AaronKutch commented 1 year ago

I found that when moving from v1.2.0 to v1.2.0-multiden that the chain-id problem and assign-consensus-keys problem doesn't happen, but the hermes error does. I later figured out that I should probably add "provider_reward_denoms": [] to the proposal when the multiden patch is present, but adding it doesn't seem to change anything. edit: v1.1.0-multiden also has same properties

AaronKutch commented 1 year ago

I figured out the hermes error, needed to add the line delete(blockedAddrs, authtypes.NewModuleAddress(providertypes.ConsumerRewardsPool).String()) (although there should be something better than a nullptr exception and some good guesses on what I'm missing vs Gaia in order to guide me in the right direction).

shaspitz commented 1 year ago

Hey @AaronKutch, thanks for the info. Note this repo hasn't been in production until recently, consequently there may be state migrations required between some of the versions you're mentioning. In particular v1.2.0-multiden is the current canonical version for consumers, and v1.1.0-multiden for providers. The v2 release branch is not production ready just yet, and v2 will require state migrations if your chain/testnet is upgrading from one version to another. Could I get more context on the enviroment that you're trying to use ICS? Are you just using the consumer module? Testnet? docker? etc. Thanks

AaronKutch commented 1 year ago

We decided to go with v1.1.0-multiden since it has been tested on Gaia, although I will periodically test new versions locally to foresee any issues. I didn't know a different version was canonical for consumers, so I will switch to v1.2.0-multiden for our first upcoming consumer. The context is that we were planning to have an unusual number of modules and sidecars for our chain. Our main worry was that a single failing component will bring down the entire chain, among other problems with the previous approach. When ICS came out, we decided upon turning our main chain into a relatively minimal and stable producer, and have each major component on its own consumer chain. We are trying to launch a testnet by next week.

shaspitz commented 1 year ago

Yea for now use the multiden versions mentioned, and when v2.0.0 comes out, that'd be the preferred versions to use. There will be instructions re state migrations. Following the v2 release, we plan to split out the semver for consumer and provider so that they'll each have their own release cycle. It'll be a lot more strait forward

shaspitz commented 1 year ago

Hey @AaronKutch I'm going to close this issue as we've released v2.0.0. Please open a new issue with a specific problem and a way to reproduce if you're still running into migration related challenges.