search

cosmos / cosmos-sdk

:chains: A Framework for Building High Value Public Blockchains :sparkles:
https://cosmos.network/
Apache License 2.0
6.17k stars 3.57k forks source link

[Bug]: SDK v0.50.3 upgrade from v0.47.x panic #19435

Closed RaulBernal closed 6 months ago

RaulBernal commented 6 months ago

Is there an existing issue for this?

What happened?

I tested the v0.47 -> v0.50.2 (check this solved issue https://github.com/cosmos/cosmos-sdk/issues/18733) using Ignite v28.0 I was able to upgrade with that code, adding the Store for circuit & feeibc but with the new scaffold with Ignite v0.28.1.1 I can't upgrade: 


raul@iMac-de-Raul v0.50 % ./bcnad start
panic: failed to load latest version: version of store feeibc mismatch root store's version; expected 99 got 0; new stores should be added using StoreUpgrades

goroutine 1 [running]:
github.com/BitCannaGlobal/bcna/cmd/bcnad/cmd.newApp({0x1084c1348, 0x1400016c230}, {0x1084d4cd0, 0x14000610870}, {0x0, 0x0}, {0x108484860, 0x140014e76c0})
        /Users/raul/SDK/v0.50/bcna/cmd/bcnad/cmd/commands.go:137 +0xa8
github.com/cosmos/cosmos-sdk/server.startApp(0x1400151e220, 0x108472a98, {0x1084731e8?, 0x0?, 0x0?})
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:642 +0x110
github.com/cosmos/cosmos-sdk/server.start(_, {{0x0, 0x0, 0x0}, {0x1084da830, 0x14001532a20}, 0x0, {0x14000ecada0, 0xe}, {0x1084f83a0, ...}, ...}, ...)
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:250 +0xd8
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2.1()
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:175 +0x58
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0x1400151e220, 0x140011377f8)
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:600 +0x16c
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2(0x14000cc4300?, {0x109fe5de0?, 0x0?, 0x0?})
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:174 +0x194
github.com/spf13/cobra.(*Command).execute(0x1400137c900, {0x109fe5de0, 0x0, 0x0})
        /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0x840
github.com/spf13/cobra.(*Command).ExecuteC(0x1400124ec00)
        /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1032
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x140000aa6e8?, {0x0, 0x0}, {0x14001107560, 0x11})
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/cmd/execute.go:34 +0x154
main.main()
        /Users/raul/SDK/v0.50/bcna/cmd/bcnad/main.go:15 +0x3c

raul@iMac-de-Raul v0.50 % ./bcnad start
panic: failed to load latest version: version of store circuit mismatch root store's version; expected 99 got 0; new stores should be added using StoreUpgrades

goroutine 1 [running]:
github.com/BitCannaGlobal/bcna/cmd/bcnad/cmd.newApp({0x1062c9348, 0x14000b0f570}, {0x1062dccd0, 0x1400131a060}, {0x0, 0x0}, {0x10628c860, 0x14000eefdc0})
        /Users/raul/SDK/v0.50/bcna/cmd/bcnad/cmd/commands.go:137 +0xa8
github.com/cosmos/cosmos-sdk/server.startApp(0x1400144c800, 0x10627aa98, {0x10627b1e8?, 0x0?, 0x0?})
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:642 +0x110
github.com/cosmos/cosmos-sdk/server.start(_, {{0x0, 0x0, 0x0}, {0x1062e2830, 0x1400145faa0}, 0x0, {0x14000e825b0, 0xe}, {0x1063003a0, ...}, ...}, ...)
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:250 +0xd8
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2.1()
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:175 +0x58
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0x1400144c800, 0x140011177f8)
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:600 +0x16c
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2(0x14000b0ad00?, {0x107dedde0?, 0x0?, 0x0?})
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/start.go:174 +0x194
github.com/spf13/cobra.(*Command).execute(0x14000f54900, {0x107dedde0, 0x0, 0x0})
        /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0x840
github.com/spf13/cobra.(*Command).ExecuteC(0x14000b72c00)
        /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1032
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x140000aa6e8?, {0x0, 0x0}, {0x1400102c660, 0x11})
        /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.2/server/cmd/execute.go:34 +0x154
main.main()
        /Users/raul/SDK/v0.50/bcna/cmd/bcnad/main.go:15 +0x3c

Cosmos SDK Version

v0.50.3

How to reproduce?

schedule a v047 -> v050 upgrade using a proposal. Scaffolding an Ignite v28.1.1 chain and trying to upgrade.

(I was able to do it with a scaffold from v28.0) This is that scaffolded code running the upgrade: 

bcnad start 
12:01PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000bc6240, circuit}" version=100
12:01PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x1400191d240, feeibc}" version=100
12:01PM INF starting node with ABCI CometBFT in-process module=server
12:01PM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
12:01PM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
12:01PM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
12:01PM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
12:01PM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
12:01PM INF service start impl=EventBus module=events msg="Starting EventBus service"
12:01PM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
12:01PM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
12:01PM INF ABCI Handshake App Info hash=F37E306C930DD54CBFF7495E78941C63CB2F83012A1BE6D83495650697BB64ED height=99 module=consensus protocol-version=0 software-version=3.0.0-rc3-03dfb95c
12:01PM INF ABCI Replay Blocks appHeight=99 module=consensus stateHeight=99 storeHeight=100
12:01PM INF Replay last block using real app module=consensus
12:01PM INF applying upgrade "StickyFingers" at height: 100 module=x/upgrade
12:01PM INF Cosmos-SDK v0.50.x is here... module=server
12:01PM INF Module: auth, Version: 4 module=server
12:01PM INF Module: authz, Version: 2 module=server
12:01PM INF Module: bank, Version: 4 module=server
12:01PM INF Module: ibc, Version: 4 module=server
12:01PM INF Module: interchainaccounts, Version: 2 module=server
12:01PM INF Module: staking, Version: 4 module=server
12:01PM INF Module: feegrant, Version: 2 module=server
12:01PM INF Module: gov, Version: 4 module=server
12:01PM INF Module: params, Version: 1 module=server
12:01PM INF Module: vesting, Version: 1 module=server
12:01PM INF Module: bcna, Version: 1 module=server
12:01PM INF Module: capability, Version: 1 module=server
12:01PM INF Module: consensus, Version: 1 module=server
12:01PM INF Module: crisis, Version: 2 module=server
12:01PM INF Module: distribution, Version: 3 module=server
12:01PM INF Module: group, Version: 2 module=server
12:01PM INF Module: mint, Version: 2 module=server
12:01PM INF Module: slashing, Version: 3 module=server
12:01PM INF Module: transfer, Version: 3 module=server
12:01PM INF Module: evidence, Version: 1 module=server
12:01PM INF Module: genutil, Version: 1 module=server
12:01PM INF Module: upgrade, Version: 2 module=server
12:01PM INF adding a new module: 06-solomachine module=server
12:01PM INF adding a new module: 07-tendermint module=server
12:01PM INF adding a new module: circuit module=server
12:01PM INF adding a new module: feeibc module=server
12:01PM INF migrating module gov from version 4 to version 5 module=server
12:01PM INF migrating module ibc from version 4 to version 5 module=server
12:01PM INF successfully migrated client to self-manage params module=x/ibc/client
12:01PM INF successfully migrated connection to self-manage params module=x/ibc/connection
12:01PM INF migrating module interchainaccounts from version 2 to version 3 module=server
12:01PM INF successfully migrated ica/host submodule to self-manage params module=x/ibc-interchainaccounts
12:01PM INF successfully migrated ica/controller submodule to self-manage params module=x/ibc-interchainaccounts
12:01PM INF adding a new module: runtime module=server
12:01PM INF migrating module slashing from version 3 to version 4 module=server
12:01PM INF migrating module staking from version 4 to version 5 module=server
12:01PM INF migrating module transfer from version 3 to version 4 module=server
12:01PM INF successfully migrated transfer app self-manage params module=x/ibc-transfer
12:01PM INF migrating module transfer from version 4 to version 5 module=server
12:01PM INF successfully added metadata to IBC voucher denominations module=x/ibc-transfer
12:01PM INF migrating module auth from version 4 to version 5 module=server
12:01PM INF finalized block block_app_hash=6724392F556C369F5A1CF5C3B18782F5D3F0F4A7450FDE456F0B9B36B6D20CA6 height=100 module=consensus num_txs_res=0 num_val_updates=0
12:01PM INF executed block app_hash=6724392F556C369F5A1CF5C3B18782F5D3F0F4A7450FDE456F0B9B36B6D20CA6 height=100 module=consensus
12:01PM INF committed state block_app_hash=F37E306C930DD54CBFF7495E78941C63CB2F83012A1BE6D83495650697BB64ED height=100 module=consensus
12:01PM INF Completed ABCI Handshake - CometBFT and App are synced appHash=6724392F556C369F5A1CF5C3B18782F5D3F0F4A7450FDE456F0B9B36B6D20CA6 appHeight=99 module=consensus
12:01PM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.2
12:01PM INF This node is a validator addr=3332A1D571F4F0489C4168A9075DA71F1A764945 module=consensus pubKey=PubKeyEd25519{4975463257214EA7DB6ED9265B554D4B745C5AFEE23908E5FC574FBAFA84F393}
12:01PM INF indexed block events height=100 module=txindex
12:01PM INF P2P Node ID ID=2c714c9ec7a0194300f0ffc5891d0fb01d6dc91c file=/Users/raul/.bcna/config/node_key.json module=p2p
12:01PM INF Adding persistent peers addrs=[] module=p2p
RaulBernal commented 6 months ago

Another issue I've noticed is that if I want to manually bump the SDK version to v0.50.3 at the "old v28.0" scaffold I get this error: 

Cosmos SDK's version is: v0.50.3

✘ Cannot build app:                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

error while running command go install github.com/bufbuild/buf/cmd/buf github.com/cosmos/gogoproto/protoc-gen-gocosmos google.golang.org/grpc/cmd/protoc-gen-go-grpc google.golang.org/protobuf/cmd/protoc-gen-go github.com/cosmos/cosmos-proto/cmd/protoc-gen-go-pulsar github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2 golang.org/x/tools/cmd/goimports: # github.com/bufbuild/buf/private/pkg/protodescriptor
../../../go/pkg/mod/github.com/bufbuild/buf@v1.27.2/private/pkg/protodescriptor/protodescriptor.go:52:24: cannot use fileDescriptorProto (variable of type *descriptorpb.FileDescriptorProto) as FileDescriptor value in assignment: *descriptorpb.FileDescriptorProto does not implement FileDescriptor (wrong type for method GetEdition)                                                                                                                                                    
                have GetEdition() descriptorpb.Edition                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                want GetEdition() string                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
../../../go/pkg/mod/github.com/bufbuild/buf@v1.27.2/private/pkg/protodescriptor/protodescriptor.go:71:32: impossible type assertion: fileDescriptor.(*descriptorpb.FileDescriptorProto)                                                                                                                                                                                                                                                                                                        
        *descriptorpb.FileDescriptorProto does not implement FileDescriptor (wrong type for method GetEdition)                                                                                                                                                                                                                                                                                                                                                                                         
                have GetEdition() descriptorpb.Edition                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                want GetEdition() string                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
../../../go/pkg/mod/github.com/bufbuild/buf@v1.27.2/private/pkg/protodescriptor/protodescriptor.go:97:33: cannot use proto.String(edition) (value of type *string) as *descriptorpb.Edition value in assignment                                                                                                                                                                                                                                                                                
../../../go/pkg/mod/github.com/bufbuild/buf@v1.27.2/private/pkg/protodescriptor/protodescriptor.go:151:36: cannot use fileDescriptorProto (variable of type *descriptorpb.FileDescriptorProto) as FileDescriptor value in argument to ValidateFileDescriptor: *descriptorpb.FileDescriptorProto does not implement FileDescriptor (wrong type for method GetEdition)                                                                                                                           
                have GetEdition() descriptorpb.Edition                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                want GetEdition() string                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
: exit status 1

So I can't test the old scaffold with SDK v0.50.3

RaulBernal commented 6 months ago

To put more context in the issue and discard a SDK problem I tried to downgrade the v28.1.1 scaffold from SDK v0.50.3 to SDK v0.50.2 and also downgrading the IBC-go from v8.1.0 to v8.0.0 being the result the same issue.

RaulBernal commented 6 months ago

v28.0 scaffold is: https://github.com/BitCannaGlobal/bcna/tree/previous_ignite_v28_sdk0.50.2

v28.1.1 scaffold is: https://github.com/BitCannaGlobal/bcna/blob/v4/app/upgrades.go https://github.com/BitCannaGlobal/bcna/tree/v4-no-nft

Here there is a comparation: https://github.com/BitCannaGlobal/bcna/pull/324/files

RaulBernal commented 6 months ago

@tac0turtle I think is not a SDK issue, maybe this issue should be reported at Ignite GitHub, I guess from one version to the last (Ignite v28.0 to v28.1.) there be something breaking. What do you think @julienrbrt ? (I name you because you are actively developing Ignite lastly)

julienrbrt commented 6 months ago

Hi, v28 -> v28.1 had some fixes in IBC module registration. Can you try directly v0.47 -> v0.50.3? That shouldn't be a problem if you add the correct new modules in your `StoreUpgrades.

Concerning your other issue: https://github.com/cosmos/cosmos-sdk/issues/19435#issuecomment-1943614479, the buf dependency was upgraded to github.com/bufbuild/buf v1.28.1, so you should upgrade that dep too I think.

RaulBernal commented 6 months ago

Hi, v28 -> v28.1 had some fixes in IBC module registration. Can you try directly v0.47 -> v0.50.3? That shouldn't be a problem if you add the correct new modules in your `StoreUpgrades.

OK I haven seen this PR (https://github.com/ignite/cli/pull/3851/files) and I add the StoreUpgrade accordingly: 

storeUpgrades := storetypes.StoreUpgrades{
Added: []string{
circuittypes.ModuleName,
ibcfeetypes.ModuleName,
ibctm.ModuleName,
solomachine.ModuleName,
// nft.ModuleName,
},
}

NFT module was deactivated in app.go_v2 , etc But still:


bcnad start          
panic: failed to load latest version: version of store feeibc mismatch root store's version; expected 99 got 0; new stores should be added using StoreUpgrades

goroutine 1 [running]: github.com/BitCannaGlobal/bcna/cmd/bcnad/cmd.newApp({0x104583288, 0x14000271a40}, {0x104596c10, 0x140001140b8}, {0x0, 0x0}, {0x1045460c0, 0x14000ac0000}) /Users/raul/SDK/v0.50/bcna/cmd/bcnad/cmd/commands.go:137 +0xa8 github.com/cosmos/cosmos-sdk/server.startApp(0x1400160a980, 0x1045341f8, {0x104534948?, 0x0?, 0x0?}) /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.3/server/start.go:642 +0x110 github.com/cosmos/cosmos-sdk/server.start(_, {{0x0, 0x0, 0x0}, {0x10459c770, 0x14000560600}, 0x0, {0x1400108e830, 0xe}, {0x1045ba420, ...}, ...}, ...) /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.3/server/start.go:250 +0xd8 github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2.1() /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.3/server/start.go:175 +0x58 github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0x1400160a980, 0x140015177f8) /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.3/server/start.go:600 +0x16c github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2(0x14000db4e00?, {0x1060d6ce0?, 0x0?, 0x0?}) /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.3/server/start.go:174 +0x194 github.com/spf13/cobra.(Command).execute(0x140013d7200, {0x1060d6ce0, 0x0, 0x0}) /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0x840 github.com/spf13/cobra.(Command).ExecuteC(0x14001245500) /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344 github.com/spf13/cobra.(Command).Execute(...) /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039 github.com/spf13/cobra.(Command).ExecuteContext(...) /Users/raul/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1032 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x140000aa6e8?, {0x0, 0x0}, {0x140010356f8, 0x11}) /Users/raul/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.3/server/cmd/execute.go:34 +0x154 main.main() /Users/raul/SDK/v0.50/bcna/cmd/bcnad/main.go:15 +0x3c


is there any more to add? 

> Concerning your other issue: [#19435 (comment)](https://github.com/cosmos/cosmos-sdk/issues/19435#issuecomment-1943614479), the buf dependency was upgraded to `github.com/bufbuild/buf v1.28.1`, so you should upgrade that dep too I think. 
That works! Thanks
RaulBernal commented 6 months ago

I've put some extra log output but as I imagine it happens after the Store issue, so it didn't help, can you point me to any log snippet to know which "new" module is throwing the panic? I've reviewed the code changes but I can't found it 

            // Print the modules with their respective ver.
            for moduleName, version := range fromVM {
                app.Logger().Info(fmt.Sprintf("Module: %s, Version: %d", moduleName, version))

            }
            versionMap, err := app.ModuleManager.RunMigrations(ctx, app.Configurator(), fromVM)
            if err != nil {
                return nil, err
            }
            app.Logger().Info(fmt.Sprintf("post migrate version map: %v", versionMap))
            // return app.ModuleManager.RunMigrations(ctx, app.Configurator(), fromVM)
            return versionMap, err
RaulBernal commented 6 months ago

I think it could be closed, related with Ignite https://github.com/ignite/cli/issues/3971