search

cosmos / cosmos-sdk

:chains: A Framework for Building High Value Public Blockchains :sparkles:
https://cosmos.network/
Apache License 2.0
6.23k stars 3.6k forks source link

Failure in migrating the auth module from 2 to 3 #14217

Closed Segfaultd closed 1 year ago

Segfaultd commented 1 year ago

Summary of Bug

When trying to upgrade our empty testnet to Cosmos SDK 0.46.6, the upgrade fails on migrating the auth module. W've been extensively reviewing our changes, and we followed every upgrade procedure from the official Cosmos SDK documentation.

Is there any thing to know or to do to be able to pass that upgrade ?

That's a duplicate issue of #13314 but the issue was closed without explicitely stating what was the solution.

4:32PM INF starting node with ABCI Tendermint in-process
4:32PM INF service start impl=multiAppConn module=proxy msg={}
4:32PM INF service start connection=query impl=localClient module=abci-client msg={}
4:32PM INF service start connection=snapshot impl=localClient module=abci-client msg={}
4:32PM INF service start connection=mempool impl=localClient module=abci-client msg={}
4:32PM INF service start connection=consensus impl=localClient module=abci-client msg={}
4:32PM INF service start impl=EventBus module=events msg={}
4:32PM INF service start impl=PubSub module=pubsub msg={}
4:32PM INF service start impl=IndexerService module=txindex msg={}
4:32PM INF ABCI Handshake App Info hash="�<k��\"38�av\x03=\x1c\x00uz+#).�_œc�ɒ��*" height=19 module=consensus protocol-version=0 software-version=1.3.0
4:32PM INF ABCI Replay Blocks appHeight=19 module=consensus stateHeight=19 storeHeight=20
4:32PM INF Replay last block using real app module=consensus
4:32PM INF applying upgrade "v1.3.0" at height: 20
4:32PM INF v1.3.0 upgrade applied
4:32PM INF migrating module authz from version 1 to version 2
4:32PM INF migrating module bank from version 2 to version 3
4:32PM INF migrating module feegrant from version 1 to version 2
4:32PM INF adding a new module: feeibc
4:32PM INF migrating module gov from version 2 to version 3
4:32PM INF migrating module staking from version 2 to version 3
4:32PM INF migrating module upgrade from version 1 to version 2
4:32PM INF migrating module auth from version 2 to version 3
panic: value is nil

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/store/types.AssertValidValue(...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/store/types/validity.go:13
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Set(0xc000dcabf0?, {0xc000d1ee80?, 0x65?, 0x65?}, {0x0?, 0xc00141c360?, 0xc0010fe300?})
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/store/gaskv/store.go:49 +0x11c
github.com/cosmos/cosmos-sdk/x/auth/migrations/v046.mapAccountAddressToAccountID({{0x2bdc230, 0xc000056380}, {0x2becdc0, 0xc000f7f4c0}, {{0xb, 0x0}, {0xc0010fe300, 0x12}, 0x14, {0x17ca0855, ...}, ...}, ...}, ...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/x/auth/migrations/v046/store.go:20 +0x1a3
github.com/cosmos/cosmos-sdk/x/auth/migrations/v046.MigrateStore(...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/x/auth/migrations/v046/store.go:30
github.com/cosmos/cosmos-sdk/x/auth/keeper.Migrator.Migrate2to3({{{0x2bc5ad8, 0xc00133c390}, {0x2beaf30, 0xc000dcabf0}, {{0x2beaf30, 0xc000dcabf0}, 0xc000014690, {0x2bc5ad8, 0xc00133c430}, {0x2bc5b28, ...}, ...}, ...}, ...}, ...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/x/auth/keeper/migrations.go:50 +0x7b
github.com/cosmos/cosmos-sdk/types/module.configurator.runModuleMigrations({{_, _}, {_, _}, {_, _}, _}, {{0x2bdc230, 0xc000056380}, {0x2becdc0, ...}, ...}, ...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/types/module/configurator.go:110 +0x335
github.com/cosmos/cosmos-sdk/types/module.Manager.RunMigrations({0xc00128f050, {0xc00104d800, 0x17, 0x17}, {0xc00104d380, 0x17, 0x17}, {0xc00104d500, 0x17, 0x17}, ...}, ...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/types/module/module.go:455 +0x4b4
github.com/lum-network/chain/app.(*App).registerUpgradeHandlers.func8({{0x2bdc230, 0xc000056380}, {0x2becdc0, 0xc000f7f4c0}, {{0xb, 0x0}, {0xc0010fe300, 0x12}, 0x14, {0x17ca0855, ...}, ...}, ...}, ...)
    /root/chain/app/app.go:649 +0x145
github.com/cosmos/cosmos-sdk/x/upgrade/keeper.Keeper.ApplyUpgrade({{0xc000479810, 0xb}, 0xc00113afc0, {0x2bc5ad8, 0xc00133c480}, {0x2beaf30, 0xc000dcabf0}, 0xc00128eb40, {0x2bbd7a0, 0xc00104bdc0}, ...}, ...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/x/upgrade/keeper/keeper.go:337 +0x19e
github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0xc000479810, 0xb}, 0xc00113afc0, {0x2bc5ad8, 0xc00133c480}, {0x2beaf30, 0xc000dcabf0}, 0xc00128eb40, {0x2bbd7a0, 0xc00104bdc0}, ...}, ...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/x/upgrade/abci.go:62 +0x66c
github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/x/upgrade/module.go:134
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x2bdc230, 0xc000056380}, {0x2becdc0, 0xc000f7f4c0}, {{0xb, 0x0}, {0xc0010fe300, 0x12}, 0x14, ...}, ...}, ...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/types/module/module.go:484 +0x398
github.com/lum-network/chain/app.(*App).BeginBlocker(_, {{0x2bdc230, 0xc000056380}, {0x2becdc0, 0xc000f7f4c0}, {{0xb, 0x0}, {0xc0010fe300, 0x12}, 0x14, ...}, ...}, ...)
    /root/chain/app/app.go:433 +0x7b
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc0010ecec0, 0x20, 0x20}, {{0xb, 0x0}, {0xc0010fe300, 0x12}, 0x14, {0x17ca0855, ...}, ...}, ...})
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/baseapp/abci.go:183 +0x843
github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc0010ecec0, 0x20, 0x20}, {{0xb, 0x0}, {0xc0010fe300, 0x12}, 0x14, {0x17ca0855, ...}, ...}, ...})
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/abci/client/local_client.go:280 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc0010ecec0, 0x20, 0x20}, {{0xb, 0x0}, {0xc0010fe300, 0x12}, 0x14, {0x17ca0855, ...}, ...}, ...})
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/proxy/app_conn.go:81 +0x55
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2bdd490?, 0xc000d0e9c0}, {0x2be4e40, 0xc000fc38d0}, 0xc000ca0f00, {0x2bec1b8, 0xc000012528}, 0x13?)
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/state/execution.go:307 +0x3dd
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc000fb4828, 0x7}}, {0xc0010fe108, 0x12}, 0x1, 0x13, {{0xc000473120, ...}, ...}, ...}, ...)
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/state/execution.go:140 +0x171
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc000fb4828, 0x7}}, {0xc0010fe108, 0x12}, 0x1, 0x13, {{0xc000473120, ...}, ...}, ...}, ...)
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/replay.go:503 +0x23c
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc000fb4828, 0x7}}, {0xc0010fe108, 0x12}, 0x1, 0x13, {{0xc000473120, ...}, ...}, ...}, ...)
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/replay.go:416 +0x7ae
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000d4d870, {0x2bed890, 0xc001073ad0})
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc000fb4828, 0x7}}, {0xc0010fe108, 0x12}, 0x1, 0x13, ...}, ...)
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc001122a00, {0x2bd8930, 0xc00120a280}, 0xc00141d090, {0x2bbecc0, 0xc0010f4d38}, 0x1?, 0x0?, 0xc00141d340, {0x2bdd490, ...}, ...)
    /root/work/pkg/mod/github.com/tendermint/tendermint@v0.34.23/node/node.go:777 +0x597
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2bfafe0, 0xc0013e4e70}, 0x0, {0x0, 0x0}, {0x2bf1458, ...}, ...}, ...)
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/server/start.go:313 +0x7db
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0012eef00?, {0xc000c99a20?, 0x0?, 0x2?})
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/server/start.go:143 +0x148
github.com/spf13/cobra.(*Command).execute(0xc0012eef00, {0xc000c99a00, 0x2, 0x2})
    /root/work/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc00128b500)
    /root/work/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:1040 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
    /root/work/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext(...)
    /root/work/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:961
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x1cc9fa0?, {0x0, 0x0}, {0xc000479810, 0xb})
    /root/work/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.6/server/cmd/execute.go:36 +0x20f
main.main()
    /root/chain/cmd/lumd/main.go:16 +0x30

Version

build_deps:
- cloud.google.com/go@v0.105.0
- cloud.google.com/go/compute@v1.10.0
- cloud.google.com/go/iam@v0.6.0
- cloud.google.com/go/storage@v1.27.0
- cosmossdk.io/errors@v1.0.0-beta.7
- cosmossdk.io/math@v1.0.0-beta.3
- filippo.io/edwards25519@v1.0.0-rc.1
- github.com/99designs/keyring@v1.2.1
- github.com/ChainSafe/go-schnorrkel@v0.0.0-20200405005733-88cbf1b4c40d
- github.com/Workiva/go-datastructures@v1.0.53
- github.com/armon/go-metrics@v0.4.0
- github.com/aws/aws-sdk-go@v1.40.45
- github.com/beorn7/perks@v1.0.1
- github.com/bgentry/go-netrc@v0.0.0-20140422174119-9fd32a8b3d3d
- github.com/bgentry/speakeasy@v0.1.0
- github.com/btcsuite/btcd@v0.22.1
- github.com/cenkalti/backoff/v4@v4.1.3
- github.com/cespare/xxhash/v2@v2.1.2
- github.com/chzyer/readline@v0.0.0-20180603132655-2972be24d48e
- github.com/coinbase/rosetta-sdk-go@v0.7.9
- github.com/confio/ics23/go@v0.7.0 => github.com/cosmos/cosmos-sdk/ics23/go@v0.8.0
- github.com/cosmos/btcutil@v1.0.4
- github.com/cosmos/cosmos-proto@v1.0.0-alpha7
- github.com/cosmos/cosmos-sdk@v0.46.6
- github.com/cosmos/go-bip39@v1.0.0
- github.com/cosmos/gogoproto@v1.4.2
- github.com/cosmos/iavl@v0.19.4
- github.com/cosmos/ibc-go/v5@v5.0.1
- github.com/cosmos/ledger-cosmos-go@v0.11.1
- github.com/cosmos/ledger-go@v0.9.2
- github.com/creachadair/taskgroup@v0.3.2
- github.com/davecgh/go-spew@v1.1.1
- github.com/desertbit/timer@v0.0.0-20180107155436-c41aec40b27f
- github.com/dvsekhvalnov/jose2go@v1.5.0
- github.com/felixge/httpsnoop@v1.0.1
- github.com/fsnotify/fsnotify@v1.5.4
- github.com/go-kit/kit@v0.12.0
- github.com/go-kit/log@v0.2.1
- github.com/go-logfmt/logfmt@v0.5.1
- github.com/godbus/dbus@v0.0.0-20190726142602-4481cbc300e2
- github.com/gogo/gateway@v1.1.0
- github.com/gogo/protobuf@v1.3.3 => github.com/regen-network/protobuf@v1.3.2-alpha.regen.4
- github.com/golang/groupcache@v0.0.0-20210331224755-41bb18bfe9da
- github.com/golang/protobuf@v1.5.2
- github.com/golang/snappy@v0.0.4
- github.com/google/btree@v1.0.1
- github.com/google/go-cmp@v0.5.9
- github.com/google/orderedcode@v0.0.1
- github.com/google/uuid@v1.3.0
- github.com/googleapis/enterprise-certificate-proxy@v0.2.0
- github.com/googleapis/gax-go/v2@v2.6.0
- github.com/gorilla/handlers@v1.5.1
- github.com/gorilla/mux@v1.8.0
- github.com/gorilla/websocket@v1.5.0
- github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0
- github.com/grpc-ecosystem/grpc-gateway@v1.16.0
- github.com/gsterjov/go-libsecret@v0.0.0-20161001094733-a6f4afe4910c
- github.com/gtank/merlin@v0.1.1
- github.com/gtank/ristretto255@v0.1.2
- github.com/hashicorp/go-cleanhttp@v0.5.2
- github.com/hashicorp/go-getter@v1.6.1
- github.com/hashicorp/go-immutable-radix@v1.3.1
- github.com/hashicorp/go-safetemp@v1.0.0
- github.com/hashicorp/go-version@v1.6.0
- github.com/hashicorp/golang-lru@v0.5.5-0.20210104140557-80c98217689d
- github.com/hashicorp/hcl@v1.0.0
- github.com/hdevalence/ed25519consensus@v0.0.0-20220222234857-c00d1f31bab3
- github.com/improbable-eng/grpc-web@v0.15.0
- github.com/jmespath/go-jmespath@v0.4.0
- github.com/klauspost/compress@v1.15.11
- github.com/lib/pq@v1.10.6
- github.com/libp2p/go-buffer-pool@v0.1.0
- github.com/magiconair/properties@v1.8.6
- github.com/manifoldco/promptui@v0.9.0
- github.com/mattn/go-colorable@v0.1.13
- github.com/mattn/go-isatty@v0.0.16
- github.com/matttproud/golang_protobuf_extensions@v1.0.2-0.20181231171920-c182affec369
- github.com/mimoo/StrobeGo@v0.0.0-20210601165009-122bf33a46e0
- github.com/minio/highwayhash@v1.0.2
- github.com/mitchellh/go-homedir@v1.1.0
- github.com/mitchellh/go-testing-interface@v1.0.0
- github.com/mitchellh/mapstructure@v1.5.0
- github.com/mtibben/percent@v0.2.1
- github.com/pelletier/go-toml/v2@v2.0.5
- github.com/pkg/errors@v0.9.1
- github.com/pmezard/go-difflib@v1.0.0
- github.com/prometheus/client_golang@v1.12.2
- github.com/prometheus/client_model@v0.2.0
- github.com/prometheus/common@v0.34.0
- github.com/prometheus/procfs@v0.8.0
- github.com/rakyll/statik@v0.1.7
- github.com/rcrowley/go-metrics@v0.0.0-20201227073835-cf1acfcdf475
- github.com/regen-network/cosmos-proto@v0.3.1
- github.com/rs/cors@v1.8.2
- github.com/rs/zerolog@v1.27.0
- github.com/spf13/afero@v1.8.2
- github.com/spf13/cast@v1.5.0
- github.com/spf13/cobra@v1.6.0
- github.com/spf13/jwalterweatherman@v1.1.0
- github.com/spf13/pflag@v1.0.5
- github.com/spf13/viper@v1.13.0
- github.com/stretchr/testify@v1.8.0
- github.com/subosito/gotenv@v1.4.1
- github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7
- github.com/tendermint/btcd@v0.1.1
- github.com/tendermint/crypto@v0.0.0-20191022145703-50d29ede1e15
- github.com/tendermint/go-amino@v0.16.0
- github.com/tendermint/tendermint@v0.34.23
- github.com/tendermint/tm-db@v0.6.7
- github.com/ulikunitz/xz@v0.5.8
- github.com/zondax/hid@v0.9.1-0.20220302062450-5552068d2266
- go.opencensus.io@v0.23.0
- golang.org/x/crypto@v0.1.0
- golang.org/x/exp@v0.0.0-20220722155223-a9213eeb770e
- golang.org/x/net@v0.1.0
- golang.org/x/oauth2@v0.1.0
- golang.org/x/sys@v0.1.0
- golang.org/x/term@v0.1.0
- golang.org/x/text@v0.4.0
- golang.org/x/xerrors@v0.0.0-20220907171357-04be3eba64a2
- google.golang.org/api@v0.100.0
- google.golang.org/genproto@v0.0.0-20221027153422-115e99e71e1c
- google.golang.org/grpc@v1.50.1
- google.golang.org/protobuf@v1.28.2-0.20220831092852-f930b1dc76e8
- gopkg.in/ini.v1@v1.67.0
- gopkg.in/yaml.v2@v2.4.0
- gopkg.in/yaml.v3@v3.0.1
- nhooyr.io/websocket@v1.8.6
- sigs.k8s.io/yaml@v1.3.0
build_tags: ledger,
commit: a7682fe71078c981d6f4ebf7cdb1a4e2c7fa2112
cosmos_sdk_version: v0.46.6
go: go version go1.19.1 linux/amd64
name: lum
server_name: lumd
version: 1.3.0
amaury1093 commented 1 year ago

It seems to me from https://github.com/cosmos/cosmos-sdk/issues/13314#issuecomment-1249936670 that the old and new binaries are using different values of the bech32 prefix. @Segfaultd Could you check that they are the same?

Segfaultd commented 1 year ago

Hello @amaurym and thank you for your comment.

You can see here that the prefixes used at Lum Chain weren't changed for the last 13 months. https://github.com/lum-network/chain/blob/master/app/prefix.go

But we have a custom coin type. Dunno if that could have an impact or not.

julienrbrt commented 1 year ago

Have you been able to resolve your issue?

Segfaultd commented 1 year ago

Hi @julienrbrt

Yes, the fix was to call SetConfig in app.go as well (since it was registered only on CLI entrypoint at first) https://github.com/lum-network/chain/commit/d9afc2595d4436a177ced5c664929004d3e3e8a8#diff-0f1d2976054440336a576d47a44a37b80cdf6701dd9113012bce0e3c425819b7R165