search

cosmos / interchain-security

Interchain Security is an open sourced IBC application which allows cosmos blockchains to lease their proof-of-stake security to one another.
https://cosmos.github.io/interchain-security/
Other
156 stars 126 forks source link

SDK changes break simulator #227

Closed mpoke closed 2 years ago

mpoke commented 2 years ago

This is visible in this PR https://github.com/cosmos/cosmos-sdk/pull/12537, i.e., 

exit status 1
[W2] Seed 2: FAILED
To reproduce run: go test ./simapp -run TestFullAppSimulation -Enabled=true -NumBlocks=50 -Genesis= -Verbose=true -Commit=true -Seed=2 -Period=10 -ExportParamsPath /tmp/sim-logs-863298177/sim_params-2.json -ExportStatePath /tmp/sim-logs-863298177/sim_state-2.json -v -timeout 24h

Running the above command on the https://github.com/cosmos/cosmos-sdk/tree/interchain-security-rebase branch results in errors, i.e., 

Simulating... block 10/50, operation 0/0.I[2022-07-12|15:49:03.573] asserting crisis invariants                  module=x/crisis inv=0/11 name=bank/nonnegative-outstanding
I[2022-07-12|15:49:03.575] asserting crisis invariants                  module=x/crisis inv=1/11 name=bank/total-supply
I[2022-07-12|15:49:03.576] asserting crisis invariants                  module=x/crisis inv=2/11 name=distribution/nonnegative-outstanding
I[2022-07-12|15:49:03.577] asserting crisis invariants                  module=x/crisis inv=3/11 name=distribution/can-withdraw
I[2022-07-12|15:49:03.619] asserting crisis invariants                  module=x/crisis inv=4/11 name=distribution/reference-count
I[2022-07-12|15:49:03.620] asserting crisis invariants                  module=x/crisis inv=5/11 name=distribution/module-account
I[2022-07-12|15:49:03.621] asserting crisis invariants                  module=x/crisis inv=6/11 name=staking/module-accounts
simulation halted due to panic on block 10
Logs to writing to /Users/marius/.simapp/simulations/2022-07-12_15:49:03.log
--- FAIL: TestAppImportExport (1.88s)
panic: invariant broken: staking: bonded and not bonded module account coins invariant
        Pool's bonded tokens: 21724093905797stake
        sum of bonded tokens: 21724093905797
not bonded token invariance:
        Pool's not bonded tokens: 4625651167973stake
        sum of not bonded tokens: 4531562441725
module accounts total (bonded + not bonded):
        Module Accounts' tokens: 26349745073770stake
        sum tokens:              26255656347522

        CRITICAL please submit the following transaction:
                 tx crisis invariant-broken staking module-accounts [recovered]
        panic: invariant broken: staking: bonded and not bonded module account coins invariant
        Pool's bonded tokens: 21724093905797stake
        sum of bonded tokens: 21724093905797
not bonded token invariance:
        Pool's not bonded tokens: 4625651167973stake
        sum of not bonded tokens: 4531562441725
module accounts total (bonded + not bonded):
        Module Accounts' tokens: 26349745073770stake
        sum tokens:              26255656347522

        CRITICAL please submit the following transaction:
                 tx crisis invariant-broken staking module-accounts [recovered]
        panic: invariant broken: staking: bonded and not bonded module account coins invariant
        Pool's bonded tokens: 21724093905797stake
        sum of bonded tokens: 21724093905797
not bonded token invariance:
        Pool's not bonded tokens: 4625651167973stake
        sum of not bonded tokens: 4531562441725
module accounts total (bonded + not bonded):
        Module Accounts' tokens: 26349745073770stake
        sum tokens:              26255656347522

        CRITICAL please submit the following transaction:
                 tx crisis invariant-broken staking module-accounts

goroutine 24 [running]:
testing.tRunner.func1.2({0x101586440, 0x14001785290})
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/testing/testing.go:1389 +0x1c8
testing.tRunner.func1()
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/testing/testing.go:1392 +0x384
panic({0x101586440, 0x14001785290})
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/runtime/panic.go:838 +0x204
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed.func2()
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:147 +0xc8
panic({0x101586440, 0x14001785290})
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/runtime/panic.go:844 +0x258
github.com/cosmos/cosmos-sdk/x/crisis/keeper.Keeper.AssertInvariants({{0x140001d2c80, 0xb, 0x10}, {{0x1296df1d8, 0x14000ed22e0}, 0x1400011c050, {0x10174f670, 0x14000ed3bb0}, {0x10174f6c0, 0x14000ed3c10}, ...}, ...}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/crisis/keeper/keeper.go:83 +0x528
github.com/cosmos/cosmos-sdk/x/crisis.EndBlocker({{0x10175d648, 0x140000400b8}, {0x101765630, 0x14000518780}, {{0x0, 0x0}, {0x1010b6a47, 0xe}, 0xa, {0x0, ...}, ...}, ...}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/crisis/abci.go:20 +0x140
github.com/cosmos/cosmos-sdk/x/crisis.AppModule.EndBlock(...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/crisis/module.go:168
github.com/cosmos/cosmos-sdk/types/module.(*Manager).EndBlock(_, {{0x10175d648, 0x140000400b8}, {0x101765630, 0x14000518780}, {{0x0, 0x0}, {0x1010b6a47, 0xe}, 0xa, ...}, ...}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/types/module/module.go:495 +0x34c
github.com/cosmos/cosmos-sdk/simapp.(*SimApp).EndBlocker(_, {{0x10175d648, 0x140000400b8}, {0x101765630, 0x14000518780}, {{0x0, 0x0}, {0x1010b6a47, 0xe}, 0xa, ...}, ...}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/simapp/app.go:439 +0x64
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0x14000ee8ea0, {0x140000400b8?})
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/baseapp/abci.go:211 +0x180
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed({0x101766ab0?, 0x14000ee84e0}, {0x10174d438?, 0x14000010018}, 0x1?, 0x0?, 0x101744ad8, {0x1400002a780, 0x17, 0x18}, ...)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/x/simulation/simulate.go:186 +0x109c
github.com/cosmos/cosmos-sdk/simapp.TestAppImportExport(0x14000ee84e0)
        /Users/marius/go/src/github.com/cosmos/cosmos-sdk/simapp/sim_test.go:112 +0x404
testing.tRunner(0x14000ee84e0, 0x101744a30)
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/testing/testing.go:1439 +0x110
created by testing.(*T).Run
        /opt/homebrew/Cellar/go/1.18.1/libexec/src/testing/testing.go:1486 +0x300
FAIL    github.com/cosmos/cosmos-sdk/simapp     2.208s
FAIL
mpoke commented 2 years ago

The issue was introduced somewhere between commits https://github.com/cosmos/cosmos-sdk/commit/57f5633fcaee355a0626bae119a00c825b5e3d5e and https://github.com/cosmos/cosmos-sdk/commit/b293d64ea1b434f0474fbfa4b7340c1bbdd3aef2 (see https://github.com/cosmos/cosmos-sdk/commits/interchain-security-rebase). It's hard to identify the exact commit since the ones in between do not build.

mpoke commented 2 years ago

The test fails because the ModuleAccountInvariants is broken, i.e., https://github.com/cosmos/cosmos-sdk/blob/c783aea68fbd856c2b188b2d467a7fa5cb4df1e6/x/staking/keeper/invariants.go#L80. Also, the test that is failing is TestAppImportExport, which first exports the state and then imports it back.

danwt commented 2 years ago

Found a problem

If we fix it will bring us closer but it might not be the only problem.

mpoke commented 2 years ago

Fixed by https://github.com/cosmos/cosmos-sdk/pull/12783