Failure to (re)delegate after validator jailed & slashed

alexanderbez commented 3 years ago

Summary of Bug

There are numerous (12+) validators on the Umee betanet that have been jailed and subsequently slashed via, what I presume to be, the gravity bridge module. I presume they got jailed and slashed due to not acting in accordance with the liveness expectations of the bridge, i.e. signing validator set updates, etc...

In any case, when the validator attempts to successfully unjail and then attempts a delegation or redelegation to themselves, they receive the following error:

Example:

Error: rpc error: code = InvalidArgument desc = recovered: calculated final stake for delegator <REDACTED> greater than current stake
        final stake:    9905000000.000000000000000000
        current stake:  9895095000.000000000000000000

Which comes from this block: https://github.com/cosmos/cosmos-sdk/blob/9fd866e3820b3510010ae172b682d71594cd8c14/x/distribution/keeper/delegation.go#L96-L132

I recall when working on this long ago with @rigelrozanski, that we knew there could be certain cases where there might be "off by one" issues, but very very unlikely. This is unfortunate as we've seen this issue for several validators and essentially make delegating to them impossible after they're unjailed.

Version

v0.43.0 (also applicable on v0.44)

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

alexanderbez commented 3 years ago

I'm not sure if this is due to the fact that jailing and slashing occurs via the gravity bridge module

ptrnull commented 3 years ago

A full stacktrace is included here:


Error: rpc error: code = InvalidArgument desc = recovered: calculated final stake for delegator umee1<edited> greater than current stake                                            
        final stake:    19405982638.090598310591510442                                                                                                                                                                                        
        current stake:  19367192000.000000000000000000                                                                                                                                                                                        
stack:                                                                                                                 
goroutine 1166453767 [running]:                                                                                                                                                                                                               
runtime/debug.Stack()                                                                                                  
        /root/.go/src/runtime/debug/stack.go:24 +0x65                                                                                                                                                                                         
github.com/cosmos/cosmos-sdk/baseapp.newDefaultRecoveryMiddleware.func1({0x186e5e0, 0xc004636850})                                                                                                                                            
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/recovery.go:71 +0x27                                                                                                                                                    
github.com/cosmos/cosmos-sdk/baseapp.newRecoveryMiddleware.func1({0x186e5e0, 0xc004636850})   
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/recovery.go:39 +0x30                                                                                                                                                    
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x186e5e0, 0xc004636850}, 0xc00105cb40)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/recovery.go:28 +0x37   
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x186e5e0, 0xc004636850}, 0x2002280)                             
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/recovery.go:33 +0x5e              
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()                                                          
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/baseapp.go:598 +0xfd                           
panic({0x186e5e0, 0xc004636850})                                                                                       
        /root/.go/src/runtime/panic.go:1038 +0x215
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x1fa42f8, 0xc000070870}, {0x1ffd438, 0xc001064100}, {{0x1ffd438, 0xc001064100}, 0xc00000f440, {0x1fa42f8, 0xc0000708b0}, {0x1fa4348, ...}, ...}, ...},
 ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/distribution/keeper/delegation.go:127 +0x714
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x1fa42f8, 0xc000070870}, {0x1ffd438, 0xc001064100}, {{0x1ffd438, 0xc001064100}, 0xc00000f440, {0x1fa42f8, 0xc0000708b0}, {0x1fa4348, ...}, ...}, ...}, 
...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/distribution/keeper/delegation.go:147 +0x25b
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Hooks.BeforeDelegationSharesModified({{{0x1fa42f8, 0xc000070870}, {0x1ffd438, 0xc001064100}, {{0x1ffd438, 0xc001064100}, 0xc00000f440, {0x1fa42f8, 0xc0000708b0}, {0x1fa4348, ...}, ...}, .
..}}, ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/distribution/keeper/hooks.go:89 +0x1ae
github.com/cosmos/cosmos-sdk/x/staking/types.MultiStakingHooks.BeforeDelegationSharesModified({_, _, _}, {{0x1fd70e8, 0xc0000b0000}, {0x2002280, 0xc01897dd00}, {{0xb, 0x0}, {0xc00f523890, ...}, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/staking/types/hooks.go:46 +0xd7
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.BeforeDelegationSharesModified(...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/staking/keeper/hooks.go:56
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.Delegate({{0x1fa42f8, 0xc000070840}, {0x1ffd438, 0xc001064100}, {0x1fdef20, 0xc000fee360}, {0x7f0cba85a040, 0xc001178dc0}, {0x1ffb010, 0xc0000b4420}, ...}, ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/staking/keeper/delegation.go:566 +0x722
github.com/cosmos/cosmos-sdk/x/staking/keeper.msgServer.Delegate({{{0x1fa42f8, 0xc000070840}, {0x1ffd438, 0xc001064100}, {0x1fdef20, 0xc000fee360}, {0x7f0cba85a040, 0xc001178dc0}, {0x1ffb010, 0xc0000b4420}, ...}}, ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/staking/keeper/msg_server.go:214 +0x63e
github.com/cosmos/cosmos-sdk/x/staking/types._Msg_Delegate_Handler.func1({0x1fd7158, 0xc021b4c6c0}, {0x1aadd80, 0xc01897da80})
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/staking/types/tx.pb.go:696 +0x78
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x1fd7158, 0xc021b4c690}, {0x4b34c6, 0x40a6cb}, 0x1b0a4e0, 0xc00d99e168)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/msg_service_router.go:113 +0xd5
github.com/cosmos/cosmos-sdk/x/staking/types._Msg_Delegate_Handler({0x1b18520, 0xc000feb550}, {0x1fd7158, 0xc021b4c690}, 0x1d95458, 0xc01baa48c0)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/staking/types/tx.pb.go:698 +0x138
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x1fd70e8, 0xc0000b0000}, {0x2002280, 0xc01897dd00}, {{0xb, 0x0}, {0xc00f523890, 0xe}, 0x33233, {0xb0ba344, ...}, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/msg_service_router.go:117 +0x34f
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x1fd70e8, 0xc0000b0000}, {0x2002280, 0xc01897dd00}, {{0xb, 0x0}, {0xc00f523890, 0xe}, 0x33233, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/baseapp.go:719 +0x46a
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc00116b6c0, 0x2, {0xc01914e000, 0xe3, 0xf0})
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/baseapp.go:679 +0xa2c
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate(...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/test_helpers.go:22
github.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x20140d0, 0xc0002899c0}, {0x0, 0x0}, {0x1fdfda8, 0xc001064100}, {0x2007ad8, ...}, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/x/auth/tx/service.go:117 +0x129
github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler({0x1a29640, 0xc001eac9c0}, {0x1fd7158, 0xc01dde0f00}, 0xc012554230, 0x0)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/types/tx/service.pb.go:743 +0x170
github.com/cosmos/cosmos-sdk/baseapp.(*GRPCQueryRouter).RegisterService.func1({{0x1fd70e8, 0xc0000b0000}, {0x2002280, 0xc01897d9c0}, {{0xb, 0x0}, {0xc00f523890, 0xe}, 0x33233, {0xb0ba344, ...}, ...}, ...}, ...)
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/grpcrouter.go:93 +0x23e
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).handleQueryGRPC(0x50e21553630ba408, 0xc001286100, {{0xc010e1e780, 0xe6, 0x1e0}, {0xc017271cb0, 0x23}, 0x33233, 0x0})
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/abci.go:576 +0x1ef
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Query(0xc00116b6c0, {{0xc010e1e780, 0xe6, 0x1e0}, {0xc017271cb0, 0x23}, 0x33233, 0x0})
        /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.43.0/baseapp/abci.go:420 +0x713
github.com/tendermint/tendermint/abci/client.(*localClient).QuerySync(0xc000281560, {{0xc010e1e780, 0xe6, 0x1e0}, {0xc017271cb0, 0x23}, 0x0, 0x0})
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.12/abci/client/local_client.go:250 +0x171
github.com/tendermint/tendermint/proxy.(*appConnQuery).QuerySync(0x2, {{0xc010e1e780, 0xe6, 0x1e0}, {0xc017271cb0, 0x23}, 0x0, 0x0})
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.12/proxy/app_conn.go:159 +0x6e
github.com/tendermint/tendermint/rpc/core.ABCIQuery(0x8, {0xc017271cb0, 0xc000192480}, {0xc010e1e780, 0x4, 0xc0039a89e0}, 0x2, 0x2)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.12/rpc/core/abci.go:20 +0xb2
reflect.Value.call({0x191ed20, 0x1d97310, 0x80}, {0x1b1cfb2, 0x4}, {0xc01892ec00, 0x5, 0x18})
        /root/.go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x191ed20, 0x1d97310, 0x21f}, {0xc01892ec00, 0x5, 0x5})
        /root/.go/src/reflect/value.go:339 +0xc5
github.com/tendermint/tendermint/rpc/jsonrpc/server.makeJSONRPCHandler.func1({0x1fb58d0, 0xc01ea61260}, 0xc014213a00)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.12/rpc/jsonrpc/server/http_json_handler.go:96 +0x1055
github.com/tendermint/tendermint/rpc/jsonrpc/server.handleInvalidJSONRPCPaths.func1({0x1fb58d0, 0xc01ea61260}, 0x7f0ce177b108)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.12/rpc/jsonrpc/server/http_json_handler.go:122 +0x5d
net/http.HandlerFunc.ServeHTTP(0xc01897d080, {0x1fb58d0, 0xc01ea61260}, 0xc0192838a0)
        /root/.go/src/net/http/server.go:2046 +0x2f
net/http.(*ServeMux).ServeHTTP(0x160, {0x1fb58d0, 0xc01ea61260}, 0xc014213a00)
        /root/.go/src/net/http/server.go:2424 +0x149
github.com/tendermint/tendermint/rpc/jsonrpc/server.maxBytesHandler.ServeHTTP({{0x1f97300, 0xc001191cc0}, 0x40d187}, {0x1fb58d0, 0xc01ea61260}, 0xc014213a00)
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.12/rpc/jsonrpc/server/http_server.go:236 +0x122
github.com/tendermint/tendermint/rpc/jsonrpc/server.RecoverAndLogHandler.func1({0x1fb5f60, 0xc00c146b60}, 0xc014213a00) 
        /root/go/pkg/mod/github.com/tendermint/tendermint@v0.34.12/rpc/jsonrpc/server/http_server.go:209 +0x38a
net/http.HandlerFunc.ServeHTTP(0x0, {0x1fb5f60, 0xc00c146b60}, 0x72bba0)
        /root/.go/src/net/http/server.go:2046 +0x2f
net/http.serverHandler.ServeHTTP({0x1fa5630}, {0x1fb5f60, 0xc00c146b60}, 0xc014213a00)
        /root/.go/src/net/http/server.go:2878 +0x43b
net/http.(*conn).serve(0xc01fa430e0, {0x1fd7158, 0xc001ee61e0})
        /root/.go/src/net/http/server.go:1929 +0xb08
created by net/http.(*Server).Serve
        /root/.go/src/net/http/server.go:3033 +0x4e8
: panic: invalid request
Usage:
  umeed tx staking delegate [validator-addr] [amount] [flags]

alexanderbez commented 3 years ago

I will also note that @mvid stated they didn't experience this issue on Somm, even though they also had jailed validators.

alexanderbez commented 3 years ago

Thanks for posting @ptrnull, but adding more panic snippets won't help here. We need folks to describe their setup and how they got to this position, i.e. how much they delegated and how much was slashed and how much was there after the slash.

alexanderbez commented 3 years ago

I've attempted to emulate the same behavior flow of what gravity would do/behave via a unit test:

Create validator
Get slashed and jailed
Unjail
(Self) delegate more

Note, the unit test PASSES. So without more info how how folks are getting into this situation, it's nearly impossible to debug.

func TestSlashingBug(t *testing.T) {
    app := simapp.Setup(t, false)
    ctx := app.BaseApp.NewContext(false, tmproto.Header{})

    addrDels := simapp.AddTestAddrsIncremental(app, ctx, 1, app.StakingKeeper.TokensFromConsensusPower(ctx, 11000000000))
    valAddrs := simapp.ConvertAddrsToValAddrs(addrDels)
    pks := simapp.CreateTestPubKeys(1)
    addr, valPubKey := valAddrs[0], pks[0]
    power := int64(10000000000)
    tstaking := teststaking.NewHelper(t, ctx, app.StakingKeeper)

    stakedAmt := tstaking.CreateValidatorWithValPower(addr, valPubKey, power, true)

    staking.EndBlocker(ctx, app.StakingKeeper)
    ctx = ctx.WithBlockHeight(ctx.BlockHeight() + 1)

    validator, ok := app.StakingKeeper.GetValidatorByConsAddr(ctx, sdk.GetConsAddress(valPubKey))
    require.True(t, ok)

    valConsAddr, err := validator.GetConsAddr()
    require.NoError(t, err)

    // slash and jail
    slashedAmt := app.StakingKeeper.Slash(
        ctx,
        valConsAddr,
        ctx.BlockHeight(),
        power,
        sdk.NewDec(1).Quo(sdk.NewDec(1000)), // the Gravity Bridge default
    )
    require.False(t, slashedAmt.IsZero())
    app.StakingKeeper.Jail(ctx, valConsAddr)

    staking.EndBlocker(ctx, app.StakingKeeper)
    ctx = ctx.WithBlockHeight(ctx.BlockHeight() + 1)

    // validator should have been jailed and slashed
    validator, ok = app.StakingKeeper.GetValidatorByConsAddr(ctx, valConsAddr)
    require.True(t, ok)
    require.True(t, validator.IsJailed())
    require.Equal(t, stakingtypes.Unbonding, validator.GetStatus())
    require.NotEqual(t, stakedAmt, validator.GetTokens())

    staking.EndBlocker(ctx, app.StakingKeeper)
    ctx = ctx.WithBlockHeight(ctx.BlockHeight() + 1)

    // unjail
    app.StakingKeeper.Unjail(ctx, valConsAddr)

    staking.EndBlocker(ctx, app.StakingKeeper)
    ctx = ctx.WithBlockHeight(ctx.BlockHeight() + 1)

    // require validator is no longer jailed
    validator, ok = app.StakingKeeper.GetValidatorByConsAddr(ctx, valConsAddr)
    require.True(t, ok)
    require.False(t, validator.IsJailed())

    // require the ability to further self-delegate
    _, err = app.StakingKeeper.Delegate(ctx, addrDels[0], sdk.NewInt(1000000), stakingtypes.Unbonded, validator, true)
    require.NoError(t, err)
}

armcmu commented 3 years ago

Today my validator was jailed. This is what I found in the logs:

8gb-hel1-2 cosmovisor[745]: 5:45PM INF indexed block height=150417 module=txindex
Sep 29 17:45:31 ubuntu-8gb-hel1-2 cosmovisor[745]: 5:45PM INF minted coins from module account amount=864149uumee from=mint module=x/bank
Sep 29 17:45:31 ubuntu-8gb-hel1-2 cosmovisor[745]: 5:45PM INF burned tokens from module account amount=360000000uumee from=bonded_tokens_pool module=x/bank
Sep 29 17:45:31 ubuntu-8gb-hel1-2 cosmovisor[745]: 5:45PM INF validator slashed by slash factor burned=360000000 module=x/staking slash_factor=0.010000000000000000 validator=umeevaloper1f897ckgjk7lrhx0gp3y438gf8dkfxfc52u98kl
Sep 29 17:45:31 ubuntu-8gb-hel1-2 cosmovisor[745]: 5:45PM INF validator jailed module=x/staking validator=umeevalcons1xvvjsqupue2dk02pauwzwkj5cskrxr6396mc5f

I have enough funds on my balance:

$ umeed q bank balances umee1f897ckgjk7lrhx0gp3y438gf8dkfxfc52czg84
balances:
- amount: "1899999100"
  denom: uumee
pagination:
  next_key: null
  total: "0"

Since I used to read on discord that guys were losing amounts of 30-50umee from a stake when the validators were out of jail, I put a huge fees of 55umee. Figured this transaction required a high fee:

umeed tx slashing unjail \
> --from umee1f897ckgjk7lrhx0gp3y438gf8dkfxfc52czg84 \
> --chain-id umee-betanet-2 \
> --fees=55000000uumee \
> --gas=auto
Enter keyring passphrase:
gas estimate: 71610
{"body":{"messages":[{"@type":"/cosmos.slashing.v1beta1.MsgUnjail","validator_addr":"umeevaloper1f897ckgjk7lrhx0gp3y438gf8dkfxfc52u98kl"}],"memo":"","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[],"fee":{"amount":[{"denom":"uumee","amount":"55000000"}],"gas_limit":"71610","payer":"","granter":""}},"signatures":[]}

confirm transaction before signing and broadcasting [y/N]: y
{"height":"153466","txhash":"BDFFA155B51EF03D75884D1931D8E2A5220804DA3FE2B103E7E55B9087B8E7BA","codespace":"","code":0,"data":"0A240A222F636F736D6F732E736C617368696E672E763162657461312E4D7367556E6A61696C","raw_log":"[{\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"/cosmos.slashing.v1beta1.MsgUnjail\"},{\"key\":\"module\",\"value\":\"slashing\"},{\"key\":\"sender\",\"value\":\"umeevaloper1f897ckgjk7lrhx0gp3y438gf8dkfxfc52u98kl\"}]}]}]","logs":[{"msg_index":0,"log":"","events":[{"type":"message","attributes":[{"key":"action","value":"/cosmos.slashing.v1beta1.MsgUnjail"},{"key":"module","value":"slashing"},{"key":"sender","value":"umeevaloper1f897ckgjk7lrhx0gp3y438gf8dkfxfc52u98kl"}]}]}],"info":"","gas_wanted":"71610","gas_used":"70086","tx":null,"timestamp":""}

Looks like we got out of jail. But, as you can see below, "VotingPower: "35640", which means from the stake lost 360umee!!!

{
  "NodeInfo": {
    "protocol_version": {
      "p2p": "8",
      "block": "11",
      "app": "0"
    },
    "id": "39208a227589203d21092df3d7a4186f219f8140",
    "listen_addr": "65.108.55.197:26656",
    "network": "umee-betanet-2",
    "version": "v0.34.13",
    "channels": "40202122233038606100",
    "moniker": "armcmu",
    "other": {
      "tx_index": "on",
      "rpc_address": "tcp://127.0.0.1:26657"
    }
  },
  "SyncInfo": {
    "latest_block_hash": "D385B919A8A5624BB62750DFB60081F754CEC89937A7330ADC44EE0AECB2B679",
    "latest_app_hash": "7A23CC01FFA8E7358F018B6FA44A72AC7BF6D9B3D736E22EC80FB532AD335AB8",
    "latest_block_height": "153470",
    "latest_block_time": "2021-09-29T16:13:36.656266603Z",
    "earliest_block_hash": "DCE7EFC72969065E6E3338E88F7A8DC1A13701A95CD9A227A914DBD0E1A7F570",
    "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
    "earliest_block_height": "1",
    "earliest_block_time": "2021-09-16T14:39:52.331523308Z",
    "catching_up": false
  },
  "ValidatorInfo": {
    "Address": "3319280381E654DB3D41EF1C275A54C42C330F51",
    "PubKey": {
      "type": "tendermint/PubKeyEd25519",
      "value": "lmOdoLVUvLr5I8dWY6kkxEszVPAXldbpT/egARhuVTA="
    },
    "VotingPower": "35640"
  }
}

In addition to the fact that 55umee was also withdrawn from the account, which I indicated in the command

umeed q bank balances umee1f897ckgjk7lrhx0gp3y438gf8dkfxfc52czg84
balances:
- amount: "1844999100"
  denom: uumee
pagination:
  next_key: null
  total: "0"

System Ubuntu 20.04

$ umeed version
0.2.0
$ gorc version
gorc 0.2.13
$ geth version
Geth
Version: 1.10.8-stable
Git Commit: 26675454bf93bf904be7a43cce6b3f550115ff90
Git Commit Date: 20210824
Architecture: amd64
Go Version: go1.16.4
Operating System: linux
GOPATH=/root/go
GOROOT=go

tanutcho commented 3 years ago

I reproduce and found out the issue occured after jail & unjail process. The reason of jailing occured after

the umee orchestrator address lack of funding (umee token)
Low on disk space (Block cannot be sync)

when orchestrator address have not enough umee token. It's shown in logs of gorc service

However, the issue cannot be solved yet due to unavailable to delegate with error


Error: rpc error: code = InvalidArgument desc = recovered: calculated final stake for delegator umee1lfgqx370k659q002k742a2wv4mlmrdh3duvlfp greater than current stake
        final stake:    5960000000.000000000000000000
        current stake:  5942138328.084146905080446595

gorc version 0.2.10

Geth Version: 1.10.8-stable Git Commit: 26675454bf93bf904be7a43cce6b3f550115ff90 Git Commit Date: 20210824 Architecture: amd64 Go Version: go1.16.4 Operating System: linux GOPATH=/root/go GOROOT=/usr/local/go

armcmu commented 3 years ago

And here's the jail again. A total of 145 blocks later, my validator and 9 other "colleagues" have gone to jail. Gorс had been off since the last incident. Only the umeed service was working and geth was looking for live peers

rSep 29 18:29:25 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"BD44AAC0CAE08396BD50BF5B9E81652F94640E1EF1562E9AE619CBE6A0059229","parts":{"hash":"8563579CA9A6C285267600AE5EC185CE91AD9956F1AD20E2FF5B5E75E5489E76","total":1}},"height":153611,"pol_round":-1,"round":0,"signature":"hdQYGyojXcrV0PKAkeOp+WTQDcQpUv7Ws0WS8Z+F5WO/Z1A3lL1fg2QBr/LmVKWp0uDJPneB4gIjSjPiN+cHDg==","timestamp":"2021-09-29T16:29:25.489461709Z"}
Sep 29 18:29:25 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF received complete proposal block hash=BD44AAC0CAE08396BD50BF5B9E81652F94640E1EF1562E9AE619CBE6A0059229 height=153611 module=consensus
Sep 29 18:29:26 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF finalizing commit of block hash=BD44AAC0CAE08396BD50BF5B9E81652F94640E1EF1562E9AE619CBE6A0059229 height=153611 module=consensus num_txs=0 root=C75CE01F7A66CF4146762424AC1FE387A88227891884F64772DB06693C17F963
Sep 29 18:29:26 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF minted coins from module account amount=864529uumee from=mint module=x/bank
Sep 29 18:29:26 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF considering signer set tx creation blockHeight=153611 lastUnbondingHeight=153592 latestSignerSetTx.Nonce=242 module=x/gravity powerDiff=0.04699623981653625 shouldCreate=false
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=36088000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=36088000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper1ps8trnn3h0fclpnnmv7rpstneq3hqdkjurmfy2
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1ptzfrzshuqk4p2u0jdkeknxdk8346yg46vul58
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=35640000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=35640000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper1f897ckgjk7lrhx0gp3y438gf8dkfxfc52u98kl
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1xvvjsqupue2dk02pauwzwkj5cskrxr6396mc5f
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=15920000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=15920000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper1lfgqx370k659q002k742a2wv4mlmrdh3dctsct
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1u0jct8nd6elga0v9mvtnaav0kuy5nf8290ecel
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=11997000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=11997000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper1c28xdkp83mu2wr0q5tgcsa6utlz4aytefq7xm8
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1xzw0vggar5x5qxyraj940e5wvx9ejay4gmef5u
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=9975000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=9975000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper1jtqufq68yxtg5d7eeuas4lvk7f6dqtvwszgv8e
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1a9shgl07gqnthjyj4ytrjl9ppczsv4g7fu4kya
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=9910000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=9910000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper14tkng2rczzuff3a35r6pndjchk7rca5fh7es6u
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1lf6ten2h09k4gchwrevy805pfa0x0jz2t867dg
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=9908000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=9908000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper16jj99kyly34z0r4x5pfwpy74wj704rl896zed6
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1qlr33tqfu7utpqxe50jcxjuef4gz0h2y4g8rnx
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=7733000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=7733000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper15k0z580ekcmt7wck6hdu2p29cquqs09vmnpxk2
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1t6l4z2vyaxwur42yh35seefjljqvvxtd2c9lct
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=6995000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=6995000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper1dytk3wwvpfyv40dx4qnsajlgramdvnt30f24nq
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1ywczsf6ufhfukfe60s3f6y6cptx44r8ggg2c72
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF slashing at current height; not scanning unbonding delegations & redelegations height=153611 module=x/staking
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF burned tokens from module account amount=6936000uumee from=bonded_tokens_pool module=x/bank
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator slashed by slash factor burned=6936000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper1jjyjvsxxmnel5hu85g0spc7qfpx8cav99t2shk
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF validator jailed module=x/staking validator=umeevalcons1yvkgm0yvph5vtlx9pacnr25xyzvm086shwet4f
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF executed block height=153611 module=state num_invalid_txs=0 num_valid_txs=0
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF commit synced commit=436F6D6D697449447B5B3835203134332031313420323338203937203131382031323320323020313020383220363120323033203131322031393620363420323232203832203231362037342031393320323239203233382031353920313135203231362031323920323034203720313433203334203537203133315D3A32353830427D
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF committed state app_hash=558F72EE61767B140A523DCB70C440DE52D84AC1E5EE9F73D881CC078F223983 height=153611 module=state num_txs=0
Sep 29 18:29:27 ubuntu-8gb-hel1-2 cosmovisor[745]: 6:29PM INF indexed block height=153611 module=txindex

This time I used a fee of only 200uumee and 36umee was missing from the stake:

INF validator slashed by slash factor burned=35640000 module=x/staking slash_factor=0.001000000000000000 validator=umeevaloper1f897ckgjk7lrhx0gp3y438gf8dkfxfc52u98kl

And now "VotingPower": "35604"

ppnv commented 3 years ago

I ran out of disk space and ended up in jail. I added SSD and wrote a command to get out of prison and now this problem. At the same time, this has not happened before, this is only the last time (I added SSD several times)

boz commented 2 years ago

I've run into this issue on the latest gravity testnet (https://github.com/Gravity-Bridge/Gravity-Bridge/issues/2). I've reset since being slashed so I can't find events, but this is what I found in the logs:

Dec 05 00:12:55 gbridge.akash.farm gravity[53043]: 12:12AM INF validator slashed by slash factor burned=182000000 module=x/staking slash_factor=0.001000000000000000 validator=gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9
Dec 06 17:03:53 gbridge.akash.farm gravity[59989]: {"level":"info","module":"x/staking","burned":"1818180000","slash_factor":"0.010000000000000000","validator":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9","time":"2021-12-06T17:03:53Z","message":"validator slashed by slash factor"}
Dec 06 17:30:30 gbridge.akash.farm gravity[59989]: {"level":"info","module":"x/staking","burned":"0","slash_factor":"0.001000000000000000","validator":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9","time":"2021-12-06T17:30:30Z","message":"validator slashed by slash factor"}
Dec 06 17:30:40 gbridge.akash.farm gravity[59989]: {"level":"info","module":"x/staking","burned":"0","slash_factor":"0.001000000000000000","validator":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9","time":"2021-12-06T17:30:40Z","message":"validator slashed by slash factor"}
Dec 06 17:35:20 gbridge.akash.farm gravity[59989]: {"level":"info","module":"x/staking","burned":"0","slash_factor":"0.001000000000000000","validator":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9","time":"2021-12-06T17:35:20Z","message":"validator slashed by slash factor"}
Dec 06 17:40:08 gbridge.akash.farm gravity[59989]: {"level":"info","module":"x/staking","burned":"0","slash_factor":"0.001000000000000000","validator":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9","time":"2021-12-06T17:40:08Z","message":"validator slashed by slash factor"}
Dec 06 17:40:37 gbridge.akash.farm gravity[59989]: {"level":"info","module":"x/staking","burned":"0","slash_factor":"0.001000000000000000","validator":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9","time":"2021-12-06T17:40:37Z","message":"validator slashed by slash factor"}
Dec 06 17:44:45 gbridge.akash.farm gravity[59989]: {"level":"info","module":"x/staking","burned":"0","slash_factor":"0.001000000000000000","validator":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9","time":"2021-12-06T17:44:45Z","message":"validator slashed by slash factor"}
Dec 06 17:46:50 gbridge.akash.farm gravity[59989]: {"level":"info","module":"x/staking","burned":"0","slash_factor":"0.001000000000000000","validator":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9","time":"2021-12-06T17:46:50Z","message":"validator slashed by slash factor"}

Looks like it's burned

 182000000 + 
1818180000
----------
2000180000

the discrepancy from the panic is

180180000000 - 
179999820000
------------
   180180000

Some details about the environment that I can recall:

my gentx had commission=0, I updated it after the chain started.
I was able to withdraw once successfully before it failed.
I don't remember if I did the successful withdraw before or after I bumped commission.
I am able to withdraw as a separate delegator delegating to this validator.

I see now that the gravity module is slashing, it may be an issue with gravity https://github.com/Gravity-Bridge/Gravity-Bridge/blob/1e58f549904f70342dd80b59a9218ad056f2054f/module/x/gravity/keeper/evidence.go#L78-L82

TxCorpi0x commented 1 year ago

@alexanderbez, I'm just curious is this issue fixed? I have a similar problem with jailed and unjailed validator rewards calculations:

github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Query.func1
github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/abci.go:409
runtime.gopanic runtime/panic.go:838
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards
github.com/cosmos/cosmos-sdk@v0.45.4/x/distribution/keeper/delegation.go:127
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.DelegationTotalRewards.func1
github.com/cosmos/cosmos-sdk@v0.45.4/x/distribution/keeper/grpc_query.go:185
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.IterateDelegations
github.com/cosmos/cosmos-sdk@v0.45.4/x/staking/keeper/alias_functions.go:127
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.DelegationTotalRewards
github.com/cosmos/cosmos-sdk@v0.45.4/x/distribution/keeper/grpc_query.go:179
github.com/cosmos/cosmos-sdk/x/distribution/types._Query_DelegationTotalRewards_Handler
github.com/cosmos/cosmos-sdk@v0.45.4/x/distribution/types/query.pb.go:1216
github.com/cosmos/cosmos-sdk/baseapp.(*GRPCQueryRouter).RegisterService.func1
github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/grpcrouter.go:85
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).handleQueryGRPC
github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/abci.go:577
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Query
github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/abci.go:421
github.com/tendermint/tendermint/abci/client.(*localClient).QuerySync
github.com/tendermint/tendermint@v0.34.19/abci/client/local_client.go:256
github.com/tendermint/tendermint/proxy.(*appConnQuery).QuerySync
github.com/tendermint/tendermint@v0.34.19/proxy/app_conn.go:159
github.com/tendermint/tendermint/rpc/core.ABCIQuery
github.com/tendermint/tendermint@v0.34.19/rpc/core/abci.go:20
github.com/tendermint/tendermint/rpc/client/local.(*Local).ABCIQueryWithOptions
github.com/tendermint/tendermint@v0.34.19/rpc/client/local/local.go:87
github.com/cosmos/cosmos-sdk/client.Context.queryABCI
github.com/cosmos/cosmos-sdk@v0.45.4/client/query.go:94
github.com/cosmos/cosmos-sdk/client.Context.QueryABCI
github.com/cosmos/cosmos-sdk@v0.45.4/client/query.go:57
github.com/cosmos/cosmos-sdk/client.Context.Invoke
github.com/cosmos/cosmos-sdk@v0.45.4/client/grpc_query.go:81
github.com/cosmos/cosmos-sdk/x/distribution/types.(*queryClient).DelegationTotalRewards
github.com/cosmos/cosmos-sdk@v0.45.4/x/distribution/types/query.pb.go:1027
github.com/cosmos/cosmos-sdk/x/distribution/types.request_Query_DelegationTotalRewards_0
github.com/cosmos/cosmos-sdk@v0.45.4/x/distribution/types/query.pb.gw.go:330
github.com/cosmos/cosmos-sdk/x/distribution/types.RegisterQueryHandlerClient.func6
github.com/cosmos/cosmos-sdk@v0.45.4/x/distribution/types/query.pb.gw.go:824
github.com/grpc-ecosystem/grpc-gateway/runtime.(*ServeMux).ServeHTTP
github.com/grpc-ecosystem/grpc-gateway@v1.16.0/runtime/mux.go:240
github.com/gorilla/mux.(*Router).ServeHTTP
github.com/gorilla/mux@v1.8.0/mux.go:210
github.com/tendermint/tendermint/rpc/jsonrpc/server.maxBytesHandler.ServeHTTP
github.com/tendermint/tendermint@v0.34.19/rpc/jsonrpc/server/http_server.go:236
github.com/tendermint/tendermint/rpc/jsonrpc/server.RecoverAndLogHandler.func1
github.com/tendermint/tendermint@v0.34.19/rpc/jsonrpc/server/http_server.go:209
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2084
net/http.serverHandler.ServeHTTP
net/http/server.go:2916
net/http.(*conn).serve
net/http/server.go:1966
calculated final stake for delegator sge1zqtautpy4nfzj4mpu8t26sy6j77f0trkd9aqqh greater than current stake
final stake: 200000000.000000000000000000
current stake: 199980001.905634721132374241

This is the difference between the final and current stake:

200000000.000000000000000000 - 
199980001.905634721132374241 
= 
19998.09940

Is it a cosmos code bug or a gravity bug?

cosmos / cosmos-sdk