search

oursky / likecoin-chain

LikeCoin chain node
https://likecoin.bigdipper.live
GNU General Public License v3.0
1 stars 2 forks source link

[Mainnet halted] unexpected validator in unbonding queue; status was not unbonding #259

Closed mitchellchou closed 2 years ago

mitchellchou commented 2 years ago

Background

  1. The chain halted at around 3am HKT 27 May
  2. Seems a validator who was already unbonding was somehow jailed in unbonding period, triggering double unbonding, so the second one failed with panic.

Reference

  1. Hotfix PR: https://github.com/likecoin/likecoin-chain/pull/74
  2. Discord thread: https://discord.com/channels/763001015712350231/966221298525143121/979461155212886096
elise-ng commented 2 years ago

Found problem is due to sdk using plain string comparison when managing the queue:

https://github.com/cosmos/cosmos-sdk/blob/d0043914ba7c37c3a0d7039d2c2a2aca6b38a93b/x/staking/keeper/validator.go#L357 https://github.com/cosmos/cosmos-sdk/blob/d0043914ba7c37c3a0d7039d2c2a2aca6b38a93b/x/staking/keeper/validator.go#L375

which might lead to failure to dequeue processed validators if there is mixed address prefix

the proposed hotfix is to silencing errors if we see that the queued validator is already in the desired state - unbonded or deleted (due to all delegations withdrew after unbonded):

https://github.com/cosmos/cosmos-sdk/blob/d0043914ba7c37c3a0d7039d2c2a2aca6b38a93b/x/staking/keeper/validator.go#L431

https://github.com/cosmos/cosmos-sdk/blob/d0043914ba7c37c3a0d7039d2c2a2aca6b38a93b/x/staking/keeper/validator.go#L435

rickmak commented 2 years ago

Close via https://github.com/likecoin/likecoin-chain/pull/75 Chain back online on 2022-05-27T10:01:00.728507999Z at block height 4011381