zenodeapp
commented
9 months ago I am currently trying to see where exactly the problem could lie. I'm no Golang developer. Though trying to debug this with some understanding of programming.
See the verifyUpdates function in the validator_set.go file. The if-statement at line 448 seems to trigger this Error.
https://github.com/cometbft/cometbft/blob/main/types/validator_set.go#L427
func verifyUpdates(
updates []*Validator,
vals *ValidatorSet,
removedPower int64,
) (tvpAfterUpdatesBeforeRemovals int64, err error) {
delta := func(update *Validator, vals *ValidatorSet) int64 {
_, val := vals.GetByAddress(update.Address)
if val != nil {
return update.VotingPower - val.VotingPower
}
return update.VotingPower
}
updatesCopy := validatorListCopy(updates)
sort.Slice(updatesCopy, func(i, j int) bool {
return delta(updatesCopy[i], vals) < delta(updatesCopy[j], vals)
})
tvpAfterRemovals := vals.TotalVotingPower() - removedPower
for _, upd := range updatesCopy {
tvpAfterRemovals += delta(upd, vals)
if tvpAfterRemovals > MaxTotalVotingPower {
return 0, ErrTotalVotingPowerOverflow
}
}
return tvpAfterRemovals + removedPower, nil
}Some observations:
- In line 445:
tvpAfterRemovals := vals.TotalVotingPower() - removedPower
does not lead to the Panic error it could create (vals.TotalVotingPower() => TotalVotingPower() => updateTotalVotingPower()).
func (vals *ValidatorSet) TotalVotingPower() int64 {
if vals.totalVotingPower == 0 {
vals.updateTotalVotingPower()
}
return vals.totalVotingPower
}and
func (vals *ValidatorSet) updateTotalVotingPower() {
sum := int64(0)
for _, val := range vals.Validators {
// mind overflow
sum = safeAddClip(sum, val.VotingPower)
if sum > MaxTotalVotingPower {
panic(fmt.Sprintf(
"Total voting power should be guarded to not exceed %v; got: %v",
MaxTotalVotingPower,
sum))
}
}
vals.totalVotingPower = sum
}This makes me believe this part is not the culprit. Unless for some reason the cached totalVotingPower in the ValidatorSet struct is a value that became too big at a certain point and doesn't get updated to the correct value now, because it only gets updated whenever the totalVotingPower equals 0, else it keeps getting it from cache. (see if vals.totalVotingPower == 0).
Though unsure if this could be the issue.
-
Another potential cause was that the Voting Powers were wrong (as in having numbers that are too big; for instance because of using the incorrect denominator). Though I believe this is not the case for when I query the status of my Node (
localhost/status) or status of all nodes (localhost/validators), I see VPs based on L1 (which is the biggest denominator and main symbol for the token). Which, when added together wouldn't by far get to the max value of 64bit/8 (MaxTotalVotingPower = int64(math.MaxInt64) / 8). -
If supposedly for removedPower in line 445 a big negative number would be inserted then that may lead to
tvpAfterRemovalsto become a huge positive number in line 445. I haven't researched this well enough to see if this happens or could happen. It's unlikely, nevertheless possible. -
Or something weird is happening in
tvpAfterRemovals += delta(upd, vals). But not seeing what that could be at the moment.
Just tried to research this a bit and see if anything could be of use here. I'll continue checking into this, though again, I'm no Golang developer and not really sure how I could actually debug this.
yihuang
alpha-omega-labs
Hello, We had success in migration of genesis.json with your help and used your source code for GenesisL1 upgrade, thank you! However faced some issue: after ~71% of updated voting power went online, block production ended with error:
INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"FA39B16C041A8FD631784A9CC19EB5F4F0FE4FD8824226175ED201FCE76FA3B8","parts":{"hash ":"85BA0BC72B40D89610F688B70AC47D3D07B56D3A672B456CB91A453F84227452","total":1}},"height":6751391,"pol_round":-1,"round":1,"signature":"zzMzxlERtfDtUyRmu3d3LJTPP9u7Fx B72GN77vcAKVguYS0r8FvliUIcJXPN8PBUohWKAbSO0QXj2abz8KPYBQ==","timestamp":"2023-09-07T19:04:44.225325519Z"} server=node 3:04PM INF received complete proposal block hash=FA39B16C041A8FD631784A9CC19EB5F4F0FE4FD8824226175ED201FCE76FA3B8 height=6751391 module=consensus server=node 3:04PM INF finalizing commit of block hash={} height=6751391 module=consensus num_txs=0 root=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 server=n ode 3:04PM INF minted coins from module account amount=2684871870626760975el1 from=mint module=x/bank 3:04PM INF executed block height=6751391 module=state num_invalid_txs=0 num_valid_txs=0 server=node 3:04PM ERR CONSENSUS FAILURE!!! err="failed to apply block; error commit failed for application: error changing validator set: total voting power of resulting valset exceeds max 1152921504606846975" module=consensus server=node stack="goroutine 138 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/te ndermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint@v0.34.29/consensus/state.go:732 +0x4c\npanic({0x26c4320, 0xc001db119 0})\n\truntime/panic.go:884 +0x213\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc0001f9880, 0x67049f)\n\tgithub.com/tendermint/tendermint@v0. 34.29/consensus/state.go:1670 +0xfab\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc0001f9880, 0x67049f)\n\tgithub.com/tendermint/tendermin t@v0.34.29/consensus/state.go:1570 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()\n\tgithub.com/tendermint/tendermint@v0.34.29/consen sus/state.go:1505 +0xaa\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc0001f9880, 0x67049f, 0x1)\n\tgithub.com/tendermint/tendermint@v0.34.29/con sensus/state.go:1543 +0xccf\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc0001f9880, 0xc001d5c140, {0xc0912c2ed0, 0x28})\n\tgithub.com/tendermint/te ndermint@v0.34.29/consensus/state.go:2164 +0x18dc\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc0001f9880, 0xc001d5c140, {0xc0912c2ed0?, 0xc004ef dc08?})\n\tgithub.com/tendermint/tendermint@v0.34.29/consensus/state.go:1962 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc0001f9880, {{0x3c 77020, 0xc00291b500}, {0xc0912c2ed0, 0x28}})\n\tgithub.com/tendermint/tendermint@v0.34.29/consensus/state.go:861 +0x3ff\ngithub.com/tendermint/tendermint/consensus.(* State).receiveRoutine(0xc0001f9880, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.29/consensus/state.go:768 +0x3f0\ncreated by github.com/tendermint/tendermint/conse nsus.(*State).OnStart\n\tgithub.com/tendermint/tendermint@v0.34.29/consensus/state.go:379 +0x12d\n" 3:04PM INF service stop impl={"Logger":{}} module=consensus msg={} server=node wal=/root/.genesisd/data/cs.wal/wal 3:04PM INF service stop impl={"Dir":"/root/.genesisd/data/cs.wal","Head":{"ID":"aFqa318hpH7x:/root/.genesisd/data/cs.wal/wal","Path":"/root/.genesisd/data/cs.wal/wal" },"ID":"group:aFqa318hpH7x:/root/.genesisd/data/cs.wal/wal","Logger":{}} module=consensus msg={} server=node wal=/root/.genesisd/data/cs.wal/walIt looks like there is some miscalculation of voting power somewhere, maybe related to denomination. Can you please point where it can be? Thank you