search

crypto-com / thaler

Thaler Experimental Network; For Crypto.org Chain: github.com/crypto-org-chain/chain-main
https://thaler-testnet.crypto.com
Other
158 stars 80 forks source link

Problem: non-determinism in reward distribution #1835

Open tomtau opened 4 years ago

tomtau commented 4 years ago

(being investigated on 0.5) some nodes randomly after reward distribution ended up with a different app hash

yihuang commented 4 years ago

current observations from logs, where issue happens:

Two possible reasons for that:

  1. period_bonus is already inconsistent.
  2. period_bonus is good, but the reward distribution code is non-determinism.

By reading the code, I think the second case is quite unlikely. Currently we added more logs to try to rule out the first case. If it's the first case, then problem space further split into two directions:

  1. fee accumulation non-determinism
  2. slashing accumulation non-determinism
tomtau commented 4 years ago

for fee accumulation, it's only here: https://github.com/crypto-com/chain/blob/release/v0.5/chain-abci/src/app/mod.rs#L239 and the TX order should be the same / deterministic (unless something strange happens with tokio async reading/writing on the socket: https://github.com/tendermint/rust-abci/blob/develop/src/server.rs#L38 )