testnet - not able to sync from the scratch

[tmeinlschmidt]

version 0.10.12 state sync set as described here: https://docs.bnbchain.org/docs/beaconchain/develop/node/synctypes

app + genesis from the release

panic: unmarshal to types.Validator failed after 160 bytes (insufficient bytes decoding []byte of length 1000000000000): 0A14666F9C66C385A14842F8730C4C906DAE8B22B9981214666F9C66C385A14842F8730C4C906DAE8B22B9981A251624DE64209142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC28023280A094A58D1D3A80A094A58D1D42090A07476168696E67614888E53C6A0E220C08D3F6DEF80510E1F4CA9A01721411896BEC37DAC5D41DC3FDB204DAFC0FCFC999A2920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D9A0180C0DFDA8EE906

goroutine 386 [running]:
github.com/cosmos/cosmos-sdk/x/stake/types.MustUnmarshalValidator(_, {_, _, _})
    github.com/cosmos/cosmos-sdk@v0.25.0/x/stake/types/validator.go:145 +0x12e
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.GetValidator({{0x15f50a8, 0xc0004f5be0}, {0x15f50a8, 0xc0004f5bf0}, {0x15f50d0, 0xc0004f5c90}, 0xc000c62690, {0x161de88, 0xc0005ae0e0}, 0xc000bd6050, ...}, ...)
    github.com/cosmos/cosmos-sdk@v0.25.0/x/stake/keeper/validator.go:25 +0x191
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.GetValidatorByConsAddr({{0x15f50a8, 0xc0004f5be0}, {0x15f50a8, 0xc0004f5bf0}, {0x15f50d0, 0xc0004f5c90}, 0xc000c62690, {0x161de88, 0xc0005ae0e0}, 0xc000bd6050, ...}, ...)
    github.com/cosmos/cosmos-sdk@v0.25.0/x/stake/keeper/validator.go:44 +0x218
github.com/bnb-chain/node/app.(*ValAddrCache).GetDistributionAddr(_, {{0x1609b08, 0xc00013c020}, {0x16236b8, 0xc00bb70740}, {{0xa, 0x0, {}, {0x0, 0x0, ...}, ...}, ...}, ...}, ...)
    github.com/bnb-chain/node/app/fee_distribution.go:62 +0x150
github.com/bnb-chain/node/app.distributeFeeBEP159({{0x1609b08, 0xc00013c020}, {0x16236b8, 0xc00bb70740}, {{0xa, 0x0, {}, {0x0, 0x0, 0x0}, ...}, ...}, ...}, ...)
    github.com/bnb-chain/node/app/fee_distribution.go:83 +0x54a
github.com/bnb-chain/node/app.(*BinanceChain).EndBlocker(_, {{0x1609b08, 0xc00013c020}, {0x16236b8, 0xc00bb70740}, {{0xa, 0x0, {}, {0x0, 0x0, ...}, ...}, ...}, ...}, ...)
    github.com/bnb-chain/node/app/app.go:907 +0xc3e
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc000d005a0, {0x24fbf5e, {}, {0x0, 0x0, 0x0}, 0x0})
    github.com/cosmos/cosmos-sdk@v0.25.0/baseapp/baseapp.go:934 +0x1dd
github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).EndBlockSync(0xc00069a2c0, {0x24fbf5e, {}, {0x0, 0x0, 0x0}, 0x0})
    github.com/cosmos/cosmos-sdk@v0.25.0/server/concurrent/async_local_client.go:408 +0x257
github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc01e067e00, {0x24fbf5e, {}, {0x0, 0x0, 0x0}, 0x0})
    github.com/tendermint/tendermint@v0.35.9/proxy/app_conn.go:115 +0x4e
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x160a550, 0xc000edce60}, {0x161a720, 0xc000e01df0}, 0xc005d5b2c0, 0x14, {0x1625fb8, 0xc0006124c8})
    github.com/tendermint/tendermint@v0.35.9/state/execution.go:298 +0x5f0
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xa, 0x0}, {0xc00a8dbe90, 0x6}}, {0xc000134978, 0x14}, 0x24fbf5d, 0x1f1600, {{0xc01d8fbde0, ...}, ...}, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/state/execution.go:128 +0x17b
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc001383a00)
    github.com/tendermint/tendermint@v0.35.9/blockchain/v0/reactor.go:401 +0xcfb
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).SwitchToBlockchain
    github.com/tendermint/tendermint@v0.35.9/blockchain/v0/reactor.go:154 +0x2e6

[isaacyuno]

Hi there, I just tested but can't reproduce the error. Could you try the steps here and sync again to see if it goes well? https://docs.bnbchain.org/docs/beaconchain/develop/node/synctypes/#recover-from-state-sync-failure

[tmeinlschmidt]

Nope. I'm not able to get it to work. Can you please share your config.toml? To start syncing from the scratch, including full history? I always ended up with the same as above. btw - running in docker, but I assume there's no difference.

[forcodedancing]

@tmeinlschmidt Can you try this release? The issue should be fixed now. https://github.com/bnb-chain/node/releases/tag/v0.10.13

[tmeinlschmidt]

0.10.13 works if I have state_sync_height = 0. If I try to sync from the scratch using state_sync_height = -1, I'm getting this:

I[2023-04-27|07:26:46.874] Relaying block for order book                module=dex height=3715
I[2023-04-27|07:26:46.874] replayed all tx. Starting match              module=dex height=3715
I[2023-04-27|07:26:46.874] No order comes in for the block              module=dexkeeper 
E[2023-04-27|07:26:46.883] Failed to load the latest breathe block height from module=dexkeeper timeNow=2020-07-09T08:08:08.821095341Z
I[2023-04-27|07:26:46.883] get last breathe block height                module=main height=0
I[2023-04-27|07:26:46.884] reload latest snapshot                       module=statesync height=0
I[2023-04-27|07:26:46.891] ABCI Handshake App Info                      module=consensus height=3715 hash=C8313E653D6B2DF45B17FAE8D4D209E03F010D97717A60ED768AE3F4A9069710 software-version= protocol-version=0
I[2023-04-27|07:26:46.891] ABCI Replay Blocks                           module=consensus appHeight=3715 storeHeight=3716 stateHeight=3715
I[2023-04-27|07:26:46.891] Replay last block using real app             module=consensus 

and error

panic: Failed to process committed block (3716:19E761CE47854135F8345FA897E8AFF86BB3D1E28B3C8CE6477F3752C20082A8): Wrong Block.Header.LastResultsHash.  Expected D00CF911DBF4BB4FF0AFBC6C312DB1292340D3962C6729B71FC9BFB9BC86BA3F, got 2E287AB6399A81813D7904B17CF4A3011D29D2C8155CD5C44BECD67CF4D54CC6

goroutine 271 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc00b6e2820)
    github.com/tendermint/tendermint@v0.35.9/blockchain/v0/reactor.go:404 +0x15a8
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
    github.com/tendermint/tendermint@v0.35.9/blockchain/v0/reactor.go:127 +0x7f
ERROR: error during handshake: error on replay: Wrong Block.Header.LastResultsHash.  Expected D00CF911DBF4BB4FF0AFBC6C312DB1292340D3962C6729B71FC9BFB9BC86BA3F, got 2E287AB6399A81813D7904B17CF4A3011D29D2C8155CD5C44BECD67CF4D54CC6
ERROR: error during handshake: error on replay: Wrong Block.Header.LastResultsHash.  Expected D00CF911DBF4BB4FF0AFBC6C312DB1292340D3962C6729B71FC9BFB9BC86BA3F, got 2E287AB6399A81813D7904B17CF4A3011D29D2C8155CD5C44BECD67CF4D54CC6
ERROR: error during handshake: error on replay: Wrong Block.Header.LastResultsHash.  Expected D00CF911DBF4BB4FF0AFBC6C312DB1292340D3962C6729B71FC9BFB9BC86BA3F, got 2E287AB6399A81813D7904B17CF4A3011D29D2C8155CD5C44BECD67CF4D54CC6

[nmiceli-simtlix]

@tmeinlschmidt It happened to me as well. When syncing from scratch in testnet, always panics at block #3715 That error is not related to a specific version. I was able to reproduce the error in 4-5 different versions.

They need to fix it somehow.

[forcodedancing]

@tmeinlschmidt @nmiceli-simtlix Can you use state sync? Due to some historical reasons, testnet cannot sync from genesis block to the latest block via executing blocks one by one.

[tmeinlschmidt]

hm, with state_sync_height = 0 it works, any other height number causes panic: runtime error. Weird. We wanted to have testnet with some history.