search

bnb-chain / node

Mozilla Public License 2.0
80 stars 38 forks source link

bnbchaind crash after restart bnbchaind when syncing #927

Closed lcgogo closed 9 months ago

lcgogo commented 1 year ago

I stop bnbchaind which is syncing block and start it again, but met followed issure

I[2023-03-05|08:57:15.927] Starting ABCI with Tendermint                module=main
E[2023-03-05|08:57:16.017] Failed to load the latest breathe block height from module=dexkeeper timeNow=2023-03-05T08:57:16.01787795Z
I[2023-03-05|08:57:16.017] Loading order book snapshot from last breathe block module=main blockHeight=0
I[2023-03-05|08:57:16.017] No breathe block is ever saved. just created match engines for all the pairs. module=main
I[2023-03-05|08:57:16.018] Initialized Block Store for replay           module=dex fromHeight=0 toHeight=0
E[2023-03-05|08:57:16.018] Failed to load the latest breathe block height from module=dexkeeper timeNow=2023-03-05T08:57:16.018279715Z
I[2023-03-05|08:57:16.018] get last breathe block height                module=main height=0
I[2023-03-05|08:57:16.018] reload latest snapshot                       module=statesync height=0
I[2023-03-05|08:57:16.050] ABCI Handshake App Info                      module=consensus height=0 hash= software-version= protocol-version=0
I[2023-03-05|08:57:16.050] ABCI Replay Blocks                           module=consensus appHeight=0 storeHeight=300478996 stateHeight=300478996
panic: Value missing for hash a947a3dcea4c0f0b57fa2f003bf2d037479cf7fd66bb2f99a1396a355c4878bc corresponding to nodeKey n�G���L
          W�/;��7G���f�/��9j5\Hx�

goroutine 1 [running]:
github.com/tendermint/iavl.(*nodeDB).GetNode(0xc000c84960, {0xc000cce0c0, 0x20, 0x20})
    github.com/tendermint/iavl@v0.12.4/nodedb.go:68 +0x39c
github.com/tendermint/iavl.(*Node).getRightNode(0xc000000000, 0xc000cc8c80, 0x0)
    github.com/tendermint/iavl@v0.12.4/node.go:413 +0x1c5
github.com/tendermint/iavl.(*Node).get(0xc000000000, 0x0?, {0x1e40a40, 0x13, 0x13})
    github.com/tendermint/iavl@v0.12.4/node.go:190 +0x119
github.com/tendermint/iavl.(*ImmutableTree).Get(0x0?, {0x1e40a40, 0x13, 0x13})
    github.com/tendermint/iavl@v0.12.4/immutable_tree.go:155 +0x4b
github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc000ce1640?, {0x1e40a40?, 0x1e40a40?, 0x13?})
    github.com/cosmos/cosmos-sdk@v0.25.0/store/iavlstore.go:153 +0x25
github.com/cosmos/cosmos-sdk/store.(*cacheKVStore).Get(0xc000c5e840, {0x1e40a40, 0x13, 0x13})
    github.com/cosmos/cosmos-sdk@v0.25.0/store/cachekvstore.go:50 +0x136
github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetNextAccountNumber({{_, _}, _, _}, {{0x169e9e0, 0xc000044038}, {0x16a81f8, 0xc00053dd00}, {{0x0, 0x0, ...}, ...}, ...})
    github.com/cosmos/cosmos-sdk@v0.25.0/x/auth/mapper.go:162 +0xfc
github.com/bnb-chain/node/app.(*BinanceChain).initChainerFn.func1({{0x169e9e0, 0xc000044038}, {0x16a81f8, 0xc00053dd00}, {{0x0, 0x0, {}, {0x0, 0x0, 0x0}, ...}, ...}, ...}, ...)
    github.com/bnb-chain/node/app/app.go:711 +0x2fb
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc00027ba40, {{0xda23816, 0xed44a093e, 0x0}, {0xc000ab41c8, 0x14}, 0xc00053dc40, {0x1ecd1e8, 0x0, 0x0}, ...})
    github.com/cosmos/cosmos-sdk@v0.25.0/baseapp/baseapp.go:352 +0x178
github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).InitChainSync(0xc0000fe580, {{0xda23816, 0xed44a093e, 0x0}, {0xc000ab41c8, 0x14}, 0xc00053dc40, {0x1ecd1e8, 0x0, 0x0}, ...})
    github.com/cosmos/cosmos-sdk@v0.25.0/server/concurrent/async_local_client.go:385 +0xfe
github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0x477d1b?, {{0xda23816, 0xed44a093e, 0x0}, {0xc000ab41c8, 0x14}, 0xc00053dc40, {0x1ecd1e8, 0x0, 0x0}, ...})
    github.com/tendermint/tendermint@v0.35.9/proxy/app_conn.go:103 +0x55
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xa, 0x0}, {0xc000b8c9e8, 0x6}}, {0xc000ab4210, 0x14}, 0x11e8f214, 0x2f030a40, {{0xc000cce560, ...}, ...}, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:313 +0xe58
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000c150c8, {0x16a9f48, 0xc000b730a0})
    github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:271 +0x4e8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xa, 0x0}, {0xc000b8c9e8, 0x6}}, {0xc000ab4210, 0x14}, 0x11e8f214, 0x2f030a40, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/node/node.go:313 +0x1a7
github.com/tendermint/tendermint/node.NewNode(0xc0009a6dc0, {0x169d620, 0xc0001ff180}, 0xc00053e410, {0x1693fa0, 0xc00053a2d0}, 0x0?, 0x55?, 0xc00053e610, {0x169f3b8, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/node/node.go:654 +0x3d1
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc0004e88b8, 0x153af80)
    github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:133 +0x6aa
github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc000b24c80?, {0xc000af5140?, 0x0?, 0x4?})
    github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:44 +0x7f
github.com/spf13/cobra.(*Command).execute(0xc000b24c80, {0xc000af5100, 0x4, 0x4})
    github.com/spf13/cobra@v0.0.5/command.go:826 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000b0b180)
    github.com/spf13/cobra@v0.0.5/command.go:914 +0x2ef
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra@v0.0.5/command.go:864
github.com/tendermint/tendermint/libs/cli.Executor.Execute({0xc000b0b180?, 0x153b230?})
    github.com/tendermint/tendermint@v0.35.9/libs/cli/setup.go:89 +0x36
main.main()
    github.com/bnb-chain/node/cmd/bnbchaind/main.go:54 +0x29d
kris-ss commented 1 year ago

Hello, please confirm the following. 1) Is this on the mainnent? 2) How did you stop the node? 3) What is the bnbchaind version that you are currently running

Thank you

lcgogo commented 1 year ago

Hello, please confirm the following.

1. Is this on the mainnent?

2. How did you stop the node?

3. What is the bnbchaind version that you are currently running

Thank you

I test it several times. The issue occurs at the very beginning of syncing (20-60 secs) when the bnbchaind startup from a blank data folder. However, after sync for a longer time. The restart is ok.

  1. yes, it's mainnet
  2. I use kill $pid
  3. 0.10.7
forcodedancing commented 1 year ago

@lcgogo Which sync mode you are using? Can you share the config file you tested? I had tried on my local machine, cannot reproduce it now.

lcgogo commented 1 year ago

the same config.toml in

wget   $(curl -s https://api.github.com/repos/bnb-chain/node/releases/latest |grep browser_ |grep mainnet_config |cut -d\" -f4)
unzip mainnet_config.zip
DATA_DIR=/data/bnb
bnbchaind --home ${DATA_DIR} init --home-client ${DATA_DIR} --moniker bnbnode
# cp mainnet config to /data/bnb/config folder
bnbchaind start --home ${DATA_DIR}
forcodedancing commented 1 year ago

Still cannot reproduce the error following your steps.

image

1) Can you make sure your config files are correctly replaced with all the downloaded config files ?

image

2) You can also try following steps

wget   $(curl -s https://api.github.com/repos/bnb-chain/node/releases/latest |grep browser_ |grep mainnet_config |cut -d\" -f4)
unzip mainnet_config.zip
mkdir HOME_DIR
mkdir HOME_DIR/data
mkdir HOME_DIR/config
cp * mainnet_dowload_dir/asset/config HOME_DIR/config
bnbchaind start --home HOME_DIR
lcgogo commented 1 year ago

Download the latest bin file and config zip and running in another host and clean data folder

root@ip-10-2-1-26:/data/bnb# wget   $(curl -s https://api.github.com/repos/bnb-chain/node/releases/latest |grep browser_ |grep mainnet_config |cut -d\" -f4)
--2023-03-06 14:38:49--  https://github.com/bnb-chain/node/releases/download/v0.10.7/mainnet_config.zip
Resolving github.com (github.com)... 192.30.255.113
Connecting to github.com (github.com)|192.30.255.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/133295130/55e2c8ca-d071-47a5-8548-95f54d8439f1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230306%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230306T143850Z&X-Amz-Expires=300&X-Amz-Signature=4e0fe7933d47de36459f0b77bf64ea3b830f418a69acd14a1c5437f886d15d96&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=133295130&response-content-disposition=attachment%3B%20filename%3Dmainnet_config.zip&response-content-type=application%2Foctet-stream [following]
--2023-03-06 14:38:50--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/133295130/55e2c8ca-d071-47a5-8548-95f54d8439f1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230306%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230306T143850Z&X-Amz-Expires=300&X-Amz-Signature=4e0fe7933d47de36459f0b77bf64ea3b830f418a69acd14a1c5437f886d15d96&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=133295130&response-content-disposition=attachment%3B%20filename%3Dmainnet_config.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13200 (13K) [application/octet-stream]
Saving to: ‘mainnet_config.zip’

mainnet_config.zip            100%[==============================================>]  12.89K  --.-KB/s    in 0.001s

2023-03-06 14:38:50 (17.8 MB/s) - ‘mainnet_config.zip’ saved [13200/13200]

root@ip-10-2-1-26:/data/bnb# unzip mainnet_config.zip
Archive:  mainnet_config.zip
   creating: asset/mainnet/
  inflating: asset/mainnet/config.toml
  inflating: asset/mainnet/genesis.json
  inflating: asset/mainnet/app.toml
root@ip-10-2-1-26:/data/bnb# HOME_DIR=/data/bnb
root@ip-10-2-1-26:/data/bnb# mkdir $HOME_DIR
mkdir: cannot create directory ‘/data/bnb’: File exists
root@ip-10-2-1-26:/data/bnb# mkdir $HOME_DIR/data
root@ip-10-2-1-26:/data/bnb# mkdir $HOME_DIR/config
root@ip-10-2-1-26:/data/bnb# cp * mainnet_dowload_dir/asset/config $HOME_DIR/config
cp: -r not specified; omitting directory 'asset'
cp: -r not specified; omitting directory 'config'
cp: -r not specified; omitting directory 'data'
cp: -r not specified; omitting directory 'linux'
cp: cannot stat 'mainnet_dowload_dir/asset/config': No such file or directory
root@ip-10-2-1-26:/data/bnb# ls
asset  bnbchaind  config  data  linux  linux_binary.zip  mainnet_config.zip
root@ip-10-2-1-26:/data/bnb# ls config
bnbchaind  linux_binary.zip  mainnet_config.zip
root@ip-10-2-1-26:/data/bnb# rm -rf config
root@ip-10-2-1-26:/data/bnb# mkdir config
root@ip-10-2-1-26:/data/bnb# ls
asset  bnbchaind  config  data  linux  linux_binary.zip  mainnet_config.zip
root@ip-10-2-1-26:/data/bnb# cp asset/mainnet/
app.toml      config.toml   genesis.json
root@ip-10-2-1-26:/data/bnb# cp asset/mainnet/* config/
root@ip-10-2-1-26:/data/bnb# ls config
app.toml  config.toml  genesis.json
root@ip-10-2-1-26:/data/bnb# ./bnbchaind start --home $HOME_DIR
I[2023-03-06|14:40:06.880] Starting ABCI with Tendermint                module=main
E[2023-03-06|14:40:06.917] Failed to load the latest breathe block height from module=dexkeeper timeNow=2023-03-06T14:40:06.917358424Z
I[2023-03-06|14:40:06.917] Loading order book snapshot from last breathe block module=main blockHeight=0
I[2023-03-06|14:40:06.917] No breathe block is ever saved. just created match engines for all the pairs. module=main
I[2023-03-06|14:40:06.917] Initialized Block Store for replay           module=dex fromHeight=0 toHeight=0
E[2023-03-06|14:40:06.917] Failed to load the latest breathe block height from module=dexkeeper timeNow=2023-03-06T14:40:06.917763429Z
I[2023-03-06|14:40:06.917] get last breathe block height                module=main height=0
I[2023-03-06|14:40:06.917] reload latest snapshot                       module=statesync height=0
I[2023-03-06|14:40:06.960] ABCI Handshake App Info                      module=consensus height=0 hash= software-version= protocol-version=0

when the bnbchaind start to download data in folder, (will take about 30s to several mins according to peer connection)

I open another terminal to find the data folder is increasing then kill the process

root@ip-10-2-1-26:/data/bnb# du -d 1 -h
64K ./asset
72K ./config
160M    ./linux
7.7M    ./data
262M    .
root@ip-10-2-1-26:/data/bnb# du -d 1 -h
64K ./asset
72K ./config
160M    ./linux
418M    ./data       <======= the data is increasing now
671M    .
root@ip-10-2-1-26:/data/bnb# du -sh
792M    .
root@ip-10-2-1-26:/data/bnb# ps -ef | grep bnb
root     1827334 1823739 80 14:43 pts/1    00:00:43 ./bnbchaind start --home /data/bnb
root     1827417 1827209  0 14:44 pts/3    00:00:00 grep --color=auto bnb
root@ip-10-2-1-26:/data/bnb# kill 1827334
root@ip-10-2-1-26:/data/bnb#

At the same time bnbchaind is stopped in old terminal, then I start it manually and it crashed.

mPending=2083
I[2023-03-06|14:44:35.154] start write recovery chunk                   module=statesync isComplete=false hash=c05c83b035ea53f941136dd5cd171a0248ab31b8baadaceb1ef0299b4349245d startIdx=33860415 numOfNodes=32671 chunkCompletion=0
I[2023-03-06|14:44:35.365] finished write recovery chunk                module=statesync isComplete=false hash=c05c83b035ea53f941136dd5cd171a0248ab31b8baadaceb1ef0299b4349245d startIdx=33860415 numOfNodes=32671 chunkCompletion=0
I[2023-03-06|14:44:35.370] try to request state                         module=statesync peer=5c96d4f0174dda7757bcda11a57ecc91851a90f0 req="300681914, 370300d5924f2032ac0bece17eed46980e5609c3796f940a9b0e075835312b94"
I[2023-03-06|14:44:35.370] try to request state                         module=statesync peer=0fd066de431d13e5613aba53b3ed25c5a7a0b67c req="300681914, ec3f91d8fa05e0d161f89e6780e79fcaff81f3f1bd9de544071b010c3ebb1e69"
I[2023-03-06|14:44:35.373] processing chunk                             module=statesync peer=ca1245471ff1f2224e1076dfd94d7d2d66077cf5 resp="[bcChunkResponseMessage c79a45e9d28bcf0282e98900e0b2fadb6ec2428124cf7aa354afa121ee2ae1cd]" numPending=2082
I[2023-03-06|14:44:35.381] start write recovery chunk                   module=statesync isComplete=false hash=c79a45e9d28bcf0282e98900e0b2fadb6ec2428124cf7aa354afa121ee2ae1cd startIdx=13456922 numOfNodes=38642 chunkCompletion=0      <============ stopped immediately
root@ip-10-2-1-26:/data/bnb#
root@ip-10-2-1-26:/data/bnb# ./bnbchaind start --home $HOME_DIR       <======== manual start again
I[2023-03-06|14:44:42.491] Starting ABCI with Tendermint                module=main
E[2023-03-06|14:44:44.589] Failed to load the latest breathe block height from module=dexkeeper timeNow=2023-03-06T14:44:44.589752665Z
I[2023-03-06|14:44:44.589] Loading order book snapshot from last breathe block module=main blockHeight=0
I[2023-03-06|14:44:44.589] No breathe block is ever saved. just created match engines for all the pairs. module=main
I[2023-03-06|14:44:44.589] Initialized Block Store for replay           module=dex fromHeight=0 toHeight=0
E[2023-03-06|14:44:44.590] Failed to load the latest breathe block height from module=dexkeeper timeNow=2023-03-06T14:44:44.590186Z
I[2023-03-06|14:44:44.590] get last breathe block height                module=main height=0
I[2023-03-06|14:44:44.590] reload latest snapshot                       module=statesync height=0
I[2023-03-06|14:44:44.634] ABCI Handshake App Info                      module=consensus height=0 hash= software-version= protocol-version=0
I[2023-03-06|14:44:44.634] ABCI Replay Blocks                           module=consensus appHeight=0 storeHeight=0 stateHeight=300681914
panic: Value missing for hash 979747c1451e342c98a77786d8088fc73aa9e0f4426db4a6daa4d217a4689b37 corresponding to nodeKey n��G�E4,��w���:���Bm��ڤ��h�7

goroutine 1 [running]:
github.com/tendermint/iavl.(*nodeDB).GetNode(0xc00018ae10, {0xc0001c8140, 0x0, 0x0})
    github.com/tendermint/iavl@v0.12.4/nodedb.go:68 +0x39c
github.com/tendermint/iavl.(*Node).getRightNode(0xc0001d80c0, 0xc000bbc240, 0x0)
    github.com/tendermint/iavl@v0.12.4/node.go:413 +0x1c5
github.com/tendermint/iavl.(*Node).get(0xc0001d80c0, 0x0, {0x1da81f0, 0x13, 0x13})
    github.com/tendermint/iavl@v0.12.4/node.go:190 +0x119
github.com/tendermint/iavl.(*ImmutableTree).Get(0x0, {0x1da81f0, 0x13, 0x13})
    github.com/tendermint/iavl@v0.12.4/immutable_tree.go:155 +0x4b
github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0x15d4500, {0x1da81f0, 0x1da81f0, 0x13})
    github.com/cosmos/cosmos-sdk@v0.25.0/store/iavlstore.go:153 +0x25
github.com/cosmos/cosmos-sdk/store.(*cacheKVStore).Get(0xc000be8600, {0x1da81f0, 0x15ed4f8, 0x13})
    github.com/cosmos/cosmos-sdk@v0.25.0/store/cachekvstore.go:50 +0x139
github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetNextAccountNumber({{_, _}, _, _}, {{0x15d3b28, 0xc0001ac020}, {0x15ed4f8, 0xc000d9e500}, {{0x0, 0x0, ...}, ...}, ...})
forcodedancing commented 1 year ago

@lcgogo I think the node is in state sync mode and downloading data from other nodes. It could be an issue and will be fixed later. For setting up your nodes, you can wait the node to catch up the latest height without restart.

lcgogo commented 1 year ago

Thanks, will keep an eye on the issue.

nmiceli-simtlix commented 1 year ago

Hello. This needs further investigation. I am experiencing the same issue. This is for testnet, tried both v0.10.8 and v0.10.9. My nodes are at chainhead and have been running for a while with no issues. When I restart my nodes, they crash as soon as they start.

I[2023-03-28|23:12:29.660] Starting ABCI with Tendermint                module=main 
I[2023-03-28|23:12:30.503] Loading order book snapshot from last breathe block module=main blockHeight=37727120
I[2023-03-28|23:12:30.566] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:175-0B3M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.566] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:3LAU-544 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.566] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ABC-001 QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.567] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ABC-59C QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.567] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ABCD-AA5M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.567] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ALELE-979M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.568] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AT10-AF2 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.568] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AT25-C88 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-03-28|23:12:30.568] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AT9-4F1 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.569] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AT9-693 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.569] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AVA-524 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.569] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BBEX-6BAM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.570] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BDX-129M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.570] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BDX-129M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.570] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BEP-5DDM QuoteAssetSymbol:BNB ListPrice:1000000.00000000 TickSize:10.00000000 LotSize:0.00000001}"
I[2023-03-28|23:12:30.571] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BHT-17FM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.571] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNBM-742M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.572] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNBM-742M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.572] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:BTCB-101 ListPrice:0.02000000 TickSize:0.00000010 LotSize:0.00100000}"
I[2023-03-28|23:12:30.573] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:BUSD-BAF ListPrice:18.00000000 TickSize:0.00000100 LotSize:0.00100000}"
I[2023-03-28|23:12:30.573] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:ETH-D5B ListPrice:0.07000000 TickSize:0.00000100 LotSize:0.00100000}"
I[2023-03-28|23:12:30.574] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:TUSD-5D7 ListPrice:17.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.574] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:USDS-5BD ListPrice:17.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.574] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BOKM-8D7M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.575] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BTCB-101 QuoteAssetSymbol:BUSD-BAF ListPrice:9000.00000000 TickSize:0.01000000 LotSize:0.00001000}"
I[2023-03-28|23:12:30.575] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BTCKFE-353M QuoteAssetSymbol:BNB ListPrice:0.00160000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.576] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BTCKFE-353M QuoteAssetSymbol:BUSD-BAF ListPrice:0.10000000 TickSize:0.00000100 LotSize:1.00000000}"
I[2023-03-28|23:12:30.576] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BUSD-227 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.576] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:CBIX-7A1 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.577] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:CRAD-D5DM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.578] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:CUB-EE5 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.01000000 LotSize:0.00000100}"
I[2023-03-28|23:12:30.578] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:DEEZ-BE4M QuoteAssetSymbol:BNB ListPrice:0.00130000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.579] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:DWU-B6BM QuoteAssetSymbol:BUSD-BAF ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.579] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ELON-72BM QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.580] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ELON-72BM QuoteAssetSymbol:BUSD-BAF ListPrice:100000.00000000 TickSize:1.00000000 LotSize:0.10000000}"
I[2023-03-28|23:12:30.580] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ETH-D5B QuoteAssetSymbol:BUSD-BAF ListPrice:200.00000000 TickSize:0.00100000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.581] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:EWE-31DM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.581] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:FABT-A4DM QuoteAssetSymbol:BNB ListPrice:0.00200000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.581] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:FEHU-639M QuoteAssetSymbol:BNB ListPrice:0.00050000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-03-28|23:12:30.582] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:GAEA-E4AM QuoteAssetSymbol:BNB ListPrice:0.66000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.582] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:GBC-F29M QuoteAssetSymbol:BUSD-BAF ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.583] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:GLT-2E8M QuoteAssetSymbol:BNB ListPrice:0.00150000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.584] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:GLT-2E8M QuoteAssetSymbol:BUSD-BAF ListPrice:0.75000000 TickSize:0.00000100 LotSize:1.00000000}"
I[2023-03-28|23:12:30.584] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:HEM-452 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.585] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:HEM-452 QuoteAssetSymbol:BUSD-227 ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.01000000}"
I[2023-03-28|23:12:30.585] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:HMT-7FEM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.585] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:HMT-7FEM QuoteAssetSymbol:BUSD-BAF ListPrice:40.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.586] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ICC-C6BM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.586] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:IMC-CFCM QuoteAssetSymbol:BUSD-BAF ListPrice:2.50000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.587] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:INNBC-1A1M QuoteAssetSymbol:BNB ListPrice:0.03000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.587] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:INNBC-1A1M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.10000000}"
I[2023-03-28|23:12:30.588] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:IRMO-C9D QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00100000 LotSize:0.00001000}"
I[2023-03-28|23:12:30.588] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ITTT-730M QuoteAssetSymbol:BNB ListPrice:0.00010000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.589] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:KAI-04E QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.589] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:LPHP-88F QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.589] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MAE-082M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.590] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MATIC-A7D QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.590] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MILAD-ECEM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.590] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MILAD-ECEM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.591] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MIMI-02FM QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.591] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MINIA-7A2M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.591] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MINIA-7A2M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.592] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MJBEP8-409M QuoteAssetSymbol:BNB ListPrice:0.50000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.593] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-094 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.593] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-1D0 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.593] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-2D6 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.593] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-477 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.594] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-4FF QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.594] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-68A QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.595] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-D1B QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.595] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-E62 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.595] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-0A3M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.596] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-132M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.596] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-1F6M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.596] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-229M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.596] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-25CM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.597] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-327M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.597] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-3B4M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.597] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-3F7M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.597] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-41CM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.598] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-5C9M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.598] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-C59M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.598] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-D1FM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.599] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-DA2M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.599] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-DC1M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.599] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-E1DM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.599] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-E53M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.600] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSFA-3D2M QuoteAssetSymbol:BNB ListPrice:0.15000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.600] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MST-EB4M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.600] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:NWCT-61FM QuoteAssetSymbol:BNB ListPrice:0.30000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.601] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:PRV1-E9BM QuoteAssetSymbol:BNB ListPrice:0.00000500 TickSize:0.00000001 LotSize:1000.00000000}"
I[2023-03-28|23:12:30.601] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:QWE-684M QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.601] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:RCO-30EM QuoteAssetSymbol:BNB ListPrice:0.04500000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.602] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:RCO-30EM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.01000000}"
I[2023-03-28|23:12:30.602] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:RUBY-366M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.602] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:RUBY-366M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.602] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SAT-30B QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.603] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SDDS-A84M QuoteAssetSymbol:BUSD-BAF ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.603] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SEFA-A0AM QuoteAssetSymbol:BNB ListPrice:0.00000001 TickSize:0.00000001 LotSize:100000.00000000}"
I[2023-03-28|23:12:30.603] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SLV-C4EM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.603] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SLV-C4EM QuoteAssetSymbol:BUSD-BAF ListPrice:26.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.604] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:START-561M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.604] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:START-BA0M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.604] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:STEEP-423M QuoteAssetSymbol:BNB ListPrice:0.00010000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-03-28|23:12:30.604] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:STL-7C6M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.605] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:STLM-DD7M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.605] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:TDZR-1DCM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.605] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:TES-BCCM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.606] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:TTT-29DM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.606] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:TTT-3FAM QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.606] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:U2U-DF5M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.607] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:VNB-708M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.607] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:VTBC-CB0M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.607] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:WOOPT-FADM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.608] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:WTR-ED7M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.608] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:WWWC-12CM QuoteAssetSymbol:BUSD-BAF ListPrice:0.20000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.608] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:XIN-F25 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.609] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:XRP-FB8 QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.609] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:YGEM-B61M QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.609] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:YGEM-B61M QuoteAssetSymbol:BUSD-BAF ListPrice:0.00100000 TickSize:0.00000001 LotSize:0.00010000}"
I[2023-03-28|23:12:30.610] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:YNT2-C51M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.610] Recovered active orders. Snapshot is fully loaded module=main 
I[2023-03-28|23:12:35.573] get last breathe block height                module=main height=37727120
panic: state.AppHash does not match AppHash after replay. Got
DB76256058CAA8D1B5D5AEBAA2CEB8617A3DDF513C46A033A51A75E12A65C138, expected 87B2757FEC5EB1C8F12954C284842970B1E6AFB7FF63D2104B956B44D772708E.

State: {{{10 0} 0.32.3} Binance-Chain-Ganges 37763484 2035763 A115F98366E8AB8A17E99A65F7E5FC35372867258BE4E3FDCDD728C446A71407:1:6617813F2D4C 2023-03-28 23:12:07.195105944 +0000 UTC ValidatorSet{
  Proposer: Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-5007747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-1437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:6492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:242252066116}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-2257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-2507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-3007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:492252066116}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-3257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:7242252066116}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-5007747933884}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:-1985020661157}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:4992252066116}
} ValidatorSet{
  Proposer: Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:-3985020661157}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-2437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:5492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:-757747933884}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-3257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-3507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-4007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:-1507747933884}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-4257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:6242252066116}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:7992252066116}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:-3985020661157}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:3992252066116}
} ValidatorSet{
  Proposer: Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-5257747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-3437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:4492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:-1757747933884}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-4257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-4507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-5007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:-3507747933884}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-5257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:5242252066116}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:6992252066116}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:8014979338843}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:2992252066116}
} 35809060 {{1048576 -1} {100000} {[ed25519]}} 1 [] [135 178 117 127 236 94 177 200 241 41 84 194 132 132 41 112 177 230 175 183 255 99 210 16 75 149 107 68 215 114 112 142]}

Did you reset Tendermint without resetting your application's data?

goroutine 1 [running]:
github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromState({_, _, _}, {{{0xa, 0x0}, {0xc0205f3158, 0x6}}, {0xc0205de468, 0x14}, 0x240399c, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:489 +0x17a
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xa, 0x0}, {0xc0205f3158, 0x6}}, {0xc0205de468, 0x14}, 0x240399c, 0x1f1033, {{0xc020702500, ...}, ...}, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:370 +0x3b7
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc021198fc8, {0x1618f10, 0xc01df76850})
    github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:271 +0x4c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xa, 0x0}, {0xc0205f3158, 0x6}}, {0xc0205de468, 0x14}, 0x240399c, 0x1f1033, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/node/node.go:313 +0x1a7
github.com/tendermint/tendermint/node.NewNode(0xc00016c840, {0x15e9b70, 0xc02073c820}, 0xc02070b9d0, {0x15d7160, 0xc0000a6fa0}, 0x0, 0xc0005aaaa0, 0xc02070bba0, {0x15f9e50, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/node/node.go:654 +0x3d1
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc00011e2b8, 0x1470078)
    github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:133 +0x66a
github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc000b25b80, {0xc000594ce0, 0x0, 0x2})
    github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:44 +0x7f
github.com/spf13/cobra.(*Command).execute(0xc000b25b80, {0xc000594ca0, 0x2, 0x2})
    github.com/spf13/cobra@v0.0.5/command.go:826 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000b24280)
    github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra@v0.0.5/command.go:864
github.com/tendermint/tendermint/libs/cli.Executor.Execute({0xc000b24280, 0x1470318})
    github.com/tendermint/tendermint@v0.35.9/libs/cli/setup.go:89 +0x36
main.main()
    github.com/bnb-chain/node/cmd/bnbchaind/main.go:54 +0x28e
nmiceli-simtlix commented 1 year ago

Is anyone looking into this issue? I can confirm this only happens when running a node on testnet. I've tried changing a few options in config.toml, but nothing fixes the issue.

Here's a more detailed log:

I[2023-04-05|21:05:16.653] Relaying block for order book                module=dex height=38061251
I[2023-04-05|21:05:16.653] replayed all tx. Starting match              module=dex height=38061251
I[2023-04-05|21:05:16.653] No order comes in for the block              module=dexkeeper 
I[2023-04-05|21:05:16.653] Relaying block for order book                module=dex height=38061252
I[2023-04-05|21:05:16.653] replayed all tx. Starting match              module=dex height=38061252
I[2023-04-05|21:05:16.653] No order comes in for the block              module=dexkeeper 
I[2023-04-05|21:05:16.904] Located day to load breathe block height     module=dexkeeper epochDay="\ufffdK\u0000\u0000\u0000\u0000\u0000\u0000"
I[2023-04-05|21:05:16.904] Loaded breathe block height                  module=dexkeeper height=38028184
I[2023-04-05|21:05:16.904] get last breathe block height                module=main height=38028184
I[2023-04-05|21:05:16.905] reload latest snapshot                       module=statesync height=38028184
I[2023-04-05|21:05:16.936] ABCI Handshake App Info                      module=consensus height=38061252 hash=A651516385C1210304753EACD653D8137108D58CD5414CA35088611FE1738BED software-version= protocol-version=0
I[2023-04-05|21:05:16.937] ABCI Replay Blocks                           module=consensus appHeight=38061252 storeHeight=38061252 stateHeight=38061252
panic: state.AppHash does not match AppHash after replay. Got
A651516385C1210304753EACD653D8137108D58CD5414CA35088611FE1738BED, expected 212E408D77C3F0DB1BEED74FFADB002CE250354F6E6F38F49956DDB32DC0E694.

State: {{{10 0} 0.32.3} Binance-Chain-Ganges 38061252 2036586 AD8832D49C55AF0572BFCC1D0FEF1B3F180155B22E3BCB8A07A47C62E051E04E:1:26AF04C17B4B 2023-04-05 21:04:54.55707933 +0000 UTC ValidatorSet{
  Proposer: Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:-5507747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:562500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:-5507747933884}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:2242252066116}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-1007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:4492252066116}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-1257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:-4757747933884}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-3007747933884}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:2014979338843}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:6992252066116}
} ValidatorSet{
  Proposer: Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:-5757747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:7492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:1242252066116}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-1257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-1507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-2007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:2492252066116}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-2257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:-5757747933884}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-4007747933884}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:14979338843}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:5992252066116}
} ValidatorSet{
  Proposer: Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-5007747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-1437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:6492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:242252066116}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-2257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-2507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-3007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:492252066116}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-3257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:7242252066116}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-5007747933884}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:-1985020661157}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:4992252066116}
} 35809060 {{1048576 -1} {100000} {[ed25519]}} 1 [] [33 46 64 141 119 195 240 219 27 238 215 79 250 219 0 44 226 80 53 79 110 111 56 244 153 86 221 179 45 192 230 148]}

Did you reset Tendermint without resetting your application's data?

goroutine 1 [running]:
github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromState({_, _, _}, {{{0xa, 0x0}, {0xc016e4ec58, 0x6}}, {0xc016e5e210, 0x14}, 0x244c4c4, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:489 +0x17a
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xa, 0x0}, {0xc016e4ec58, 0x6}}, {0xc016e5e210, 0x14}, 0x244c4c4, 0x1f136a, {{0xc016e5a580, ...}, ...}, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:370 +0x3b7
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00c5eafc8, {0x1618f10, 0xc0001ac230})
    github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:271 +0x4c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xa, 0x0}, {0xc016e4ec58, 0x6}}, {0xc016e5e210, 0x14}, 0x244c4c4, 0x1f136a, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/node/node.go:313 +0x1a7
github.com/tendermint/tendermint/node.NewNode(0xc000152c60, {0x15e9b70, 0xc016dfe780}, 0xc016dadfd0, {0x15d7160, 0xc0009aa6e0}, 0x0, 0x146d660, 0xc016e161a0, {0x15f9e50, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/node/node.go:654 +0x3d1
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc0005962e8, 0x1470078)
    github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:133 +0x66a
github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc000186280, {0xc00026f300, 0x0, 0x2})
    github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:44 +0x7f
github.com/spf13/cobra.(*Command).execute(0xc000186280, {0xc00026f2c0, 0x2, 0x2})
    github.com/spf13/cobra@v0.0.5/command.go:826 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001fd900)
    github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
    github.com/spf13/cobra@v0.0.5/command.go:864
github.com/tendermint/tendermint/libs/cli.Executor.Execute({0xc0001fd900, 0x1470318})
    github.com/tendermint/tendermint@v0.35.9/libs/cli/setup.go:89 +0x36
main.main()
    github.com/bnb-chain/node/cmd/bnbchaind/main.go:54 +0x28e
timurgum commented 1 year ago

caught exactly the same error when updating to v0.10.11, how to solve this problem ?

forcodedancing commented 1 year ago

Hello. This needs further investigation. I am experiencing the same issue. This is for testnet, tried both v0.10.8 and v0.10.9. My nodes are at chainhead and have been running for a while with no issues. When I restart my nodes, they crash as soon as they start.

I[2023-03-28|23:12:29.660] Starting ABCI with Tendermint                module=main 
I[2023-03-28|23:12:30.503] Loading order book snapshot from last breathe block module=main blockHeight=37727120
I[2023-03-28|23:12:30.566] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:175-0B3M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.566] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:3LAU-544 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.566] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ABC-001 QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.567] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ABC-59C QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.567] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ABCD-AA5M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.567] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ALELE-979M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.568] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AT10-AF2 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.568] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AT25-C88 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-03-28|23:12:30.568] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AT9-4F1 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.569] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AT9-693 QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.569] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:AVA-524 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.569] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BBEX-6BAM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.570] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BDX-129M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.570] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BDX-129M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.570] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BEP-5DDM QuoteAssetSymbol:BNB ListPrice:1000000.00000000 TickSize:10.00000000 LotSize:0.00000001}"
I[2023-03-28|23:12:30.571] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BHT-17FM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.571] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNBM-742M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.572] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNBM-742M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.572] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:BTCB-101 ListPrice:0.02000000 TickSize:0.00000010 LotSize:0.00100000}"
I[2023-03-28|23:12:30.573] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:BUSD-BAF ListPrice:18.00000000 TickSize:0.00000100 LotSize:0.00100000}"
I[2023-03-28|23:12:30.573] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:ETH-D5B ListPrice:0.07000000 TickSize:0.00000100 LotSize:0.00100000}"
I[2023-03-28|23:12:30.574] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:TUSD-5D7 ListPrice:17.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.574] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BNB QuoteAssetSymbol:USDS-5BD ListPrice:17.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.574] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BOKM-8D7M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.575] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BTCB-101 QuoteAssetSymbol:BUSD-BAF ListPrice:9000.00000000 TickSize:0.01000000 LotSize:0.00001000}"
I[2023-03-28|23:12:30.575] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BTCKFE-353M QuoteAssetSymbol:BNB ListPrice:0.00160000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.576] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BTCKFE-353M QuoteAssetSymbol:BUSD-BAF ListPrice:0.10000000 TickSize:0.00000100 LotSize:1.00000000}"
I[2023-03-28|23:12:30.576] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:BUSD-227 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.576] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:CBIX-7A1 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.577] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:CRAD-D5DM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.578] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:CUB-EE5 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.01000000 LotSize:0.00000100}"
I[2023-03-28|23:12:30.578] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:DEEZ-BE4M QuoteAssetSymbol:BNB ListPrice:0.00130000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.579] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:DWU-B6BM QuoteAssetSymbol:BUSD-BAF ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.579] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ELON-72BM QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.580] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ELON-72BM QuoteAssetSymbol:BUSD-BAF ListPrice:100000.00000000 TickSize:1.00000000 LotSize:0.10000000}"
I[2023-03-28|23:12:30.580] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ETH-D5B QuoteAssetSymbol:BUSD-BAF ListPrice:200.00000000 TickSize:0.00100000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.581] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:EWE-31DM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.581] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:FABT-A4DM QuoteAssetSymbol:BNB ListPrice:0.00200000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.581] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:FEHU-639M QuoteAssetSymbol:BNB ListPrice:0.00050000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-03-28|23:12:30.582] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:GAEA-E4AM QuoteAssetSymbol:BNB ListPrice:0.66000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.582] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:GBC-F29M QuoteAssetSymbol:BUSD-BAF ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.583] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:GLT-2E8M QuoteAssetSymbol:BNB ListPrice:0.00150000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.584] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:GLT-2E8M QuoteAssetSymbol:BUSD-BAF ListPrice:0.75000000 TickSize:0.00000100 LotSize:1.00000000}"
I[2023-03-28|23:12:30.584] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:HEM-452 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.585] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:HEM-452 QuoteAssetSymbol:BUSD-227 ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.01000000}"
I[2023-03-28|23:12:30.585] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:HMT-7FEM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.585] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:HMT-7FEM QuoteAssetSymbol:BUSD-BAF ListPrice:40.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.586] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ICC-C6BM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.586] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:IMC-CFCM QuoteAssetSymbol:BUSD-BAF ListPrice:2.50000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.587] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:INNBC-1A1M QuoteAssetSymbol:BNB ListPrice:0.03000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.587] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:INNBC-1A1M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.10000000}"
I[2023-03-28|23:12:30.588] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:IRMO-C9D QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00100000 LotSize:0.00001000}"
I[2023-03-28|23:12:30.588] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:ITTT-730M QuoteAssetSymbol:BNB ListPrice:0.00010000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.589] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:KAI-04E QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.589] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:LPHP-88F QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.589] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MAE-082M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.590] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MATIC-A7D QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.590] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MILAD-ECEM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.590] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MILAD-ECEM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.591] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MIMI-02FM QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.591] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MINIA-7A2M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.591] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MINIA-7A2M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.592] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MJBEP8-409M QuoteAssetSymbol:BNB ListPrice:0.50000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.593] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-094 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.593] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-1D0 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.593] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-2D6 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.593] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-477 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.594] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-4FF QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.594] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-68A QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.595] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-D1B QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.595] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MOCKT-E62 QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.595] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-0A3M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.596] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-132M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.596] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-1F6M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.596] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-229M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.596] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-25CM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.597] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-327M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.597] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-3B4M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.597] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-3F7M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.597] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-41CM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.598] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-5C9M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.598] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-C59M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.598] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-D1FM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.599] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-DA2M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.599] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-DC1M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.599] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-E1DM QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.599] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSDK-E53M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.600] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MSFA-3D2M QuoteAssetSymbol:BNB ListPrice:0.15000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.600] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:MST-EB4M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.600] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:NWCT-61FM QuoteAssetSymbol:BNB ListPrice:0.30000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.601] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:PRV1-E9BM QuoteAssetSymbol:BNB ListPrice:0.00000500 TickSize:0.00000001 LotSize:1000.00000000}"
I[2023-03-28|23:12:30.601] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:QWE-684M QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.601] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:RCO-30EM QuoteAssetSymbol:BNB ListPrice:0.04500000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.602] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:RCO-30EM QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.01000000}"
I[2023-03-28|23:12:30.602] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:RUBY-366M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.602] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:RUBY-366M QuoteAssetSymbol:BUSD-BAF ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.602] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SAT-30B QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.603] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SDDS-A84M QuoteAssetSymbol:BUSD-BAF ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.603] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SEFA-A0AM QuoteAssetSymbol:BNB ListPrice:0.00000001 TickSize:0.00000001 LotSize:100000.00000000}"
I[2023-03-28|23:12:30.603] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SLV-C4EM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.603] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:SLV-C4EM QuoteAssetSymbol:BUSD-BAF ListPrice:26.00000000 TickSize:0.00010000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.604] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:START-561M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.604] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:START-BA0M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.604] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:STEEP-423M QuoteAssetSymbol:BNB ListPrice:0.00010000 TickSize:0.00000001 LotSize:10.00000000}"
I[2023-03-28|23:12:30.604] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:STL-7C6M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.605] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:STLM-DD7M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.605] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:TDZR-1DCM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.605] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:TES-BCCM QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.606] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:TTT-29DM QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.606] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:TTT-3FAM QuoteAssetSymbol:BNB ListPrice:2.00000000 TickSize:0.00000010 LotSize:0.10000000}"
I[2023-03-28|23:12:30.606] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:U2U-DF5M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.607] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:VNB-708M QuoteAssetSymbol:BNB ListPrice:0.10000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.607] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:VTBC-CB0M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.607] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:WOOPT-FADM QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00000001 LotSize:100.00000000}"
I[2023-03-28|23:12:30.608] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:WTR-ED7M QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.608] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:WWWC-12CM QuoteAssetSymbol:BUSD-BAF ListPrice:0.20000000 TickSize:0.00000100 LotSize:0.01000000}"
I[2023-03-28|23:12:30.608] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:XIN-F25 QuoteAssetSymbol:BNB ListPrice:1.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.609] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:XRP-FB8 QuoteAssetSymbol:BNB ListPrice:0.01000000 TickSize:0.00000001 LotSize:1.00000000}"
I[2023-03-28|23:12:30.609] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:YGEM-B61M QuoteAssetSymbol:BNB ListPrice:0.00100000 TickSize:0.00010000 LotSize:0.00010000}"
I[2023-03-28|23:12:30.609] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:YGEM-B61M QuoteAssetSymbol:BUSD-BAF ListPrice:0.00100000 TickSize:0.00000001 LotSize:0.00010000}"
I[2023-03-28|23:12:30.610] Successfully Loaded order snapshot           module=main pair="{BaseAssetSymbol:YNT2-C51M QuoteAssetSymbol:BNB ListPrice:10.00000000 TickSize:0.00001000 LotSize:0.00100000}"
I[2023-03-28|23:12:30.610] Recovered active orders. Snapshot is fully loaded module=main 
I[2023-03-28|23:12:35.573] get last breathe block height                module=main height=37727120
panic: state.AppHash does not match AppHash after replay. Got
DB76256058CAA8D1B5D5AEBAA2CEB8617A3DDF513C46A033A51A75E12A65C138, expected 87B2757FEC5EB1C8F12954C284842970B1E6AFB7FF63D2104B956B44D772708E.

State: {{{10 0} 0.32.3} Binance-Chain-Ganges 37763484 2035763 A115F98366E8AB8A17E99A65F7E5FC35372867258BE4E3FDCDD728C446A71407:1:6617813F2D4C 2023-03-28 23:12:07.195105944 +0000 UTC ValidatorSet{
  Proposer: Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-5007747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-1437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:6492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:242252066116}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-2257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-2507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-3007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:492252066116}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-3257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:7242252066116}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-5007747933884}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:-1985020661157}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:4992252066116}
} ValidatorSet{
  Proposer: Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:-3985020661157}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-2437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:5492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:-757747933884}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-3257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-3507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-4007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:-1507747933884}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-4257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:6242252066116}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:7992252066116}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:-3985020661157}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:3992252066116}
} ValidatorSet{
  Proposer: Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-5257747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-3437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:4492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:-1757747933884}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-4257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-4507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-5007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:-3507747933884}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-5257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:5242252066116}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:6992252066116}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:8014979338843}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:2992252066116}
} 35809060 {{1048576 -1} {100000} {[ed25519]}} 1 [] [135 178 117 127 236 94 177 200 241 41 84 194 132 132 41 112 177 230 175 183 255 99 210 16 75 149 107 68 215 114 112 142]}

Did you reset Tendermint without resetting your application's data?

goroutine 1 [running]:
github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromState({_, _, _}, {{{0xa, 0x0}, {0xc0205f3158, 0x6}}, {0xc0205de468, 0x14}, 0x240399c, ...}, ...)
  github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:489 +0x17a
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xa, 0x0}, {0xc0205f3158, 0x6}}, {0xc0205de468, 0x14}, 0x240399c, 0x1f1033, {{0xc020702500, ...}, ...}, ...}, ...)
  github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:370 +0x3b7
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc021198fc8, {0x1618f10, 0xc01df76850})
  github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:271 +0x4c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xa, 0x0}, {0xc0205f3158, 0x6}}, {0xc0205de468, 0x14}, 0x240399c, 0x1f1033, ...}, ...)
  github.com/tendermint/tendermint@v0.35.9/node/node.go:313 +0x1a7
github.com/tendermint/tendermint/node.NewNode(0xc00016c840, {0x15e9b70, 0xc02073c820}, 0xc02070b9d0, {0x15d7160, 0xc0000a6fa0}, 0x0, 0xc0005aaaa0, 0xc02070bba0, {0x15f9e50, ...}, ...)
  github.com/tendermint/tendermint@v0.35.9/node/node.go:654 +0x3d1
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc00011e2b8, 0x1470078)
  github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:133 +0x66a
github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc000b25b80, {0xc000594ce0, 0x0, 0x2})
  github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:44 +0x7f
github.com/spf13/cobra.(*Command).execute(0xc000b25b80, {0xc000594ca0, 0x2, 0x2})
  github.com/spf13/cobra@v0.0.5/command.go:826 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000b24280)
  github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
  github.com/spf13/cobra@v0.0.5/command.go:864
github.com/tendermint/tendermint/libs/cli.Executor.Execute({0xc000b24280, 0x1470318})
  github.com/tendermint/tendermint@v0.35.9/libs/cli/setup.go:89 +0x36
main.main()
  github.com/bnb-chain/node/cmd/bnbchaind/main.go:54 +0x28e

This issue is a different one. It is fixed in this release https://github.com/bnb-chain/node/releases/tag/v0.10.12, please have a try.

forcodedancing commented 1 year ago

Is anyone looking into this issue? I can confirm this only happens when running a node on testnet. I've tried changing a few options in config.toml, but nothing fixes the issue.

Here's a more detailed log:

I[2023-04-05|21:05:16.653] Relaying block for order book                module=dex height=38061251
I[2023-04-05|21:05:16.653] replayed all tx. Starting match              module=dex height=38061251
I[2023-04-05|21:05:16.653] No order comes in for the block              module=dexkeeper 
I[2023-04-05|21:05:16.653] Relaying block for order book                module=dex height=38061252
I[2023-04-05|21:05:16.653] replayed all tx. Starting match              module=dex height=38061252
I[2023-04-05|21:05:16.653] No order comes in for the block              module=dexkeeper 
I[2023-04-05|21:05:16.904] Located day to load breathe block height     module=dexkeeper epochDay="\ufffdK\u0000\u0000\u0000\u0000\u0000\u0000"
I[2023-04-05|21:05:16.904] Loaded breathe block height                  module=dexkeeper height=38028184
I[2023-04-05|21:05:16.904] get last breathe block height                module=main height=38028184
I[2023-04-05|21:05:16.905] reload latest snapshot                       module=statesync height=38028184
I[2023-04-05|21:05:16.936] ABCI Handshake App Info                      module=consensus height=38061252 hash=A651516385C1210304753EACD653D8137108D58CD5414CA35088611FE1738BED software-version= protocol-version=0
I[2023-04-05|21:05:16.937] ABCI Replay Blocks                           module=consensus appHeight=38061252 storeHeight=38061252 stateHeight=38061252
panic: state.AppHash does not match AppHash after replay. Got
A651516385C1210304753EACD653D8137108D58CD5414CA35088611FE1738BED, expected 212E408D77C3F0DB1BEED74FFADB002CE250354F6E6F38F49956DDB32DC0E694.

State: {{{10 0} 0.32.3} Binance-Chain-Ganges 38061252 2036586 AD8832D49C55AF0572BFCC1D0FEF1B3F180155B22E3BCB8A07A47C62E051E04E:1:26AF04C17B4B 2023-04-05 21:04:54.55707933 +0000 UTC ValidatorSet{
  Proposer: Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:-5507747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:562500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:-5507747933884}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:2242252066116}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-1007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:4492252066116}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-1257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:-4757747933884}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-3007747933884}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:2014979338843}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:6992252066116}
} ValidatorSet{
  Proposer: Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:-5757747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:7492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:1242252066116}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-1257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-1507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-2007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:2492252066116}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-2257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:-5757747933884}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-4007747933884}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:14979338843}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:5992252066116}
} ValidatorSet{
  Proposer: Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-5007747933884}
  Validators:
    Validator{06FD60078EB4C2356137DD50036597DB267CF616 PubKeyEd25519{E17CBE9C20CDCFDF876B3B12978D3264A007FCAAA71C4CDB701D9EBC0323F44F} VP:1000000000000 A:-1437500000000}
    Validator{18E69CC672973992BB5F76D049A5B2C5DDF77436 PubKeyEd25519{184E7B103D34C41003F9B864D5F8C1ADDA9BD0436B253BB3C844BC739C1E77C9} VP:1000000000000 A:6492252066116}
    Validator{344C39BB8F4512D6CAB1F6AAFAC1811EF9D8AFDF PubKeyEd25519{4D420AEA843E92A0CFE69D89696DFF6827769F9CB52A249AF537CE89BF2A4B74} VP:1000000000000 A:242252066116}
    Validator{37EF19AF29679B368D2B9E9DE3F8769B35786676 PubKeyEd25519{BD03DE9F8AB29E2800094E153FAC6F696CFA512536C9C2F804DCB2C2C4E4AED6} VP:1000000000000 A:-2257747933884}
    Validator{62633D9DB7ED78E951F79913FDC8231AA77EC12B PubKeyEd25519{8F4A74A07351895DDF373057B98FAE6DFAF2CD21F37A063E19601078FE470D53} VP:1000000000000 A:-2507747933884}
    Validator{91844D296BD8E591448EFC65FD6AD51A888D58FA PubKeyEd25519{C80E9ABEF7FF439C10C68FE8F1303DEDDFC527718C3B37D8BA6807446E3C827A} VP:1000000000000 A:-3007747933884}
    Validator{9CCDDD479C0AD8DCD01D754DD95FE15384E8BBDC PubKeyEd25519{4D1BE64F0E9A466C2E66A53433928192783E29F8FA21BEB2133499B5EF770F60} VP:2000000000000 A:492252066116}
    Validator{B3727172CE6473BC780298A2D66C12F1A14F5B2A PubKeyEd25519{9142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC} VP:1000000000000 A:-3257747933884}
    Validator{B6F20C7FAA2B2F6F24518FA02B71CB5F4A09FBA3 PubKeyEd25519{49B288E4EBBB3A281C2D546FC30253D5BAF08993B6E5D295FB787A5B314A298E} VP:1000000000000 A:7242252066116}
    Validator{D4CECEF238E778C7063552C3A7AB95DA35C3FB47 PubKeyEd25519{99308AA365C40554BC89982AF505D85DA95251445D5DD4A9BB37DD2584FD92D3} VP:1000000000000 A:-5007747933884}
    Validator{F15AB82564D6F450CA6F30A074568A5E6DEE7B9B PubKeyEd25519{DF33205D0974D06F3F5D32E6FD98E5DEBD7EC39A796455F93C63AEEF0DADB31A} VP:2000000000000 A:-1985020661157}
    Validator{F42F1D05AC568D12E26B9655395E7FBBD46BC5BB PubKeyEd25519{01776920FF0B0F38D78CF95C033C21ADF7045785114E392A7544179652E0A612} VP:1000000000000 A:4992252066116}
} 35809060 {{1048576 -1} {100000} {[ed25519]}} 1 [] [33 46 64 141 119 195 240 219 27 238 215 79 250 219 0 44 226 80 53 79 110 111 56 244 153 86 221 179 45 192 230 148]}

Did you reset Tendermint without resetting your application's data?

goroutine 1 [running]:
github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromState({_, _, _}, {{{0xa, 0x0}, {0xc016e4ec58, 0x6}}, {0xc016e5e210, 0x14}, 0x244c4c4, ...}, ...)
  github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:489 +0x17a
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xa, 0x0}, {0xc016e4ec58, 0x6}}, {0xc016e5e210, 0x14}, 0x244c4c4, 0x1f136a, {{0xc016e5a580, ...}, ...}, ...}, ...)
  github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:370 +0x3b7
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00c5eafc8, {0x1618f10, 0xc0001ac230})
  github.com/tendermint/tendermint@v0.35.9/consensus/replay.go:271 +0x4c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xa, 0x0}, {0xc016e4ec58, 0x6}}, {0xc016e5e210, 0x14}, 0x244c4c4, 0x1f136a, ...}, ...)
  github.com/tendermint/tendermint@v0.35.9/node/node.go:313 +0x1a7
github.com/tendermint/tendermint/node.NewNode(0xc000152c60, {0x15e9b70, 0xc016dfe780}, 0xc016dadfd0, {0x15d7160, 0xc0009aa6e0}, 0x0, 0x146d660, 0xc016e161a0, {0x15f9e50, ...}, ...)
  github.com/tendermint/tendermint@v0.35.9/node/node.go:654 +0x3d1
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc0005962e8, 0x1470078)
  github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:133 +0x66a
github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc000186280, {0xc00026f300, 0x0, 0x2})
  github.com/cosmos/cosmos-sdk@v0.25.0/server/start.go:44 +0x7f
github.com/spf13/cobra.(*Command).execute(0xc000186280, {0xc00026f2c0, 0x2, 0x2})
  github.com/spf13/cobra@v0.0.5/command.go:826 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001fd900)
  github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
  github.com/spf13/cobra@v0.0.5/command.go:864
github.com/tendermint/tendermint/libs/cli.Executor.Execute({0xc0001fd900, 0x1470318})
  github.com/tendermint/tendermint@v0.35.9/libs/cli/setup.go:89 +0x36
main.main()
  github.com/bnb-chain/node/cmd/bnbchaind/main.go:54 +0x28e

The issue is fixed in this release https://github.com/bnb-chain/node/releases/tag/v0.10.12, please have a try.

forcodedancing commented 1 year ago

caught exactly the same error when updating to v0.10.11, how to solve this problem ?

The latest release fixes the issue, please download it https://github.com/bnb-chain/node/releases/tag/v0.10.12

tmeinlschmidt commented 1 year ago

btw - after upgrading to 0.10.12 (because of issues in 0.10.11) I ended up with

I[2023-04-25|15:24:20.461] Starting ABCI with Tendermint                module=main 
failed to load rootMultiStore: wanted to load target 310798570 but only found up to 310798569

and it dies.

config used from 0.10.12, no changes except for state_sync_height = -1

nmiceli-simtlix commented 1 year ago

Upgraded to v0.10.12. It now panics with this error:

I[2023-04-25|15:34:43.257] FeeCalculation distributeFeeBEP159           module=main height=38780764 prevBlockFee=
I[2023-04-25|15:34:43.259] Sync params proposals.                       module=paramHub 
I[2023-04-25|15:34:43.290] Executed block                               module=state height=38780764 validTxs=0 invalidTxs=0
I[2023-04-25|15:34:43.298] Committed state                              module=state height=38780764 txs=0 appHash=9705FBDD81D7B7AD794EDF5D70639D6F1926E5FFF471AACDC060BCE95621D97D
I[2023-04-25|15:34:43.315] symbols to match                             module=dexkeeper symbols=[]
I[2023-04-25|15:34:43.315] No order comes in for the block              module=dexkeeper 
I[2023-04-25|15:34:43.315] FeeCalculation distributeFeeBEP159           module=main height=38780765 prevBlockFee=
I[2023-04-25|15:34:43.316] Sync params proposals.                       module=paramHub 
I[2023-04-25|15:34:43.322] Executed block                               module=state height=38780765 validTxs=4 invalidTxs=0
I[2023-04-25|15:34:43.327] Committed state                              module=state height=38780765 txs=4 appHash=99E85E612EF228C3E3370E17457FB51A0579A936761C264A318A1F61B78556A1
I[2023-04-25|15:34:43.344] symbols to match                             module=dexkeeper symbols=[]
I[2023-04-25|15:34:43.344] No order comes in for the block              module=dexkeeper 
I[2023-04-25|15:34:43.345] FeeCalculation distributeFeeBEP159           module=main height=38780766 prevBlockFee=
panic: unmarshal to types.Validator failed after 160 bytes (insufficient bytes decoding []byte of length 1000000000000): 0A14666F9C66C385A14842F8730C4C906DAE8B22B9981214666F9C66C385A14842F8730C4C906DAE8B22B9981A251624DE64209142AFCC691B7CC05D26C7B0BE0C8B46418294171730E079F384FDE2FA50BAFC28023280A094A58D1D3A80A094A58D1D42090A07476168696E67614888E53C6A0E220C08D3F6DEF80510E1F4CA9A01721411896BEC37DAC5D41DC3FDB204DAFC0FCFC999A2920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D920180A094A58D1D9A0180C0DFDA8EE906

goroutine 454 [running]:
github.com/cosmos/cosmos-sdk/x/stake/types.MustUnmarshalValidator(_, {_, _, _})
    github.com/cosmos/cosmos-sdk@v0.25.0/x/stake/types/validator.go:145 +0x12e
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.GetValidator({{0x16e3370, 0xc000a6b730}, {0x16e3370, 0xc000a6b740}, {0x16e3398, 0xc000a6b7e0}, 0xc000ae9b90, {0x16f1760, 0xc000d740c0}, 0xc000d44100, ...}, ...)
    github.com/cosmos/cosmos-sdk@v0.25.0/x/stake/keeper/validator.go:25 +0x189
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.GetValidatorByConsAddr({{0x16e3370, 0xc000a6b730}, {0x16e3370, 0xc000a6b740}, {0x16e3398, 0xc000a6b7e0}, 0xc000ae9b90, {0x16f1760, 0xc000d740c0}, 0xc000d44100, ...}, ...)
    github.com/cosmos/cosmos-sdk@v0.25.0/x/stake/keeper/validator.go:44 +0x1f8
github.com/bnb-chain/node/app.(*ValAddrCache).GetDistributionAddr(_, {{0x16e9ba0, 0xc000046090}, {0x16f3518, 0xc001d87740}, {{0xa, 0x0, {}, {0x0, 0x0, ...}, ...}, ...}, ...}, ...)
    github.com/bnb-chain/node/app/fee_distribution.go:62 +0x150
github.com/bnb-chain/node/app.distributeFeeBEP159({{0x16e9ba0, 0xc000046090}, {0x16f3518, 0xc001d87740}, {{0xa, 0x0, {}, {0x0, 0x0, 0x0}, ...}, ...}, ...}, ...)
    github.com/bnb-chain/node/app/fee_distribution.go:83 +0x545
github.com/bnb-chain/node/app.(*BinanceChain).EndBlocker(_, {{0x16e9ba0, 0xc000046090}, {0x16f3518, 0xc001d87740}, {{0xa, 0x0, {}, {0x0, 0x0, ...}, ...}, ...}, ...}, ...)
    github.com/bnb-chain/node/app/app.go:907 +0xc05
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc000ba8780, {0x24fbf5e, {}, {0x0, 0x0, 0x0}, 0x0})
    github.com/cosmos/cosmos-sdk@v0.25.0/baseapp/baseapp.go:934 +0x1dd
github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).EndBlockSync(0xc0000d66e0, {0x24fbf5e, {}, {0x0, 0x0, 0x0}, 0x0})
    github.com/cosmos/cosmos-sdk@v0.25.0/server/concurrent/async_local_client.go:408 +0x257
github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc002faf740?, {0x24fbf5e, {}, {0x0, 0x0, 0x0}, 0x0})
    github.com/tendermint/tendermint@v0.35.9/proxy/app_conn.go:115 +0x4e
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x16ea5e8, 0xc00b07f580}, {0x16f0740, 0xc00012bd40}, 0xc00c15b680, 0x14?, {0x16f4178, 0xc000014e68})
    github.com/tendermint/tendermint@v0.35.9/state/execution.go:298 +0x6e2
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xa, 0x0}, {0xc000d3b890, 0x6}}, {0xc000bb2d50, 0x14}, 0x24fbf5d, 0x1f1600, {{0xc02789e640, ...}, ...}, ...}, ...)
    github.com/tendermint/tendermint@v0.35.9/state/execution.go:128 +0x178
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc00b41fd40)
    github.com/tendermint/tendermint@v0.35.9/blockchain/v0/reactor.go:401 +0xd1b
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).SwitchToBlockchain
    github.com/tendermint/tendermint@v0.35.9/blockchain/v0/reactor.go:154 +0x305
forcodedancing commented 1 year ago

@tmeinlschmidt Try the latest release https://github.com/bnb-chain/node/releases/tag/v0.10.13 An issue is introduced in v0.10.12 (failed to unmarshal validator error) and it has been fixed in v0.10.13.

nmiceli-simtlix commented 1 year ago

@forcodedancing the unmarshal validator error has now been fixed with the latest update.

Now, when I restart my node, it won't get connected to peers and resume syncing unless I delete the addrbook.json file before starting. When doing that, it resumes fine, no more panics.