Mainnet sync fails and crashes the node

[RaghavSood]

We are trying to do a mainnet sync using the 0.5.10 binary, along with the accompanying config.

The node syncs up to a certain point, then crashes with:

log file:

I[2019-06-15|23:20:12.572] Relaying block for order book                module=dex height=7402602
I[2019-06-15|23:20:12.572] replayed all tx. Starting match              module=dex height=7402602
I[2019-06-15|23:20:12.572] No new orders for any pair, give up matching module=dexkeeper
I[2019-06-15|23:20:12.572] No order comes in for the block              module=dexkeeper
I[2019-06-15|23:20:12.572] Relaying block for order book                module=dex height=7402603
I[2019-06-15|23:20:12.573] does not recover invalid tx                  module=dex txHash=3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE err="{\"codespace\":1,\"code\":1,\"abci_code\":65537,\"message\":\"recovered: open /home/ubuntu/.bnbchaind/data/application.db/151355.ldb: too many open files\\nstack:\\ngoroutine 836 [running]:\\nruntime/debug.Stack(0xc000de86b8, 0xfa79c0, 0xc0a7209410)\\n\\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx.func1(0xc000de9aa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:810 +0x6e\\npanic(0xfa79c0, 0xc0a7209410)\\n\\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc00000e488, 0xc05bc756b0, 0x29, 0x30, 0x29, 0xc05bc756b0, 0x8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc00fe58210, 0xc05bc75650, 0x21, 0x21, 0x0, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc0000bf900, 0xc11848d8a0, 0x20, 0x20, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:76 +0x29e\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getRightNode(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc14cdce0b0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de89e8, 0xc000de8aa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0a7d79d90, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8a48, 0xc000de8b08)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0a7d79ce0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8aa8, 0xc000de8b68)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a1130, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b08, 0xc000de8bc8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a1080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b68, 0xc000de8c28)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a0fd0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x249249249249, 0x249249249249)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc03413cb00, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x10000000042d1a1, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc02fa55080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc031fa3e20, 0xc000de8ce0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc011520210, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x20, 0xf483c0, 0xc11848d820, 0xc11848d820)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*ImmutableTree).Get(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc01102b920, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xc11848d820, 0x8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:131 +0x86\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).GetAccount(0xc053124140, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc12b08e980, 0xc11848d3c0, 0x14, 0x14, 0xf34460, 0xc000de8eb8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc
12b08e980, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0x1042900, 0x13f2840)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0xc12b08ec00, 0x13a9d78)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.getCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:159 +0xb1\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.addCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:199 +0xc2\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.inputOutputCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc0a7208510, 0x1, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:239 +0x43b\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.BaseKeeper.InputOutputCoins(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:73\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.handleMsgSend(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f4ca0, 0xc000d6a1e0, 0xc0a7208510, 0x1, 0x1, 0xc0a7208570, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:24 +0xed\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.NewHandler.func1(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f31a0, 0xc0a72085a0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:12 +0x226\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(0xc000891590, 0x13e9e80, 0xc0a7209080, 0xc12b08ea00, 0x11, 0xc031fa3840, 0x1, 0x1, 0xc1178339c0, 0x40, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:743 +0x2f3\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx(0xc000891590, 0x1100105, 0xc03ce4d2b0, 0xc8, 0xc8, 0x13cda80, 0xc12e319680, 0xc1178339c0, 0x40, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:833 +0x24c\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc000891590, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:651 +0x297\\ngithub.com/binance-chain/node/app.(*BinanceChain).DeliverTx(0xc0000b6840, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/app/app.go:400 +0x9b\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker.func1(0xc003117970, 0xc000de9fa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc003117970)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).OnStart\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 +0x64\\n\"}" height=7402603
I[2019-06-15|23:20:12.573] replayed all tx. Starting match              module=dex height=7402603
I[2019-06-15|23:20:12.573] No new orders for any pair, give up matching module=dexkeeper
I[2019-06-15|23:20:12.573] No order comes in for the block              module=dexkeeper
I[2019-06-15|23:20:12.736] ABCI Handshake App Info                      module=consensus height=7402603 hash=19A0D6CC663D5BA6DB170E08CFCC06EABD2D70AF01CD225F2E05E7FCA0338762 software-version= protocol-version=0
I[2019-06-15|23:20:12.745] ABCI Replay Blocks                           module=consensus appHeight=7402603 storeHeight=7402604 stateHeight=7402603
I[2019-06-15|23:20:12.745] Replay last block using real app             module=consensus
I[2019-06-15|23:20:14.391] Starting ABCI with Tendermint                module=main

stderr:

ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash.  Expected 19A0D6CC663D5BA6DB170E08CFCC06EABD2D70AF01CD225F2E05E7FCA0338762, got 76B5139088EA904E4EF6B3C0EA57AE04937B7CC926D080BE49820C31A9BBDA9A

This happened earlier using the 0.5.8.1 node as well, but on a different block:

ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash.  Expected B41A8F465A899CDDD1ACD3D0962DE7B81691856A710494EFA68DC96BE938675B, got 061047A25E344572F2CF7931B953A863AC2F05B7CD8CA87089284BB484355387

We started a full resync after that using the updated node version, but it is still happening.

After the node crashes, it is restarted by the process manager - the logs then indicate it rolls back a few thousand blocks (it crashes on 7402603, and starts syncing from 73XXXXX) - It then reaches 7402603 and crashes again.

[ackratos]

Does two times crash you see "Open too many files" both?

If that's the case could you please paste your machine's 'ulimit -a' result here? If the open files in result is less than 1024 please change system config:

vim /etc/security/limits.conf

add in the end

• soft nofile 4096 * hard nofile 4096

Raghav Sood notifications@github.com于2019年6月16日 周日上午7:28写道：

We are trying to do a mainnet sync using the 0.5.10 binary, along with the accompanying config.

The node syncs up to a certain point, then crashes with:

log file:

I[2019-06-15|23:20:12.572] Relaying block for order book module=dex height=7402602 I[2019-06-15|23:20:12.572] replayed all tx. Starting match module=dex height=7402602 I[2019-06-15|23:20:12.572] No new orders for any pair, give up matching module=dexkeeper I[2019-06-15|23:20:12.572] No order comes in for the block module=dexkeeper I[2019-06-15|23:20:12.572] Relaying block for order book module=dex height=7402603 I[2019-06-15|23:20:12.573] does not recover invalid tx module=dex txHash=3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE err="{\"codespace\":1,\"code\":1,\"abci_code\":65537,\"message\":\"recovered: open /home/ubuntu/.bnbchaind/data/application.db/151355.ldb: too many open files\nstack:\ngoroutine 836 [running]:\nruntime/debug.Stack(0xc000de86b8, 0xfa79c0, 0xc0a7209410)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(BaseApp).RunTx.func1(0xc000de9aa8)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:810 <http://ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(BaseApp).RunTx.func1(0xc000de9aa8)%5C%5Cn%5C%5Ct/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:810> +0x6e\npanic(0xfa79c0, 0xc0a7209410)\n\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(GoLevelDB).Get(0xc00000e488, 0xc05bc756b0, 0x29, 0x30, 0x29, 0xc05bc756b0, 0x8)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(prefixDB).Get(0xc00fe58210, 0xc05bc75650, 0x21, 0x21, 0x0, 0x0, 0x0)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(nodeDB).GetNode(0xc0000bf900, 0xc11848d8a0, 0x20, 0x20, 0x0)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:76 +0x29e\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).getRightNode(...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc14cdce0b0 <http://ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).getRightNode(...)%5C%5Cn%5C%5Ct/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363%5C%5Cngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc14cdce0b0>, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de89e8, 0xc000de8aa8)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc0a7d79d90, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8a48, 0xc000de8b08)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc0a7d79ce0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8aa8, 0xc000de8b68)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc0374a1130, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b08, 0xc000de8bc8)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc0374a1080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b68, 0xc000de8c28)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc0374a0fd0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x249249249249, 0x249249249249)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc03413cb00, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x10000000042d1a1, 0x0)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc02fa55080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc031fa3e20, 0xc000de8ce0)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(Node).get(0xc011520210, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x20, 0xf483c0, 0xc11848d820, 0xc11848d820)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(ImmutableTree).Get(...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(IavlStore).Get(0xc01102b920 <http://ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(ImmutableTree).Get(...)%5C%5Cn%5C%5Ct/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98%5C%5Cngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(IavlStore).Get(0xc01102b920>, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xc11848d820, 0x8)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:131 +0x86\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(accountStoreCache).GetAccount(0xc053124140, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(accountCache).getAccountFromCache(0xc12b08e980, 0xc11848d3c0, 0x14, 0x14, 0xf34460, 0xc000de8eb8)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(accountCache).GetAccount(0xc 12b08e980 http://ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc12b08e980, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(accountCache).getAccountFromCache(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0x1042900, 0x13f2840)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(accountCache).GetAccount(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0xc12b08ec00, 0x13a9d78)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0xc11848d3c0, 0x14, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.getCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:159 +0xb1\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.addCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:199 +0xc2\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.inputOutputCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc0a7208510, 0x1, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:239 +0x43b\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.BaseKeeper.InputOutputCoins(...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:73\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.handleMsgSend(0x13e9e80 http://ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.BaseKeeper.InputOutputCoins(...)%5C%5Cn%5C%5Ct/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:73%5C%5Cngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.handleMsgSend(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f4ca0, 0xc000d6a1e0, 0xc0a7208510, 0x1, 0x1, 0xc0a7208570, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:24 +0xed\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.NewHandler.func1(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f31a0, 0xc0a72085a0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:12 +0x226\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(BaseApp).runMsgs(0xc000891590, 0x13e9e80, 0xc0a7209080, 0xc12b08ea00, 0x11, 0xc031fa3840, 0x1, 0x1, 0xc1178339c0, 0x40, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:743 +0x2f3\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(BaseApp).RunTx(0xc000891590, 0x1100105, 0xc03ce4d2b0, 0xc8, 0xc8, 0x13cda80, 0xc12e319680, 0xc1178339c0, 0x40, 0x0, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:833 +0x24c\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(BaseApp).DeliverTx(0xc000891590, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:651 +0x297\ngithub.com/binance-chain/node/app.(BinanceChain).DeliverTx(0xc0000b6840, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/app/app.go:400 +0x9b\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(asyncLocalClient).deliverTxWorker.func1(0xc003117970, 0xc000de9fa8)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(asyncLocalClient).deliverTxWorker(0xc003117970)\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 http://ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc003117970)%5C%5Cn%5C%5Ct/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(asyncLocalClient).OnStart\n\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 <http://github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(asyncLocalClient).OnStart%5C%5Cn%5C%5Ct/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93> +0x64\n\"}" height=7402603 I[2019-06-15|23:20:12.573] replayed all tx. Starting match module=dex height=7402603 I[2019-06-15|23:20:12.573] No new orders for any pair, give up matching module=dexkeeper I[2019-06-15|23:20:12.573] No order comes in for the block module=dexkeeper I[2019-06-15|23:20:12.736] ABCI Handshake App Info module=consensus height=7402603 hash=19A0D6CC663D5BA6DB170E08CFCC06EABD2D70AF01CD225F2E05E7FCA0338762 software-version= protocol-version=0 I[2019-06-15|23:20:12.745] ABCI Replay Blocks module=consensus appHeight=7402603 storeHeight=7402604 stateHeight=7402603 I[2019-06-15|23:20:12.745] Replay last block using real app module=consensus I[2019-06-15|23:20:14.391] Starting ABCI with Tendermint module=main

stderr:

ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash. Expected 19A0D6CC663D5BA6DB170E08CFCC06EABD2D70AF01CD225F2E05E7FCA0338762, got 76B5139088EA904E4EF6B3C0EA57AE04937B7CC926D080BE49820C31A9BBDA9A

This happened earlier using the 0.5.8.1 node as well, but on a different block:

ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash. Expected B41A8F465A899CDDD1ACD3D0962DE7B81691856A710494EFA68DC96BE938675B, got 061047A25E344572F2CF7931B953A863AC2F05B7CD8CA87089284BB484355387

We started a full resync after that using the updated node version, but it is still happening.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/binance-chain/node-binary/issues/62?email_source=notifications&email_token=AALAI2JBTUMXKHCFXM4PAR3P2V3IZA5CNFSM4HYQAP6KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GZW55LA, or mute the thread https://github.com/notifications/unsubscribe-auth/AALAI2MK7IEDO3G4T2FFMNTP2V3IZANCNFSM4HYQAP6A .

--

[RaghavSood]

Does two times crash you see "Open too many files" both?

If that's the case could you please paste your machine's 'ulimit -a' result here? If the open files in result is less than 1024 please change system config:

vim /etc/security/limits.conf # add in the end * soft nofile 4096 * hard nofile 4096

We did not see any file handle related errors, but I have increased the limit anyways - it still crashes on the same block, with the same log file and stderr messages.

[ackratos]

We did not see any file handle related errors, but I have increased the limit anyways - it still crashes on the same block, with the same log file and stderr messages.

The file handle related errors comes from here in your log:

I[2019-06-15|23:20:12.573] does not recover invalid tx                  module=dex txHash=3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE err="{\"codespace\":1,\"code\":1,\"abci_code\":65537,\"message\":\"recovered: open /home/ubuntu/.bnbchaind/data/application.db/151355.ldb: too many open files\\nstack:\\ngoroutine 836 [running]:\\nruntime/debug.Stack(0xc000de86b8, 0xfa79c0, 0xc0a7209410)\\n\\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx.func1(0xc000de9aa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:810 +0x6e\\npanic(0xfa79c0, 0xc0a7209410)\\n\\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc00000e488, 0xc05bc756b0, 0x29, 0x30, 0x29, 0xc05bc756b0, 0x8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc00fe58210, 0xc05bc75650, 0x21, 0x21, 0x0, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc0000bf900, 0xc11848d8a0, 0x20, 0x20, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:76 +0x29e\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getRightNode(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc14cdce0b0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de89e8, 0xc000de8aa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0a7d79d90, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8a48, 0xc000de8b08)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0a7d79ce0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0xc000de8aa8, 0xc000de8b68)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a1130, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b08, 0xc000de8bc8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a1080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc000de8b68, 0xc000de8c28)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc0374a0fd0, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x249249249249, 0x249249249249)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc03413cb00, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0x1, 0x10000000042d1a1, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc02fa55080, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc031fa3e20, 0xc000de8ce0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc011520210, 0xc01e52a640, 0xc11848d820, 0x1c, 0x20, 0x20, 0xf483c0, 0xc11848d820, 0xc11848d820)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*ImmutableTree).Get(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc01102b920, 0xc11848d820, 0x1c, 0x20, 0x1c, 0xc11848d820, 0x8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:131 +0x86\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).GetAccount(0xc053124140, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc12b08e980, 0xc11848d3c0, 0x14, 0x14, 0xf34460, 0xc000de8eb8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc
12b08e980, 0xc11848d3c0, 0x14, 0x14, 0x0, 0x0)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0x1042900, 0x13f2840)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc12b08ec00, 0xc11848d3c0, 0x14, 0x14, 0xc12b08ec00, 0x13a9d78)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.getCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:159 +0xb1\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.addCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc11848d3c0, 0x14, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:199 +0xc2\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.inputOutputCoins(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13db640, 0xc000958840, 0x1272558, 0xc000bf6230, 0xc0a7208510, 0x1, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:239 +0x43b\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.BaseKeeper.InputOutputCoins(...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/keeper.go:73\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.handleMsgSend(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f4ca0, 0xc000d6a1e0, 0xc0a7208510, 0x1, 0x1, 0xc0a7208570, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:24 +0xed\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank.NewHandler.func1(0x13e9e80, 0xc0a72090e0, 0xc12b08ea00, 0x13, 0x13f31a0, 0xc0a72085a0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/bank/handler.go:12 +0x226\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(0xc000891590, 0x13e9e80, 0xc0a7209080, 0xc12b08ea00, 0x11, 0xc031fa3840, 0x1, 0x1, 0xc1178339c0, 0x40, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:743 +0x2f3\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx(0xc000891590, 0x1100105, 0xc03ce4d2b0, 0xc8, 0xc8, 0x13cda80, 0xc12e319680, 0xc1178339c0, 0x40, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:833 +0x24c\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc000891590, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:651 +0x297\\ngithub.com/binance-chain/node/app.(*BinanceChain).DeliverTx(0xc0000b6840, 0xc03ce4d2b0, 0xc8, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/app/app.go:400 +0x9b\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker.func1(0xc003117970, 0xc000de9fa8)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc003117970)\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).OnStart\\n\\t/Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 +0x64\\n\"}" height=7402603

Your local database has already inconsistent with other peers as the first time you execute this tx 3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE your execution result is error: open too many files while all other nodes on chain is a good transaction: https://explorer.binance.org/tx/3161EE8A251CF5A0FD363AA164B7EDA4BA512C4CCD72A434A51E54D3BDED1FDE.

To recover your database you'd better clean your $HOME/data folder, and increase the open file handle limit before you resync.

[RaghavSood]

Right, that makes sense - I've started a fresh sync, but is there any way to rollback just the broken chain state? The chain is already at 13m+ blocks, and resyncing from scratch after errors like this, which might crop up in other scenarios (running out of disk space, for instance) is not usually feasible.

[chainwhisper]

Right, that makes sense - I've started a fresh sync, but is there any way to rollback just the broken chain state? The chain is already at 13m+ blocks, and resyncing from scratch after errors like this, which might crop up in other scenarios (running out of disk space, for instance) is not usually feasible.

I think you have to restart from the beginning.

[ackratos]

we can release some data fixing tools to cover such case soon. but its not rollback, anyone cannot rollback blockchain.

if you doesn't care about historical data, there is a state_sync option https://docs.binance.org/fullnode.html#state-sync can help you directly sync to latest block.

[RaghavSood]

we can release some data fixing tools to cover such case soon. but its not rollback, anyone cannot rollback blockchain.

if you doesn't care about historical data, there is a state_sync option https://docs.binance.org/fullnode.html#state-sync can help you directly sync to latest block.

Right, of course, I'm not talking about an actual rollback of blocks, just the local state to a known-good point so that we don't need to resync from scratch if the chain head fails.

We unfortunately require the full history, so need to sync without pruning

[RaghavSood]

The issue still persists - A fresh resync from scratch after increasing the open file limit to 4096 synced up to the exact same block (7402603) before crashing again, with the same stack trace in the logs.

I increased the limit to 35000, and it is still failing on that specific block with the same error message.

Any more ideas?

Running on:

$ uname -a
Linux binance-mainnet-s 4.15.0-1033-gcp #35-Ubuntu SMP Fri May 17 13:27:54 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[RaghavSood]

The main process at least does not seem to be using up all that many descriptors:

$ while sleep 1; do ls /proc/2000/fd | wc -l | tee -a temp.out; done
321
324
327
331
334
338
338
339
339
339
339
342
361
366
371
376
379
384
387
392
397
401
406
410
414
418
345
# crashed here

The node also doesn't seem to be using up the full limit:

$ ulimit -n
35000
$ cat /proc/2957/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             39869                39869                processes
Max open files            1024                 4096                 files
Max locked memory         16777216             16777216             bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       39869                39869                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

[chainwhisper]

How much memory does your machine have?

Can you share the error log here please?

[RaghavSood]

I've attached the latest bnc.log, and the stderr and stdout logs.

The machine is a GCP VM with 4 cores/10 GB memory - we are using a 500 GB SSD for the blockchain data.

logs.zip

[chainwhisper]

@RaghavSood I saw in the log that your node is already synced to height 8762856.

[RaghavSood]

@huangsuyu It was, that was the initial crash we received 2 days ago - we resynced from scratch after that, and then started crashing on block 7402603, which is when I opened this ticket.

[RaghavSood]

Apologies, I grabbed the logs from the wrong node (we have two, both are crashing with the same error).

Here's the updated ones: bnc-s.zip

[chainwhisper]

Apologies, I grabbed the logs from the wrong node (we have two, both are crashing with the same error).

Here's the updated ones: bnc-s.zip

Thanks!

Is bnbchaind running as a systemd?

why it restarted after crash 3mins later?