drunken005
commented
4 years ago The bnbchaind process exits after 'numPending=480' to 'numPending=1'
Closed drunken005 closed 4 years ago
drunken005
commented
4 years ago The bnbchaind process exits after 'numPending=480' to 'numPending=1'
ackratos
commented
4 years ago I think this isn't the first time you run the fullnode. If you deliberately deleted the $BNBHOME/data/STATESYNC.LOCK and want state sync again. You should also delete $BNBHOME/data and $BNBHOME/config/priv_validator_key.json
Could you describe more about how your node is initialized? Did it run for a while? What did you do before this time startup failure?
drunken005
commented
4 years ago It is not the first time, because the first time in the synchronization process, when numPending=1, the process automatically exits, the above error occurs when the script is run again.
drunken005
commented
4 years ago Also tried what you said delete $BNBHOME/data and $BNBHOME/config/priv_validator_key.json @ackratos
ackratos
commented
4 years ago It is not the first time, because the first time in the synchronization process, when numPending=1, the process automatically exits, the above error occurs when the script is run again.
could you please share the error on first-time synchronization? That's the root cause I think. The error message you paste is just because the second time you start the broken node, some legacy data make it failed to start.
ackratos
commented
4 years ago Also tried what you said delete $BNBHOME/data and $BNBHOME/config/priv_validator_key.json @ackratos
you mean the error message is still there after you delete these two things?
drunken005
commented
4 years ago It is not the first time, because the first time in the synchronization process, when numPending=1, the process automatically exits, the above error occurs when the script is run again.
could you please share the error on first-time synchronization? That's the root cause I think. The error message you paste is just because the second time you start the broken node, some legacy data make it failed to start.
E[2019-07-31|18:47:33.919] Connection failed @ recvRoutine (reading byte) module=p2p peer=8825b32e3abec71d772abf009ba1956d452be1fa@3.92.86.128:27146 conn=MConn{3.92.86.128:27146} err=EOF
E[2019-07-31|18:47:33.919] Stopping peer for error module=p2p peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}" err=EOF
I[2019-07-31|18:47:33.929] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:33.929] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:34.429] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:34.639] added a peer to state sync reactor module=statesync peer=8825b32e3abec71d772abf009ba1956d452be1fa
E[2019-07-31|18:47:34.639] Connection failed @ recvRoutine (reading byte) module=p2p peer=8825b32e3abec71d772abf009ba1956d452be1fa@3.92.86.128:27146 conn=MConn{3.92.86.128:27146} err=EOF
E[2019-07-31|18:47:34.639] Stopping peer for error module=p2p peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}" err=EOF
I[2019-07-31|18:47:34.649] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:34.649] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:34.741] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{3.81.223.15:27146} 2726550182cbc5f4618c27e49c730752a96901e8 out}"
I[2019-07-31|18:47:35.172] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:35.350] added a peer to state sync reactor module=statesync peer=8825b32e3abec71d772abf009ba1956d452be1fa
E[2019-07-31|18:47:35.350] Connection failed @ recvRoutine (reading byte) module=p2p peer=8825b32e3abec71d772abf009ba1956d452be1fa@3.92.86.128:27146 conn=MConn{3.92.86.128:27146} err=EOF
E[2019-07-31|18:47:35.350] Stopping peer for error module=p2p peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}" err=EOF
I[2019-07-31|18:47:35.360] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:35.360] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:35.919] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:36.068] added a peer to state sync reactor module=statesync peer=8825b32e3abec71d772abf009ba1956d452be1fa
E[2019-07-31|18:47:36.068] Connection failed @ recvRoutine (reading byte) module=p2p peer=8825b32e3abec71d772abf009ba1956d452be1fa@3.92.86.128:27146 conn=MConn{3.92.86.128:27146} err=EOF
E[2019-07-31|18:47:36.068] Stopping peer for error module=p2p peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}" err=EOF
I[2019-07-31|18:47:36.078] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:36.078] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:36.639] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:36.835] processing chunk module=statesync peer=8c379d4d3b9995c712665dc9a9414dbde5b30483 resp="[bcChunkResponseMessage ff229ac2add8be2bbfb2b67cce1f44f1a4d10540bf220613530aaaacdf9a6d45]" numPending=1
I[2019-07-31|18:47:36.838] start write recovery chunk module=statesync isComplete=true hash=ff229ac2add8be2bbfb2b67cce1f44f1a4d10540bf220613530aaaacdf9a6d45 startIdx=1384230 numOfNodes=1 chunkCompletion=2
I[2019-07-31|18:47:36.838] finished write recovery chunk module=statesync isComplete=true hash=ff229ac2add8be2bbfb2b67cce1f44f1a4d10540bf220613530aaaacdf9a6d45 startIdx=1384230 numOfNodes=1 chunkCompletion=2
I[2019-07-31|18:47:36.903] added a peer to state sync reactor module=statesync peer=8825b32e3abec71d772abf009ba1956d452be1fa
E[2019-07-31|18:47:36.903] Connection failed @ recvRoutine (reading byte) module=p2p peer=8825b32e3abec71d772abf009ba1956d452be1fa@3.92.86.128:27146 conn=MConn{3.92.86.128:27146} err=EOF
E[2019-07-31|18:47:36.903] Stopping peer for error module=p2p peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}" err=EOF
I[2019-07-31|18:47:36.913] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:36.913] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:37.350] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:37.886] added a peer to state sync reactor module=statesync peer=aea74b16d28d06cbfbb1179c177e8cd71315cce4
E[2019-07-31|18:47:37.960] Connection failed @ recvRoutine (reading byte) module=p2p peer=aea74b16d28d06cbfbb1179c177e8cd71315cce4@52.199.237.19:27146 conn=MConn{52.199.237.19:27146} err=EOF
E[2019-07-31|18:47:37.961] Stopping peer for error module=p2p peer="Peer{MConn{52.199.237.19:27146} aea74b16d28d06cbfbb1179c177e8cd71315cce4 out}" err=EOF
I[2019-07-31|18:47:37.967] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{52.199.237.19:27146} aea74b16d28d06cbfbb1179c177e8cd71315cce4 out}"
I[2019-07-31|18:47:37.967] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{52.199.237.19:27146} aea74b16d28d06cbfbb1179c177e8cd71315cce4 out}"
I[2019-07-31|18:47:38.068] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:38.903] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{3.92.86.128:27146} 8825b32e3abec71d772abf009ba1956d452be1fa out}"
I[2019-07-31|18:47:39.886] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{52.199.237.19:27146} aea74b16d28d06cbfbb1179c177e8cd71315cce4 out}"At this point the program exits
ackratos
commented
4 years ago At this point the program exits
cannot see the exact error. Did you capture stdout/stderr output? If you run your node by nohup in background. There should be a nohup.log file.
Could you please check whether there is any panic stack trace?
drunken005
commented
4 years ago Yes, it's nohup run, This information is from nohup's log file. @ackratos This is the last log after the process exits
drunken005
commented
4 years ago At this point the program exits
cannot see the exact error. Did you capture stdout/stderr output? If you run your node by
nohupin background. There should be a nohup.log file. Could you please check whether there is any panic stack trace?
I see nohup.out
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x120364e, 0x16)
/usr/local/go/src/runtime/panic.go:617 +0x72
runtime.sysMap(0xc1f4000000, 0x18000000, 0x2090db8)
/usr/local/go/src/runtime/mem_linux.go:170 +0xc7
runtime.(*mheap).sysAlloc(0x2077c00, 0x14e24000, 0x2077c10, 0xa712)
/usr/local/go/src/runtime/malloc.go:633 +0x1cd
runtime.(*mheap).grow(0x2077c00, 0xa712, 0x0)
/usr/local/go/src/runtime/mheap.go:1222 +0x42
runtime.(*mheap).allocSpanLocked(0x2077c00, 0xa712, 0x2090dc8, 0x20301a00000000)
/usr/local/go/src/runtime/mheap.go:1150 +0x37f
runtime.(*mheap).alloc_m(0x2077c00, 0xa712, 0x101, 0x7f0ce2722fff)
/usr/local/go/src/runtime/mheap.go:977 +0xc2
runtime.(*mheap).alloc.func1()
/usr/local/go/src/runtime/mheap.go:1048 +0x4c
runtime.(*mheap).alloc(0x2077c00, 0xa712, 0x7f0d03000101, 0x7f0cfdf26e70)
/usr/local/go/src/runtime/mheap.go:1047 +0x8a
runtime.largeAlloc(0x14e24000, 0xffffffffffff0100, 0x42)
/usr/local/go/src/runtime/malloc.go:1055 +0x99
runtime.mallocgc.func1()
/usr/local/go/src/runtime/malloc.go:950 +0x46
runtime.systemstack(0x0)
/usr/local/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1153
cd /data
git lfs clone https://github.com/binance-chain/node-binary.git
cp -rf /data/node-binary/fullnode/testnet/0.6.1/config /data/bnb/testnet
nohup /data/node-binary/fullnode/testnet/0.6.1/linux/bnbchaind start --home /data/bnb/testnet &
```bashwhat's your current memory, please try to increase your memory to above 16G. If the problem still exists, feel free to re-open this issue
run:
return error message:
logs:
config.toml
app.toml
How to fix this error？cc @HaoyangLiu