search

node-real / bsc-erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
118 stars 44 forks source link

Cannot synchronize blocks. v1.2.10 #413

Closed rayn316 closed 1 week ago

rayn316 commented 1 week ago

System information

Erigon version: v1.2.10

OS & Version: Linux

Commit hash:

Erigon Command (with flags/config):

Consensus Layer:

Consensus Layer Command (with flags/config):

Chain/Network:

Expected behaviour

Actual behaviour

Logs of different nodes getting stuck are divided into different sections.

Jun 20 18:27:30 erigon[1642960]: [INFO] [06-20|18:27:30.718] [2/15 Headers] No block headers to write in this log period number=39774898 blk/second=0.000 alloc=3.7GB sys=5.9GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:37:40 erigon[3379155]: [INFO] [06-20|18:37:40.716] [2/15 Headers] No block headers to write in this log period number=39774849 blk/second=0.000 alloc=4.7GB sys=10.9GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:37:40 erigon[3379155]: [INFO] [06-20|18:37:40.716] Req/resp stats                           req=6 reqMin=39774930 reqMax=39775122 skel=29 skelMin=39774848 skelMax=39775040 resp=0 respMin=0 respMax=0 dups=0
Jun 20 18:38:10 erigon[3379155]: [INFO] [06-20|18:38:10.717] [2/15 Headers] No block headers to write in this log period number=39774849 blk/second=0.000 alloc=5.7GB sys=10.9GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:38:10 erigon[3379155]: [INFO] [06-20|18:38:10.717] Req/resp stats                           req=5 reqMin=39774930 reqMax=39775140 skel=29 skelMin=39774848 skelMax=39775040 resp=1 respMin=39775141 respMax=39775141 dups=0
Jun 20 18:21:33 erigon[683556]: [INFO] [06-20|18:21:33.393] [2/15 Headers] Wrote block headers       number=39774588 blk/second=6.333 alloc=6.0GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:22:03 erigon[683556]: [INFO] [06-20|18:22:03.392] [2/15 Headers] Wrote block headers       number=39774775 blk/second=6.233 alloc=5.1GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:22:33 erigon[683556]: [INFO] [06-20|18:22:33.392] [2/15 Headers] No block headers to write in this log period number=39774775 blk/second=0.000 alloc=6.5GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:23:03 erigon[683556]: [INFO] [06-20|18:23:03.392] [2/15 Headers] No block headers to write in this log period number=39774775 blk/second=0.000 alloc=4.9GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:23:06 erigon[683556]: [WARN] [06-20|18:23:06.707] [rpc] served                             conn=194.67.206.45:33272 method=eth_coinbase reqid=2 t=44.533µs err="etherbase must be explicitly specified"
Jun 20 18:23:06 erigon[683556]: [WARN] [06-20|18:23:06.722] [rpc] served                             conn=185.87.48.4:49204 method=eth_coinbase reqid=2 t=29.184µs err="etherbase must be explicitly specified"
Jun 20 18:23:33 erigon[683556]: [INFO] [06-20|18:23:33.392] [2/15 Headers] No block headers to write in this log period number=39774775 blk/second=0.000 alloc=6.6GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:24:03 erigon[683556]: [INFO] [06-20|18:24:03.392] [2/15 Headers] No block headers to write in this log period number=39774775 blk/second=0.000 alloc=4.2GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:24:15 erigon[683556]: [INFO] [06-20|18:24:15.981] [p2p] GoodPeers                          eth66=61 eth67=18
Jun 20 18:24:17 erigon[683556]: [INFO] [06-20|18:24:17.523] [txpool] stat                            pending=31859 baseFee=0 queued=15476 alloc=6.2GB sys=11.8GB
Jun 20 18:24:22 erigon[683556]: [INFO] [06-20|18:24:22.214] [mem] memory stats                       Rss=51.8GB Size=0B Pss=51.8GB SharedClean=6.0MB SharedDirty=0B PrivateClean=36.0GB PrivateDirty=15.8GB Referenced=46.9GB Anonymous=15.8GB Swap=1.2GB alloc=6.8GB sys=11.8GB
Jun 20 18:24:33 erigon[683556]: [INFO] [06-20|18:24:33.392] [2/15 Headers] Wrote block headers       number=39774823 blk/second=1.600 alloc=5.5GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:25:03 erigon[683556]: [INFO] [06-20|18:25:03.392] [2/15 Headers] No block headers to write in this log period number=39774823 blk/second=0.000 alloc=4.3GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:25:06 erigon[683556]: [WARN] [06-20|18:25:06.332] [rpc] served                             conn=185.87.48.4:37212 method=eth_coinbase reqid=2 t=43.972µs err="etherbase must be explicitly specified"
Jun 20 18:25:06 erigon[683556]: [WARN] [06-20|18:25:06.390] [rpc] served                             conn=194.67.206.50:55440 method=eth_coinbase reqid=2 t=50.745µs err="etherbase must be explicitly specified"
Jun 20 18:25:33 erigon[683556]: [INFO] [06-20|18:25:33.392] [2/15 Headers] No block headers to write in this log period number=39774823 blk/second=0.000 alloc=4.1GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:26:03 erigon[683556]: [INFO] [06-20|18:26:03.392] [2/15 Headers] No block headers to write in this log period number=39774823 blk/second=0.000 alloc=5.6GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:26:33 erigon[683556]: [INFO] [06-20|18:26:33.392] [2/15 Headers] No block headers to write in this log period number=39774823 blk/second=0.000 alloc=3.9GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:27:03 erigon[683556]: [INFO] [06-20|18:27:03.392] [2/15 Headers] No block headers to write in this log period number=39774823 blk/second=0.000 alloc=4.1GB sys=11.8GB invalidHeaders=0 rejectedBadHeaders=0
Jun 20 18:27:06 erigon[683556]: [WARN] [06-20|18:27:06.879] [rpc] served                             conn=194.67.206.43:34844 method=eth_coinbase reqid=2 t=34.845µs err="etherbase must be explicitly specified"
Jun 20 18:27:06 erigon[683556]: [WARN] [06-20|18:27:06.924] [rpc] served                             conn=194.67.206.50:37756 method=eth_coinbase reqid=2 t=32.44µs err="etherbase must be explicitly specified"
Jun 20 18:27:15 erigon[683556]: [INFO] [06-20|18:27:15.981] [p2p] GoodPeers                          eth66=60 eth67=18
Jun 20 18:27:17 erigon[683556]: [INFO] [06-20|18:27:17.517] [txpool] stat                            pending=33876 baseFee=0 queued=15941 alloc=5.7GB sys=11.8GB
Jun 20 18:27:21 erigon[683556]: [INFO] [06-20|18:27:21.925] [mem] memory stats                       Rss=50.5GB Size=0B Pss=50.5GB SharedClean=6.1MB SharedDirty=0B PrivateClean=34.7GB PrivateDirty=15.8GB Referenced=45.6GB Anonymous=15.8GB Swap=1.2GB alloc=6.9GB sys=11.8GB

Steps to reproduce the behaviour

Backtrace

[backtrace]

image

rayn316 commented 1 week ago

Synchronization will take place after rebooting the node.

rayn316 commented 1 week ago

It seems that it synchronizes after a restart, but it gets stuck again after completing the synchronization once. image

blxdyx commented 1 week ago

Show start command

rayn316 commented 1 week ago

Show start command

/node/archive/bsc/bin/erigon --datadir=/node/archive/bsc/erigon/ --chain=bsc --prune=disabled --db.pagesize=16k --db.size.limit=16TB --rpc.gascap=600000000 --rpc.evmtimeout=30s --rpc.batch.limit=100 --rpc.returndata.limit=104857600 --txpool.globalslots 1000000 --txpool.globalbasefeeslots 3000000 --txpool.globalqueue 3000000 --txpool.pricelimit=1 --http=true --http.api eth,net,web3,debug,trace,txpool --http.addr 0.0.0.0 --http.corsdomain '*' --http.vhosts '*' --ws  --authrpc.addr 127.0.0.1 --private.api.addr='' --maxpeers=300 --bodies.cache 6442450944 --batchSize=1024M --torrent.download.slots=10 --torrent.download.rate=100mb
blxdyx commented 1 week ago

remove --maxpeers

rayn316 commented 1 week ago

Thank you for your help. The nodes have synchronized. I will continue to monitor the situation.

durch commented 1 week ago

@blxdyx we have four nodes, all of them stuck around the same block height, this has been syncing fine until about 3 hrs ago, and was keeping up with the head without problems. Its with No block bodies to write in this log period block number=3977856. Similar range on all of them, restart of one node did not seem to help, attaching our config below:


chain = "bsc"                                                                                                                                                                                                                       
datadir = "/bsc/data/erigon"                                                                                                                                                                                                        
"db.pagesize" = "16k"                                                                                                                                                                                                               
"db.size.limit" = "10TB"                                                                                                                                                                                                                                                                                                                                                                                                                                         
snapshots = true                                                                                                                                                                                                                    
"txpool.disable" = true                                                                                                                                                                                                                                                                                                                                                                                                                                     
port = 30303                                                                                                                                                                                                                        
"p2p.allowed-ports" = [30303, 30304, 30305, 30306, 30307]                                                                                                                                                                           
"p2p.protocol" = 66 
"authrpc.port" = 8551                                                                                                                                                                                                                                                                                                                                                                                                                                           
http = true                                                                                                                                                                                                                         "http.port" = 8545                                                                                                                                                                                                                  
"http.addr" = "0.0.0.0"                                                                                                                                                                                                             
"http.api" = "eth,erigon,web3,net,trace,debug"                                                                                                                                                                                      
"http.corsdomain" = "*"                                                                                                                                                                                                             
"http.vhosts" = "*"                                                                                                                                                                                                                 
"http.compression" = true                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
metrics = true                                                                                                                                                                                                                      
"metrics.port" = 6060                                                                                                                                                                                                               
"metrics.addr" = "0.0.0.0"                                                                                                                                                                                                                                                                                                                                                                                                                      
"private.api.addr" = "127.0.0.1:9090"                                                                                                                                                                                                                                                                                                                                                                                                                      
"torrent.port" = 42069                                                                                                                                                                                                              
"torrent.download.rate" = "512mb"                                                                                                                                                                                                  
"torrent.upload.rate" = "512mb"
durch commented 1 week ago

Looks like there is some progress on one of the nodes, lets see if it recoveres

durch commented 1 week ago

Happy to report that all are nodes have finally synced, altough it would be good to understand the cause of the delay

blxdyx commented 1 week ago

remove this: p2p.protocol= 66 , bsc only support 68 now