search

bnb-chain / bsc

A BNB Smart Chain client based on the go-ethereum fork
GNU Lesser General Public License v3.0
2.75k stars 1.57k forks source link

Deep froze chain segment takes a long long time, the node is stopped there #1684

Closed chengjiao closed 1 year ago

chengjiao commented 1 year ago

each time when the nodes runs for 3 days or so, it will step into "Deep froze chain segment", and stuck there for about 2h-3h. it can be recovered each time, but normally I cannot wait, so ctrl+c and start again.

what does this mean? my hardware is 6000 iops, the througput is 350, is this enough?

and is it possible to have a automatic script to stop it every 1h, then start it ?

chengjiao commented 1 year ago

and how to tune mgasps value to be above 50?

currently the vaule is 17 on my node

chengjiao commented 1 year ago

adding the log when repro, the time is 06-09|07:13:10 for now, but the log still exists at 06-09|06:41:17

INFO [06-09|06:40:09.961] Deep froze chain segment blocks=19 elapsed=1.295ms number=28,853,498 hash=0bf8ef..7da98e

ERROR[06-09|06:41:08.613] Expired request does not exist peer=be438d4fa38fe9c6685d9d8470bd7d61e667d665f9252cdf71cd357dc62d761f

ERROR[06-09|06:41:09.365] Expired request does not exist peer=be4541c38403788834ee91db1809036ea168bc56224769a2416670e943e7af8c

ERROR[06-09|06:41:09.406] Expired request does not exist peer=46f42ccd36cb51901ca9b3ab5865b99cf3342ec5ff348001d83e4b0f3b0bab54

ERROR[06-09|06:41:09.452] Expired request does not exist peer=beeeb2192a524dd96ebcb4e788c8770bdde8265184ad9285a7e9947b5ff1d208

ERROR[06-09|06:41:09.738] Expired request does not exist peer=be3a6e6bc4579631701309838c80736594311155c07b66c0ff8f27a3b2ea0bfa

INFO [06-09|06:41:17.689] Downloader queue stats receiptTasks=0 blockTasks=5 itemSize=51.41KiB throttle=5100

WARN [06-09|06:41:17.689] Synchronisation failed, dropping peer peer=46f42ccd36cb51901ca9b3ab5865b99cf3342ec5ff348001d83e4b0f3b0bab54 err="no peers available or all tried for download"

MatusKysel commented 1 year ago

Deep froze chain segment has nothing to do with your issue. It seems you are missing peers to sync up with no peers available or all tried for download. Maybe you have connectivity problems?

0xLyon commented 1 year ago

@chengjiao As per @MatusKysel comment, the mentioned "Deep froze chain segment" is part of the node's normal operations and only informational on data being written to files from the DB.

Please kind try the below steps for peers related issues:

Ensure you are using the latest release from here: https://github.com/bnb-chain/bsc/releases

chengjiao commented 1 year ago

thanks for the suggestion, I've tried the steps you mentioned and restart the node for now. I will check the status after several hours.

chengjiao commented 1 year ago

the node has recovered back to normal, thx very much.

But another issue, I am streaming the debug trace of the block, sometimes I get the slow rpc call,

debug_traceBlockByNumber 29040054 2.0715224742889404s debug_traceBlockByNumber 29040056 4.136905193328857s debug_traceBlockByNumber 29040057 1.452056884765625s

normally the time consumpted should be within 1s, less than 1s.

What I should do to improve the speed to get block trace? The EBS is 6000iops, and the throughput is 350.

MatusKysel commented 1 year ago

@chengjiao https://github.com/bnb-chain/bsc/issues/1664 please follow this issue. We need more time to investigate. if you have more logs or anything pleas share it there