search

node-real / bsc-erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
124 stars 42 forks source link

err="DumpBlocks and No block headers to write in this log period issue #316

Closed danlix2000 closed 6 months ago

danlix2000 commented 7 months ago

OS = Ubuntu 22.04.4 LTS Erigon v1.1.13 used snapshots - erigon_data_20240205.lz4.xxx node has been syncing for more than 1 week allowed below ports 4000 , 4001 - TCP & UDP 30303 , 30304 - TCP & UDP 42069 , TCP & UDP

Node CPU - 32 , RAM = 256 GB Hard disk - 15 TB - NVME

The first issue is

[snapshots] retire blocks err="DumpBlocks: DumpHeaders: header missed in db: block_num=35000000, hash=dba069f603db6f3bb4f3335eb3784fe33e1e3f22f852c56a102021469e72330b" fromBlock=35000000 toBlock=35500000

this is logs log file.

[INFO] [03-07|05:59:50.699] Served conn=127.0.0.1:42098 t=68.071µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|05:59:52.306] [7/15 Execution] Executed blocks number=36752697 blk/s=3.2 tx/s=560.5 Mgas/s=57.9 gasState=0.00 batch=3.9MB alloc=4.3GB sys=8.1GB [INFO] [03-07|06:00:16.215] [7/15 Execution] Completed on block=36752754 [INFO] [03-07|06:00:16.215] [7/15 Execution] DONE in=1m4.119927628s [INFO] [03-07|06:00:16.215] [8/15 HashState] Promoting plain state from=36752570 to=36752754 [INFO] [03-07|06:00:16.215] [8/15 HashState] Incremental promotion from=36752570 to=36752754 codes=true csbucket=AccountChangeSet [INFO] [03-07|06:00:18.406] [8/15 HashState] Incremental promotion from=36752570 to=36752754 codes=false csbucket=AccountChangeSet [INFO] [03-07|06:00:36.455] [8/15 HashState] Incremental promotion from=36752570 to=36752754 codes=false csbucket=StorageChangeSet [INFO] [03-07|06:00:50.999] Served conn=127.0.0.1:42098 t=59.195µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|06:00:52.105] [p2p] GoodPeers eth66=1 [INFO] [03-07|06:01:06.986] [8/15 HashState] ETL [2/2] Loading into=HashedStorage current_prefix=e9dae3d7 [INFO] [03-07|06:01:09.210] [9/15 IntermediateHashes] Generating intermediate hashes from=36752570 to=36752754 [INFO] [03-07|06:01:39.694] [9/15 IntermediateHashes] Calculating Merkle root current key=a729fe74 [INFO] [03-07|06:01:51.301] Served conn=127.0.0.1:42098 t=92.739µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|06:02:07.764] [10/15 CallTraces] Pruned call trace intermediate table from=36662570 to=36662753 [INFO] [03-07|06:02:51.603] Served conn=127.0.0.1:42098 t=80.575µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|06:02:52.104] [p2p] GoodPeers eth66=1 [INFO] [03-07|06:03:27.954] [12/15 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=93c2260e [INFO] [03-07|06:03:39.956] [13/15 LogIndex] processing from=36752571 to=36752754 [INFO] [03-07|06:03:51.905] Served conn=127.0.0.1:42098 t=69.473µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|06:04:37.470] [14/15 TxLookup] ETL [2/2] Loading into=BlockTransactionLookup current_prefix=9c649525 [INFO] [03-07|06:04:52.104] [p2p] GoodPeers eth66=1 [INFO] [03-07|06:04:52.205] Served conn=127.0.0.1:42098 t=67.53µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|06:05:02.713] RPC Daemon notified of new headers from=36752570 to=36752754 hash=0xf0d20ed86282ea9cfed66fcf8b03e69506712f0e9da42b3b2cff790c7f9459bd header sending=255.563µs log sending=261ns [INFO] [03-07|06:05:02.713] Commit cycle in=6.553260472s [INFO] [03-07|06:05:02.713] Timings (slower than 50ms) Headers=52.692s BlockHashes=317ms Bodies=1m37.586s Senders=104ms Execution=1m4.119s HashState=52.994s IntermediateHashes=58.277s CallTraces=29.07s AccountHistoryIndex=21.157s StorageHistoryIndex=42.241s LogIndex=27.345s TxLookup=48.857s [INFO] [03-07|06:05:02.713] Tables PlainState=293.9GB AccountChangeSet=677.4GB StorageChangeSet=1.5TB BlockTransaction=59.0GB TransactionLog=2.8TB FreeList=167.4MB ReclaimableSpace=669.6GB [WARN] [03-07|06:05:02.715] [snapshots] retire blocks err="DumpBlocks: DumpHeaders: header missed in db: block_num=35000000, hash=dba069f603db6f3bb4f3335eb3784fe33e1e3f22f852c56a102021469e72330b" fromBlock=35000000 toBlock=35500000 [INFO] [03-07|06:05:37.768] [2/15 Headers] Waiting for headers... from=36752754 [INFO] [03-07|06:05:52.508] Served conn=127.0.0.1:42098 t=53.343µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|06:05:57.769] [2/15 Headers] Wrote block headers number=36752826 blk/second=3.600 alloc=3.3GB sys=8.1GB [INFO] [03-07|06:06:17.769] [2/15 Headers] Wrote block headers number=36752894 blk/second=3.400 alloc=3.4GB sys=8.1GB [INFO] [03-07|06:06:29.312] [2/15 Headers] Processed highest inserted=36752934 age=53s [INFO] [03-07|06:06:29.615] [5/15 Bodies] Processing bodies... from=36752754 to=36752934 [INFO] [03-07|06:06:49.616] [5/15 Bodies] Downloading block bodies block_num=36752809 delivery/sec=401.4KB wasted/sec=0B remaining=125 delivered=107 cache=3.9MB alloc=3.6GB sys=8.1GB [INFO] [03-07|06:06:52.105] [p2p] GoodPeers eth66=1 [INFO] [03-07|06:06:52.808] Served conn=127.0.0.1:42098 t=81.226µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|06:07:09.616] [5/15 Bodies] Downloading block bodies block_num=36752827 delivery/sec=82.9KB wasted/sec=0B remaining=107 delivered=125 cache=3.9MB alloc=3.0GB sys=8.1GB

other issue is - No block headers

[INFO] [03-07|08:54:43.162] Served conn=127.0.0.1:42098 t=81.829µs method=eth_syncing reqid=1 params=[] [INFO] [03-07|08:54:49.194] [2/15 Headers] No block headers to write in this log period block number=36753898 [INFO] [03-07|08:54:49.194] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-07|08:54:52.104] [p2p] GoodPeers [INFO] [03-07|08:55:09.194] [2/15 Headers] No block headers to write in this log period block number=36753898 [INFO] [03-07|08:55:09.194] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0

service file

[Unit] Description=BSC Execution Client (Mainnet) After=network.target Wants=network.target [Service] User=ubuntu Group=root Type=simple Restart=on-failure TimeoutStopSec=120 RestartSec=60 LimitNOFILE=32768 ExecStart=/home/ubuntu/erigon/build/bin/erigon --datadir=/data/erigon --sentry.drop-useless-peers --p2p.protocol=66 --p2p.allowed-ports=30303,30304 --txpool.disable --rpc.gascap=150000000 --chain=bsc --http --state.cache=4000000 --http.addr=localhost --http.port=8545 --http.api=eth,erigon,web3,net,debug,trace,txpool,engine --authrpc.port=8551 --http.trace --private.api.addr=127.0.0.1:9090 --port=30303 --http.corsdomain= --rpc.batch.concurrency=4 --authrpc.addr 127.0.0.1 --batchSize=512M --torrent.download.rate=5000mb --snapshots=true --snap.keepblocks=true --ws --ws.compression --http.compression --http.vhosts= --torrent.port=42069 --authrpc.jwtsecret=/var/lib/jwtsecret/jwt.hex --internalcl=false --authrpc.vhosts=* --authrpc.addr=localhost --metrics --trace.maxtraces 400 --rpc.returndata.limit 300000 --rpc.batch.limit 10000 --db.pagesize=16k --nat=extip:x.x.x.x --bootnodes=enode://433c8bfdf53a3e2268ccb1b829e47f629793291cbddf0c76ae626da802f90532251fc558e2e0d10d6725e759088439bf1cd4714716b03a259a35d4b2e4acfa7f@52.69.102.73:30311,enode://571bee8fb902a625942f10a770ccf727ae2ba1bab2a2b64e121594a99c9437317f6166a395670a00b7d93647eacafe598b6bbcef15b40b6d1a10243865a3e80f@35.73.84.120:30311,enode://fac42fb0ba082b7d1eebded216db42161163d42e4f52c9e47716946d64468a62da4ba0b1cac0df5e8bf1e5284861d757339751c33d51dfef318be5168803d0b5@18.203.152.54:30311,enode://3063d1c9e1b824cfbb7c7b6abafa34faec6bb4e7e06941d218d760acdd7963b274278c5c3e63914bd6d1b58504c59ec5522c56f883baceb8538674b92da48a96@34.250.32.100:30311,enode://ad78c64a4ade83692488aa42e4c94084516e555d3f340d9802c2bf106a3df8868bc46eae083d2de4018f40e8d9a9952c32a0943cd68855a9bc9fd07aac982a6d@34.204.214.24:30311

[Install] WantedBy=default.target

I have another node with --downloader.verify flag but the same issue happening there also.

Still, the node is not synced to the head.

blxdyx commented 7 months ago

The reason is erigon will move block data from db to snapshots. The minimal block in your db is bigger than the max in snapshots. Use the latest release. Or download the latest snapshots. https://github.com/bnb-chain/bsc-snapshots/blob/main/README.md#erigon_snapshots_20240301tarlz4-md504238a8fb079904f4f1c8a9862054d7e If don't work , try stage_headers reset.

danlix2000 commented 7 months ago

upgraded to the latest and did a stage_headers reset. still did not see err="DumpBlocks: DumpHeaders error

but still shows [2/15 Headers] No block headers to write in this log period block number=36902300

blxdyx commented 7 months ago

Show more logs

danlix2000 commented 6 months ago

Mar 14 17:57:12 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.247] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:12 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.248] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:12 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.249] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:12 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.250] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:12 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.251] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:12 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.253] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:12 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.255] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:38 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.256] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:38 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:16.257] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:38 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:17.069] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:38 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:17.073] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:38 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:17.078] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:38 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:17.080] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:38 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|17:55:17.081] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 17:57:38 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|17:55:57.665] [2/15 Headers] No block headers to write in this log period block number=36958229 Mar 14 18:03:25 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|17:56:26.579] Req/resp stats req=1346 reqMin=36958042 reqMax=36968215 skel=9 skelMin=36958228 skelMax=36958420 resp=8132 respMin=34143908 respMax=36968216 dups=590176 Mar 14 18:03:47 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|17:56:34.899] [p2p] GoodPeers eth66=14 Mar 14 18:04:30 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:04:01.128] [p2p] GoodPeers eth66=2 Mar 14 18:49:17 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:14:03.447] [p2p] GoodPeers Mar 14 18:49:32 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:49:25.768] [p2p] GoodPeers Mar 14 18:50:14 amer-wash-bsc-erigon erigon[111591]: [WARN] [03-14|18:50:02.723] [downloader] Rejected header marked as bad hash=0x38502b43d783fe713bce73bd4cdd3f428752022cde0d8888ba2fd0c7884ad2e8 height=36958230 Mar 14 18:50:30 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:50:22.125] [p2p] GoodPeers Mar 14 18:50:49 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:50:29.790] [2/15 Headers] No block headers to write in this log period block number=36958229 Mar 14 18:51:07 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:50:41.558] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=36958225 respMax=36958263 dups=5 Mar 14 18:52:10 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:52:04.244] [2/15 Headers] No block headers to write in this log period block number=36958229 Mar 14 18:52:30 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:52:16.116] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=36958264 respMax=36958300 dups=0 Mar 14 19:04:19 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|18:52:21.970] [p2p] GoodPeers Mar 14 19:04:58 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:04:44.239] [p2p] GoodPeers Mar 14 19:05:56 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:05:48.706] [2/15 Headers] No block headers to write in this log period block number=36958229 Mar 14 19:06:23 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:06:01.292] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=36958301 respMax=36958339 dups=0 Mar 14 19:06:36 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:06:24.009] [p2p] GoodPeers Mar 14 19:08:12 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:08:03.384] [2/15 Headers] No block headers to write in this log period block number=36958229 Mar 14 19:08:39 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:08:19.155] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=36958340 respMax=36958377 dups=0 Mar 14 19:08:57 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:08:22.643] [p2p] GoodPeers Mar 14 19:10:39 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:10:24.993] [p2p] GoodPeers Mar 14 19:11:11 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:10:54.343] [2/15 Headers] No block headers to write in this log period block number=36958229 Mar 14 19:11:46 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:11:13.706] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=1 respMin=36958378 respMax=36958390 dups=0 Mar 14 19:12:44 amer-wash-bsc-erigon erigon[111591]: [INFO] [03-14|19:12:22.286] [p2p] GoodPeers

blxdyx commented 6 months ago

Use the latest release.

danlix2000 commented 6 months ago

we were in 1.1.14

Now upgraded to [INFO] [03-15|03:32:44.643] Build info git_branch=HEAD git_tag=v1.1.15 git_commit=9a42b90a23dd6d1eeb5015cfea52584326ff6dcd

danlix2000 commented 6 months ago

after the upgrade. I saw this error .

[EROR] [03-15|03:50:34.773] slash validator failed block hash=0x34dcbac09f9ef399228645ea1d7fa19c00cd0111beda7dcfe76ad06f0e19d6d6 address=0xD1d6bF74282782B0b3eb1413c901D6eCF02e8e28 error="expected system tx (hash 0x5ccb2806c40744fc1c8f18a5f689b604409892ee1fa00ed3265bd3378951f806, nonce 1770018, to 0x0000000000000000000000000000000000001001, value 0x0, gas 9223372036854775807, gasPrice 0x0, data c96be4cb000000000000000000000000d1d6bf74282782b0b3eb1413c901d6ecf02e8e28), actual tx (hash 0xc95d89d4f8349caa1bc6781f771146a7d6ba9929716f38388f9f26fd64ff783f, nonce 1770018, to 0x0000000000000000000000000000000000001000, value 0x154841bf906faee, gas 9223372036854775807, gasPrice 0x0, data f340fa01000000000000000000000000685b1ded8013785d6623cc18d214320b6bb64759)"

more logs:

[INFO] [03-15|03:47:44.347] [5/15 Bodies] Downloading block bodies block_num=36964149 delivery/sec=15.8MB wasted/sec=0B remaining=15855 delivered=5966 cache=6.6MB alloc=3.9GB sys=5.4GB [INFO] [03-15|03:48:04.345] [5/15 Bodies] Downloading block bodies block_num=36967875 delivery/sec=16.6MB wasted/sec=0B remaining=12129 delivered=9705 cache=5.3MB alloc=3.1GB sys=6.0GB [INFO] [03-15|03:48:24.345] [5/15 Bodies] Downloading block bodies block_num=36969522 delivery/sec=17.0MB wasted/sec=0B remaining=10482 delivered=15369 cache=232.1MB alloc=4.8GB sys=6.0GB [INFO] [03-15|03:48:44.345] [5/15 Bodies] Downloading block bodies block_num=36974333 delivery/sec=14.9MB wasted/sec=0B remaining=5715 delivered=20591 cache=9.1MB alloc=4.1GB sys=6.0GB [INFO] [03-15|03:49:04.346] [5/15 Bodies] Downloading block bodies block_num=36974371 delivery/sec=16.5MB wasted/sec=0B remaining=5633 delivered=25624 cache=256.0MB alloc=3.1GB sys=6.1GB [INFO] [03-15|03:49:23.523] [p2p] GoodPeers eth66=54 [INFO] [03-15|03:49:24.345] [5/15 Bodies] Downloading block bodies block_num=36978953 delivery/sec=4.1MB wasted/sec=0B remaining=1051 delivered=26716 cache=37.0MB alloc=3.6GB sys=6.1GB [INFO] [03-15|03:49:44.345] [5/15 Bodies] Downloading block bodies block_num=36978953 delivery/sec=1.8MB wasted/sec=0B remaining=1051 delivered=27068 cache=73.8MB alloc=3.9GB sys=6.1GB [INFO] [03-15|03:50:09.408] [5/15 Bodies] Processed highest=36980005 [INFO] [03-15|03:50:09.408] [5/15 Bodies] DONE in=3m5.064391617s [INFO] [03-15|03:50:09.409] [6/15 Senders] Started from=36958229 to=36980005 [INFO] [03-15|03:50:27.695] [7/15 Execution] Blocks execution from=36958229 to=36980005 [EROR] [03-15|03:50:34.773] slash validator failed block hash=0x34dcbac09f9ef399228645ea1d7fa19c00cd0111beda7dcfe76ad06f0e19d6d6 address=0xD1d6bF74282782B0b3eb1413c901D6eCF02e8e28 error="expected system tx (hash 0x5ccb2806c40744fc1c8f18a5f689b604409892ee1fa00ed3265bd3378951f806, nonce 1770018, to 0x0000000000000000000000000000000000001001, value 0x0, gas 9223372036854775807, gasPrice 0x0, data c96be4cb000000000000000000000000d1d6bf74282782b0b3eb1413c901d6ecf02e8e28), actual tx (hash 0xc95d89d4f8349caa1bc6781f771146a7d6ba9929716f38388f9f26fd64ff783f, nonce 1770018, to 0x0000000000000000000000000000000000001000, value 0x154841bf906faee, gas 9223372036854775807, gasPrice 0x0, data f340fa01000000000000000000000000685b1ded8013785d6623cc18d214320b6bb64759)" [INFO] [03-15|03:50:49.938] [7/15 Execution] Executed blocks number=36958236 blk/s=0.3 tx/s=74.7 Mgas/s=5.3 gasState=0.00 batch=150.2KB alloc=4.2GB sys=6.6GB [INFO] [03-15|03:51:08.774] [7/15 Execution] Executed blocks number=36958245 blk/s=0.5 tx/s=99.0 Mgas/s=9.4 gasState=0.00 batch=424.0KB alloc=4.4GB sys=6.6GB [INFO] [03-15|03:51:23.523] [p2p] GoodPeers eth66=67 [INFO] [03-15|03:51:29.719] [7/15 Execution] Executed blocks number=36958260 blk/s=0.7 tx/s=138.9 Mgas/s=13.0 gasState=0.00 batch=905.0KB alloc=4.6GB sys=6.6GB [INFO] [03-15|03:51:48.079] [7/15 Execution] Executed blocks number=36958277 blk/s=0.9 tx/s=182.9 Mgas/s=16.2 gasState=0.00 batch=1.5MB alloc=4.9GB sys=6.6GB [INFO] [03-15|03:52:08.489] [7/15 Execution] Executed blocks number=36958298 blk/s=1.0 tx/s=195.6 Mgas/s=19.2 gasState=0.00 batch=2.3MB alloc=5.1GB sys=6.6GB [INFO] [03-15|03:52:28.415] [7/15 Execution] Executed blocks number=36958319 blk/s=1.1 tx/s=217.4 Mgas/s=20.3 gasState=0.00 batch=3.1MB alloc=2.8GB sys=6.6GB [INFO] [03-15|03:52:48.126] [7/15 Execution] Executed blocks number=36958344 blk/s=1.3 tx/s=263.9 Mgas/s=25.9 gasState=0.00 batch=4.1MB alloc=3.2GB sys=6.6GB [INFO] [03-15|03:53:08.081] [7/15 Execution] Executed blocks number=36958377 blk/s=1.7 tx/s=317.6 Mgas/s=31.0 gasState=0.00 batch=5.5MB alloc=3.6GB sys=6.6GB [INFO] [03-15|03:53:23.523] [p2p] GoodPeers eth66=68 [INFO] [03-15|03:53:45.379] [7/15 Execution] Executed blocks number=36958400 blk/s=0.6 tx/s=114.9 Mgas/s=12.1 gasState=0.00 batch=6.5MB alloc=2.8GB sys=6.7GB [INFO] [03-15|03:53:48.162] [7/15 Execution] Executed blocks number=36958405 blk/s=1.8 tx/s=385.6 Mgas/s=40.0 gasState=0.00 batch=6.8MB alloc=2.8GB sys=6.7GB [INFO] [03-15|03:54:07.723] [7/15 Execution] Executed blocks number=36958441 blk/s=1.8 tx/s=353.6 Mgas/s=35.2 gasState=0.00 batch=8.3MB alloc=3.2GB sys=6.7GB

blxdyx commented 6 months ago

Yes, it's normal.

danlix2000 commented 6 months ago

we were in v1.1.15

node is synced.

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' http://localhost:8545 {"jsonrpc":"2.0","id":1,"result":false}

but blocks are lagging.

now continuously getting below issue

[INFO] [03-16|10:12:56.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:12:56.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=1 skelMin=37015956 skelMax=37016148 resp=1 respMin=34410808 respMax=34410808 dups=0 [INFO] [03-16|10:13:05.200] Served conn=127.0.0.1:53752 t=101.135µs method=eth_syncing reqid=1 params=[] [INFO] [03-16|10:13:16.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:13:16.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:13:36.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:13:36.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:13:56.581] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:13:56.581] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:14:08.526] [p2p] GoodPeers [INFO] [03-16|10:14:16.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:14:16.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:14:35.502] Served conn=127.0.0.1:50172 t=93.56µs method=eth_syncing reqid=1 params=[] [INFO] [03-16|10:14:36.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:14:36.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:14:56.581] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:14:56.581] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:15:16.581] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:15:16.581] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:15:36.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:15:36.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:15:56.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:15:56.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:16:05.805] Served conn=127.0.0.1:41756 t=75.306µs method=eth_syncing reqid=1 params=[] [INFO] [03-16|10:16:08.526] [p2p] GoodPeers [INFO] [03-16|10:16:16.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:16:16.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:16:27.981] [snapshots] Blocks Stat blocks=36600k indices=36600k alloc=3.8GB sys=9.8GB [INFO] [03-16|10:16:36.581] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:16:36.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:16:56.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:16:56.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:17:16.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:17:16.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 [INFO] [03-16|10:17:36.108] Served conn=127.0.0.1:46544 t=88.019µs method=eth_syncing reqid=1 params=[] [INFO] [03-16|10:17:36.582] [2/15 Headers] No block headers to write in this log period block number=37015957 [INFO] [03-16|10:17:36.582] Req/resp stats req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0

blxdyx commented 6 months ago

show you start command. suggest to remove: --sentry.drop-useless-peers=true

danlix2000 commented 6 months ago

this is my service file. ExecStart=/home/ubuntu/erigon/build/bin/erigon --datadir=/data/erigon --sentry.drop-useless-peers --p2p.protocol=66 --p2p.allowed-ports=30303,30304 --txpool.disable --rpc.gascap=150000000 --chain=bsc --http --state.cache=4000000 --http.addr=localhost --http.port=8545 --http.api=eth,erigon,web3,net,debug,trace,txpool,engine --authrpc.port=8551 --http.trace --private.api.addr=127.0.0.1:9090 --port=30303 --http.corsdomain=* --rpc.batch.concurrency=4 --authrpc.addr 127.0.0.1 --batchSize=512M --torrent.download.rate=5000mb --snapshots=true --snap.keepblocks=true --ws --ws.compression --http.compression --http.vhosts=* --torrent.port=42069 --authrpc.jwtsecret=/var/lib/jwtsecret/jwt.hex --internalcl=false --authrpc.vhosts=* --authrpc.addr=localhost --trace.maxtraces 400 --rpc.returndata.limit 300000 --rpc.batch.limit 10000 --db.pagesize=16k --nat=extip:x.x8.1x.x --bootnodes=enode://433c8bfdf53a3e2268ccb1b829e47f629793291cbddf0c76ae626da802f90532251fc558e2e0d10d6725e759088439bf1cd4714716b03a259a35d4b2e4acfa7f@52.69.102.73:30311,enode://571bee8fb902a625942f10a770ccf727ae2ba1bab2a2b64e121594a99c9437317f6166a395670a00b7d93647eacafe598b6bbcef15b40b6d1a10243865a3e80f@35.73.84.120:30311,enode://fac42fb0ba082b7d1eebded216db42161163d42e4f52c9e47716946d64468a62da4ba0b1cac0df5e8bf1e5284861d757339751c33d51dfef318be5168803d0b5@18.203.152.54:30311,enode://3063d1c9e1b824cfbb7c7b6abafa34faec6bb4e7e06941d218d760acdd7963b274278c5c3e63914bd6d1b58504c59ec5522c56f883baceb8538674b92da48a96@34.250.32.100:30311,enode://ad78c64a4ade83692488aa42e4c94084516e555d3f340d9802c2bf106a3df8868bc46eae083d2de4018f40e8d9a9952c32a0943cd68855a9bc9fd07aac982a6d@34.204.214.24:30311

danlix2000 commented 6 months ago

should I remove --sentry.drop-useless-peers in the above service file? but in node-real/bsc-erigon readme file mention that runs the command with sentry.drop-useless-peers

blxdyx commented 6 months ago

should I remove --sentry.drop-useless-peers in the above service file? but in node-real/bsc-erigon readme file mention that runs the command with sentry.drop-useless-peers

Yes, the readme haven't update.

danlix2000 commented 6 months ago

I removed flag and did daemon-reload and restarted bsc service. but after some time I still getting [03-18|10:06:15.319] [2/15 Headers] No block headers to write in this log period block number=36592657

danlix2000 commented 6 months ago

Can I get some healthy BSC boot nodes to add to my service file? It may be helpful for my issues

danlix2000 commented 6 months ago

can we upgrade to the latest version (v1.2.2)

blxdyx commented 6 months ago

Yes, suggest use v1.2.2

danlix2000 commented 6 months ago

is there any solution for "No block bodies to write in this log period "

[INFO] [03-20|09:49:51.194] [5/15 Bodies] No block bodies to write in this log period block number=37130641 [INFO] [03-20|09:50:11.195] [5/15 Bodies] No block bodies to write in this log period block number=37130641 [INFO] [03-20|09:50:30.821] Served conn=127.0.0.1:50822 t=86.306µs method=eth_syncing reqid=1 params=[] [INFO] [03-20|09:50:31.194] [5/15 Bodies] No block bodies to write in this log period block number=37130641 [INFO] [03-20|09:50:31.194] [5/15 Bodies] DONE in=1m40.000339291s [INFO] [03-20|09:50:31.195] RPC Daemon notified of new headers from=37130641 to=37130811 hash=0x0560402471e6751ac19e6726841b3aa9e4a50e29d3494380d513410b4b4c5b58 header sending=198.453µs log sending=251ns [INFO] [03-20|09:50:31.195] Timings (slower than 50ms) Headers=76ms BlockHashes=52ms Bodies=1m40s [INFO] [03-20|09:50:31.195] [2/15 Headers] Waiting for headers... from=37130811 [INFO] [03-20|09:50:31.263] [2/15 Headers] Processed highest inserted=37130844 age=3s [INFO] [03-20|09:50:31.331] [5/15 Bodies] Processing bodies... from=37130641 to=37130844 [INFO] [03-20|09:50:51.332] [5/15 Bodies] No block bodies to write in this log period block number=37130641 [INFO] [03-20|09:50:59.938] [p2p] GoodPeers eth67=10 eth68=33 eth66=90 [INFO] [03-20|09:51:11.331] [5/15 Bodies] No block bodies to write in this log period block number=37130641 [INFO] [03-20|09:51:31.331] [5/15 Bodies] No block bodies to write in this log period block number=37130641 [INFO] [03-20|09:51:51.331] [5/15 Bodies] No block bodies to write in this log period block number=37130641 [INFO] [03-20|09:52:01.122] Served conn=127.0.0.1:54318 t=105.264µs method=eth_syncing reqid=1 params=[]

blxdyx commented 6 months ago

add flags: --bodies.cache=214748364800 --batchSize=4096M

danlix2000 commented 6 months ago

added the above flags but still facing the issue.

can I get some enode ids for boot nodes?

[INFO] [03-20|15:14:05.587] [p2p] GoodPeers eth68=32 eth66=99 eth67=1 [INFO] [03-20|15:14:06.786] Served conn=127.0.0.1:40482 t=86.747µs method=eth_syncing reqid=1 params=[] [INFO] [03-20|15:14:11.213] [5/15 Bodies] No block bodies to write in this log period block number=37136960 [INFO] [03-20|15:14:31.214] [5/15 Bodies] No block bodies to write in this log period block number=37136960 [INFO] [03-20|15:14:51.214] [5/15 Bodies] No block bodies to write in this log period block number=37136960 [INFO] [03-20|15:15:11.213] [5/15 Bodies] No block bodies to write in this log period block number=37136960 [INFO] [03-20|15:15:11.213] [5/15 Bodies] DONE in=1m40.000467366s [INFO] [03-20|15:15:11.214] RPC Daemon notified of new headers from=37136960 to=37137296 hash=0xbd8999ba2022153a99c3f2e54a8ea32ad5f5e5ea402134524dccb0dd5e16cc92 header sending=426.802µs log sending=290ns [INFO] [03-20|15:15:11.214] Timings (slower than 50ms) Headers=77ms Bodies=1m40s [INFO] [03-20|15:15:11.214] [2/15 Headers] Waiting for headers... from=37137296 [INFO] [03-20|15:15:11.288] [2/15 Headers] Processed highest inserted=37137330 age=1s [INFO] [03-20|15:15:11.308] [5/15 Bodies] Processing bodies... from=37136960 to=37137330 [INFO] [03-20|15:15:31.309] [5/15 Bodies] No block bodies to write in this log period block number=37136960 [INFO] [03-20|15:15:37.106] Served conn=127.0.0.1:38100 t=75.265µs method=eth_syncing reqid=1 params=[] [INFO] [03-20|15:15:51.309] [5/15 Bodies] No block bodies to write in this log period block number=37136960 [INFO] [03-20|15:16:05.587] [p2p] GoodPeers eth66=99 eth67=1 eth68=34 [INFO] [03-20|15:16:11.308] [5/15 Bodies] No block bodies to write in this log period block number=37136960

blxdyx commented 6 months ago

So restart can't fix that. What about try unwind some block?