search

erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.12k stars 1.11k forks source link

BSC - No block headers to write in this log period block number #5703

Closed breezytm closed 1 year ago

breezytm commented 1 year ago

System information

Erigon version: erigon version 2.27.0-stable-6d8b304e

OS & Version: Linux

Commit hash : 6d8b304

Expected behaviour

Syncing to the latest block

Actual behaviour

Error: No block headers to write in this log period

bsc-node-000            | [INFO] [10-11|15:02:13.458] [p2p] GoodPeers                          eth66=86
bsc-node-000            | [INFO] [10-11|15:02:13.470] [txpool] stat                            block=0 pending=1 baseFee=0 queued=16 alloc=3.8GB sys=65.4GB
bsc-node-000            | [INFO] [10-11|15:02:31.125] [2/16 Headers] No block headers to write in this log period block number=20304127 blk/second=0.000
bsc-node-000            | [INFO] [10-11|15:02:51.125] [2/16 Headers] No block headers to write in this log period block number=20304127 blk/second=0.000
bsc-node-000            | [INFO] [10-11|15:03:11.126] [2/16 Headers] No block headers to write in this log period block number=20304127 blk/second=0.000
bsc-node-000            | [INFO] [10-11|15:03:13.468] [txpool] stat                            block=0 pending=1 baseFee=0 queued=16 alloc=3.3GB sys=65.4GB
bsc-node-000            | [INFO] [10-11|15:03:31.125] [2/16 Headers] No block headers to write in this log period block number=20304127 blk/second=0.000
bsc-node-000            | [INFO] [10-11|15:03:51.126] [2/16 Headers] No block headers to write in this log period block number=20304127 blk/second=0.000

Steps to reproduce the behaviour

    command:
      - erigon
      - --datadir=/root/.local/share/erigon
      - --chain=bsc
      - --port=29505
      - --private.api.addr=0.0.0.0:9090
      - --http=false
      - --db.pagesize=16kb
      - --prune=hrtc
      - --torrent.port=29506
      - --torrent.download.rate=1000mb
      - --torrent.download.slots=6
      - --downloader.verify
      - --batchSize=512M
      - --etl.bufferSize=512M
      - --healthcheck
      - --verbosity=3
luwang920 commented 1 year ago

[2/16 Headers] No block headers to write in this log period block number=37076755 How to fix it ?Thanks

alexmance commented 1 year ago

We had the same issue as you @luwang920.

We were able to move forward (or it seems so) with the following command: ./integration stage_exec --datadir=/mnt/erigon/ --chain='bor-mainnet' --unwind 100 (we tried first with 10, then 20 blocks, nothing changed. Jumped to 100 and saw some changes)

version: erigon version 2.31.0-stable-a53e8a3e

luwang920 commented 1 year ago

Hi alexmance Thanks , it didn't work for me .. I tried to roll back several times...

alexmance commented 1 year ago

Hi @luwang920 Just take a look at our servers.. and after 4 hours it crashed again and we are stuck in the same block... sorry man. We decided to re-sync from the official snapshot today, as it has been a few days with this behavior...

nikolinsko commented 1 year ago

this is still an issue for me on

[INFO] [12-28|13:00:34.744] Build info                               git_branch=heads/v2.32.0 git_tag=v2.32.0 git_commit=021891a311430ffefde77fcceda567ba324e9abe
[INFO] [12-28|12:44:24.951] [2/16 Headers] No block headers to write in this log period block number=24199022
[INFO] [12-28|12:44:24.951] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=17 skelMin=24199023 skelMax=24493935 resp=32 respMin=24199023 respMax=24293803 dups=941
[DBUG] [12-28|12:44:24.951] [Downloader] Queue sizes                 anchors=0 links=97497 persisted=65535
[DBUG] [12-28|12:44:26.084] [Downloader] Request skeleton            anchors=0 top seen height=24293804 highestInDb=24199022
[DBUG] [12-28|12:44:27.085] [Downloader] Request skeleton            anchors=0 top seen height=24293804 highestInDb=24199022
[DBUG] [12-28|12:44:28.086] [Downloader] Request skeleton            anchors=0 top seen height=24293804 highestInDb=24199022
[DBUG] [12-28|12:44:29.179] [Downloader] Request skeleton            anchors=0 top seen height=24293805 highestInDb=24199022
[DBUG] [12-28|12:44:30.180] [Downloader] Request skeleton            anchors=0 top seen height=24293805 highestInDb=24199022
[DBUG] [12-28|12:44:31.442] [Downloader] Request skeleton            anchors=0 top seen height=24293806 highestInDb=24199022
[DBUG] [12-28|12:44:32.536] [Downloader] Request skeleton            anchors=0 top seen height=24293806 highestInDb=24199022
[DBUG] [12-28|12:44:33.536] [Downloader] Request skeleton            anchors=0 top seen height=24293806 highestInDb=24199022
[DBUG] [12-28|12:44:35.213] [Downloader] Request skeleton            anchors=0 top seen height=24293807 highestInDb=24199022
[DBUG] [12-28|12:44:36.894] [Downloader] Request skeleton            anchors=0 top seen height=24293807 highestInDb=24199022
[DBUG] [12-28|12:44:36.986] [Downloader] new anchor too far in the past: 0, latest header in db: 24199022
[DBUG] [12-28|12:44:38.093] [Downloader] Request skeleton            anchors=0 top seen height=24293808 highestInDb=24199022
[DBUG] [12-28|12:44:39.139] [Downloader] Request skeleton            anchors=0 top seen height=24293808 highestInDb=24199022
[DBUG] [12-28|12:44:39.231] [Downloader] new anchor too far in the past: 0, latest header in db: 24199022
[INFO] [12-28|12:44:39.484] [txpool] stat                            block=24199023 pending=10000 baseFee=636 queued=2000 alloc=4.7GB sys=9.2GB
egorFiNE commented 1 year ago

Still happens daily to me with the BSC node on 2.32. unwind usually helps.

ChinW commented 1 year ago

also met the same log, I am not sure if it is just expected?

I have tried to run unwind=10 and then restarted the erigon, still see below log

[INFO] [01-05|21:14:41.714] [2/16 Headers] Waiting for headers...    from=22999999
[INFO] [01-05|21:15:01.715] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:15:21.716] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:15:39.629] [txpool] stat                            block=22999999 pending=0 baseFee=0 queued=15 alloc=3.0GB sys=3.3GB
[INFO] [01-05|21:15:41.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:16:01.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:16:21.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:16:21.715] Req/resp stats                           req=3 reqMin=24522988 reqMax=24523180 skel=0 skelMin=0 skelMax=0 resp=1 respMin=24523181 respMax=24523181 dups=0
[INFO] [01-05|21:16:39.601] [p2p] GoodPeers                          eth66=1 eth67=0
[INFO] [01-05|21:16:39.629] [txpool] stat                            block=22999999 pending=0 baseFee=0 queued=16 alloc=2.3GB sys=3.4GB
[INFO] [01-05|21:16:41.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:16:41.714] Req/resp stats                           req=2 reqMin=24522988 reqMax=24523180 skel=0 skelMin=0 skelMax=0 resp=1 respMin=0 respMax=0 dups=1
[INFO] [01-05|21:17:01.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:17:01.714] Req/resp stats                           req=3 reqMin=24522988 reqMax=24523180 skel=0 skelMin=0 skelMax=0 resp=1 respMin=0 respMax=0 dups=1
[INFO] [01-05|21:17:21.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:17:21.714] Req/resp stats                           req=10 reqMin=24521285 reqMax=24523205 skel=0 skelMin=0 skelMax=0 resp=14 respMin=24521478 respMax=24523206 dups=4

my build info Build info git_branch=stable git_tag=v2.32.0 git_commit=021891a311430ffefde77fcceda567ba324e9abe

luwang920 commented 1 year ago

also met the same log, I am not sure if it is just expected?

I have tried to run unwind=10 and then restarted the erigon, still see below log

[INFO] [01-05|21:14:41.714] [2/16 Headers] Waiting for headers...    from=22999999
[INFO] [01-05|21:15:01.715] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:15:21.716] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:15:39.629] [txpool] stat                            block=22999999 pending=0 baseFee=0 queued=15 alloc=3.0GB sys=3.3GB
[INFO] [01-05|21:15:41.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:16:01.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:16:21.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:16:21.715] Req/resp stats                           req=3 reqMin=24522988 reqMax=24523180 skel=0 skelMin=0 skelMax=0 resp=1 respMin=24523181 respMax=24523181 dups=0
[INFO] [01-05|21:16:39.601] [p2p] GoodPeers                          eth66=1 eth67=0
[INFO] [01-05|21:16:39.629] [txpool] stat                            block=22999999 pending=0 baseFee=0 queued=16 alloc=2.3GB sys=3.4GB
[INFO] [01-05|21:16:41.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:16:41.714] Req/resp stats                           req=2 reqMin=24522988 reqMax=24523180 skel=0 skelMin=0 skelMax=0 resp=1 respMin=0 respMax=0 dups=1
[INFO] [01-05|21:17:01.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:17:01.714] Req/resp stats                           req=3 reqMin=24522988 reqMax=24523180 skel=0 skelMin=0 skelMax=0 resp=1 respMin=0 respMax=0 dups=1
[INFO] [01-05|21:17:21.714] [2/16 Headers] No block headers to write in this log period block number=22999999
[INFO] [01-05|21:17:21.714] Req/resp stats                           req=10 reqMin=24521285 reqMax=24523205 skel=0 skelMin=0 skelMax=0 resp=14 respMin=24521478 respMax=24523206 dups=4

my build info Build info git_branch=stable git_tag=v2.32.0 git_commit=021891a311430ffefde77fcceda567ba324e9abe

I was stucking in this error for several days.
Unwind , and stop at the same block, Unwind , and stop at the same block again.. Finally , I downloaded the latest offical snapshot, resync .. My node is polygon archive mainnet , it took me 3 days to download and unzip the snapshot .....terrible...

Br Lu

nandubatchu commented 1 year ago

@egorFiNE can you help me with the unwind steps you followed for bor-mainnet?

egorFiNE commented 1 year ago

@nandubatchu we gave up on erigon for bor-mainnet. It took something close to 20 days to sync and then threw "no block headers" daily or even more often.

mikolajsobolewski commented 1 year ago

@nandubatchu we gave up on erigon for bor-mainnet. It took something close to 20 days to sync and then threw "no block headers" daily or even more often.

We also think to give up on erigon for BSC. We have the same errors in mainnet and have no answers from Erigon developers.

egorFiNE commented 1 year ago

@mikolajsobolewski on BSC it's not that much of a PITA. It happens like twice per week. You unwind ten blocks, restart and in about 20 minutes or less it's up and running again.

mikolajsobolewski commented 1 year ago

@mikolajsobolewski on BSC it's not that much of a PITA. It happens like twice per week. You unwind ten blocks, restart and in about 20 minutes or less it's up and running again.

But if we use their geth fork always receives new blocks. I don't know how it doest that, but fact is fact. So i think, that it is the erigon error. We don't want to restart the node 5-6 times per day. it's really strange since 2022 Oct no one work solution.

egorFiNE commented 1 year ago

Updated to 2.34.0 and here we go again. :-(

nebojsa-ristovic commented 1 year ago

Same here for both Polygon and BSC Erigon node.

egorFiNE commented 1 year ago

...aaaand again.

egorFiNE commented 1 year ago

Now I'm having this issue multiple times per day on 2.34.0 on BSC BUT a simple restart helps, no need to unwind.

AskAlexSharov commented 1 year ago

Probably related: https://github.com/ledgerwatch/erigon/issues/6532

seonggwonyoon commented 1 year ago

Same on 2.36.1 (polygon and bsc)

akshat-pyor commented 1 year ago

@egorFiNE

Now I'm having this issue multiple times per day on 2.34.0 on BSC BUT a simple restart helps, no need to unwind.

facing the same issue (No block headers to write in this log period block number ) on version 2.39.0. for bsc. what is the way out here given that i restarted it multiple times.

Vitawt-The-Duke commented 1 year ago

Hello. I've been stuck with the same problem and fixed it with updating to the latest binary and unwinding: (The client stopped getting new data for more than a couple of weeks I suppose) My flow:

Timestamp 2023-03-13_20:47:02
Latest block at current node is: 26110021
Latest block at mainnet is: 26439285
diff is: 329264

Node parameters:

128GB of RAM
zraid0 (4x stripe SSD)
AMD Ryzen 9 5950X 16-Core Processor

Startup parameters:

#!/bin/bash
exec /go/erigon/build/bin/erigon --chain="bsc" \
    --snapshots=true \
    --txpool.disable \
    --db.pagesize="16k" \
    --datadir="/data/bsc" \
    --http --http.vhosts="*" --http.addr="0.0.0.0" --http.port="8545" \
    --http.api="net,eth,web3" \
    --ws \
    --metrics --metrics.addr="0.0.0.0" --metrics.port="6060"

tried to fix: -update the client to the latest version -unwind and it helps!!

[WARN] [03-13|20:36:58.215] no log dir set, console logging only
[INFO] [03-13|20:36:58.216] Starting metrics server                  addr=http://0.0.0.0:6060/debug/metrics/prometheus
[WARN] [03-13|20:36:58.221] no log dir set, console logging only
[INFO] [03-13|20:36:58.221] Build info                               git_branch=HEAD git_tag=v2.40.1 git_commit=94b4cee5a898a47b9241ce824622af6350e5dc5b
[INFO] [03-13|20:36:58.221] Starting Erigon on BSC mainnet...
[INFO] [03-13|20:36:58.224] Maximum peer count                       ETH=100 total=100
[INFO] [03-13|20:36:58.224] starting HTTP APIs                       APIs=net,eth,web3
[INFO] [03-13|20:36:58.224] torrent verbosity                        level=WRN
[INFO] [03-13|20:37:00.327] Set global gas cap                       cap=50000000
[INFO] [03-13|20:37:00.333] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
[INFO] [03-13|20:37:00.333] Opening Database                         label=chaindata path=/data/bsc/chaindata
[INFO] [03-13|20:37:00.339] Initialised chain configuration          config="{ChainID: 56 Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Gibbs: 23846001, Engine: parlia}" genesis=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b
[INFO] [03-13|20:37:00.339] Effective                                prune_flags= snapshot_flags="--snapshots=true" history.v3=false
[INFO] [03-13|20:37:03.960] Initialising Ethereum protocol           network=56
[INFO] [03-13|20:37:04.242] Starting private RPC server              on=127.0.0.1:9090
[INFO] [03-13|20:37:04.242] new subscription to logs established
[INFO] [03-13|20:37:04.243] rpc filters: subscribing to Erigon events
[INFO] [03-13|20:37:04.243] new subscription to newHeaders established
[INFO] [03-13|20:37:04.244] Reading JWT secret                       path=/data/bsc/jwt.hex
[INFO] [03-13|20:37:04.244] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true
[INFO] [03-13|20:37:04.244] HTTP endpoint opened                     url=[::]:8545 ws=true ws.compression=true grpc=false
[INFO] [03-13|20:37:04.250] Started P2P networking                   version=68 self=enode://209ec59bf1d372caecb61836089e006c34f31dadefdc7e5abecebb45bd5735bc7ccf6b00c209a344fb584b2f6fe5bc83168fcf51643067ef2e7c262fd54a7eef@127.0.0.1:30304 name=erigon/v2.40.1-stable-94b4cee5/linux-amd64/go1.20.2
[INFO] [03-13|20:37:04.282] Started P2P networking                   version=67 self=enode://209ec59bf1d372caecb61836089e006c34f31dadefdc7e5abecebb45bd5735bc7ccf6b00c209a344fb584b2f6fe5bc83168fcf51643067ef2e7c262fd54a7eef@127.0.0.1:30303 name=erigon/v2.40.1-stable-94b4cee5/linux-amd64/go1.20.2
[INFO] [03-13|20:37:04.284] [1/15 Snapshots] Fetching torrent files metadata
[INFO] [03-13|20:37:04.287] [snapshots] Blocks Stat                  blocks=26020k indices=26020k alloc=2.8GB sys=3.0GB
[INFO] [03-13|20:37:04.287] [2/15 Headers] Waiting for headers...    from=26438754
[INFO] [03-13|20:37:24.288] [2/15 Headers] Wrote block headers       number=26439044 blk/second=14.500 alloc=3.1GB sys=3.3GB
[INFO] [03-13|20:37:44.288] [2/15 Headers] No block headers to write in this log period block number=26439044
[INFO] [03-13|20:38:44.288] [2/15 Headers] No block headers to write in this log period block number=26439044
[INFO] [03-13|20:39:03.960] [p2p] GoodPeers                          eth66=26
[INFO] [03-13|20:39:04.287] [2/15 Headers] No block headers to write in this log period block number=26439044
[INFO] [03-13|20:39:24.287] [2/15 Headers] No block headers to write in this log period block number=26439044

have updates after rewind:

[INFO] [03-13|20:48:24.288] [2/15 Headers] Wrote block headers       number=26439236 blk/second=9.600 alloc=2.6GB sys=3.6GB

Morning update (13 hrs passed)

[INFO] [03-14|10:14:46.840] [5/15 Bodies] No block bodies to write in this log period block number=26341579
[INFO] [03-14|10:15:03.960] [p2p] GoodPeers                          eth66=9 eth67=1 eth68=14
[INFO] [03-14|10:15:06.841] [5/15 Bodies] No block bodies to write in this log period block number=26341579
[INFO] [03-14|10:15:26.841] [5/15 Bodies] No block bodies to write in this log period block number=26341579
[INFO] [03-14|10:15:46.840] [5/15 Bodies] No block bodies to write in this log period block number=26341579

today update(1 day+ passed):

[INFO] [03-15|11:46:39.294] [7/15 Execution] Executed blocks         number=26263449 blk/s=12.2 tx/s=1748.5 Mgas/s=183.2 gasState=0.19 batch=237.3MB alloc=6.1GB sys=10.2GB
[INFO] [03-15|11:46:59.355] [7/15 Execution] Executed blocks         number=26263673 blk/s=11.2 tx/s=1626.2 Mgas/s=173.9 gasState=0.19 batch=245.9MB alloc=4.2GB sys=10.2GB
[INFO] [03-15|11:47:19.440] [7/15 Execution] Executed blocks         number=26263908 blk/s=11.7 tx/s=1764.3 Mgas/s=186.3 gasState=0.20 batch=254.7MB alloc=5.6GB sys=10.2GB
[INFO] [03-15|11:47:22.606] Committed State                          gas reached=111327847129 gasTarget=549755813888
[INFO] [03-15|11:47:26.056] [p2p] GoodPeers                          eth67=5 eth68=11 eth66=14
[INFO] [03-15|11:47:52.734] [] ETL [2/2] Loading                     into=PlainCodeHash current_prefix=bbaaa420
[INFO] [03-15|11:48:03.423] Write to db                              progress=0.1M/3.8M current table=PlainState
[INFO] [03-15|11:48:39.094] [] ETL [2/2] Loading                     into=PlainState current_prefix=2ab0e9e4
[INFO] [03-15|11:49:09.093] [] ETL [2/2] Loading                     into=PlainState current_prefix=36a23dfd

Update! It works!

Latest block at current node is: 26257353
Latest block at mainnet is: 26485629
diff is: 228276
jajaislanina commented 1 year ago

These are our Polygon nodes running Erigon 2.39. (current block over time)

image

Same condition happens on other networks (BSC, Sepolia etc.)

Right now, when we detect that the node is stalling (flat line on the graph) we need to restart it and it will start processing after ~10 minutes. This is happening at least once per day - sometimes multiple times even.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 40 days with no activity. Remove stale label or comment, or this will be closed in 7 days.

yperbasis commented 1 year ago

BSC is no longer supported by Erigon. Please consider https://github.com/node-real/bsc-erigon instead.

d0zingcat commented 1 year ago

In my situation, I utilized an older snapshot (e.g., erigon-05-23), and configured --nodiscover and --bootnodes=xxxx (the enodes were chosen from here based on their recentness, ensuring no lag between these nodes and other nodes on the mainnet). After removing them and adding --snapshots=false, the blocks began to sync correctly.

This solution was found in this document. An excerpt from the document is:

curl localhost:8545 -X POST --data '{"jsonrpc":"2.0","method":"eth_syncing","id":1}' -H "Content-Type: application/json"

In my particular situation, the result was:

{"jsonrpc":"2.0","id":1,"result":false}

However, the block number was 43057508, which could not possibly be the most recent one!