[2/15 Headers] No block headers to write in this log ( stuck with bsc archive erigon setup with snapshot)

tardigralice commented 3 months ago

i have issue with bsc erigon archive setup.

Snapshot: erigon_data_20240803
Dedicated server : AMD 5950x / Ubuntu 22 / 128gb ram / 11 TB NVME
Erigon version: 1.2.14-dev
I downloaded and concatenated all the snapshot files to mdbx.dat (around 8TB) in /home/erigonbsc/mainnet/chaindata
I downloaded the mdbx.dat file for the parlia folder /home/erigonbsc/mainnet/parlia

[INFO] [08-11|12:28:46.845] Starting Erigon on BSC mainnet...
[INFO] [08-11|12:28:46.846] Maximum peer count                       ETH=100 total=100
[INFO] [08-11|12:28:46.846] starting HTTP APIs                       port=18789 APIs=eth,debug,net,trace,web3,erigon
[INFO] [08-11|12:28:46.846] torrent verbosity                        level=WRN
[INFO] [08-11|12:28:46.846] [torrent] Public IP                      ip=XX.XX.XX.XX
NFO] [08-11|12:28:46.847] Set global gas cap                       cap=50000000
[INFO] [08-11|12:28:46.850] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=50mb upload.rate=4mb
[INFO] [08-11|12:28:46.850] Opening Database                         label=chaindata path=/home/erigonbsc/mainnet/chaindata
[INFO] [08-11|12:28:46.851] [db] open                                label=chaindata sizeLimit=12TB pageSize=16384
[INFO] [08-11|12:28:46.858] Initialised chain configuration          config="{ChainID: 56 Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Planck: 27281024, Luban: 29020050, Plato: 30720096, Hertz: 31302048, Hertzfix: 34140700, ShanghaiTime: 1705996800, KeplerTime 1705996800, FeynmanTime 1713419340, FeynmanFixTime 1713419340, CancunTime 1718863500, HaberTime 1718863500, HaberFixTime <nil>, Bohr <nil>, Engine: parlia}" genesis=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b
[INFO] [08-11|12:29:10.842] Initialising Ethereum protocol           network=56
[INFO] [08-11|12:29:10.842] Opening Database                         label=parlia path=/home/erigonbsc/mainnet/parlia
[INFO] [08-11|12:29:10.844] Opening Database                         label=blob path=/home/erigonbsc/mainnet/blobs/blob
[INFO] [08-11|12:29:11.198] Starting private RPC server              on=127.0.0.1:9090
[INFO] [08-11|12:29:11.198] new subscription to logs established
[INFO] [08-11|12:29:11.198] rpc filters: subscribing to Erigon events
[INFO] [08-11|12:29:11.198] New txs subscriber joined
[INFO] [08-11|12:29:11.198] new subscription to newHeaders established
[INFO] [08-11|12:29:11.198] Reading JWT secret                       path=/home/erigonbsc/mainnet/jwt.hex
[INFO] [08-11|12:29:11.199] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true
[INFO] [08-11|12:29:11.199] JsonRpc endpoint opened                  ws=true ws.compression=true grpc=false websocket.url=127.0.0.1:8546 http.url=127.0.0.1:18789
[INFO] [08-11|12:29:11.222] Started P2P networking                   version=68 self=enode://d8d28c96d7006a8886cad93964aee0e544804499e575a9b45da0047f9c0b9553365a5bf238712a74e8840e0fd082046c5f9c4021176605d6af6818d5a459d308@xx.xx.xx.xx:30303 name=erigon/v1.2.14/linux-amd64/go1.21.6
[INFO] [08-11|12:29:11.240] [1/15 Snapshots] Requesting downloads
[WARN] [08-11|12:29:11.464] [1/15 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=3999999 min_in_db=40888600
[INFO] [08-11|12:29:11.464] [snapshots:download] Stat                blocks=4000k indices=4000k alloc=4.1GB sys=4.3GB
[INFO] [08-11|12:29:11.464] [2/15 Headers] Waiting for headers...    from=41027349
[INFO] [08-11|12:29:11.800] [txpool] Started
[INFO] [08-11|12:29:41.647] [2/15 Headers] No block headers to write in this log period number=41027349 blk/second=0.000 alloc=1.2GB sys=4.4GB invalidHeaders=115 rejectedBadHeaders=0
[INFO] [08-11|12:30:11.635] [2/15 Headers] No block headers to write in this log period number=41027349 blk/second=0.000 alloc=1.2GB sys=4.4GB invalidHeaders=118 rejectedBadHeaders=0
[INFO] [08-11|12:30:41.649] [2/15 Headers] No block headers to write in this log period number=41027349 blk/second=0.000 alloc=1.5GB sys=4.4GB invalidHeaders=115 rejectedBadHeaders=0
[INFO] [08-11|12:31:11.471] [2/15 Headers] No block headers to write in this log period number=41027349 blk/second=0.000 alloc=2.6GB sys=4.4GB invalidHeaders=116 rejectedBadHeaders=0
[INFO] [08-11|12:31:41.564] [2/15 Headers] No block headers to write in this log period number=41027349 blk/second=0.000 alloc=1.6GB sys=4.4GB invalidHeaders=120 rejectedBadHeaders=0
[INFO] [08-11|12:31:41.564] Req/resp stats                           req=579 reqMin=41027347 reqMax=41269906 skel=26 skelMin=41027348 skelMax=41064404 resp=1093 respMin=41027348 respMax=41269907 dups=155692
[EROR] [08-11|12:33:33.936] Staged Sync                              err="runtime error: index out of range [1] with length 1, trace: [stageloop.go:105 panic.go:920 panic.go:114 btree_generic.go:522 btree_generic.go:779 header_algos.go:385 stage_headers.go:249 stage_headers.go:131 default_stages.go:55 sync.go:507 sync.go:377 stageloop.go:149 stageloop.go:71 asm_amd64.s:1650]"
[INFO] [08-11|12:33:34.716] [2/15 Headers] Waiting for headers...    from=41027349
[INFO] [08-11|12:34:04.777] [2/15 Headers] No block headers to write in this log period number=41027349 blk/second=0.000 alloc=1.6GB sys=4.4GB invalidHeaders=199 rejectedBadHeaders=0

chetanbothra commented 3 months ago

getting the same [INFO] [08-12|06:16:41.483] [2/15 Headers] No block headers to write in this log period number=41290997 blk/second=0.000 alloc=3.1GB sys=5.3GB invalidHeaders=0 rejectedBadHeaders=0

blxdyx commented 3 months ago

remove the flag --maxpeers=100

JB273 commented 3 months ago

remove the flag --maxpeers=100

Didn't help here, also --maxpeers=10 didn't work

I think something wrong with this: [EROR] [08-11|12:33:33.936] Staged Sync err="runtime error: index out of range [1] with length 1, trace: [stageloop.go:105 panic.go:920 panic.go:114 btree_generic.go:522 btree_generic.go:779 header_algos.go:385 stage_headers.go:249 stage_headers.go:131 default_stages.go:55 sync.go:507 sync.go:377 stageloop.go:149 stageloop.go:71 asm_amd64.s:1650]"

blxdyx commented 3 months ago

Could show some dbug logs?

blxdyx commented 3 months ago

remove the flag --maxpeers=100

Didn't help here, also --maxpeers=10 didn't work

I think something wrong with this: [EROR] [08-11|12:33:33.936] Staged Sync err="runtime error: index out of range [1] with length 1, trace: [stageloop.go:105 panic.go:920 panic.go:114 btree_generic.go:522 btree_generic.go:779 header_algos.go:385 stage_headers.go:249 stage_headers.go:131 default_stages.go:55 sync.go:507 sync.go:377 stageloop.go:149 stageloop.go:71 asm_amd64.s:1650]"

@JB273 Try the latest main branch.

JB273 commented 3 months ago

Could show some dbug logs? @blxdyx Now on main branch (no git checkout):

Aug 14 14:00:38 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:38.492] QuerySeeds read nodes from the node DB   count=0
Aug 14 14:00:38 rpc02-var-m bsc-mainnet.sh[2939937]: [INFO] [08-14|14:00:38.492] Started P2P networking                   version=68 self=enode://5455a08e72abbd9a03e66f25bd7663c8ea52a10d03fc98975766c87932cb50141074fef977bf217f06567af6e4457d1f0ccff625bbc829902b97dbd90e7a8e2b@127.0.0.1:30303 name=erigon/v1.2.14/linux-amd64/go1.21.9
Aug 14 14:00:38 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:38.492] QuerySeeds read nodes from the node DB   count=0
Aug 14 14:00:38 rpc02-var-m bsc-mainnet.sh[2939937]: [INFO] [08-14|14:00:38.530] [1/15 Snapshots] Requesting downloads
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [WARN] [08-14|14:00:40.766] [1/15 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=39899999 min_in_db=40888600
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [INFO] [08-14|14:00:40.767] [snapshots:download] Stat                blocks=39900k indices=39900k alloc=3.3GB sys=5.1GB
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:40.767] [1/15 Snapshots] DONE                    in=2.268081962s
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [INFO] [08-14|14:00:40.767] [2/15 Headers] Waiting for headers...    from=41027349
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:40.767] [2/15 Headers] Requested header          from=41357926 length=192
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:40.767] [downloader] Request skeleton            anchors=1 highestInDb=41027349
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:40.767] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:40.831] [2/15 Headers] Requested header          from=41357734 length=192
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:40.850] [2/15 Headers] Requested header          from=41027540 length=192
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:40.850] slow peer or too many requests, dropping its old requests name=Geth/v1.4.12-83a9b13...
Aug 14 14:00:40 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:40.850] slow peer or too many requests, dropping its old requests name=Geth/v1.4.12-83a9b13...
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.476] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.477] [2/15 Headers] Requested header          from=41034488 length=192
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.477] slow peer or too many requests, dropping its old requests name=Geth/v1.4.12-83a9b13...
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.477] slow peer or too many requests, dropping its old requests name=Geth/v1.4.13-26a4d4f...
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.477] slow peer or too many requests, dropping its old requests name=Geth/v1.4.12-83a9b13...
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.898] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.899] [2/15 Headers] Requested header          from=41047033 length=192
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.899] slow peer or too many requests, dropping its old requests name=Geth/v1.4.13-26a4d4f...
Aug 14 14:00:41 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:41.899] [downloader] Request skeleton            anchors=191 highestInDb=41027349
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.318] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.319] [2/15 Headers] Requested header          from=41027541 length=192
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.319] slow peer or too many requests, dropping its old requests name=Geth/v1.4.13-26a4d4f...
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.319] slow peer or too many requests, dropping its old requests name=Geth/v1.4.12-83a9b13...
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.319] slow peer or too many requests, dropping its old requests name=Geth/v1.4.13-26a4d4f...
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.319] [downloader] Request skeleton            anchors=191 highestInDb=41027349
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.321] [downloader] Request skeleton            anchors=190 highestInDb=41027349
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.324] [downloader] Request skeleton            anchors=190 highestInDb=41027349
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.326] [downloader] Request skeleton            anchors=189 highestInDb=41027349
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.329] [2/15 Headers] Requested header          from=41063245 length=192
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.329] [downloader] Request skeleton            anchors=188 highestInDb=41027349
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.329] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.875] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 14 14:00:42 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:42.875] [2/15 Headers] Requested header          from=41027540 length=192
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.296] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.297] [2/15 Headers] Requested header          from=41027541 length=192
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.300] [2/15 Headers] Requested header          from=41027734 length=192
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.302] [2/15 Headers] Requested header          from=41027927 length=192
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.305] [2/15 Headers] Requested header          from=41028120 length=192
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.308] [2/15 Headers] Requested header          from=41028506 length=192
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.331] [downloader] Request skeleton            anchors=170 highestInDb=41027349
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.334] [2/15 Headers] Requested header          from=41028313 length=192
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.334] [downloader] Request skeleton            anchors=169 highestInDb=41027349
Aug 14 14:00:43 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:43.334] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 14 14:00:44 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:44.294] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 14 14:00:44 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:44.773] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 14 14:00:44 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:44.774] [downloader] Request skeleton            anchors=60 highestInDb=41027349
Aug 14 14:00:44 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:44.774] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 14 14:00:45 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:45.350] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 14 14:00:45 rpc02-var-m bsc-mainnet.sh[2939937]: [DBUG] [08-14|14:00:45.843] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"

blxdyx commented 3 months ago

@JB273 Remove the datadir/downloader, you miss some snapshots, it will redownload missed snapshots. That's why you can't find the ancestor

JB273 commented 3 months ago

@JB273 Remove the datadir/downloader, you miss some snapshots, it will redownload missed snapshots. That's why you can't find the ancestor

Changed to this now, will let know here if this helps UPDATE: It seems still the same errors on the DBUG logs...

Aug 16 11:54:50 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:50.428] [downloader] Request skeleton            anchors=48 highestInDb=41027349
Aug 16 11:54:50 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:50.429] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 16 11:54:50 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:50.844] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:50 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:50.844] [2/15 Headers] Requested header          from=41055525 length=192
Aug 16 11:54:50 rpc02-var-m bsc-mainnet.sh[3218034]: [INFO] [08-16|11:54:50.851] [2/15 Headers] No block headers to write in this log period number=41027349 blk/second=0.000 alloc=4.9GB sys=8.9GB invalidHeaders=60 rejectedBadHeaders=0
Aug 16 11:54:51 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:51.069] [2/15 Headers] Requested header          from=41027541 length=192
Aug 16 11:54:51 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:51.142] [2/15 Headers] Requested header          from=41028312 length=192
Aug 16 11:54:51 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:51.258] [2/15 Headers] Requested header          from=41039313 length=192
Aug 16 11:54:51 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:51.426] [2/15 Headers] Requested header          from=41027540 length=192
Aug 16 11:54:51 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:51.429] [downloader] Request skeleton            anchors=51 highestInDb=41027349
Aug 16 11:54:51 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:51.429] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 16 11:54:51 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:51.561] [2/15 Headers] Requested header          from=41055139 length=192
Aug 16 11:54:52 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:52.059] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:52 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:52.474] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:52 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:52.475] [2/15 Headers] Requested header          from=41027541 length=192
Aug 16 11:54:52 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:52.475] [downloader] Request skeleton            anchors=49 highestInDb=41027349
Aug 16 11:54:52 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:52.475] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 16 11:54:52 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:52.896] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:53 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:53.306] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:53 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:53.306] [2/15 Headers] Requested header          from=41412865 length=192
Aug 16 11:54:53 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:53.720] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:53 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:53.721] [downloader] Request skeleton            anchors=49 highestInDb=41027349
Aug 16 11:54:53 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:53.721] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.131] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.139] [2/15 Headers] Requested header          from=41027540 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.249] [2/15 Headers] Requested header          from=41412868 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.252] [2/15 Headers] Requested header          from=41027350 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.263] [2/15 Headers] Requested header          from=41028122 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.305] [2/15 Headers] Requested header          from=41027736 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.341] [2/15 Headers] Requested header          from=41412232 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.825] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.825] [2/15 Headers] Requested header          from=41412839 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.825] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.825] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.827] [2/15 Headers] Requested header          from=41027540 length=192
Aug 16 11:54:54 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:54.860] [2/15 Headers] Requested header          from=41412426 length=192
Aug 16 11:54:55 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:55.303] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:55 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:55.712] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.262] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.262] [2/15 Headers] Requested header          from=41030049 length=192
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.262] slow peer or too many requests, dropping its old requests name=Geth/v1.4.13-26a4d4f...
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.262] slow peer or too many requests, dropping its old requests name=Geth/v1.4.13-26a4d4f...
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.262] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.263] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.265] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.266] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.267] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.269] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.270] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.272] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.724] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.725] [2/15 Headers] Requested header          from=41055139 length=192
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.725] [downloader] Request skeleton            anchors=52 highestInDb=41027349
Aug 16 11:54:56 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:56.725] [2/15 Headers] Requested skeleton        from=41027348 length=192
Aug 16 11:54:57 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:57.132] Couldn't add port mapping                proto=tcp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
Aug 16 11:54:57 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:57.132] Couldn't add port mapping                proto=udp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
Aug 16 11:54:57 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:57.136] [downloader] Verification failed for header hash=0xfcc0324b978906c89cf002e3fcf46aeee903e650dc4accbfb3c15d8bbd899ad7 height=41027350 err="header = 40888599, hash = 0xa4bd4610eb24a5962908baa0ca16353b5401fabdb6d9561733fcc3f0d96c78f5, err = unknown ancestor"
Aug 16 11:54:57 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:57.161] [p2p] Dial scheduler                     protocol=68 peers=6/3 tried=9 static=6 connect: connection refused=3
Aug 16 11:54:57 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:57.162] [p2p] Discovery table                    protocol=68 version=v4 len=172 live=69 unsol=16 ips=214 db=0 reval=114 invalid IP in response record: loopback address from non-loopback host=2 RPC timeout=8 invalid IP in response record: invalid IP=1 unsolicited reply=1
Aug 16 11:54:57 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:57.168] [p2p] Server                             protocol=68 peers=6 trusted=0 inbound=0

UPDATE: It looks it just cannot get past this block...

[EROR] [08-16|12:22:08.245] Staged Sync                              err="runtime error: index out of range [1] with length 1, trace: [stageloop.go:105 panic.go:920 panic.go:114 btree_generic.go:522 btree_generic.go:779 header_algos.go:385 stage_headers.go:249 stage_headers.go:131 default_stages.go:55 sync.go:507 sync.go:377 stageloop.go:149 stageloop.go:71 asm_amd64.s:1650]"
[INFO] [08-16|12:22:09.183] [2/15 Headers] Waiting for headers...    from=41027349

blxdyx commented 2 months ago

try reset the header.

JB273 commented 2 months ago

try reset the header.

Look like did this the trick... Node is in sync now

blxdyx commented 2 months ago

Aug 16 11:54:57 rpc02-var-m bsc-mainnet.sh[3218034]: [DBUG] [08-16|11:54:57.161] [p2p] Dial scheduler protocol=68 peers=6/3 tried=9 static=6 connect: connection refused=3

Reset header can solve most problem of stage header

node-real / bsc-erigon

[2/15 Headers] No block headers to write in this log ( stuck with bsc archive erigon setup with snapshot) #478