search

node-real / bsc-erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
118 stars 44 forks source link

Infinite loop of "snapshots build, gather headers block=xxxxxx" #424

Closed aploium closed 1 week ago

aploium commented 3 weeks ago

System information

Erigon version: erigon version 1.2.11-dev

OS & Version: Linux

Commit hash: official released binary file

Erigon Command (with flags/config): 

chain: bsc

txpool.disable: true
sentry.log-peer-info: true

log.dir.verbosity: debug

p2p.protocol: "68"

db.pagesize: 16k

http.addr: "0.0.0.0"
http.port: 8585
http.api: eth,net,web3,debug,txpool,bsc,admin

ws: false
ws.compression: false

torrent.download.rate: 500mb
torrent.upload.rate: 1mb
torrent.verbosity: 3
torrent.conns.perfile: 30

rpc.returndata.limit: 990000

nat: any

trustedpeers: []
staticpeers:
  - "enode://fe0bb07eae29e8cfaa5bb15b0db8c386a45b7da2c94e1dabd7ca58b6327eee0c27bdcea4f08db19ea07b9a1391e5496a28c675c6eee578154edae4fa44640c5d@54.228.2.74:30311"

Actual behaviour

the erigon stuck in Infinite loop printing logs like this:

[INFO] [07-05|10:01:37.045] logging to file system                   log dir=/erigon_data/logs file prefix=erigon log level=dbug json=false
[INFO] [07-05|10:01:37.047] Build info                               git_branch= git_tag= git_commit=
[INFO] [07-05|10:01:37.047] Starting Erigon on BSC mainnet...
[INFO] [07-05|10:01:37.050] Maximum peer count                       ETH=100 total=100
[INFO] [07-05|10:01:37.050] starting HTTP APIs                       port=8585 APIs=eth,net,web3,debug,txpool,bsc,admin
[INFO] [07-05|10:01:37.050] torrent verbosity                        level=INF
[INFO] [07-05|10:01:37.052] [torrent] Public IP                      ip=......
[INFO] [07-05|10:01:37.053] Set global gas cap                       cap=50000000
[INFO] [07-05|10:01:37.057] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=500mb upload.rate=1mb
[INFO] [07-05|10:01:37.059] Opening Database                         label=chaindata path=/erigon_data/chaindata
[INFO] [07-05|10:01:37.062] [db] open                                label=chaindata sizeLimit=12TB pageSize=16384
[INFO] [07-05|10:01:37.071] Initialised chain configuration          config="{ChainID: 56 Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Planck: 27281024, Luban: 29020050, Plato: 30720096, Hertz: 31302048, Hertzfix: 34140700, ShanghaiTime: 1705996800, KeplerTime 1705996800, FeynmanTime 1713419340, FeynmanFixTime 1713419340, CancunTime 1718863500, HaberTime 1718863500, HaberFixTime 1720591588, Engine: parlia}" genesis=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b
[DBUG] [07-05|10:01:39.375] [db] open                                label=downloader sizeLimit=16GB pageSize=4096
[INFO] [07-05|10:01:41.382] UPnP device at 192.168.50.227: mapping internal UDP port 42069: success: external port 42069
[INFO] [07-05|10:01:41.382] UPnP device at 192.168.50.227: mapping internal TCP port 42069: success: external port 42069
[DBUG] [07-05|10:02:00.726] [snapshots.webseed] providers            http=0 disk=0
[INFO] [07-05|10:02:00.731] Initialising Ethereum protocol           network=56
[INFO] [07-05|10:02:00.731] Opening Database                         label=parlia path=/erigon_data/parlia
[DBUG] [07-05|10:02:00.734] [db] open                                label=consensus sizeLimit=2TB pageSize=16384
[INFO] [07-05|10:02:00.734] Opening Database                         label=blob path=/erigon_data/blobs/blob
[DBUG] [07-05|10:02:00.737] [db] open                                label=blob sizeLimit=2TB pageSize=4096
[INFO] [07-05|10:02:02.059] Starting private RPC server              on=127.0.0.1:9090
[INFO] [07-05|10:02:02.059] new subscription to logs established
[INFO] [07-05|10:02:02.060] rpc filters: subscribing to Erigon events
[DBUG] [07-05|10:02:02.061] Establishing event subscription channel with the RPC daemon ...
[INFO] [07-05|10:02:02.061] new subscription to newHeaders established
[INFO] [07-05|10:02:02.061] Reading JWT secret                       path=/erigon_data/jwt.hex
[INFO] [07-05|10:02:02.062] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=false
[INFO] [07-05|10:02:02.062] JsonRpc endpoint opened                  ws=true ws.compression=false grpc=false websocket.url=[::]:8546 http.url=[::]:8585
[DBUG] [07-05|10:02:02.063] [db] open                                label=diagnostics sizeLimit=16GB pageSize=4096
[DBUG] [07-05|10:02:02.065] [diskutils] Implemented only for darwin
[DBUG] [07-05|10:02:02.068] [db] open                                label=sentry sizeLimit=8GB pageSize=4096
[INFO] [07-05|10:02:02.070] NAT ExternalIP resolved                  ip=......
[DBUG] [07-05|10:02:02.070] QuerySeeds read nodes from the node DB   count=0
[DBUG] [07-05|10:02:02.071] QuerySeeds read nodes from the node DB   count=0
[INFO] [07-05|10:02:02.071] Started P2P networking                   version=68 self=enode://....@....:30303 name=erigon/v1.2.11/linux-amd64/go1.22.2
[INFO] [07-05|10:02:02.082] [1/15 Snapshots] Requesting downloads
[INFO] [07-05|10:02:02.095] Mapped network port                      proto=tcp extport=30303 intport=30303 interface=NAT-PMP(192.168.50.1)
[INFO] [07-05|10:02:02.123] Mapped network port                      proto=udp extport=30303 intport=30303 interface=NAT-PMP(192.168.50.1)
[DBUG] [07-05|10:02:02.905] AnchorTree add anchor                    anchor.blockHeight=40194852 success=false
[WARN] [07-05|10:02:03.193] [1/15 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=38699999 min_in_db=39818000
[INFO] [07-05|10:02:03.193] [snapshots:download] Stat                blocks=38700k indices=38700k alloc=3.4GB sys=5.3GB
[DBUG] [07-05|10:02:03.193] [1/15 Snapshots] DONE                    in=1.1165831s
[INFO] [07-05|10:02:03.193] [2/15 Headers] Waiting for headers...    from=39908401
[DBUG] [07-05|10:02:03.193] [2/15 Headers] Requested header          from=40194851 length=192
[DBUG] [07-05|10:02:03.193] [downloader] Request skeleton            anchors=1 highestInDb=39908401
[DBUG] [07-05|10:02:03.194] [2/15 Headers] Requested skeleton        from=39908400 length=192
[DBUG] [07-05|10:02:03.722] AnchorTree add anchor                    anchor.blockHeight=40194660 success=false
[DBUG] [07-05|10:02:03.722] [2/15 Headers] Requested header          from=40194659 length=192
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39908593 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39908786 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39908979 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39909172 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39909365 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39909558 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39909751 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39909944 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39910137 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39910330 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39910523 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39910716 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39910909 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39911102 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39911295 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39911488 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39911681 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39911874 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39912067 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39912260 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39912453 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39912646 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39912839 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39913032 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39913225 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39913418 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39913611 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39913804 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39913997 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39914190 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39914383 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39914576 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39914769 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39914962 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39915155 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39915348 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39915541 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39915734 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39915927 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39916120 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39916313 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39916506 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39916699 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39916892 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39917085 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39917278 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39917471 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39917664 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39917857 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39918050 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39918243 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39918436 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39918629 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39918822 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39919015 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39919208 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39919401 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39919594 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39919787 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39919980 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39920173 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39920366 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39920559 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39920752 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39920945 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39921138 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39921331 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39921524 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39921717 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39921910 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39922103 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39922296 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39922489 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39922682 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39922875 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39923068 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39923261 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39923454 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39923647 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39923840 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39924033 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39924226 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39924419 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39924612 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39924805 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39924998 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39925191 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39925384 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39925577 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39925770 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39925963 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39926156 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39926349 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39926542 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39926735 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39926928 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39927121 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39927314 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39927507 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39927700 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39927893 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39928086 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39928279 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39928472 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39928665 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39928858 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39929051 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39929244 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39929437 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39929630 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39929823 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39930016 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39930209 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39930402 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39930595 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39930788 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39930981 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39931174 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39931367 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39931560 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39931753 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39931946 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39932139 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39932332 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39932525 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39932718 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39932911 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39933104 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39933297 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39933490 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39933683 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39933876 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39934069 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39934262 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39934455 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39934648 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39934841 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39935034 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39935227 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39935420 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39935613 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39935806 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39935999 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39936192 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39936385 success=false
[DBUG] [07-05|10:02:03.811] AnchorTree add anchor                    anchor.blockHeight=39936578 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39936771 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39936964 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39937157 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39937350 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39937543 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39937736 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39937929 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39938122 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39938315 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39938508 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39938701 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39938894 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39939087 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39939280 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39939473 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39939666 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39939859 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39940052 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39940245 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39940438 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39940631 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39940824 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39941017 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39941210 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39941403 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39941596 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39941789 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39941982 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39942175 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39942368 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39942561 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39942754 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39942947 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39943140 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39943333 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39943526 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39943719 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39943912 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39944105 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39944298 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39944491 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39944684 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39944877 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39945070 success=false
[DBUG] [07-05|10:02:03.812] AnchorTree add anchor                    anchor.blockHeight=39945263 success=false
[DBUG] [07-05|10:02:03.812] [2/15 Headers] Requested header          from=39908592 length=192
[DBUG] [07-05|10:02:03.932] AnchorTree add anchor                    anchor.blockHeight=40194468 success=false
[DBUG] [07-05|10:02:04.027] [2/15 Headers] Requested header          from=39909364 length=192
[INFO] [07-05|10:02:04.038] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:04.517] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[DBUG] [07-05|10:02:04.517] AnchorTree add anchor                    anchor.blockHeight=39908593 success=false
[DBUG] [07-05|10:02:04.517] [2/15 Headers] Requested header          from=39908592 length=192
[DBUG] [07-05|10:02:04.517] slow peer or too many requests, dropping its old requests name=Geth/v1.4.10-f0c7795...
[DBUG] [07-05|10:02:04.517] slow peer or too many requests, dropping its old requests name=Geth/v1.4.10-f0c7795...
[DBUG] [07-05|10:02:04.517] [downloader] Request skeleton            anchors=192 highestInDb=39908401
[DBUG] [07-05|10:02:04.517] [downloader] Request skeleton            anchors=192 highestInDb=39908401
[DBUG] [07-05|10:02:04.518] [downloader] Request skeleton            anchors=192 highestInDb=39908401
[DBUG] [07-05|10:02:04.519] [downloader] Request skeleton            anchors=192 highestInDb=39908401
[DBUG] [07-05|10:02:04.519] [downloader] Request skeleton            anchors=192 highestInDb=39908401
[DBUG] [07-05|10:02:04.520] [downloader] Request skeleton            anchors=191 highestInDb=39908401
[DBUG] [07-05|10:02:04.520] [downloader] Request skeleton            anchors=190 highestInDb=39908401
[DBUG] [07-05|10:02:04.521] [downloader] Request skeleton            anchors=190 highestInDb=39908401
[DBUG] [07-05|10:02:04.521] [downloader] Request skeleton            anchors=190 highestInDb=39908401
[DBUG] [07-05|10:02:04.522] [downloader] Request skeleton            anchors=189 highestInDb=39908401
[INFO] [07-05|10:02:04.532] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:04.631] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[DBUG] [07-05|10:02:04.631] AnchorTree add anchor                    anchor.blockHeight=39908980 success=false
[DBUG] [07-05|10:02:04.631] [downloader] Request skeleton            anchors=189 highestInDb=39908401
[DBUG] [07-05|10:02:04.632] [downloader] Request skeleton            anchors=188 highestInDb=39908401
[DBUG] [07-05|10:02:04.632] [downloader] Request skeleton            anchors=187 highestInDb=39908401
[DBUG] [07-05|10:02:04.633] [downloader] Request skeleton            anchors=187 highestInDb=39908401
[INFO] [07-05|10:02:04.647] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:04.878] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[DBUG] [07-05|10:02:04.878] AnchorTree add anchor                    anchor.blockHeight=39908594 success=false
[DBUG] [07-05|10:02:04.878] [2/15 Headers] Requested header          from=39908593 length=192
[DBUG] [07-05|10:02:04.878] slow peer or too many requests, dropping its old requests name=Geth/v1.4.10-f0c7795...
[DBUG] [07-05|10:02:04.879] slow peer or too many requests, dropping its old requests name=Geth/v1.4.10-f0c7795...
[DBUG] [07-05|10:02:04.879] [downloader] Request skeleton            anchors=188 highestInDb=39908401
[DBUG] [07-05|10:02:04.879] [2/15 Headers] Requested skeleton        from=39908400 length=192
[DBUG] [07-05|10:02:04.879] AnchorTree add anchor                    anchor.blockHeight=39908787 success=false
[INFO] [07-05|10:02:04.898] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:04.998] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[DBUG] [07-05|10:02:04.998] [2/15 Headers] Requested header          from=39908786 length=192
[DBUG] [07-05|10:02:04.998] slow peer or too many requests, dropping its old requests name=Geth/v1.4.10-f0c7795...
[DBUG] [07-05|10:02:04.998] slow peer or too many requests, dropping its old requests name=Geth/v1.4.10-f0c7795...
[INFO] [07-05|10:02:05.010] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:05.110] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:05.131] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:05.231] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:05.242] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:05.354] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:05.365] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:05.464] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:05.477] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:05.576] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:05.586] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:05.684] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:05.736] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:05.843] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[DBUG] [07-05|10:02:05.879] [downloader] Request skeleton            anchors=99 highestInDb=39908401
[DBUG] [07-05|10:02:05.879] [2/15 Headers] Requested skeleton        from=39908400 length=192
[DBUG] [07-05|10:02:05.882] AnchorTree add anchor                    anchor.blockHeight=39908595 success=false
[DBUG] [07-05|10:02:05.882] [2/15 Headers] Requested header          from=39908594 length=192
[DBUG] [07-05|10:02:05.897] AnchorTree add anchor                    anchor.blockHeight=40194276 success=false
[DBUG] [07-05|10:02:05.897] [2/15 Headers] Requested header          from=40194275 length=192
[DBUG] [07-05|10:02:05.904] AnchorTree add anchor                    anchor.blockHeight=39908593 success=false
[DBUG] [07-05|10:02:05.904] [2/15 Headers] Requested header          from=39908592 length=192
[DBUG] [07-05|10:02:06.095] AnchorTree add anchor                    anchor.blockHeight=39908403 success=false
[DBUG] [07-05|10:02:06.095] [2/15 Headers] Requested header          from=39908402 length=192
[DBUG] [07-05|10:02:06.111] AnchorTree add anchor                    anchor.blockHeight=40194084 success=false
[DBUG] [07-05|10:02:06.111] [2/15 Headers] Requested header          from=40194083 length=192
[INFO] [07-05|10:02:06.126] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:06.254] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[DBUG] [07-05|10:02:06.258] AnchorTree add anchor                    anchor.blockHeight=39908403 success=false
[DBUG] [07-05|10:02:06.258] [2/15 Headers] Requested header          from=39908402 length=192
[INFO] [07-05|10:02:06.272] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:06.400] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:06.414] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:06.542] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:06.556] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:06.682] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:06.697] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:06.823] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:06.838] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:06.964] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[DBUG] [07-05|10:02:06.964] AnchorTree add anchor                    anchor.blockHeight=40193892 success=false
[DBUG] [07-05|10:02:06.964] [2/15 Headers] Requested header          from=40193891 length=192
[DBUG] [07-05|10:02:06.964] [downloader] Request skeleton            anchors=65 highestInDb=39908401
[DBUG] [07-05|10:02:06.964] [2/15 Headers] Requested skeleton        from=39908400 length=192
[INFO] [07-05|10:02:06.980] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:07.107] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:07.120] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:07.337] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:07.348] [parlia] snapshots build, gather headers block=39900000
[DBUG] [07-05|10:02:07.450] [downloader] Verification failed for header hash=0xf9dd77a960f210ac816ed29213ece933889f20f0397fd98f38a826938f266626 height=39908402 err="unknown ancestor"
[INFO] [07-05|10:02:07.460] [parlia] snapshots build, gather headers block=39900000

Steps to reproduce the behaviour

There is two way to reproduce:

First way: from fresh official snapshot

  1. create a new folder with no existing data in it
  2. download and copy the official erigon snapshot to it's chain data. I've tested verison 0520 and 0629 of this snapshot, all can reproduce this.
  3. start the erigon, wait it to complete downloading the snapshot from torrent
  4. This Infinite loop occurs.

Second way: from existing data

  1. stop erigon
  2. build/bin/integration stage_headers --reset --config config.yaml --datadir=/erigon_data --chain bsc
  3. start erigon
  4. wait
  5. this issues occurs

Related issues

I found two other people here have the same issues: #404 #381

EDIT 07/05 10:05: add full log in debug level

STdevK commented 2 weeks ago

Can you share the first method node start-up command?

aploium commented 2 weeks ago

Can you share the first method node start-up command?

The first and second method use same start-up config (given above), there is no extra settings in command 

erigon --config config.yaml --datadir /erigon_data

Update: I've kept the node started, and it still stucking 3 days in this loop

STdevK commented 2 weeks ago

From the Second way, can you unwind some blocks and try again?

  1. Stop erigon node
  2. run make integration
  3. run ./build/bin/integration stage_exec --datadir ./data/ --chain bsc --unwind 10
  4. Start erigon node
darkhorse-spb commented 2 weeks ago

same problem here

aploium commented 2 weeks ago

From the Second way, can you unwind some blocks and try again?

  1. Stop erigon node
  2. run make integration
  3. run ./build/bin/integration stage_exec --datadir ./data/ --chain bsc --unwind 10
  4. Start erigon node

Hello, I've did this 10 hrs ago, and it just stucked in logs like this

[INFO] [07-11|09:07:58.777] [2/15 Headers] No block headers to write in this log period number=39908401 blk/second=0.000 alloc=3.5GB sys=6.9GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [07-11|09:07:58.778] Req/resp stats                           req=121 reqMin=40260056 reqMax=40260368 skel=25 skelMin=39908400 skelMax=39945456 resp=121 respMin=40260249 respMax=40260369 dups=0
[INFO] [07-11|09:08:28.777] [2/15 Headers] No block headers to write in this log period number=39908401 blk/second=0.000 alloc=3.7GB sys=6.9GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [07-11|09:08:28.777] Req/resp stats                           req=123 reqMin=40259933 reqMax=40260247 skel=25 skelMin=39908400 skelMax=39945456 resp=123 respMin=40260126 respMax=40260248 dups=0
[INFO] [07-11|09:08:58.777] [2/15 Headers] No block headers to write in this log period number=39908401 blk/second=0.000 alloc=4.0GB sys=6.9GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [07-11|09:08:58.777] Req/resp stats                           req=122 reqMin=40259811 reqMax=40260124 skel=26 skelMin=39908400 skelMax=39945456 resp=123 respMin=40260004 respMax=40260125 dups=1
[INFO] [07-11|09:09:27.242] [p2p] GoodPeers                          eth68=48
[INFO] [07-11|09:09:27.304] [mem] memory stats                       Rss=4.8GB Size=0B Pss=4.8GB SharedClean=2.7MB SharedDirty=0B PrivateClean=409.1MB PrivateDirty=4.4GB Referenced=4.7GB Anonymous=4.4GB Swap=170.9MB alloc=4.2GB sys=6.9GB
[INFO] [07-11|09:09:28.777] [2/15 Headers] No block headers to write in this log period number=39908401 blk/second=0.000 alloc=4.2GB sys=6.9GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [07-11|09:09:28.777] Req/resp stats                           req=123 reqMin=40259688 reqMax=40260002 skel=25 skelMin=39908400 skelMax=39945456 resp=125 respMin=40259881 respMax=40260003 dups=2
[INFO] [07-11|09:09:58.777] [2/15 Headers] No block headers to write in this log period number=39908401 blk/second=0.000 alloc=3.4GB sys=6.9GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [07-11|09:09:58.777] Req/resp stats                           req=124 reqMin=40259564 reqMax=40259879 skel=25 skelMin=39908400 skelMax=39945456 resp=132 respMin=40259757 respMax=40259880 dups=8

I notice the block no 40259881, my first try and second try are both this block number, could this block be something special?

blxdyx commented 2 weeks ago

Try rm datadir/snapshots/prohibit_new_downloads.lock datdir/snapshots/snapshots-lock.json and restart.

blxdyx commented 2 weeks ago

@aploium Please show print stages

[WARN] [07-05|10:02:03.193] [1/15 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=38699999 min_in_db=39818000

Use the latest main branch. and rm datadir/snapshots/prohibit_new_downloads.lock datdir/snapshots/snapshots-lock.json and restart, it will download the missed snapshots.

blxdyx commented 1 week ago

reopen if still have issue