[After Upgrade to v2.50] ReadBodyWithTransactions can't find block num

c0deright commented 1 year ago

System information

Erigon version: 2.50.1

OS & Version: Ubuntu 20.04

Erigon Command (with flags/config):

[..]
[Service]
Type=simple
ExecStart=erigon-client --datadir /data/goerli2/erigon --port 30305 --p2p.allowed-ports 30306 --http --http.addr 0.0.0.0 --http.port 8547 --http.api "trace,eth,net,web3" --http.vhosts='*' --maxpeers 30 --private.api.addr '' --torrent.port 42069 --torrent.download.rate 100mb --prune hrtc --prune.h.older 2300000 --prune.r.older 2300000 --prune.t.older 2300000 --prune.c.older 2300000 --authrpc.port 8553 --chain goerli --authrpc.jwtsecret /data/goerli2/jwtsecret
KillSignal=SIGINT

Restart=always
PrivateTmp=true
TimeoutStopSec=3600s
TimeoutStartSec=10s
StartLimitInterval=120s
StartLimitBurst=5

Concensus Layer: prysm v4.0.8

Concensus Layer Command (with flags/config):

[..]
[Service]
Type=simple
ExecStart=prysm-client --datadir=/data/goerli2/prysm --grpc-gateway-port 3502 --rpc-port 4002 --p2p-tcp-port 13002 --p2p-udp-port 12002 --monitoring-port 8082 --execution-endpoint=http://localhost:8553 --jwt-secret=/data/goerli2/jwtsecret --block-batch-limit 1024 --goerli --genesis-state=/data/goerli2/prysm/genesis.ssz --checkpoint-sync-url https://goerli-sync.invis.tools/
StandardOutput=append:/data/goerli2/prysm/log.txt

Chain/Network: goerli

Expected behaviour

Erigon does sync like it did with v2.48.1 before we upgraded to v2.50.0

Actual behaviour

Erigon doesn't sync with v2.50.0 and v2.50.1 and reports errors

Steps to reproduce the behaviour

Upgrade from v2.48.1 to v2.50.0 on goerli testnet

Backtrace

[INFO] [10-04|11:12:25.526] logging to file system                   log dir=/data/goerli2/erigon/logs file prefix=erigon log level=info json=false
[INFO] [10-04|11:12:25.526] Build info                               git_branch= git_tag= git_commit=
[INFO] [10-04|11:12:25.526] Starting Erigon on Görli testnet... 
[INFO] [10-04|11:12:25.529] Maximum peer count                       ETH=30 total=30
[INFO] [10-04|11:12:25.529] starting HTTP APIs                       APIs=trace,eth,net,web3
[INFO] [10-04|11:12:25.529] torrent verbosity                        level=WRN
[INFO] [10-04|11:12:27.633] Set global gas cap                       cap=50000000
[INFO] [10-04|11:12:27.635] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=100mb upload.rate=4mb
[INFO] [10-04|11:12:27.635] Opening Database                         label=chaindata path=/data/goerli2/erigon/chaindata
[INFO] [10-04|11:12:30.037] Initialised chain configuration          config="{ChainID: 5, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 1561651, Muir Glacier: <nil>, Berlin: 4460644, London: 5062605, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: 10790000, Merge Netsplit: <nil>, Shanghai: 1678832736, Cancun: <nil>, Prague: <nil>, Engine: clique}" genesis=0xbf7e331f7f7c1dd2e05159666b3bf8bc7a8a3a9eb1d518969eab529dd9b88c1a
[INFO] [10-04|11:12:50.044] [snapshots] Creating .torrent files      progress=56/56 alloc=812.3MB sys=860.4MB
[INFO] [10-04|11:12:51.826] Initialising Ethereum protocol           network=5
[INFO] [10-04|11:12:51.826] Opening Database                         label=clique path=/data/goerli2/erigon/clique/db/clique
[INFO] [10-04|11:13:03.913] new subscription to logs established 
[INFO] [10-04|11:13:03.913] rpc filters: subscribing to Erigon events 
[INFO] [10-04|11:13:03.913] new subscription to newHeaders established 
[INFO] [10-04|11:13:03.913] New txs subscriber joined 
[INFO] [10-04|11:13:03.915] HTTP endpoint opened                     url=[::]:8547 ws=false ws.compression=true grpc=false
[INFO] [10-04|11:13:03.915] Reading JWT secret                       path=/data/goerli2/jwtsecret
[INFO] [10-04|11:13:03.919] HTTP endpoint opened for Engine API      url=127.0.0.1:8553 ws=true ws.compression=true
[WARN] [10-04|11:13:03.923] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
[INFO] [10-04|11:13:03.924] Started P2P networking                   version=67 self=enode://344895ebf2c1ebe18298b2f189054ec7aaf673b6d3bab908f2c5f2179e098a93c69c89be9a1565c459a7694bcaad1ba8f0532d8d8b4e549e76f5941d2bc02a32@127.0.0.1:0 name=erigon/v2.50.1/linux-amd64/go1.20.7
[WARN] [10-04|11:13:03.926] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
[INFO] [10-04|11:13:03.927] [1/12 Snapshots] Fetching torrent files metadata 
[INFO] [10-04|11:13:03.927] Started P2P networking                   version=68 self=enode://344895ebf2c1ebe18298b2f189054ec7aaf673b6d3bab908f2c5f2179e098a93c69c89be9a1565c459a7694bcaad1ba8f0532d8d8b4e549e76f5941d2bc02a32@127.0.0.1:0 name=erigon/v2.50.1/linux-amd64/go1.20.7
[INFO] [10-04|11:13:03.928] [txpool] Started 
[INFO] [10-04|11:13:04.745] [snapshots] Blocks Stat                  blocks=9509k indices=9509k alloc=1.5GB sys=1.6GB
[INFO] [10-04|11:13:06.034] [snapshots] Retire Blocks                range=9509k-9510k
[INFO] [10-04|11:13:06.081] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0xa529bea74782d3153eb7b16dd0e970361827267ea54fb48bb8f88de029c494ba requestId=0
[INFO] [10-04|11:13:11.846] [snapshots] Creating .torrent files      progress=56/56 alloc=1.1GB sys=1.6GB
[INFO] [10-04|11:13:24.506] [snapshots] Compression                  ratio=0.00 file=v1-009509-009510-transactions
[INFO] [10-04|11:13:51.450] [snapshots] Blocks Stat                  blocks=9510k indices=9510k alloc=1.0GB sys=1.9GB
[INFO] [10-04|11:15:42.535] [EngineBlockDownloader] Processed        highest=9806982
[INFO] [10-04|11:15:42.535] Beginning downloaded blocks insertion 
[INFO] [10-04|11:15:42.606] [EngineBlockDownloader] Finished downloading blocks from=9806977 to=9806982
[INFO] [10-04|11:15:48.762] [NewPayload] Handling new payload        height=9806999 hash=0xbcccd53d416fd77b3211b2b30b93b377db5651b1c280a42cb8233d8614fc0f9d
[INFO] [10-04|11:15:48.762] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0xf8ba2490abd913dd21b8d3dfa5f1ca9ce1db914e518adca3c11afe76120977d2 requestId=0
[INFO] [10-04|11:15:50.162] [EngineBlockDownloader] Processed        highest=9806998
[INFO] [10-04|11:15:50.162] Beginning downloaded blocks insertion 
[INFO] [10-04|11:15:50.243] [EngineBlockDownloader] Finished downloading blocks from=9806982 to=9806998
[INFO] [10-04|11:15:50.308] [EngineBlockDownloader] block verification skipped 
[INFO] [10-04|11:15:51.817] [p2p] GoodPeers                          eth68=1
[INFO] [10-04|11:16:00.597] [NewPayload] Handling new payload        height=9807000 hash=0x8c578fbbbeb01014f97dfa8c9552c0a8cc39c908ad5aebe4c91adda23f3d62cd
[INFO] [10-04|11:16:01.691] [3/12 Senders] Started                   from=9636433 to=9807000
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636434 hash=0xeab6c3d821a81851bb4f4486e0cd08e859ce46ea33966500be37f021d3b2ec47
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636435 hash=0x4cf39ef158b918873824a556350e76298fd2485ba4715d28b23b2d6c0f641755
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636436 hash=0x0164c85f1d145b740000a6382ac4436c585f6cdfd4ea33ca45f33caf13612233
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636437 hash=0xdca0a245da80cd68ff0e8c64d1bb7d51419e733d554f883657292f6eb89efc1f
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636438 hash=0xce65598e79b7584cf3318f13eecda7653ebbd47eefb63936e5558ef4eb6f7871
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636439 hash=0xf171ff90a324f685e3710de3fc5270d128741718292d482ca68e35fb78f6cd5a
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636440 hash=0x5a261ff39e60adb2c78accf67179a3a37b9bb7aab85a50af461534569ab0b852
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636441 hash=0x58920a16da68b62c0b4fc16796ec1c6b58258582896098aaca51c249de8b4c6c
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636442 hash=0xbb7d121415a15dfbff5a45aa8b45ea92dc95fa86c3886232773c946e5a9ea9cd
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636443 hash=0xfd5dca55f8044cc50d6fc25c7f5dc5541bda4ebdf20ce03c77e60deffc8c38d6
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636444 hash=0xd284fc0f8e3a7bf5c27e5ed58d9ff77a2fcefc154ab69b0c7f35ffd87dcae8a6
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636445 hash=0x2740b437c8b741e9109b8b9014440ddcc3b62ef9a44339509b881a8290432a6b
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636446 hash=0x1f62f97d55ba3a023797bd6a1acb06730ee8a926546d83a8a8912257bc966ed8
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636447 hash=0x2d5588dff5c67f10b9cb2f74923bc9340c860ea0b0259965936ef6a35588e8fa
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636448 hash=0x3842bfd4329f392db3c966ac947b39c7d4c639543a3f37a867bd9188df421deb
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636449 hash=0x5085794194479cb7e583c141c67141c343c2c8396a2253fb6ce1ae76f7de1ed5
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636450 hash=0x21af93ff4dba4b14c7ebe4d683f5bccaaedc2686055114f48e7d06a49a61aaee
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636451 hash=0xcaa68ddb2be0620f826d041594ba96e6d69165311831169fe107ad2f442c022d
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636452 hash=0x2fba86935e0990bdec49f777fd7bc9f36415abd2d9c33032c1323ad48fcb851b
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636453 hash=0x13b13930a5cb1ffc19603e42926fbb37d77544d8ac908b3b71ba69bc7c076865
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636454 hash=0x4eeb03e1674f80a3a678817d9b894cf6a7f6c72131e2be5441d2644b763fe5be
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636455 hash=0x122266657947172547f16c6b699e907c519f9bdedb943795d1bc673b2625581d
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636456 hash=0x754187821768b0983f23c1791bd81616d55bc39e865b62acebd29b92d32a6b96
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636457 hash=0xd3a88d402a1d47b1454b43b6b3727ae3da1b7f001adefeec04bcaf8f93507601
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636458 hash=0x66bba50b7520392e976126697763556ef375c86de3e12d111fbcae59c3310e0d
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636459 hash=0xb97e595e686db546bc5d96f548c1f0252fc4447c1c055e252b17d5251afbf5ff
[WARN] [10-04|11:16:01.691] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636460 hash=0x05bd34ba89c1f7d722009eb060b96ae79b22a43d48f6d7108ce577ecbd56cb1c
[..]
[WARN] [10-04|11:16:01.709] [3/12 Senders] ReadBodyWithTransactions can't find block num=9637608 hash=0xa90d6dbb534205674a3097c5c8bc45b92a0df12cfdf124416f42bcf50d5848a2
[WARN] [10-04|11:16:01.709] [3/12 Senders] ReadBodyWithTransactions can't find block num=9637609 hash=0x71cd2e538c042fe0f8b35552e09553cb3fe9657485fea87c209e7386075be4ce
[WARN] [10-04|11:16:01.709] [3/12 Senders] ReadBodyWithTransactions can't find block num=9637610 hash=0x36d3c2228f4298da39dbcf5f60c46c96856f6ea7046f4b75d0d2c154259c91dd
[WARN] [10-04|11:16:01.709] [3/12 Senders] ReadBodyWithTransactions can't find block num=9637611 hash=0x30fa4e745396662a8fa6601df5e8cfa71576f80dae9b2d3c232ec2d6338bc2e3
[WARN] [10-04|11:16:01.709] [3/12 Senders] ReadBodyWithTransactions can't find block num=9637612 hash=0xa99aed1cdaf9840e44fa460d597767b6299af60d5f432f4e90fa6b284dc07701
[INFO] [10-04|11:16:03.916] [txpool] stat                            pending=5 baseFee=0 queued=140 alloc=1.8GB sys=2.7GB
[INFO] [10-04|11:16:31.691] [3/12 Senders] Recovery                  block_number=9667512 ch=9997/10000
[INFO] [10-04|11:17:01.702] [3/12 Senders] Recovery                  block_number=9695818 ch=9984/10000
[INFO] [10-04|11:17:31.706] [3/12 Senders] Recovery                  block_number=9726165 ch=9977/10000
[INFO] [10-04|11:18:01.703] [3/12 Senders] Recovery                  block_number=9755516 ch=10000/10000
[INFO] [10-04|11:18:31.691] [3/12 Senders] Recovery                  block_number=9784332 ch=10000/10000
[INFO] [10-04|11:18:50.964] [3/12 Senders] DONE                      in=2m49.273216585s
[INFO] [10-04|11:18:50.964] [4/12 Execution] Blocks execution        from=9636433 to=9807000
[EROR] [10-04|11:18:50.964] [4/12 Execution] Empty block             blocknum=9636434
[INFO] [10-04|11:18:50.964] [4/12 Execution] Completed on            block=9636433
[WARN] [10-04|11:18:50.964] bad forkchoice                           head=0xce85d8afa5a910884025de9c3098fded14c06dbabfcca26e9838117c651e08f7 hash=0x8c578fbbbeb01014f97dfa8c9552c0a8cc39c908ad5aebe4c91adda23f3d62cd
[INFO] [10-04|11:18:51.817] [p2p] GoodPeers                          eth67=1 eth68=1
[INFO] [10-04|11:19:00.580] [NewPayload] Handling new payload        height=9807013 hash=0x36ee3e2568179e7b1c38363bbc919c97fd62e41089955f8363451dac72089451
[INFO] [10-04|11:19:00.581] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0x765003b61f017c45602b83a71fd76a5484fffcb2f099ef2b9dad8ce52943071d requestId=0
[INFO] [10-04|11:19:02.830] [EngineBlockDownloader] Processed        highest=9807012
[INFO] [10-04|11:19:02.830] Beginning downloaded blocks insertion 
[INFO] [10-04|11:19:02.925] [EngineBlockDownloader] Finished downloading blocks from=9807000 to=9807012
[INFO] [10-04|11:19:02.998] [EngineBlockDownloader] block verification skipped 
[INFO] [10-04|11:19:03.914] [txpool] stat                            pending=11 baseFee=0 queued=554 alloc=3.7GB sys=7.2GB
[INFO] [10-04|11:19:16.075] [NewPayload] Handling new payload        height=9807014 hash=0xb11d93573acca80f29a9c7c20043b51400548d5cde52b88ab03ea3c4e781a054
[INFO] [10-04|11:19:17.122] [3/12 Senders] Started                   from=9636433 to=9807014
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636434 hash=0xeab6c3d821a81851bb4f4486e0cd08e859ce46ea33966500be37f021d3b2ec47
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636435 hash=0x4cf39ef158b918873824a556350e76298fd2485ba4715d28b23b2d6c0f641755
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636436 hash=0x0164c85f1d145b740000a6382ac4436c585f6cdfd4ea33ca45f33caf13612233
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636437 hash=0xdca0a245da80cd68ff0e8c64d1bb7d51419e733d554f883657292f6eb89efc1f
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636438 hash=0xce65598e79b7584cf3318f13eecda7653ebbd47eefb63936e5558ef4eb6f7871
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636439 hash=0xf171ff90a324f685e3710de3fc5270d128741718292d482ca68e35fb78f6cd5a
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636440 hash=0x5a261ff39e60adb2c78accf67179a3a37b9bb7aab85a50af461534569ab0b852
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636441 hash=0x58920a16da68b62c0b4fc16796ec1c6b58258582896098aaca51c249de8b4c6c
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636442 hash=0xbb7d121415a15dfbff5a45aa8b45ea92dc95fa86c3886232773c946e5a9ea9cd
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636443 hash=0xfd5dca55f8044cc50d6fc25c7f5dc5541bda4ebdf20ce03c77e60deffc8c38d6
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636444 hash=0xd284fc0f8e3a7bf5c27e5ed58d9ff77a2fcefc154ab69b0c7f35ffd87dcae8a6
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636445 hash=0x2740b437c8b741e9109b8b9014440ddcc3b62ef9a44339509b881a8290432a6b
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636446 hash=0x1f62f97d55ba3a023797bd6a1acb06730ee8a926546d83a8a8912257bc966ed8
[WARN] [10-04|11:19:17.123] [3/12 Senders] ReadBodyWithTransactions can't find block num=9636447 hash=0x2d5588dff5c67f10b9cb2f74923bc9340c860ea0b0259965936ef6a35588e8fa
[..]

matcra587 commented 1 year ago

We've observed that this issue arises when the node is not set as an archive node. On the other hand, when configured as an archive node, version 2.50.2 runs without any problems

qwertzlbert commented 1 year ago

We observed the same issue on mainnet with version 2.51.0 , but only with nodes running with a pruned config and started like the following:

--chain=mainnet --datadir=/srv/ethereum/erigon/mainnet/data --private.api.addr=localhost:9090 --metrics --metrics.port=26660 --http.addr 0.0.0.0 --ws --http --rpc.batch.concurrency=12 --db.pagesize=16kb --prune=htc --prune.r.before=11184524 --http.api=eth,erigon,web3,net,trace,txpool

Any update on that topic would be appreciated. Thanks

RatanRSur commented 1 year ago

I'm seeing the same issue on mainnet. Version: erigon version 2.51.0-a952e91f

My config:

prune.r.before: "11052984"
prune: "htc"
http: "true"
http.api: ["eth","erigon","engine", "debug", "net"]
datadir: "/home/val/mnt"

logs:

[INFO] [10-11|13:36:11.748] [NewPayload] Handling new payload        height=18328723 hash=0x057bcab966f2552622cedcb299f2601063dbf5e613fad2a48497a8ea3a1aef3c
[INFO] [10-11|13:36:11.748] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0x6361abb0101a440c0d165768cdc8ebe594d1a3955bd793e3281f9e0cc671af30 requestId=0
[INFO] [10-11|13:36:45.393] [EngineBlockDownloader] Processed        highest=18328722
[INFO] [10-11|13:36:45.393] Beginning downloaded blocks insertion 
[INFO] [10-11|13:36:45.397] [EngineBlockDownloader] Finished downloading blocks from=18328720 to=18328722
[INFO] [10-11|13:36:52.914] [EngineBlockDownloader] blocks verification successful 
[INFO] [10-11|13:36:58.032] [p2p] GoodPeers                          eth68=3 eth66=2 eth67=2
[INFO] [10-11|13:37:00.822] [NewPayload] Handling new payload        height=18328727 hash=0x96c03ad08abf2695c559d212a0ac3aca88b1654b5e9f402edcd36b649edcedee
[INFO] [10-11|13:37:00.822] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0x1258e0b4f2fe3f10c208b23e6720311fd5456d226c2856b4797e8d49731bff6e requestId=0
[INFO] [10-11|13:37:01.401] [txpool] stat                            pending=9784 baseFee=0 queued=667 alloc=4.0GB sys=4.5GB
[INFO] [10-11|13:37:05.022] [EngineBlockDownloader] Processed        highest=18328726
[INFO] [10-11|13:37:05.022] Beginning downloaded blocks insertion 
[INFO] [10-11|13:37:05.034] [EngineBlockDownloader] Finished downloading blocks from=18328723 to=18328726
[INFO] [10-11|13:37:05.038] [EngineBlockDownloader] block verification skipped 
[INFO] [10-11|13:37:11.458] [NewPayload] Handling new payload        height=18328728 hash=0x0aabc6c9ef80263d243c48092d13802fc31640d167c7a78dc34619fb64d96e01
[INFO] [10-11|13:37:11.680] [3/12 Senders] Started                   from=18328692 to=18328728
[WARN] [10-11|13:37:11.681] [3/12 Senders] ReadBodyWithTransactions can't find block num=18328693 hash=0xdd01e15b1d59cb7bb69406849857c707317d8494422fb15338c81eefe1bb5340
[INFO] [10-11|13:37:11.780] [4/12 Execution] Blocks execution        from=18328692 to=18328728
[EROR] [10-11|13:37:11.780] [4/12 Execution] Empty block             blocknum=18328693
[INFO] [10-11|13:37:11.780] [4/12 Execution] Completed on            block=18328692
[WARN] [10-11|13:37:11.780] bad forkchoice                           head=0xf0ac68f1db375d345d0dd303267febcb142e5f768a2b25a7f70c1175f301f71b hash=0x0aabc6c9ef80263d243c48092d13802fc31640d167c7a78dc34619fb64d96e01
[INFO] [10-11|13:37:11.858] [3/12 Senders] Started                   from=18328692 to=18328727
[WARN] [10-11|13:37:11.858] [3/12 Senders] ReadBodyWithTransactions can't find block num=18328693 hash=0xdd01e15b1d59cb7bb69406849857c707317d8494422fb15338c81eefe1bb5340
[INFO] [10-11|13:37:12.013] [4/12 Execution] Blocks execution        from=18328692 to=18328727
[EROR] [10-11|13:37:12.013] [4/12 Execution] Empty block             blocknum=18328693
[INFO] [10-11|13:37:12.013] [4/12 Execution] Completed on            block=18328692
[WARN] [10-11|13:37:12.013] bad forkchoice                           head=0xf0ac68f1db375d345d0dd303267febcb142e5f768a2b25a7f70c1175f301f71b hash=0x96c03ad08abf2695c559d212a0ac3aca88b1654b5e9f402edcd36b649edcedee
[INFO] [10-11|13:37:12.089] [3/12 Senders] Started                   from=18328692 to=18328726
[WARN] [10-11|13:37:12.089] [3/12 Senders] ReadBodyWithTransactions can't find block num=18328693 hash=0xdd01e15b1d59cb7bb69406849857c707317d8494422fb15338c81eefe1bb5340
[INFO] [10-11|13:37:12.237] [4/12 Execution] Blocks execution        from=18328692 to=18328726
[EROR] [10-11|13:37:12.237] [4/12 Execution] Empty block             blocknum=18328693
[INFO] [10-11|13:37:12.238] [4/12 Execution] Completed on            block=18328692

qwertzlbert commented 1 year ago

Since 2.52.0 the status has changed, instead of the ReadBodyWithTransactions can't find block the node will start to sync again, but will never finish and seems to be stuck at block 0x10f2076 and the consensus client won't be able to use it (teku in our case).

Erigon log:

Oct 13 10:57:48 erigon[2701335]: [INFO] [10-13|10:57:48.165] [EngineBlockDownloader] Processed        highest=18341021
Oct 13 10:57:48 erigon[2701335]: [INFO] [10-13|10:57:48.165] Beginning downloaded blocks insertion
Oct 13 10:57:48 erigon[2701335]: [INFO] [10-13|10:57:48.165] [EngineBlockDownloader] Finished downloading blocks from=18341020 to=18341021
Oct 13 10:57:48 erigon[2701335]: [INFO] [10-13|10:57:48.166] [EngineBlockDownloader] block verification skipped
Oct 13 10:58:00 erigon[2701335]: [INFO] [10-13|10:58:00.241] [NewPayload] Handling new payload        height=18341023 hash=0x5509b0c6feee955843659c25adb82289d6f7125f8bfef994dfc8854db8371e67
Oct 13 10:58:12 erigon[2701335]: [INFO] [10-13|10:58:12.089] [NewPayload] Handling new payload        height=18341024 hash=0x3638f064c1edc17453653d4a4d8148458e35eecbcde2dc17f0d06d40381a7046
Oct 13 10:58:24 erigon[2701335]: [INFO] [10-13|10:58:24.897] [NewPayload] Handling new payload        height=18341025 hash=0x6b0701eae1bf5697dfef5c2fc5e75d0af04be947d82b13bb9df80f281cf2f0d9
Oct 13 10:58:36 erigon[2701335]: [INFO] [10-13|10:58:36.035] [NewPayload] Handling new payload        height=18341026 hash=0xbcda834a3bf1df27d698f06baf44fefb375346174daf671963cd6ac1f789e0d9
Oct 13 10:58:48 erigon[2701335]: [INFO] [10-13|10:58:48.102] [NewPayload] Handling new payload        height=18341027 hash=0x3e0d94a0e75b212dea8e5c92c92655b09ebeb88abae62011cc7cf99fbdfa21db
Oct 13 10:58:59 erigon[2701335]: [INFO] [10-13|10:58:59.606] [NewPayload] Handling new payload        height=18341028 hash=0xdf3d45bce1aec7b871a60ec5825ff7af30a4a5347885b35471b1a1942f9b294a
Oct 13 10:59:11 erigon[2701335]: [INFO] [10-13|10:59:11.849] [NewPayload] Handling new payload        height=18341029 hash=0x7410c2678ac1f69f05f4d01eda77a9e0fc468fb0961f9dd1d79e4d11eb265556
Oct 13 10:59:24 erigon[2701335]: [INFO] [10-13|10:59:24.712] [NewPayload] Handling new payload        height=18341030 hash=0x954c34f25ea89a8a18b1084f7a09f31782d42c0b8fa8f183572b17f5b11ffcc4
Oct 13 10:59:31 erigon[2701335]: [INFO] [10-13|10:59:31.555] [p2p] GoodPeers                          eth68=32 eth67=32 eth66=1
Oct 13 10:59:32 erigon[2701335]: [INFO] [10-13|10:59:32.560] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=3.3GB sys=9.7GB
Oct 13 10:59:37 erigon[2701335]: [INFO] [10-13|10:59:37.003] [NewPayload] Handling new payload        height=18341031 hash=0x983c08b0b1d66e89e4512ae900a5cbcc8405ff99d773f4ee584b403d909e7b51

Teku log (for reference only):

Oct 13 10:57:48 teku[3223992]: 10:57:48.298 INFO  - Syncing completed
Oct 13 10:58:00 teku[3223992]: 10:58:00.345 WARN  - Unable to execute the current chain head block payload because the Execution Client is syncing. Activating optimistic sync of the beacon chain node
Oct 13 10:58:00 teku[3223992]: 10:58:00.707 INFO  - Syncing     *** Target slot: 7530888, Head slot: 7530888, Waiting for execution layer sync, Connected peers: 50
Oct 13 10:58:11 teku[3223992]: 10:58:11.021 INFO  - Syncing     *** Target slot: 7530889, Head slot: 7530888, Waiting for execution layer sync, Connected peers: 69
Oct 13 10:58:23 teku[3223992]: 10:58:23.046 INFO  - Syncing     *** Target slot: 7530890, Head slot: 7530889, Waiting for execution layer sync, Connected peers: 82
Oct 13 10:58:35 teku[3223992]: 10:58:35.048 INFO  - Syncing     *** Target slot: 7530891, Head slot: 7530890, Waiting for execution layer sync, Connected peers: 99
Oct 13 10:58:47 teku[3223992]: 10:58:47.051 INFO  - Syncing     *** Target slot: 7530892, Head slot: 7530891, Waiting for execution layer sync, Connected peers: 98
Oct 13 10:58:59 teku[3223992]: 10:58:59.054 INFO  - Syncing     *** Target slot: 7530893, Head slot: 7530892, Waiting for execution layer sync, Connected peers: 99
Oct 13 10:59:11 teku[3223992]: 10:59:11.062 INFO  - Syncing     *** Target slot: 7530894, Head slot: 7530893, Waiting for execution layer sync, Connected peers: 99
Oct 13 10:59:23 teku[3223992]: 10:59:23.057 INFO  - Syncing     *** Target slot: 7530895, Head slot: 7530894, Waiting for execution layer sync, Connected peers: 98
Oct 13 10:59:35 teku[3223992]: 10:59:35.057 INFO  - Syncing     *** Target slot: 7530896, Head slot: 7530895, Waiting for execution layer sync, Connected peers: 99
Oct 13 10:59:47 teku[3223992]: 10:59:47.062 INFO  - Syncing     *** Target slot: 7530897, Head slot: 7530896, Waiting for execution layer sync, Connected peers: 100
Oct 13 10:59:59 teku[3223992]: 10:59:59.067 INFO  - Syncing     *** Target slot: 7530898, Head slot: 7530897, Waiting for execution layer sync, Connected peers: 99
Oct 13 11:00:11 teku[3223992]: 11:00:11.064 INFO  - Syncing     *** Target slot: 7530899, Head slot: 7530898, Waiting for execution layer sync, Connected peers: 99
Oct 13 11:00:23 teku[3223992]: 11:00:23.061 INFO  - Syncing     *** Target slot: 7530900, Head slot: 7530899, Waiting for execution layer sync, Connected peers: 99
Oct 13 11:00:35 teku[3223992]: 11:00:35.066 INFO  - Syncing     *** Target slot: 7530901, Head slot: 7530900, Waiting for execution layer sync, Connected peers: 99
Oct 13 11:00:47 teku[3223992]: 11:00:47.075 INFO  - Syncing     *** Target slot: 7530902, Head slot: 7530901, Waiting for execution layer sync, Connected peers: 99
Oct 13 11:00:59 teku[3223992]: 11:00:59.069 INFO  - Syncing     *** Target slot: 7530903, Head slot: 7530902, Waiting for execution layer sync, Connected peers: 98
Oct 13 11:01:11 teku[3223992]: 11:01:11.070 INFO  - Syncing     *** Target slot: 7530904, Head slot: 7530903, Waiting for execution layer sync, Connected peers: 98
Oct 13 11:01:23 teku[3223992]: 11:01:23.105 INFO  - Syncing     *** Target slot: 7530905, Head slot: 7530904, Waiting for execution layer sync, Connected peers: 99
Oct 13 11:01:35 teku[3223992]: 11:01:35.080 INFO  - Syncing     *** Target slot: 7530906, Head slot: 7530905, Waiting for execution layer sync, Connected peers: 100
Oct 13 11:01:47 teku[3223992]: 11:01:47.070 INFO  - Syncing     *** Target slot: 7530907, Head slot: 7530906, Waiting for execution layer sync, Connected peers: 100
Oct 13 11:01:59 teku[3223992]: 11:01:59.072 INFO  - Syncing     *** Target slot: 7530908, Head slot: 7530907, Waiting for execution layer sync, Connected peers: 99
Oct 13 11:02:11 teku[3223992]: 11:02:11.092 INFO  - Syncing     *** Target slot: 7530909, Head slot: 7530908, Waiting for execution layer sync, Connected peers: 98
Oct 13 11:02:23 teku[3223992]: 11:02:23.076 INFO  - Syncing     *** Target slot: 7530910, Head slot: 7530909, Waiting for execution layer sync, Connected peers: 100
Oct 13 11:02:35 teku[3223992]: 11:02:35.078 INFO  - Syncing     *** Target slot: 7530911, Head slot: 7530910, Waiting for execution layer sync, Connected peers: 100
Oct 13 11:02:35 teku[3223992]: 11:02:35.502 INFO  - Epoch Event *** Epoch: 235341, Justified checkpoint: 235339, Finalized checkpoint: 235338, Finalized root: f1be2435874a7238e2e3e499f9780c3150f78238ba8e793badab021f65e77db0
Oct 13 11:02:51 teku[3223992]: 11:02:51.927 INFO  - Syncing     *** Target slot: 7530912, Head slot: 7530911, Waiting for execution layer sync, Connected peers: 100
Oct 13 11:02:59 teku[3223992]: 11:02:59.082 INFO  - Syncing     *** Target slot: 7530913, Head slot: 7530912, Waiting for execution layer sync, Connected peers: 99
Oct 13 11:03:00 teku[3223992]: 11:03:00.575 INFO  - Syncing completed
Oct 13 11:03:12 teku[3223992]: 11:03:12.138 WARN  - Unable to execute the current chain head block payload because the Execution Client is syncing. Activating optimistic sync of the beacon chain node

As new versions were released in the meantime we will continue to try these...

Update:

same as above also happens with 2.52.4
We will update Teku now as 23.10.0 added support for certain messages and above issue might be related to that

Update 2:

after upgrading Teku to 23.10.0 the full nodes are still not able to sync and teku won't connect, but normal archive nodes are able to work again though. This however seems to be unrelated to the issue in this ticket and I just updated the comment in case others face similar issues

RatanRSur commented 1 year ago

also seeing same new behavior after updating

c0deright commented 1 year ago

Still not syncing with v.2.52.5.

The error is gone, the log indicates erigon is downloading recent blocks but it skips block verification and the eth_blockNumber api call returns a block number from 3 weeks ago when we upgraded from v2.48.1 to 2.50.x.

qwertzlbert commented 1 year ago

I just upgrade to v2.53.2 and it seems like the nodes are working again. Erigon is syncing again and Consensus layer client is also able to connect again.

odilatinam commented 9 months ago

I had a backup pruned node crash in mid December. When I upgraded from v2.48.1 to v2.57.0 I ran into a similar issue, except it doesn't resolve itself and just keeps going in a loop. My main archive node was ok with the same upgrade (but it didn't crash in mid December and was up to date before upgrading on January 20).

Jan 22 08:37:46 p5530 erigon[2600]: [INFO] [01-22|08:37:46.215] [3/12 Senders] Recovery                  block_number=19044799 ch=0/10000
Jan 22 08:38:16 p5530 erigon[2600]: [INFO] [01-22|08:38:16.215] [3/12 Senders] Recovery                  block_number=19051026 ch=0/10000
Jan 22 08:38:46 p5530 erigon[2600]: [INFO] [01-22|08:38:46.215] [3/12 Senders] Recovery                  block_number=19057009 ch=0/10000
Jan 22 08:39:04 p5530 erigon[2600]: [INFO] [01-22|08:39:04.800] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-4049101880
Jan 22 08:39:06 p5530 erigon[2600]: [INFO] [01-22|08:39:06.260] [3/12 Senders] DONE                      in=17m20.046197842s
Jan 22 08:39:06 p5530 erigon[2600]: [INFO] [01-22|08:39:06.262] [4/12 Execution] Blocks execution        from=18799999 to=19061054
Jan 22 08:39:06 p5530 erigon[2600]: [EROR] [01-22|08:39:06.265] [4/12 Execution] Empty block             blocknum=18800000
Jan 22 08:39:06 p5530 erigon[2600]: [INFO] [01-22|08:39:06.266] [4/12 Execution] Completed on            block=18799999
Jan 22 08:39:06 p5530 erigon[2600]: [WARN] [01-22|08:39:06.268] bad forkchoice                           head=0xf144cf0a6aaca8f32498d659ef7c58f4256d2397dca536b47bd0cb7a8290fa4b hash=0x5b84253ced73b9334fae258dd0ad2fcb89bc8ce07492165eb1305f9815e36fef
Jan 22 08:39:14 p5530 erigon[2600]: [INFO] [01-22|08:39:14.145] [NewPayload] Handling new payload        height=19061142 hash=0x22ea0222c3785fe5e4ef492f70932bd3076c89848924de2261b329813e6799ab
Jan 22 08:39:14 p5530 erigon[2600]: [INFO] [01-22|08:39:14.146] [EngineBlockDownloader] Downloading PoS headers... hash=0x7b39cc0da80fe360d870cfae80584fd80dbb85615b8fab9e73cbc24ad9ab3fc3 requestId=0
Jan 22 08:40:12 p5530 erigon[2600]: [INFO] [01-22|08:40:12.983] [NewPayload] Handling new payload        height=19061143 hash=0x5520b0e749e538ca8e07c6d32e66f7c37b8c23c9de8ae34e03589e178ef32966
Jan 22 08:40:12 p5530 erigon[2600]: [INFO] [01-22|08:40:12.983] [NewPayload] Handling new payload        height=19061144 hash=0x9c5d061944495dc797e0c87907607a2e116ffd43fb9cb29ddbe3c321606f1275
Jan 22 08:40:12 p5530 erigon[2600]: [INFO] [01-22|08:40:12.984] [NewPayload] Handling new payload        height=19061145 hash=0xc9077d37f3e1a6b136bfcdc104a546c3bccb002e2b102ffcef31337cb56a58cb
Jan 22 08:40:13 p5530 erigon[2600]: [INFO] [01-22|08:40:13.006] [EngineBlockDownloader] Processing bodies... from=19061054 to=19061141
Jan 22 08:40:13 p5530 erigon[2600]: [INFO] [01-22|08:40:13.040] [NewPayload] Handling new payload        height=19061146 hash=0x050f353169056d46d8e7693a749069ee5e0e8f5590eb7d5d100ddb14c9be14f0
Jan 22 08:40:13 p5530 erigon[2600]: [INFO] [01-22|08:40:13.299] [NewPayload] Handling new payload        height=19061147 hash=0xc22bf81aef535c6b8df157f6e23c080ab985d82bfecb52ed9898e329a4fb373a
Jan 22 08:40:18 p5530 erigon[2600]: [INFO] [01-22|08:40:18.586] [p2p] GoodPeers                          eth67=1 eth68=1
Jan 22 08:40:20 p5530 erigon[2600]: [INFO] [01-22|08:40:20.787] [EngineBlockDownloader] Processed        highest=19061141
Jan 22 08:40:20 p5530 erigon[2600]: [INFO] [01-22|08:40:20.787] Beginning downloaded blocks insertion
Jan 22 08:40:20 p5530 erigon[2600]: [INFO] [01-22|08:40:20.941] [EngineBlockDownloader] Finished downloading blocks from=19061054 to=19061141
Jan 22 08:40:21 p5530 erigon[2600]: [INFO] [01-22|08:40:21.003] [EngineBlockDownloader] block verification skipped
Jan 22 08:40:21 p5530 erigon[2600]: [INFO] [01-22|08:40:21.756] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=4.9GB sys=9.5GB
Jan 22 08:40:25 p5530 erigon[2600]: [INFO] [01-22|08:40:25.730] [NewPayload] Handling new payload        height=19061148 hash=0x346cf410087d59d0c0eeac81ac68c5491d3ff4e6e74201e932c273cc470883f4
Jan 22 08:40:25 p5530 erigon[2600]: [INFO] [01-22|08:40:25.730] [EngineBlockDownloader] Downloading PoS headers... hash=0xc22bf81aef535c6b8df157f6e23c080ab985d82bfecb52ed9898e329a4fb373a requestId=0
Jan 22 08:40:29 p5530 erigon[2600]: [INFO] [01-22|08:40:29.018] [EngineBlockDownloader] Processed        highest=19061147
Jan 22 08:40:29 p5530 erigon[2600]: [INFO] [01-22|08:40:29.018] Beginning downloaded blocks insertion
Jan 22 08:40:29 p5530 erigon[2600]: [INFO] [01-22|08:40:29.087] [EngineBlockDownloader] Finished downloading blocks from=19061142 to=19061147
Jan 22 08:40:29 p5530 erigon[2600]: [INFO] [01-22|08:40:29.149] [EngineBlockDownloader] block verification skipped
Jan 22 08:40:36 p5530 erigon[2600]: [INFO] [01-22|08:40:36.432] [NewPayload] Handling new payload        height=19061149 hash=0xa3e1203c47a31d11eb504b3f24eff245e443073514b3b9255656076e1b4ab505
Jan 22 08:40:46 p5530 erigon[2600]: [INFO] [01-22|08:40:46.962] [3/12 Senders] Started                   from=18799999 to=19061149
Jan 22 08:40:46 p5530 erigon[2600]: [WARN] [01-22|08:40:46.962] [3/12 Senders] ReadBodyWithTransactions can't find block num=18800000 hash=0x15fca36cac3a1adddc81cd5032b748d883cad94dc1fe192af5ab3779e8375511
Jan 22 08:40:46 p5530 erigon[2600]: [WARN] [01-22|08:40:46.962] [3/12 Senders] ReadBodyWithTransactions can't find block num=18800001 hash=0xe68ce66b7c72b67c6a7628829c8ffa71bed7914da025cbd10e127dc1cc40355a
Jan 22 08:40:46 p5530 erigon[2600]: [WARN] [01-22|08:40:46.962] [3/12 Senders] ReadBodyWithTransactions can't find block num=18800002 hash=0xcc9dc5f7b2b90758471f494bab7c615ac0073ff758026c2966408462dbfcd3d6

Before I delete the chaindata and rebuild it from snapshots, is there something you want me to try? This is the backup pruned node service

[Service]
User=erigon
Group=erigon
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/erigon \
  --chain=mainnet \
  --datadir=/var/lib/eth/erigon \
  --http.api=engine,eth,net \
  --authrpc.jwtsecret=/var/lib/jwtsecret/jwt.hex \
  --private.api.addr= \
  --torrent.port=42068 \
  --torrent.upload.rate=25mb \
  --torrent.download.rate=200mb \
  --torrent.staticpeers=enode://669772c10d9bdbea9096abb904608f23954c7e50c6fc5d23c255ad2137de3b1869c4d7ed073097c1c2bff98ab9285bae99ca342bee935ee72793dcaab0fd6752@192.168.8.22:42069 \
  --torrent.maxpeers=1 \
  --staticpeers=enode://669772c10d9bdbea9096abb904608f23954c7e50c6fc5d23c255ad2137de3b1869c4d7ed073097c1c2bff98ab9285bae99ca342bee935ee72793dcaab0fd6752@192.168.8.22:30303 \
  --maxpeers=1 \
  --port=30301 \
  --p2p.allowed-ports=30301,30302 \
  --prune=htcr \
  --metrics

drjake007 commented 7 months ago

I'm having the same issue - stuck in a loop. I upgraded from 2.58.1 to 2.59.0 and rebuilt the node (just deleted chaindata directory) in order to take advantage of the reduced database size specified in PR #9123. I'm on 2.59.3 now and still stuck in loop. Do I have to rebuild again? Any other data directories need to be removed besides chaindata before rebuild?

[Service]
User=erigon
Group=erigon
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/erigon/build/bin/erigon \
  --chain=mainnet \
  --datadir=/var/lib/erigon \
  --http.api=engine,eth,net \
  --authrpc.jwtsecret=/var/lib/jwtsecret/jwt.hex \
  --private.api.addr= \
  --prune.r.before=11052984 \
  --prune htc \   
  --metrics \
  --maxpeers=40 \
  --log.console.verbosity=debug

Mar 29 11:02:01 ckubuntu erigon[71884]: [INFO] [03-29|11:02:01.170] [3/12 Senders] Recovery                  block_number=19538209 ch=4/10000
Mar 29 11:02:02 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:02.059] Error in DNS random node sync            tree=all.mainnet.ethdisco.net err="lookup FAX6WE4IONJ7F4P7DWMJQQZTHY.all.mainnet.ethdisco.net on 127.0.0.53:53: no such host"
Mar 29 11:02:07 ckubuntu erigon[71884]: [INFO] [03-29|11:02:07.285] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-411300847
Mar 29 11:02:07 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:07.812] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:02:08 ckubuntu erigon[71884]: [INFO] [03-29|11:02:08.892] [3/12 Senders] DONE                      in=9m37.72230006s
Mar 29 11:02:08 ckubuntu erigon[71884]: [INFO] [03-29|11:02:08.893] [4/12 Execution] Blocks execution        from=19249999 to=19541233
Mar 29 11:02:08 ckubuntu erigon[71884]: [EROR] [03-29|11:02:08.893] [4/12 Execution] Empty block             blocknum=19250000
Mar 29 11:02:08 ckubuntu erigon[71884]: [INFO] [03-29|11:02:08.894] [4/12 Execution] Completed on            block=19249999
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.894] [4/12 Execution] DONE                    in=1.365783ms
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.894] [5/12 HashState] DONE                    in=4.95µs
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.894] [6/12 IntermediateHashes] DONE           in=201.324µs
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.894] [7/12 CallTraces] DONE                   in=3.326µs
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.894] [8/12 AccountHistoryIndex] DONE          in=1.503µs
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.894] [9/12 StorageHistoryIndex] DONE          in=1.252µs
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.894] [10/12 LogIndex] DONE                    in=2.615µs
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.895] [11/12 TxLookup] DONE                    in=462.511µs
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.895] [12/12 Finish] DONE                      in=18.886µs
Mar 29 11:02:08 ckubuntu erigon[71884]: [WARN] [03-29|11:02:08.895] bad forkchoice                           head=0xaa69de29f2b185d99fec391a53282fe99bc4d5698610b93dbf345daf9aa43b60 hash=0xea9b87474c5b3d58d8337b28816cfcc8f326637524d8cdef5139c02859c78723
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unknown node=99 unsolicited reply=1037
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=1
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2415 static=0 i/o timeout=39 connect: no route to host=2 connect: connection refused=14
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 ecies: invalid message=1
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2436 static=0 connect: connection refused=15 connect: no route to host=2 i/o timeout=33
Mar 29 11:02:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:02:12 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:12.439] [txpool] Commit                          written_kb=1544 in=32.864019ms
Mar 29 11:02:23 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:23.051] [ForkChoiceUpdated] sending forkChoiceMessage head=0x1798b48a505d3513d3325c6e6328f3d59a4fec88d7f289ffc4449e5aba8208cf
Mar 29 11:02:23 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:23.060] [ForkChoiceUpdated] Handling fork choice headerHash=0x1798b48a505d3513d3325c6e6328f3d59a4fec88d7f289ffc4449e5aba8208cf
Mar 29 11:02:23 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:23.060] [ForkChoiceUpdated] Fork choice: need to download header with hash 1798b48a505d3513d3325c6e6328f3d59a4fec88d7f289ffc4449e5aba8208cf
Mar 29 11:02:23 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:23.060] [ForkChoiceUpdated] got reply            payloadStatus="&{Status:SYNCING ValidationError:<nil> LatestValidHash:<nil> CriticalError:<nil>}"
Mar 29 11:02:23 ckubuntu erigon[71884]: [INFO] [03-29|11:02:23.060] [EngineBlockDownloader] Downloading PoS headers... hash=0x1798b48a505d3513d3325c6e6328f3d59a4fec88d7f289ffc4449e5aba8208cf requestId=0
Mar 29 11:02:23 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:23.069] [downloader] Set posAnchor               blockHeight=1
Mar 29 11:02:23 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:23.070] [downloader] Request header              numer=0 length=192
Mar 29 11:02:23 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:23.070] [downloader] Sent request                height=0
Mar 29 11:02:23 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:23.558] [downloader] Collecting...               from=19541268 to=19541077 len=192
Mar 29 11:02:24 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:24.317] [downloader] Synced                      requestId=0
Mar 29 11:02:24 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:24.318] [downloader] Collecting...               from=19541268 to=19541077 len=192
Mar 29 11:02:24 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:24.318] [downloader] posAnchor is nil
Mar 29 11:02:24 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:24.318] [downloader] Collecting...               from=19541268 to=19541077 len=192
Mar 29 11:02:24 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:24.318] [downloader] posAnchor is nil
Mar 29 11:02:24 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:24.320] [db] open                                lable=inMem sizeLimit=2TB pageSize=4096
Mar 29 11:02:24 ckubuntu erigon[71884]: [INFO] [03-29|11:02:24.331] [EngineBlockDownloader] Processing bodies... from=19541233 to=19541268
Mar 29 11:02:27 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:27.562] [txpool] Commit                          written_kb=1480 in=34.206608ms
Mar 29 11:02:31 ckubuntu erigon[71884]: [INFO] [03-29|11:02:31.748] [EngineBlockDownloader] Processed        highest=19541268
Mar 29 11:02:31 ckubuntu erigon[71884]: [INFO] [03-29|11:02:31.748] Beginning downloaded blocks insertion
Mar 29 11:02:32 ckubuntu erigon[71884]: [INFO] [03-29|11:02:32.527] [EngineBlockDownloader] Finished downloading blocks from=19541233 to=19541268
Mar 29 11:02:42 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:42.674] [txpool] Commit                          written_kb=1272 in=23.19844ms
Mar 29 11:02:57 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:57.789] [txpool] Commit                          written_kb=1540 in=16.413199ms
Mar 29 11:02:59 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:59.097] [ForkChoiceUpdated] sending forkChoiceMessage head=0x1798b48a505d3513d3325c6e6328f3d59a4fec88d7f289ffc4449e5aba8208cf
Mar 29 11:02:59 ckubuntu erigon[71884]: [DBUG] [03-29|11:02:59.097] [ForkChoiceUpdated] Handling fork choice headerHash=0x1798b48a505d3513d3325c6e6328f3d59a4fec88d7f289ffc4449e5aba8208cf
Mar 29 11:03:00 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:00.098] treating forkChoiceUpdated as asynchronous as it is taking too long
Mar 29 11:03:00 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:00.099] [ForkChoiceUpdated] got reply            payloadStatus="&{Status:SYNCING ValidationError:<nil> LatestValidHash:<nil> CriticalError:<nil>}"
Mar 29 11:03:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:01.829] UnwindTo                                 block=19259999
Mar 29 11:03:06 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:06.642] [1/12 Snapshots] DONE                    in=10.891µs
Mar 29 11:03:06 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:06.642] [2/12 BlockHashes] DONE                  in=1.493µs
Mar 29 11:03:06 ckubuntu erigon[71884]: [INFO] [03-29|11:03:06.642] [3/12 Senders] Started                   from=19249999 to=19541268
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250000 hash=0xfacc7efd9f9c59c426a0545f59520fcc7234b6fe677882ff9144d3260c13d956
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250001 hash=0x5a2a03ae70d7e797fcea1551f15d4ad30d000c693c60bf2b4b711186f832ece3
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250002 hash=0x5391067b1240c8aab63a3c86f2564ae4d75442a655f4ae97d468d3f4fdbfa7de
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250003 hash=0x6650e0d20af229902be54991e6ec3f327aec8d65926466db8e159136465f00d4
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250004 hash=0xba007218b1627c6b007ffd5ae9db5f1129d71cf8b8b902b6f4e03d9086b719e1
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250005 hash=0x82a1344ffbd7dd23fa5f507dce319876540f07249032c022c4ec5e15f8b7afe3
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250006 hash=0x54dd9715cea38b95b64c0dd5359e96cd9e184e9549e07f0fc94ceef8f7becfe8
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250007 hash=0x57f556524beb25c5310b4bb7fb74afdb276383f3bc4de8c5e8478e89ed188fd3
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250008 hash=0x7ae07ab2fa6dc88510fa2e5bb5d32c6527434e212dce2a8e5547f29d9a806608
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.643] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250009 hash=0x08044a160ff9f5588ca0ee732c9c9a9ae56c0e9db4fc551511ff024d0989e949
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.644] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250010 hash=0x6b5b1a49a1e3d2eb46c412b6b5c46507f0e31924d83e987b635a3ca72ea5fcba
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.644] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250011 hash=0x65ae02521d6d6b18ad21f5614c0aa0a1da572d81bc68e734071440ebbba514e0
Mar 29 11:03:06 ckubuntu erigon[71884]: [WARN] [03-29|11:03:06.644] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250012 hash=0x93245d1e6bb8f9dea5d5d150cf95850bab46f7065a92fc71cb057769c308274a
.
.
.
Mar 29 11:03:07 ckubuntu erigon[71884]: [WARN] [03-29|11:03:07.146] [3/12 Senders] ReadBodyWithTransactions can't find block num=19259993 hash=0xc09a11e812d143919f457fe88b030d807c7a30a3bf46157c3203ec4b3b2a12e1
Mar 29 11:03:07 ckubuntu erigon[71884]: [WARN] [03-29|11:03:07.146] [3/12 Senders] ReadBodyWithTransactions can't find block num=19259994 hash=0xa19e421ce30f4d8a0aad25f98b5fb61a450587a139c351ccc45c33abe1676584
Mar 29 11:03:07 ckubuntu erigon[71884]: [WARN] [03-29|11:03:07.146] [3/12 Senders] ReadBodyWithTransactions can't find block num=19259995 hash=0xe719bc44d439ba4e7d13fd7df66113e484df465a24d6ed9a7dc9c7a9ae20348f
Mar 29 11:03:07 ckubuntu erigon[71884]: [WARN] [03-29|11:03:07.146] [3/12 Senders] ReadBodyWithTransactions can't find block num=19259996 hash=0xa1ec350692e4879d7c4e296ac8aa9f0b80be8afe3f0cde5743269df27bdd2227
Mar 29 11:03:07 ckubuntu erigon[71884]: [WARN] [03-29|11:03:07.146] [3/12 Senders] ReadBodyWithTransactions can't find block num=19259997 hash=0xa6dbdad24e60b5e03c7482f1c80d4c465012f0a50db39a8320a5d515f9146ae3
Mar 29 11:03:07 ckubuntu erigon[71884]: [WARN] [03-29|11:03:07.146] [3/12 Senders] ReadBodyWithTransactions can't find block num=19259998 hash=0x67a69e16e7bda2c3446ad2ce030d56dcf4f730a444238b0f028f587dce1ceafc
Mar 29 11:03:07 ckubuntu erigon[71884]: [WARN] [03-29|11:03:07.146] [3/12 Senders] ReadBodyWithTransactions can't find block num=19259999 hash=0xf046122a81d538a6c7a7d940f507d684491f7cf63a0afec94b32a6e4660bf4e4
Mar 29 11:03:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=1
Mar 29 11:03:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2439 static=0 connect: no route to host=2 connect: connection refused=14 i/o timeout=39
Mar 29 11:03:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unknown node=99 unsolicited reply=1037
Mar 29 11:03:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 too many peers=1
Mar 29 11:03:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2460 static=0 i/o timeout=33 connect: connection refused=15 connect: no route to host=2
Mar 29 11:03:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:03:12 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:12.924] [txpool] Commit                          written_kb=1372 in=29.089154ms
Mar 29 11:03:28 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:28.036] [txpool] Commit                          written_kb=1372 in=17.524166ms
Mar 29 11:03:36 ckubuntu erigon[71884]: [INFO] [03-29|11:03:36.644] [3/12 Senders] Recovery                  block_number=19270870 ch=1/10000
Mar 29 11:03:43 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:43.162] [txpool] Commit                          written_kb=1736 in=22.331717ms
Mar 29 11:03:58 ckubuntu erigon[71884]: [DBUG] [03-29|11:03:58.279] [txpool] Commit                          written_kb=1272 in=17.051485ms
Mar 29 11:04:06 ckubuntu erigon[71884]: [INFO] [03-29|11:04:06.643] [3/12 Senders] Recovery                  block_number=19283257 ch=0/10000
Mar 29 11:04:06 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:06.833] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:04:08 ckubuntu erigon[71884]: [INFO] [03-29|11:04:08.161] [p2p] GoodPeers                          eth68=3
Mar 29 11:04:08 ckubuntu erigon[71884]: [INFO] [03-29|11:04:08.653] [mem] memory stats                       Rss=19.5MB Size=0B Pss=19.5MB SharedClean=2.4KB SharedDirty=0B PrivateClean=12.9MB PrivateDirty=6.6MB Referenced=17.8MB Anonymous=6.6MB Swap=402.3KB alloc=3.6GB sys=9.7GB
Mar 29 11:04:08 ckubuntu erigon[71884]: [INFO] [03-29|11:04:08.871] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=3.7GB sys=9.7GB
Mar 29 11:04:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=2
Mar 29 11:04:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2463 static=0 connect: no route to host=2 connect: connection refused=14 i/o timeout=39
Mar 29 11:04:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:04:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 ecies: invalid message=1
Mar 29 11:04:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2484 static=0 i/o timeout=33 connect: connection refused=15 connect: no route to host=2
Mar 29 11:04:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:04:13 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:13.406] [txpool] Commit                          written_kb=1336 in=22.244271ms
Mar 29 11:04:28 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:28.520] [txpool] Commit                          written_kb=1156 in=13.121005ms
Mar 29 11:04:32 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:32.072] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:04:36 ckubuntu erigon[71884]: [INFO] [03-29|11:04:36.643] [3/12 Senders] Recovery                  block_number=19299198 ch=0/10000
Mar 29 11:04:43 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:43.643] [txpool] Commit                          written_kb=1072 in=14.378372ms
Mar 29 11:04:58 ckubuntu erigon[71884]: [DBUG] [03-29|11:04:58.766] [txpool] Commit                          written_kb=1156 in=15.299686ms
Mar 29 11:05:06 ckubuntu erigon[71884]: [INFO] [03-29|11:05:06.647] [3/12 Senders] Recovery                  block_number=19313831 ch=0/10000
Mar 29 11:05:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:08.535] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:05:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:08.908] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:05:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:08.908] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:05:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:08.914] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 too many peers=2
Mar 29 11:05:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:08.914] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=1
Mar 29 11:05:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:08.914] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2508 static=0 i/o timeout=33 connect: connection refused=15 connect: no route to host=2
Mar 29 11:05:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:08.914] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2487 static=0 connect: no route to host=2 connect: connection refused=14 i/o timeout=40
Mar 29 11:05:13 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:13.919] [txpool] Commit                          written_kb=1320 in=46.445592ms
Mar 29 11:05:29 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:29.012] [txpool] Commit                          written_kb=1304 in=16.11028ms
Mar 29 11:05:36 ckubuntu erigon[71884]: [INFO] [03-29|11:05:36.643] [3/12 Senders] Recovery                  block_number=19327975 ch=3/10000
Mar 29 11:05:37 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:37.837] Error in DNS random node sync            tree=all.mainnet.ethdisco.net err="lookup 4BJVXVSLCRMSYINZ6KIKF7JCQY.all.mainnet.ethdisco.net on 127.0.0.53:53: no such host"
Mar 29 11:05:44 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:44.133] [txpool] Commit                          written_kb=1060 in=14.684134ms
Mar 29 11:05:55 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:55.503] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:05:55 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:55.915] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:05:59 ckubuntu erigon[71884]: [DBUG] [03-29|11:05:59.274] [txpool] Commit                          written_kb=1344 in=33.607146ms
Mar 29 11:06:02 ckubuntu erigon[71884]: [INFO] [03-29|11:06:02.904] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-2964340468
Mar 29 11:06:06 ckubuntu erigon[71884]: [INFO] [03-29|11:06:06.643] [3/12 Senders] Recovery                  block_number=19343018 ch=0/10000
Mar 29 11:06:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:08.903] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=2
Mar 29 11:06:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:08.903] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:06:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:08.903] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2511 static=0 connect: no route to host=2 connect: connection refused=14 i/o timeout=40
Mar 29 11:06:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 EOF=1 too many peers=1
Mar 29 11:06:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2532 static=0 i/o timeout=33 connect: connection refused=15 connect: no route to host=2
Mar 29 11:06:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:06:14 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:14.406] [txpool] Commit                          written_kb=1348 in=40.937244ms
Mar 29 11:06:29 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:29.501] [txpool] Commit                          written_kb=1144 in=16.481729ms
Mar 29 11:06:36 ckubuntu erigon[71884]: [INFO] [03-29|11:06:36.649] [3/12 Senders] Recovery                  block_number=19358765 ch=5/10000
Mar 29 11:06:44 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:44.630] [txpool] Commit                          written_kb=1124 in=22.811853ms
Mar 29 11:06:59 ckubuntu erigon[71884]: [DBUG] [03-29|11:06:59.745] [txpool] Commit                          written_kb=1000 in=15.796083ms
Mar 29 11:07:06 ckubuntu erigon[71884]: [INFO] [03-29|11:07:06.643] [3/12 Senders] Recovery                  block_number=19374209 ch=0/10000
Mar 29 11:07:08 ckubuntu erigon[71884]: [INFO] [03-29|11:07:08.161] [p2p] GoodPeers                          eth68=3
Mar 29 11:07:08 ckubuntu erigon[71884]: [INFO] [03-29|11:07:08.678] [mem] memory stats                       Rss=19.7MB Size=0B Pss=19.7MB SharedClean=2.4KB SharedDirty=0B PrivateClean=12.7MB PrivateDirty=7.0MB Referenced=19.6MB Anonymous=7.0MB Swap=419.5KB alloc=4.9GB sys=9.7GB
Mar 29 11:07:08 ckubuntu erigon[71884]: [INFO] [03-29|11:07:08.871] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=4.9GB sys=9.7GB
Mar 29 11:07:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=1
Mar 29 11:07:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:07:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2535 static=0 connect: connection refused=14 i/o timeout=40 connect: no route to host=3
Mar 29 11:07:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unknown node=79 unsolicited reply=1025
Mar 29 11:07:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 too many peers=2
Mar 29 11:07:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2556 static=0 i/o timeout=33 connect: connection refused=15 connect: no route to host=2
Mar 29 11:07:14 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:14.875] [txpool] Commit                          written_kb=1164 in=23.620953ms
Mar 29 11:07:29 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:29.985] [txpool] Commit                          written_kb=960 in=12.084761ms
Mar 29 11:07:36 ckubuntu erigon[71884]: [INFO] [03-29|11:07:36.643] [3/12 Senders] Recovery                  block_number=19389711 ch=0/10000
Mar 29 11:07:45 ckubuntu erigon[71884]: [DBUG] [03-29|11:07:45.111] [txpool] Commit                          written_kb=1016 in=14.871821ms
Mar 29 11:08:00 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:00.241] [txpool] Commit                          written_kb=812 in=22.948423ms
Mar 29 11:08:06 ckubuntu erigon[71884]: [INFO] [03-29|11:08:06.644] [3/12 Senders] Recovery                  block_number=19405624 ch=0/10000
Mar 29 11:08:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:08:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2557 static=0 connect: connection refused=14 i/o timeout=40 connect: no route to host=4
Mar 29 11:08:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0
Mar 29 11:08:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 too many peers=1
Mar 29 11:08:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2579 static=0 connect: no route to host=2 i/o timeout=33 connect: connection refused=15
Mar 29 11:08:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:08:15 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:15.366] [txpool] Commit                          written_kb=1492 in=21.611846ms
Mar 29 11:08:18 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:18.665] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:08:20 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:20.527] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:08:30 ckubuntu erigon[71884]: [INFO] [03-29|11:08:30.492] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-2036869672
Mar 29 11:08:30 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:30.500] [txpool] Commit                          written_kb=1300 in=32.019831ms
Mar 29 11:08:36 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:36.586] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:08:36 ckubuntu erigon[71884]: [INFO] [03-29|11:08:36.644] [3/12 Senders] Recovery                  block_number=19421240 ch=0/10000
Mar 29 11:08:43 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:43.908] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:08:45 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:45.607] [txpool] Commit                          written_kb=1108 in=22.489148ms
Mar 29 11:08:53 ckubuntu erigon[71884]: [DBUG] [03-29|11:08:53.433] Error in DNS random node sync            tree=all.mainnet.ethdisco.net err="lookup CZSHNLBPFHLZPS7JE52HJBD2RM.all.mainnet.ethdisco.net on 127.0.0.53:53: no such host"
Mar 29 11:09:00 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:00.726] [txpool] Commit                          written_kb=1272 in=18.335863ms
Mar 29 11:09:06 ckubuntu erigon[71884]: [INFO] [03-29|11:09:06.650] [3/12 Senders] Recovery                  block_number=19436803 ch=11/10000
Mar 29 11:09:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2581 static=0 connect: no route to host=4 connect: connection refused=14 i/o timeout=40
Mar 29 11:09:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=2
Mar 29 11:09:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:09:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 too many peers=1
Mar 29 11:09:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2603 static=0 i/o timeout=34 connect: connection refused=15 connect: no route to host=2
Mar 29 11:09:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:09:15 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:15.864] [txpool] Commit                          written_kb=1320 in=29.484459ms
Mar 29 11:09:30 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:30.981] [txpool] Commit                          written_kb=1100 in=29.576753ms
Mar 29 11:09:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:34.140] Error in DNS random node sync            tree=all.mainnet.ethdisco.net err="lookup HZCN74AMQSKCX6GPGVDLROEV3I.all.mainnet.ethdisco.net on 127.0.0.53:53: no such host"
Mar 29 11:09:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:34.147] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:09:36 ckubuntu erigon[71884]: [INFO] [03-29|11:09:36.647] [3/12 Senders] Recovery                  block_number=19452840 ch=0/10000
Mar 29 11:09:38 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:38.738] Error in DNS random node sync            tree=all.mainnet.ethdisco.net err="lookup CZSHNLBPFHLZPS7JE52HJBD2RM.all.mainnet.ethdisco.net on 127.0.0.53:53: no such host"
Mar 29 11:09:46 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:46.099] [txpool] Commit                          written_kb=1216 in=23.963227ms
Mar 29 11:09:46 ckubuntu erigon[71884]: [DBUG] [03-29|11:09:46.789] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:10:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:01.225] [txpool] Commit                          written_kb=1128 in=27.43847ms
Mar 29 11:10:06 ckubuntu erigon[71884]: [INFO] [03-29|11:10:06.642] [3/12 Senders] Recovery                  block_number=19468554 ch=0/10000
Mar 29 11:10:08 ckubuntu erigon[71884]: [INFO] [03-29|11:10:08.161] [p2p] GoodPeers                          eth68=3
Mar 29 11:10:08 ckubuntu erigon[71884]: [INFO] [03-29|11:10:08.711] [mem] memory stats                       Rss=20.0MB Size=0B Pss=20.0MB SharedClean=2.4KB SharedDirty=0B PrivateClean=13.0MB PrivateDirty=7.1MB Referenced=19.9MB Anonymous=7.1MB Swap=429.7KB alloc=3.8GB sys=9.7GB
Mar 29 11:10:08 ckubuntu erigon[71884]: [INFO] [03-29|11:10:08.871] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=3.8GB sys=9.7GB
Mar 29 11:10:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2605 static=0 connect: no route to host=4 connect: connection refused=14 i/o timeout=40
Mar 29 11:10:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unknown node=99 unsolicited reply=1037
Mar 29 11:10:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:08.902] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=2
Mar 29 11:10:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 ecies: invalid message=1
Mar 29 11:10:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:10:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2624 static=0 i/o timeout=34 connect: connection refused=15 connect: no route to host=2
Mar 29 11:10:16 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:16.335] [txpool] Commit                          written_kb=1136 in=16.286547ms
Mar 29 11:10:31 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:31.049] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:10:31 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:31.500] [txpool] Commit                          written_kb=1056 in=27.000486ms
Mar 29 11:10:36 ckubuntu erigon[71884]: [INFO] [03-29|11:10:36.652] [3/12 Senders] Recovery                  block_number=19483593 ch=1/10000
Mar 29 11:10:46 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:46.343] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:10:46 ckubuntu erigon[71884]: [DBUG] [03-29|11:10:46.583] [txpool] Commit                          written_kb=1268 in=20.426757ms
Mar 29 11:10:50 ckubuntu erigon[71884]: [INFO] [03-29|11:10:50.654] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-2490153702
Mar 29 11:11:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:01.706] [txpool] Commit                          written_kb=1104 in=18.416918ms
Mar 29 11:11:06 ckubuntu erigon[71884]: [INFO] [03-29|11:11:06.643] [3/12 Senders] Recovery                  block_number=19497724 ch=0/10000
Mar 29 11:11:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=2
Mar 29 11:11:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2629 static=0 connect: no route to host=4 connect: connection refused=14 i/o timeout=40
Mar 29 11:11:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:11:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0
Mar 29 11:11:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2648 static=0 i/o timeout=34 connect: connection refused=16 connect: no route to host=2
Mar 29 11:11:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:11:16 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:16.823] [txpool] Commit                          written_kb=1040 in=15.581584ms
Mar 29 11:11:27 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:27.630] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:11:31 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:31.956] [txpool] Commit                          written_kb=1068 in=26.031288ms
Mar 29 11:11:33 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:33.543] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:11:36 ckubuntu erigon[71884]: [INFO] [03-29|11:11:36.642] [3/12 Senders] Recovery                  block_number=19512953 ch=0/10000
Mar 29 11:11:47 ckubuntu erigon[71884]: [DBUG] [03-29|11:11:47.070] [txpool] Commit                          written_kb=984 in=18.330411ms
Mar 29 11:12:02 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:02.235] [txpool] Commit                          written_kb=1172 in=60.521256ms
Mar 29 11:12:06 ckubuntu erigon[71884]: [INFO] [03-29|11:12:06.643] [3/12 Senders] Recovery                  block_number=19527498 ch=0/10000
Mar 29 11:12:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2653 static=0 i/o timeout=40 connect: no route to host=4 connect: connection refused=14
Mar 29 11:12:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:12:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=1
Mar 29 11:12:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:08.905] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:12:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:08.905] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 too many peers=1
Mar 29 11:12:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:08.905] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2672 static=0 i/o timeout=35 connect: connection refused=16 connect: no route to host=2
Mar 29 11:12:17 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:17.321] [txpool] Commit                          written_kb=1280 in=25.118609ms
Mar 29 11:12:32 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:32.433] [txpool] Commit                          written_kb=756 in=13.651305ms
Mar 29 11:12:32 ckubuntu erigon[71884]: [INFO] [03-29|11:12:32.959] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-1923453730
Mar 29 11:12:33 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:33.731] [txpool.fetch] Handling incoming message msg=POOLED_TRANSACTIONS_66 err="txn rlp too big"
Mar 29 11:12:34 ckubuntu erigon[71884]: [INFO] [03-29|11:12:34.537] [3/12 Senders] DONE                      in=9m27.894610333s
Mar 29 11:12:34 ckubuntu erigon[71884]: [INFO] [03-29|11:12:34.537] [4/12 Execution] Blocks execution        from=19249999 to=19541268
Mar 29 11:12:34 ckubuntu erigon[71884]: [EROR] [03-29|11:12:34.538] [4/12 Execution] Empty block             blocknum=19250000
Mar 29 11:12:34 ckubuntu erigon[71884]: [INFO] [03-29|11:12:34.538] [4/12 Execution] Completed on            block=19249999
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.538] [4/12 Execution] DONE                    in=1.275641ms
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.538] [5/12 HashState] DONE                    in=4.639µs
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.539] [6/12 IntermediateHashes] DONE           in=196.725µs
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.539] [7/12 CallTraces] DONE                   in=1.473µs
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.539] [8/12 AccountHistoryIndex] DONE          in=1.674µs
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.539] [9/12 StorageHistoryIndex] DONE          in=1.283µs
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.539] [10/12 LogIndex] DONE                    in=2.615µs
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.539] [11/12 TxLookup] DONE                    in=547.744µs
Mar 29 11:12:34 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:34.539] [12/12 Finish] DONE                      in=21.42µs
Mar 29 11:12:34 ckubuntu erigon[71884]: [WARN] [03-29|11:12:34.540] bad forkchoice                           head=0xaa69de29f2b185d99fec391a53282fe99bc4d5698610b93dbf345daf9aa43b60 hash=0x1798b48a505d3513d3325c6e6328f3d59a4fec88d7f289ffc4449e5aba8208cf
Mar 29 11:12:37 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:36.999] [NewPayload] sending block               height=19541358 hash=0xe9fe163796f5bd2fa6a589c43e1d5b3c26953ded3022443ba7fdc16b61081d32
Mar 29 11:12:37 ckubuntu erigon[71884]: [INFO] [03-29|11:12:37.000] [NewPayload] Handling new payload        height=19541358 hash=0xe9fe163796f5bd2fa6a589c43e1d5b3c26953ded3022443ba7fdc16b61081d32
Mar 29 11:12:37 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:37.000] [NewPayload] New payload: need to download parent height=19541358 hash=0xe9fe163796f5bd2fa6a589c43e1d5b3c26953ded3022443ba7fdc16b61081d32 parentHash=0x7ab22ad0693688962746df8df5515dcea2ec2741f8f280844fe157d199b21a24
Mar 29 11:12:37 ckubuntu erigon[71884]: [INFO] [03-29|11:12:37.000] [EngineBlockDownloader] Downloading PoS headers... hash=0x7ab22ad0693688962746df8df5515dcea2ec2741f8f280844fe157d199b21a24 requestId=0
Mar 29 11:12:37 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:37.001] [downloader] Set posAnchor               blockHeight=1
Mar 29 11:12:37 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:37.003] [downloader] Request header              numer=0 length=192
Mar 29 11:12:37 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:37.004] [downloader] Sent request                height=0
Mar 29 11:12:37 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:37.809] [downloader] Collecting...               from=19541357 to=19541166 len=192
Mar 29 11:12:38 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:38.052] [NewPayload] got reply                   payloadStatus="&{Status:SYNCING ValidationError:<nil> LatestValidHash:<nil> CriticalError:<nil>}"
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.268] [downloader] Synced                      requestId=0
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.268] [ForkChoiceUpdated] sending forkChoiceMessage head=0xe9fe163796f5bd2fa6a589c43e1d5b3c26953ded3022443ba7fdc16b61081d32
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.268] [ForkChoiceUpdated] Handling fork choice headerHash=0xe9fe163796f5bd2fa6a589c43e1d5b3c26953ded3022443ba7fdc16b61081d32
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.268] [ForkChoiceUpdated] Fork choice: need to download header with hash e9fe163796f5bd2fa6a589c43e1d5b3c26953ded3022443ba7fdc16b61081d32
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.268] [ForkChoiceUpdated] got reply            payloadStatus="&{Status:SYNCING ValidationError:<nil> LatestValidHash:<nil> CriticalError:<nil>}"
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.269] [downloader] Collecting...               from=19541357 to=19541166 len=192
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.269] [downloader] posAnchor is nil
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.270] [downloader] Collecting...               from=19541357 to=19541166 len=192
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.270] [downloader] posAnchor is nil
Mar 29 11:12:39 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:39.272] [db] open                                lable=inMem sizeLimit=2TB pageSize=4096
Mar 29 11:12:39 ckubuntu erigon[71884]: [INFO] [03-29|11:12:39.285] [EngineBlockDownloader] Processing bodies... from=19541268 to=19541357
Mar 29 11:12:47 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:47.561] [txpool] Commit                          written_kb=1504 in=19.348973ms
Mar 29 11:12:48 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:48.983] [NewPayload] sending block               height=19541359 hash=0x26528923fc8c37a33144daa811c9afb5a3df64b208c42ea366bf8cc796792422
Mar 29 11:12:48 ckubuntu erigon[71884]: [INFO] [03-29|11:12:48.983] [NewPayload] Handling new payload        height=19541359 hash=0x26528923fc8c37a33144daa811c9afb5a3df64b208c42ea366bf8cc796792422
Mar 29 11:12:48 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:48.984] [NewPayload] New payload: need to download parent height=19541359 hash=0x26528923fc8c37a33144daa811c9afb5a3df64b208c42ea366bf8cc796792422 parentHash=0xe9fe163796f5bd2fa6a589c43e1d5b3c26953ded3022443ba7fdc16b61081d32
Mar 29 11:12:48 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:48.984] [NewPayload] got reply                   payloadStatus="&{Status:SYNCING ValidationError:<nil> LatestValidHash:<nil> CriticalError:<nil>}"
Mar 29 11:12:49 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:49.144] [ForkChoiceUpdated] sending forkChoiceMessage head=0x26528923fc8c37a33144daa811c9afb5a3df64b208c42ea366bf8cc796792422
Mar 29 11:12:49 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:49.144] [ForkChoiceUpdated] Handling fork choice headerHash=0x26528923fc8c37a33144daa811c9afb5a3df64b208c42ea366bf8cc796792422
Mar 29 11:12:49 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:49.144] [ForkChoiceUpdated] Fork choice: need to download header with hash 26528923fc8c37a33144daa811c9afb5a3df64b208c42ea366bf8cc796792422
Mar 29 11:12:49 ckubuntu erigon[71884]: [DBUG] [03-29|11:12:49.144] [ForkChoiceUpdated] got reply            payloadStatus="&{Status:SYNCING ValidationError:<nil> LatestValidHash:<nil> CriticalError:<nil>}"
Mar 29 11:12:53 ckubuntu erigon[71884]: [INFO] [03-29|11:12:53.463] [EngineBlockDownloader] Processed        highest=19541357
Mar 29 11:12:53 ckubuntu erigon[71884]: [INFO] [03-29|11:12:53.463] Beginning downloaded blocks insertion
Mar 29 11:12:53 ckubuntu erigon[71884]: [INFO] [03-29|11:12:53.707] [EngineBlockDownloader] Finished downloading blocks from=19541268 to=19541357
Mar 29 11:12:53 ckubuntu erigon[71884]: [INFO] [03-29|11:12:53.713] [EngineBlockDownloader] block verification skipped
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.183] [NewPayload] sending block               height=19541360 hash=0x4cea1331177f8165a8766570c7f49bd3186f4ae145a95e610ce5221ea19d4cd7
Mar 29 11:13:01 ckubuntu erigon[71884]: [INFO] [03-29|11:13:01.183] [NewPayload] Handling new payload        height=19541360 hash=0x4cea1331177f8165a8766570c7f49bd3186f4ae145a95e610ce5221ea19d4cd7
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.184] [NewPayload] New payload: need to download parent height=19541360 hash=0x4cea1331177f8165a8766570c7f49bd3186f4ae145a95e610ce5221ea19d4cd7 parentHash=0x26528923fc8c37a33144daa811c9afb5a3df64b208c42ea366bf8cc796792422
Mar 29 11:13:01 ckubuntu erigon[71884]: [INFO] [03-29|11:13:01.184] [EngineBlockDownloader] Downloading PoS headers... hash=0x26528923fc8c37a33144daa811c9afb5a3df64b208c42ea366bf8cc796792422 requestId=0
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.184] [downloader] Set posAnchor               blockHeight=1
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.185] [downloader] Request header              numer=0 length=192
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.186] [downloader] Sent request                height=0
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.652] [downloader] Collecting...               from=19541359 to=19541168 len=192
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.652] [downloader] Synced                      requestId=0
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.669] [db] open                                lable=inMem sizeLimit=2TB pageSize=4096
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.752] [downloader] Collecting...               from=19541359 to=19541168 len=192
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.752] [downloader] posAnchor is nil
Mar 29 11:13:01 ckubuntu erigon[71884]: [INFO] [03-29|11:13:01.880] [EngineBlockDownloader] Processed        highest=19541359
Mar 29 11:13:01 ckubuntu erigon[71884]: [INFO] [03-29|11:13:01.880] Beginning downloaded blocks insertion
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.886] [downloader] Collecting...               from=19541359 to=19541168 len=192
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.886] [downloader] posAnchor is nil
Mar 29 11:13:01 ckubuntu erigon[71884]: [INFO] [03-29|11:13:01.894] [EngineBlockDownloader] Finished downloading blocks from=19541358 to=19541359
Mar 29 11:13:01 ckubuntu erigon[71884]: [INFO] [03-29|11:13:01.899] [EngineBlockDownloader] block verification skipped
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.909] [NewPayload] got reply                   payloadStatus="&{Status:VALID ValidationError:<nil> LatestValidHash:0x4cea1331177f8165a8766570c7f49bd3186f4ae145a95e610ce5221ea19d4cd7 CriticalError:<nil>}"
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.929] [ForkChoiceUpdated] sending forkChoiceMessage head=0x4cea1331177f8165a8766570c7f49bd3186f4ae145a95e610ce5221ea19d4cd7
Mar 29 11:13:01 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:01.929] [ForkChoiceUpdated] Handling fork choice headerHash=0x4cea1331177f8165a8766570c7f49bd3186f4ae145a95e610ce5221ea19d4cd7
Mar 29 11:13:02 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:02.675] [txpool] Commit                          written_kb=908 in=12.129555ms
Mar 29 11:13:02 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:02.930] treating forkChoiceUpdated as asynchronous as it is taking too long
Mar 29 11:13:02 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:02.930] [ForkChoiceUpdated] got reply            payloadStatus="&{Status:SYNCING ValidationError:<nil> LatestValidHash:<nil> CriticalError:<nil>}"
Mar 29 11:13:04 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:04.577] UnwindTo                                 block=19259999
Mar 29 11:13:04 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:04.689] Error in DNS random node sync            tree=all.mainnet.ethdisco.net err="lookup OB7YUZHRET7FHUT6T42IO63KLY.all.mainnet.ethdisco.net on 127.0.0.53:53: no such host"
Mar 29 11:13:04 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:04.774] Error in DNS random node sync            tree=all.mainnet.ethdisco.net err="lookup W5MGLPXSXEWYO62K6BHU2YRDZ4.all.mainnet.ethdisco.net on 127.0.0.53:53: no such host"
Mar 29 11:13:04 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:04.832] Error in DNS random node sync            tree=all.mainnet.ethdisco.net err="lookup OB7YUZHRET7FHUT6T42IO63KLY.all.mainnet.ethdisco.net on 127.0.0.53:53: no such host"
Mar 29 11:13:08 ckubuntu erigon[71884]: [INFO] [03-29|11:13:08.161] [p2p] GoodPeers                          eth68=3
Mar 29 11:13:08 ckubuntu erigon[71884]: [INFO] [03-29|11:13:08.519] [mem] memory stats                       Rss=17.2MB Size=0B Pss=17.2MB SharedClean=2.4KB SharedDirty=0B PrivateClean=10.7MB PrivateDirty=6.4MB Referenced=16.8MB Anonymous=6.4MB Swap=416.4KB alloc=3.0GB sys=9.7GB
Mar 29 11:13:08 ckubuntu erigon[71884]: [INFO] [03-29|11:13:08.871] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=3.1GB sys=9.7GB
Mar 29 11:13:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:08.901] [p2p] Server                             protocol=67 peers=0 trusted=0 inbound=0 too many peers=2
Mar 29 11:13:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:08.901] [p2p] Dial scheduler                     protocol=67 peers=0/13 tried=2677 static=0 i/o timeout=40 connect: no route to host=4 connect: connection refused=14
Mar 29 11:13:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:08.901] [p2p] Discovery table                    protocol=67 version=v4 len=159 live=0 unsol=500 ips=192 db=0 reval=11 RPC timeout=10 unsolicited reply=1037 unknown node=99
Mar 29 11:13:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:08.906] [p2p] Server                             protocol=68 peers=3 trusted=0 inbound=0 too many peers=2
Mar 29 11:13:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:08.906] [p2p] Discovery table                    protocol=68 version=v4 len=161 live=1 unsol=500 ips=174 db=0 reval=12 RPC timeout=10 unsolicited reply=1025 unknown node=79
Mar 29 11:13:08 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:08.906] [p2p] Dial scheduler                     protocol=68 peers=3/13 tried=2696 static=0 i/o timeout=35 connect: connection refused=16 connect: no route to host=2
Mar 29 11:13:10 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:10.278] [1/12 Snapshots] DONE                    in=11.862µs
Mar 29 11:13:10 ckubuntu erigon[71884]: [DBUG] [03-29|11:13:10.278] [2/12 BlockHashes] DONE                  in=1.533µs
Mar 29 11:13:10 ckubuntu erigon[71884]: [INFO] [03-29|11:13:10.278] [3/12 Senders] Started                   from=19249999 to=19541360
Mar 29 11:13:10 ckubuntu erigon[71884]: [WARN] [03-29|11:13:10.279] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250000 hash=0xfacc7efd9f9c59c426a0545f59520fcc7234b6fe677882ff9144d3260c13d956
Mar 29 11:13:10 ckubuntu erigon[71884]: [WARN] [03-29|11:13:10.279] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250001 hash=0x5a2a03ae70d7e797fcea1551f15d4ad30d000c693c60bf2b4b711186f832ece3
Mar 29 11:13:10 ckubuntu erigon[71884]: [WARN] [03-29|11:13:10.279] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250002 hash=0x5391067b1240c8aab63a3c86f2564ae4d75442a655f4ae97d468d3f4fdbfa7de
Mar 29 11:13:10 ckubuntu erigon[71884]: [WARN] [03-29|11:13:10.279] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250003 hash=0x6650e0d20af229902be54991e6ec3f327aec8d65926466db8e159136465f00d4
Mar 29 11:13:10 ckubuntu erigon[71884]: [WARN] [03-29|11:13:10.279] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250004 hash=0xba007218b1627c6b007ffd5ae9db5f1129d71cf8b8b902b6f4e03d9086b719e1
Mar 29 11:13:10 ckubuntu erigon[71884]: [WARN] [03-29|11:13:10.279] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250005 hash=0x82a1344ffbd7dd23fa5f507dce319876540f07249032c022c4ec5e15f8b7afe3
Mar 29 11:13:10 ckubuntu erigon[71884]: [WARN] [03-29|11:13:10.279] [3/12 Senders] ReadBodyWithTransactions can't find block num=19250006 hash=0x54dd9715cea38b95b64c0dd5359e96cd9e184e9549e07f0fc94ceef8f7becfe8

odilatinam commented 7 months ago

I'm having the same issue - stuck in a loop. I upgraded from 2.58.1 to 2.59.0 and rebuilt the node (just deleted chaindata directory) in order to take advantage of the reduced database size specified in PR #9123. I'm on 2.59.3 now and still stuck in loop. Do I have to rebuild again? Any other data directories need to be removed besides chaindata before rebuild?

I just upgraded my backup pruned node from 2.58.1 to 2.59.3 and it didn't get stuck (to test because I saw your post). I had deleted the chaindata directory for 2.58.1 after not receiving any feedback for weeks.

Two things I had done, I moved the snapshots to a hard drive before rebuilding in 2.58.1 because I noticed my ssd was almost full (100G free out of 1.9T). That may have been what caused the system to crash back in December. Also when upgrading just now, I noticed it took a few minutes for the erigon process to stop and I waited patiently for it to happen.

In comparison, my full node that is witnessing survived 2 power outages where it shut down ungracefully. It is now connected to a new battery backup (old one had failed). Based on the error messages, it seems as though the pruned node can't find something when trying to recover from an ungraceful shut down.

erigontech / erigon