Stuck at block height 17999999

Hi all

I'm wondering if I'm stuck, given a reused Beacon chain (previously running with Geth), against a fresh Erigon full sync? Appreciate any help/suggestions!

Notes

Stuck at block height 17999999
Previously using Geth as my Execution Chain, switched to Erigon a few weeks ago
Beacon chain setup has not changed since switching from Geth. Could this be related, given being stuck at block height 17999999?
GraphQL endpoint, only services up to block head 17999999
I have noticed Erigon output, syncing past block 18m - GraphQL never serves beyond this

Versions

Execution chain: Erigon 2.53.4-c4843268 Beacon chain: Nimbus v23.10.0-8b07f4 Go: go1.20.10 linux/amd64 OS: Fedora 36, Intel XEON

CPU

$ lscpu 
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  24
  On-line CPU(s) list:   0-23
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
    CPU family:          6
    Model:               63
    Thread(s) per core:  2
    Core(s) per socket:  12
    Socket(s):           1
    Stepping:            2
    CPU(s) scaling MHz:  40%
    CPU max MHz:         3100.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            4597.36
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca 
                         sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flus                         h_l1d
Virtualization features: 
  Virtualization:        VT-x

Erigon configuration

erigon \
        --chain="mainnet" \
        --datadir="/storage/ethereum/erigon/data" \
        --snapshots="true" \
        --authrpc.jwtsecret="/var/ethereum/jwtsecret" \
        --torrent.download.rate="150mb" --torrent.upload.rate="15mb" \
        --http="true" --http.api="web3,net,eth" --http.addr="192.168.2.3" --http.vhosts="*" \
        --ws="true" \
        --ws.compression="false" \
#        --ws.api="web3,net,eth" \
#        --ws.addr="192.168.2.3" \
        --graphql="true" \
        --private.api.addr="192.168.2.3:9191" \
        --authrpc.addr="127.0.0.1" --authrpc.port="8551" --authrpc.vhosts="127.0.0.1"

Output

Nov 20 09:50:23 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Nov 20 09:50:23 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:23.837] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Nov 20 09:50:23 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:23.837] Build info                               git_branch=HEAD git_tag=v2.53.4 git_commit=c4843268b788c0899cdad12d220fea4396afbb56
Nov 20 09:50:23 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:23.838] Starting Erigon on Ethereum mainnet...
Nov 20 09:50:23 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:23.839] Maximum peer count                       ETH=100 total=100
Nov 20 09:50:23 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:23.839] starting HTTP APIs                       APIs=web3,net,eth
Nov 20 09:50:23 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:23.840] torrent verbosity                        level=WRN
Nov 20 09:50:25 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:25.944] Set global gas cap                       cap=50000000
Nov 20 09:50:26 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:26.042] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=150mb upload.rate=15mb
Nov 20 09:50:26 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:26.042] Opening Database                         label=chaindata path=/storage/ethereum/erigon/data/chaindata
Nov 20 09:50:26 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:26.046] [db] chaindata                           sizeLimit=3TB pageSize=8192
Nov 20 09:50:26 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:26.047] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: <nil>, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
Nov 20 09:50:51 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:51.800] Initialising Ethereum protocol           network=1
Nov 20 09:50:51 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:51.800] Disk storage enabled for ethash DAGs     dir=/storage/ethereum/erigon/data/ethash-dags count=2
Nov 20 09:50:51 anon-r2 erigon[2745314]: 17999999
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.795] Starting private RPC server              on=192.168.2.3:9191
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.795] new subscription to logs established
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.795] rpc filters: subscribing to Erigon events
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.795] New txs subscriber joined
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.796] new subscription to newHeaders established
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.796] HTTP endpoint opened                     url=192.168.2.3:8545 ws=true ws.compression=false grpc=false
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.796] Reading JWT secret                       path=/var/ethereum/jwtsecret
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.798] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=false
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.809] [1/12 Snapshots] Fetching torrent files metadata
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.813] [txpool] Started
Nov 20 09:50:53 anon-r2 erigon[2745314]: [WARN] [11-20|09:50:53.814] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Nov 20 09:50:53 anon-r2 erigon[2745314]: [WARN] [11-20|09:50:53.816] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.818] Started P2P networking                   version=67 self=enode://[REDACTED]@127.0.0.1:30304 name=erigon/v2.53.4-c4843268/linux-amd64/go1.20.10
Nov 20 09:50:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:53.820] Started P2P networking                   version=68 self=enode://[REDACTED]@127.0.0.1:30303 name=erigon/v2.53.4-c4843268/linux-amd64/go1.20.10
Nov 20 09:50:56 anon-r2 erigon[2745314]: [INFO] [11-20|09:50:56.981] [snapshots] Blocks Stat                  blocks=18000k indices=18000k alloc=3.4GB sys=4.4GB
Nov 20 09:53:51 anon-r2 erigon[2745314]: [INFO] [11-20|09:53:51.788] [p2p] GoodPeers
Nov 20 09:53:53 anon-r2 erigon[2745314]: [INFO] [11-20|09:53:53.796] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=2.1GB sys=4.4GB

Eventually building over time, to output the same following lines;

Nov 20 05:47:29 anon-r2 erigon[2375722]: [INFO] [11-20|05:47:29.039] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=2.2GB sys=4.5GB
Nov 20 05:50:09 anon-r2 erigon[2375722]: [INFO] [11-20|05:50:09.745] [p2p] GoodPeers
Nov 20 05:50:29 anon-r2 erigon[2375722]: [INFO] [11-20|05:50:29.038] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=2.2GB sys=4.5GB
Nov 20 05:53:09 anon-r2 erigon[2375722]: [INFO] [11-20|05:53:09.745] [p2p] GoodPeers

Thanks for your help!

@AskAlexSharov --internalcl flag is not working for me.

Erigon is stuck at 16999999 and not moving for 2 days.

Any other suggestion?

@sebicas no logs - no honey

Having similar issues with goerli sync (tried restarting multiple times) {"jsonrpc":"2.0","id":67,"result":"0x81b31f"}

PS: internalcl was added later on, after it got stuck at the block 0x81b31f

ExecStart=/nodes/volume1/eth/bin/erigon \
--chain=goerli \
--datadir=/nodes/volume1/eth/goerli \
--internalcl \
--ws \
--http.vhosts="*" \
--http.addr=0.0.0.0 \
--http.port=9547 \
--http.api="admin,debug,eth,net,txpool,web3,erigon" \
--torrent.port=42068 \
--maxpeers=50 \
--rpc.txfeecap=10 \
--state.cache=8192

Screenshot 2023-11-23 at 10 22 44 AM

Try some external CL for goerli

Thanks @AskAlexSharov

It looks like progression is being made, however I've noticed the service I've setup for Erigon (using the command above, now with --internalcl), has crashed a couple of times over the last few days. I'm just leaving it to do it's thing, as it tends to start syncing again.

If more logs are desired, let me know.

The node currently seems to be dormant, sitting at 10k pending TXs;

Nov 23 12:35:09 anon-r2 erigon[3060088]: [INFO] [11-23|12:35:09.744] [p2p] GoodPeers                          eth67=14 eth66=1 eth68=15
Nov 23 12:35:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:35:18.590] P2P                                      app=caplin peers=41
Nov 23 12:35:31 anon-r2 erigon[3060088]: [INFO] [11-23|12:35:31.131] [txpool] stat                            pending=10000 baseFee=0 queued=18048 alloc=5.3GB sys=10.8GB
Nov 23 12:36:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:36:18.595] P2P                                      app=caplin peers=40
Nov 23 12:37:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:37:18.591] P2P                                      app=caplin peers=48
Nov 23 12:38:09 anon-r2 erigon[3060088]: [INFO] [11-23|12:38:09.744] [p2p] GoodPeers                          eth68=18 eth67=13
Nov 23 12:38:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:38:18.591] P2P                                      app=caplin peers=49
Nov 23 12:38:31 anon-r2 erigon[3060088]: [INFO] [11-23|12:38:31.179] [txpool] stat                            pending=10000 baseFee=0 queued=19269 alloc=6.0GB sys=10.8GB
Nov 23 12:39:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:39:18.591] P2P                                      app=caplin peers=44
Nov 23 12:40:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:40:18.594] P2P                                      app=caplin peers=41
Nov 23 12:41:09 anon-r2 erigon[3060088]: [INFO] [11-23|12:41:09.745] [p2p] GoodPeers                          eth67=12 eth66=3 eth68=18
Nov 23 12:41:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:41:18.591] P2P                                      app=caplin peers=41
Nov 23 12:41:31 anon-r2 erigon[3060088]: [INFO] [11-23|12:41:31.131] [txpool] stat                            pending=10000 baseFee=0 queued=20272 alloc=5.0GB sys=10.8GB
Nov 23 12:42:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:42:18.591] P2P                                      app=caplin peers=43
Nov 23 12:43:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:43:18.591] P2P                                      app=caplin peers=41
Nov 23 12:44:09 anon-r2 erigon[3060088]: [INFO] [11-23|12:44:09.744] [p2p] GoodPeers                          eth68=21 eth67=11
Nov 23 12:44:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:44:18.591] P2P                                      app=caplin peers=47
Nov 23 12:44:31 anon-r2 erigon[3060088]: [INFO] [11-23|12:44:31.134] [txpool] stat                            pending=10000 baseFee=0 queued=23164 alloc=5.2GB sys=10.8GB

More data leading up to above, including the service crash;

Nov 23 11:43:07 anon-r2 erigon[2971467]: [INFO] [11-23|11:43:07.408] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-3974712468
Nov 23 11:43:14 anon-r2 erigon[2971467]: [INFO] [11-23|11:43:14.345] [p2p] GoodPeers                          eth68=32 eth67=31 eth66=1
Nov 23 11:43:16 anon-r2 erigon[2971467]: [INFO] [11-23|11:43:16.348] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=000000000119909c6b175474e89094c44da98b954eedeac495271d0f0000000000000001
Nov 23 11:43:16 anon-r2 erigon[2971467]: [INFO] [11-23|11:43:16.978] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=8.0GB sys=14.8GB
Nov 23 11:43:26 anon-r2 erigon[2971467]: [INFO] [11-23|11:43:26.323] P2P                                      app=caplin peers=41
Nov 23 11:43:43 anon-r2 erigon[2971467]: [INFO] [11-23|11:43:43.826] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-3721122729
Nov 23 11:43:46 anon-r2 erigon[2971467]: [INFO] [11-23|11:43:46.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=000000000119b6cc3408d13d5deb25db529eff2ea53d3aaf79dc977b0000000000000001
Nov 23 11:44:16 anon-r2 erigon[2971467]: [INFO] [11-23|11:44:16.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=000000000119ded2ae3359ed3c567482fb0102c584c23daa2693eacf0000000000000001
Nov 23 11:44:22 anon-r2 erigon[2971467]: [INFO] [11-23|11:44:22.478] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-15744716
Nov 23 11:44:26 anon-r2 erigon[2971467]: [INFO] [11-23|11:44:26.324] P2P                                      app=caplin peers=40
Nov 23 11:44:46 anon-r2 erigon[2971467]: [INFO] [11-23|11:44:46.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011a0529a876f27f13a9eb6e621202cefdd5afc4a90e64570000000000000001
Nov 23 11:45:01 anon-r2 erigon[2971467]: [INFO] [11-23|11:45:01.986] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-1759367266
Nov 23 11:45:16 anon-r2 erigon[2971467]: [INFO] [11-23|11:45:16.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011a27902dc9050d9873f50526e467e983d435e6d8d9afb00000000000000001
Nov 23 11:45:26 anon-r2 erigon[2971467]: [INFO] [11-23|11:45:26.324] P2P                                      app=caplin peers=39
Nov 23 11:45:43 anon-r2 erigon[2971467]: [INFO] [11-23|11:45:43.923] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-163743870
Nov 23 11:45:46 anon-r2 erigon[2971467]: [INFO] [11-23|11:45:46.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011a48507d1afa7b718fb893db30a3abc0cfc608aacfebb00000000000000001
Nov 23 11:46:14 anon-r2 erigon[2971467]: [INFO] [11-23|11:46:14.345] [p2p] GoodPeers                          eth67=32 eth66=1 eth68=33
Nov 23 11:46:16 anon-r2 erigon[2971467]: [INFO] [11-23|11:46:16.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011a6935a44a4797f78d2d64af93991c01724b937b5877c50000000000000001
Nov 23 11:46:16 anon-r2 erigon[2971467]: [INFO] [11-23|11:46:16.975] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=6.6GB sys=14.8GB
Nov 23 11:46:26 anon-r2 erigon[2971467]: [INFO] [11-23|11:46:26.323] P2P                                      app=caplin peers=47
Nov 23 11:46:30 anon-r2 erigon[2971467]: [INFO] [11-23|11:46:30.844] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-806667262
Nov 23 11:46:46 anon-r2 erigon[2971467]: [INFO] [11-23|11:46:46.348] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011a8d64b7dd2ce14c40fdecf65a81d1c536d49390a7b7340000000000000001
Nov 23 11:47:09 anon-r2 erigon[2971467]: [INFO] [11-23|11:47:09.829] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-1019104797
Nov 23 11:47:16 anon-r2 erigon[2971467]: [INFO] [11-23|11:47:16.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011ab0724976fb03c32e5b8cfe2b6ccb31c09ba78ebaba410000000000000001
Nov 23 11:47:26 anon-r2 erigon[2971467]: [INFO] [11-23|11:47:26.323] P2P                                      app=caplin peers=41
Nov 23 11:47:46 anon-r2 erigon[2971467]: [INFO] [11-23|11:47:46.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011ad2c492d6c1e31e14520e676a687f0a93788b716beff50000000000000001
Nov 23 11:47:56 anon-r2 erigon[2971467]: [INFO] [11-23|11:47:56.519] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-3296237439
Nov 23 11:48:16 anon-r2 erigon[2971467]: [INFO] [11-23|11:48:16.349] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011af266675fdee6819b33a00a8efe2caecf7e4a127a68e20000000000000001
Nov 23 11:48:26 anon-r2 erigon[2971467]: [INFO] [11-23|11:48:26.324] P2P                                      app=caplin peers=45
Nov 23 11:48:38 anon-r2 systemd[1]: erigon.service: systemd-oomd killed 4 process(es) in this unit.
Nov 23 11:48:38 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Nov 23 11:48:38 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.
Nov 23 11:48:38 anon-r2 systemd[1]: erigon.service: Consumed 1d 10h 2min 4.206s CPU time.
Nov 23 11:49:23 anon-r2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 1.
Nov 23 11:49:23 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Nov 23 11:49:23 anon-r2 systemd[1]: erigon.service: Consumed 1d 10h 2min 4.206s CPU time.
Nov 23 11:49:23 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Nov 23 11:49:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:24.313] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Nov 23 11:49:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:24.313] Build info                               git_branch=HEAD git_tag=v2.53.4 git_commit=c4843268b788c0899cdad12d220fea4396afbb56
Nov 23 11:49:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:24.314] Starting Erigon on Ethereum mainnet...
Nov 23 11:49:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:24.317] Maximum peer count                       ETH=100 total=100
Nov 23 11:49:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:24.318] starting HTTP APIs                       APIs=web3,net,eth
Nov 23 11:49:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:24.319] torrent verbosity                        level=WRN
Nov 23 11:49:26 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:26.424] Set global gas cap                       cap=50000000
Nov 23 11:49:26 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:26.526] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=150mb upload.rate=15mb
Nov 23 11:49:26 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:26.740] Opening Database                         label=chaindata path=/storage/ethereum/erigon/data/chaindata
Nov 23 11:49:26 anon-r2 erigon[3060088]: mdbx_setup_dxb:15946 filesize mismatch (expect 1848647876608b/225665024p, have 2039690035200b/248985600p)
Nov 23 11:49:38 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:38.589] [db] chaindata                           sizeLimit=3TB pageSize=8192
Nov 23 11:49:38 anon-r2 erigon[3060088]: [INFO] [11-23|11:49:38.593] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: <nil>, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
Nov 23 11:50:09 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:09.753] Initialising Ethereum protocol           network=1
Nov 23 11:50:09 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:09.753] Disk storage enabled for ethash DAGs     dir=/storage/ethereum/erigon/data/ethash-dags count=2
Nov 23 11:50:09 anon-r2 erigon[3060088]: 17999999
Nov 23 11:50:31 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:31.130] Starting private RPC server              on=192.168.2.3:9191
Nov 23 11:50:31 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:31.130] new subscription to logs established
Nov 23 11:50:31 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:31.131] [Checkpoint Sync] Requesting beacon state uri=https://mainnet-checkpoint-sync.attestant.io/eth/v2/debug/beacon/states/finalized
Nov 23 11:50:31 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:31.951] [txpool] Started
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.805] [Sentinel] Sentinel started              enr=enr:-Ly4QEJTBj_5PtFyV5a6G17KD38gBRWIKWVMZattB4dOz_D5M4cceEecitKqICR0uc5_82rBZ_gyOzSb_5ev_EksneABh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC7pNqWBAAAAP__________gmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQJww5LpSGbv4qg0kbpbtgimjYWRVqB3i7_05QxKLXeVnohzeW5jbmV0cwCDdGNwgg-hg3VkcIIPoA
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.805] Static peers                             len=0
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.805] rpc filters: subscribing to Erigon events
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.806] new subscription to newHeaders established
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.806] New txs subscriber joined
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.807] Reading JWT secret                       path=/var/ethereum/jwtsecret
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.809] HTTP endpoint opened                     url=192.168.2.3:8545 ws=true ws.compression=false grpc=false
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.810] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=false
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.823] [1/12 Snapshots] Fetching torrent files metadata
Nov 23 11:50:43 anon-r2 erigon[3060088]: [WARN] [11-23|11:50:43.835] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.843] Started P2P networking                   version=68 self=enode://[REDACTED]@127.0.0.1:30303 name=erigon/v2.53.4-c4843268/linux-amd64/go1.20.10
Nov 23 11:50:43 anon-r2 erigon[3060088]: [WARN] [11-23|11:50:43.845] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Nov 23 11:50:43 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:43.857] Started P2P networking                   version=67 self=enode://[REDACTED]@127.0.0.1:30304 name=erigon/v2.53.4-c4843268/linux-amd64/go1.20.10
Nov 23 11:50:46 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:46.269] [snapshots] Blocks Stat                  blocks=18000k indices=18000k alloc=4.9GB sys=5.1GB
Nov 23 11:50:50 anon-r2 erigon[3060088]: [INFO] [11-23|11:50:50.753] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0x1ef09eb5681a9b3a495805e98a81b1ae1d19245889e02f5b88b65958ca188e11 requestId=0
Nov 23 11:51:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:51:18.590] Started Ethereum 2.0 Gossip Service      app=caplin
Nov 23 11:51:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:51:18.591] [Caplin] starting clstages loop          app=caplin
Nov 23 11:51:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:51:18.591] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=7826240
Nov 23 11:51:37 anon-r2 erigon[3060088]: [INFO] [11-23|11:51:37.996] [NewPayload] Handling new payload        height=18634250 hash=0xbb190d850e1b21be2332add165a13ec4baac005ecfcbf223b9852d4d2582597f
Nov 23 11:51:38 anon-r2 erigon[3060088]: [INFO] [11-23|11:51:38.191] [EngineBlockDownloader] Processing bodies... from=18628490 to=18634244
Nov 23 11:51:48 anon-r2 erigon[3060088]: [INFO] [11-23|11:51:48.591] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7826048 blockNumber=18633943 blk/sec=6.4 mbps/sec=1.9333 peers=11
Nov 23 11:51:48 anon-r2 erigon[3060088]: [INFO] [11-23|11:51:48.850] [NewPayload] Handling new payload        height=18634251 hash=0xda4bbc433f4760695f477c379654a9e9e804a83a185104bcdc0c23402b146c0e
Nov 23 11:52:02 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:02.176] [NewPayload] Handling new payload        height=18634252 hash=0x4fa80740012be1098a6bbab786881c49e05b5d77f2e434a003d4ee8b55809970
Nov 23 11:52:08 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:08.194] [EngineBlockDownloader] Downloading block bodies block_num=18628613 delivery/sec=2.4MB wasted/sec=0B remaining=5630 delivered=646 cache=60.4MB alloc=4.5GB sys=5.7GB
Nov 23 11:52:13 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:13.876] [NewPayload] Handling new payload        height=18634253 hash=0x78208171a118c3420dd5d3e88bcc7825ebaf833f6ae50889eee6bf051a17913d
Nov 23 11:52:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:18.590] P2P                                      app=caplin peers=15
Nov 23 11:52:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:18.592] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7825984 blockNumber=18633879 blk/sec=2.1 mbps/sec=1.1461 peers=15
Nov 23 11:52:27 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:27.533] [NewPayload] Handling new payload        height=18634254 hash=0x24d091abcbe6e6fb8794d89f84b0ab42fa82ea53be081eb59aadf8d572b9c35d
Nov 23 11:52:37 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:37.871] [NewPayload] Handling new payload        height=18634255 hash=0xaf0df659ad0061c1e90e4bfcbf5cb315fd689d329b6f95e569c825ac97d46242
Nov 23 11:52:38 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:38.192] [EngineBlockDownloader] Downloading block bodies block_num=18629079 delivery/sec=2.1MB wasted/sec=0B remaining=5164 delivered=1205 cache=69.2MB alloc=3.4GB sys=6.3GB
Nov 23 11:52:48 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:48.591] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7825856 blockNumber=18633751 blk/sec=4.3 mbps/sec=0.0000 peers=17
Nov 23 11:52:49 anon-r2 erigon[3060088]: [INFO] [11-23|11:52:49.844] [NewPayload] Handling new payload        height=18634256 hash=0xd9c38a316669d7cb7f8ee0aa520b747ad0b7ee0dd1252aa40eca3436b0e1215b
Nov 23 11:53:01 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:01.392] [NewPayload] Handling new payload        height=18634257 hash=0x5095fbe4c28bf33aefc98be31d9a0699e7b172a87e0620e4037e72db3c7d83b2
Nov 23 11:53:08 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:08.192] [EngineBlockDownloader] Downloading block bodies block_num=18630175 delivery/sec=2.9MB wasted/sec=0B remaining=4068 delivered=1981 cache=35.8MB alloc=5.3GB sys=6.4GB
Nov 23 11:53:09 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:09.744] [p2p] GoodPeers                          eth68=2
Nov 23 11:53:13 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:13.637] [NewPayload] Handling new payload        height=18634258 hash=0x16e087bf483ae8a5ff0ba40e6c2b78f4e8ea5a16601586f0a90c88b88c84bd94
Nov 23 11:53:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:18.590] P2P                                      app=caplin peers=5
Nov 23 11:53:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:18.591] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7825600 blockNumber=18633496 blk/sec=8.5 mbps/sec=0.0000 peers=5
Nov 23 11:53:25 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:25.078] [NewPayload] Handling new payload        height=18634259 hash=0x6ff1eae52f3382faed158d0fbb32c7f678fbcf1431f6a390aecf4b2d540024c1
Nov 23 11:53:31 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:31.133] [txpool] stat                            pending=195 baseFee=0 queued=1764 alloc=4.0GB sys=6.8GB
Nov 23 11:53:38 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:38.193] [EngineBlockDownloader] Downloading block bodies block_num=18630574 delivery/sec=3.8MB wasted/sec=0B remaining=3669 delivered=2840 cache=102.6MB alloc=4.3GB sys=6.8GB
Nov 23 11:53:38 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:38.802] [NewPayload] Handling new payload        height=18634260 hash=0x00a29fb70946f233922abf5e997ee06004e00bb5f0465d15a27a310624581fd8
Nov 23 11:53:48 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:48.591] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7825472 blockNumber=18633369 blk/sec=4.3 mbps/sec=0.0000 peers=13
Nov 23 11:53:50 anon-r2 erigon[3060088]: [INFO] [11-23|11:53:50.376] [NewPayload] Handling new payload        height=18634261 hash=0xab1e810b4a7c7540ea47adcc41086b0d89bf60d515a7b107b71261afc6166aff
Nov 23 11:54:02 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:02.446] [NewPayload] Handling new payload        height=18634262 hash=0x4445e67b116a2299dfd3b6a4257114ef2745266625c5dde2184d67a6915c1900
Nov 23 11:54:08 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:08.193] [EngineBlockDownloader] Downloading block bodies block_num=18631224 delivery/sec=7.1MB wasted/sec=0B remaining=3019 delivered=4137 cache=228.9MB alloc=6.5GB sys=6.8GB
Nov 23 11:54:16 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:16.258] [NewPayload] Handling new payload        height=18634263 hash=0xd096fbf89b799bdfd59738bae8ba4dceae5ed79774d19963780aad3e65626cef
Nov 23 11:54:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:18.590] P2P                                      app=caplin peers=6
Nov 23 11:54:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:18.592] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7825408 blockNumber=18633305 blk/sec=2.1 mbps/sec=0.0000 peers=6
Nov 23 11:54:27 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:27.744] [NewPayload] Handling new payload        height=18634264 hash=0x24b5b1aeebc695f2517276e2e56fa1b2cb96f52ea711b531ab55e71483b368e2
Nov 23 11:54:38 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:38.193] [EngineBlockDownloader] Downloading block bodies block_num=18632767 delivery/sec=6.6MB wasted/sec=0B remaining=1476 delivered=5144 cache=174.5MB alloc=5.0GB sys=7.0GB
Nov 23 11:54:38 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:38.827] [NewPayload] Handling new payload        height=18634265 hash=0x39294992d6105ebd283fb525bbefe7aa82910edab54df0673b76c5542dee3d61
Nov 23 11:54:48 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:48.591] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7825344 blockNumber=18633241 blk/sec=2.1 mbps/sec=0.0000 peers=13
Nov 23 11:54:50 anon-r2 erigon[3060088]: [INFO] [11-23|11:54:50.333] [NewPayload] Handling new payload        height=18634266 hash=0xebdf5c0d3541ca20e7d1f2dd5e62e15585f0a35207cfbf29b990275d6d0335ed
Nov 23 11:55:01 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:01.488] [NewPayload] Handling new payload        height=18634267 hash=0x6036a8b6464f655f97acf9e5b4c4ffa875059f03a0a72e6193898c3217638cff
Nov 23 11:55:05 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:05.012] [EngineBlockDownloader] Processed        highest=18634244
Nov 23 11:55:05 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:05.012] Beginning downloaded blocks insertion
Nov 23 11:55:14 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:14.913] [EngineBlockDownloader] Finished downloading blocks from=18628490 to=18634244
Nov 23 11:55:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:18.591] P2P                                      app=caplin peers=6
Nov 23 11:55:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:18.591] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7825153 blockNumber=18633052 blk/sec=6.4 mbps/sec=1.8616 peers=6
Nov 23 11:55:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:18.732] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
Nov 23 11:55:20 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:20.818] [Caplin] Downloading epochs from reqresp app=caplin stage=CatchUpEpochs from=244570 to=244573
Nov 23 11:55:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:24.208] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244570
Nov 23 11:55:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:24.546] [NewPayload] Handling new payload        height=18634269 hash=0x5f4b8ccd43291c2d805e17d086f10eecc33278b05be7751c9480a93405eecac1
Nov 23 11:55:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:24.546] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0x644f48f6c7032922b98c580cc71c3cb4ca36f06c84312aa91eeb2e57d7f43795 requestId=0
Nov 23 11:55:24 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:24.850] [EngineBlockDownloader] Processing bodies... from=18634244 to=18634268
Nov 23 11:55:29 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:29.330] [EngineBlockDownloader] Processed        highest=18634268
Nov 23 11:55:29 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:29.330] Beginning downloaded blocks insertion
Nov 23 11:55:29 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:29.393] [EngineBlockDownloader] Finished downloading blocks from=18634244 to=18634268
Nov 23 11:55:29 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:29.402] [EngineBlockDownloader] block verification skipped
Nov 23 11:55:33 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:33.723] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244571
Nov 23 11:55:33 anon-r2 erigon[3060088]: [WARN] [11-23|11:55:33.733] fail to process block                    reason="replay block, code: 2" slot=7826272
Nov 23 11:55:33 anon-r2 erigon[3060088]: [WARN] [11-23|11:55:33.733] bad blocks segment received              err="replay block, code: 2"
Nov 23 11:55:36 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:36.906] [NewPayload] Handling new payload        height=18634270 hash=0xfa3d3bb021c402d3477d88af7da837ba9ad197a198c853f62c7a07a03a0cb412
Nov 23 11:55:51 anon-r2 erigon[3060088]: [INFO] [11-23|11:55:51.081] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244570
Nov 23 11:56:09 anon-r2 erigon[3060088]: [INFO] [11-23|11:56:09.744] [p2p] GoodPeers                          eth68=8 eth67=2
Nov 23 11:56:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:56:18.591] P2P                                      app=caplin peers=9
Nov 23 11:56:31 anon-r2 erigon[3060088]: [INFO] [11-23|11:56:31.170] [txpool] stat                            pending=1433 baseFee=0 queued=2847 alloc=5.2GB sys=10.6GB
Nov 23 11:57:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:57:18.591] P2P                                      app=caplin peers=18
Nov 23 11:58:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:58:18.591] P2P                                      app=caplin peers=33
Nov 23 11:59:09 anon-r2 erigon[3060088]: [INFO] [11-23|11:59:09.745] [p2p] GoodPeers                          eth68=9 eth67=3
Nov 23 11:59:18 anon-r2 erigon[3060088]: [INFO] [11-23|11:59:18.595] P2P                                      app=caplin peers=28
Nov 23 11:59:31 anon-r2 erigon[3060088]: [INFO] [11-23|11:59:31.160] [txpool] stat                            pending=2877 baseFee=0 queued=6155 alloc=5.8GB sys=10.8GB
Nov 23 12:00:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:00:18.590] P2P                                      app=caplin peers=31
Nov 23 12:01:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:01:18.592] P2P                                      app=caplin peers=37
Nov 23 12:02:09 anon-r2 erigon[3060088]: [INFO] [11-23|12:02:09.744] [p2p] GoodPeers                          eth66=1 eth67=5 eth68=9
Nov 23 12:02:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:02:18.591] P2P                                      app=caplin peers=37
Nov 23 12:02:31 anon-r2 erigon[3060088]: [INFO] [11-23|12:02:31.132] [txpool] stat                            pending=3610 baseFee=0 queued=7259 alloc=5.3GB sys=10.8GB

Nevermind, it's alive again!

However, it looks like it's starting from the position it was at a couple days ago. :(

Nov 23 12:53:03 anon-r2 erigon[3060088]: [INFO] [11-23|12:53:03.514] [3/12 Senders] Started                   from=17999999 to=18634270
Nov 23 12:53:09 anon-r2 erigon[3060088]: [INFO] [11-23|12:53:09.744] [p2p] GoodPeers                          eth68=19 eth67=15
Nov 23 12:53:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:53:18.659] P2P                                      app=caplin peers=44
Nov 23 12:53:31 anon-r2 erigon[3060088]: [INFO] [11-23|12:53:31.156] [txpool] stat                            pending=10000 baseFee=0 queued=27985 alloc=8.9GB sys=11.3GB
Nov 23 12:53:33 anon-r2 erigon[3060088]: [INFO] [11-23|12:53:33.515] [3/12 Senders] Recovery                  block_number=18018723 ch=8629/10000
Nov 23 12:54:03 anon-r2 erigon[3060088]: [INFO] [11-23|12:54:03.522] [3/12 Senders] Recovery                  block_number=18036500 ch=10000/10000
Nov 23 12:54:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:54:18.591] P2P                                      app=caplin peers=38
Nov 23 12:54:33 anon-r2 erigon[3060088]: [INFO] [11-23|12:54:33.518] [3/12 Senders] Recovery                  block_number=18054524 ch=9998/10000
Nov 23 12:55:05 anon-r2 erigon[3060088]: [INFO] [11-23|12:55:05.559] [3/12 Senders] Recovery                  block_number=18073078 ch=9153/10000
Nov 23 12:55:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:55:18.592] P2P                                      app=caplin peers=37
Nov 23 12:55:33 anon-r2 erigon[3060088]: [INFO] [11-23|12:55:33.516] [3/12 Senders] Recovery                  block_number=18091826 ch=10000/10000
Nov 23 12:55:36 anon-r2 erigon[3060088]: [INFO] [11-23|12:55:36.133] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-1583398239
Nov 23 12:56:03 anon-r2 erigon[3060088]: [INFO] [11-23|12:56:03.516] [3/12 Senders] Recovery                  block_number=18112164 ch=10000/10000
Nov 23 12:56:09 anon-r2 erigon[3060088]: [INFO] [11-23|12:56:09.747] [p2p] GoodPeers                          eth68=20 eth67=15
Nov 23 12:56:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:56:18.590] P2P                                      app=caplin peers=32
Nov 23 12:56:31 anon-r2 erigon[3060088]: [INFO] [11-23|12:56:31.131] [txpool] stat                            pending=10000 baseFee=0 queued=29304 alloc=8.6GB sys=16.0GB
Nov 23 12:56:33 anon-r2 erigon[3060088]: [INFO] [11-23|12:56:33.522] [3/12 Senders] Recovery                  block_number=18126999 ch=9993/10000
Nov 23 12:57:03 anon-r2 erigon[3060088]: [INFO] [11-23|12:57:03.518] [3/12 Senders] Recovery                  block_number=18144391 ch=10000/10000
Nov 23 12:57:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:57:18.591] P2P                                      app=caplin peers=40
Nov 23 12:57:33 anon-r2 erigon[3060088]: [INFO] [11-23|12:57:33.523] [3/12 Senders] Recovery                  block_number=18164711 ch=10000/10000
Nov 23 12:58:03 anon-r2 erigon[3060088]: [INFO] [11-23|12:58:03.517] [3/12 Senders] Recovery                  block_number=18183379 ch=10000/10000
Nov 23 12:58:03 anon-r2 erigon[3060088]: [INFO] [11-23|12:58:03.743] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-3796689768
Nov 23 12:58:18 anon-r2 erigon[3060088]: [INFO] [11-23|12:58:18.592] P2P                                      app=caplin peers=43

@AskAlexSharov you are right, sorry about that.

erigon version 2.49.2-stable-a8ae8a03

Here is my Erigon config:

erigon --private.api.addr 0.0.0.0:9090 --datadir /var/lib/erigon --port 30303 --p2p.allowed-ports 30303,30304,30305 --torrent.port 42069 --nat any --log.console.verbosity info --metrics --metrics.addr 0.0.0.0 --http --http.addr 0.0.0.0 --http.port 8545 --http.vhosts=* --http.corsdomain=* --ws --rpc.returndata.limit 1000000 --authrpc.addr 0.0.0.0 --authrpc.port 8551 --authrpc.vhosts=* --authrpc.jwtsecret /var/lib/erigon/ee-secret/jwtsecret --maxpeers 1000 --db.pagesize 16K --chain mainnet --http.api web3,eth,net,engine --prune=htc --prune.r.before=11052984 --internalcl --torrent.download.rate=999mb --torrent.download.slots=6

Logs show some progress, but when I check via RCP it stills stuck at block # 16999999 for over 3 days

curl -s -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "method":"eth_blockNumber", "params": [], "id":1}' localhost:8545 | jq
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x103663f"
}

Here are the latest logs:

eth-docker-execution-1  | [INFO] [11-24|09:19:23.612] [4/12 Execution] Executed blocks         number=17170659 blk/s=14.1 tx/s=2107.3 Mgas/s=210.8 gasState=0.02 batch=19.8MB alloc=8.7GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:19:43.623] [4/12 Execution] Executed blocks         number=17170932 blk/s=13.6 tx/s=2056.9 Mgas/s=207.7 gasState=0.03 batch=28.5MB alloc=6.5GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:19:48.028] P2P                                      app=caplin peers=33
eth-docker-execution-1  | [WARN] [11-24|09:19:48.117] Served                                   conn=45.118.135.196:54320 method=eth_accounts reqid=1 t=106.691µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:20:03.709] [4/12 Execution] Executed blocks         number=17171221 blk/s=14.4 tx/s=2152.9 Mgas/s=215.3 gasState=0.03 batch=38.2MB alloc=8.5GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:20:15.847] [txpool] stat                            pending=10000 baseFee=19990 queued=30000 alloc=5.4GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:20:23.635] [4/12 Execution] Executed blocks         number=17171492 blk/s=13.6 tx/s=2043.5 Mgas/s=204.5 gasState=0.04 batch=47.2MB alloc=6.2GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:20:24.634] Served                                   conn=45.118.135.196:57378 method=eth_accounts reqid=1 t=122.331µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:20:43.674] [4/12 Execution] Executed blocks         number=17171757 blk/s=13.2 tx/s=1963.0 Mgas/s=199.5 gasState=0.05 batch=56.0MB alloc=8.2GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:20:48.029] P2P                                      app=caplin peers=33
eth-docker-execution-1  | [INFO] [11-24|09:20:56.194] [p2p] GoodPeers                          eth68=181 eth67=170 eth66=8
eth-docker-execution-1  | [WARN] [11-24|09:20:57.157] Served                                   conn=139.162.8.59:41144 method=eth_accounts reqid=0 t=108.799µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [WARN] [11-24|09:21:01.101] Served                                   conn=45.118.135.196:42944 method=eth_accounts reqid=1 t=107.353µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:21:03.735] [4/12 Execution] Executed blocks         number=17172013 blk/s=12.8 tx/s=2067.5 Mgas/s=192.8 gasState=0.05 batch=64.5MB alloc=5.8GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:21:13.992] Served                                   conn=46.114.201.24:25201 method=eth_accounts reqid=4521900295 t=102.929µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:21:15.877] [txpool] stat                            pending=10000 baseFee=19993 queued=30000 alloc=7.1GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:21:23.612] [4/12 Execution] Executed blocks         number=17172289 blk/s=13.9 tx/s=2324.3 Mgas/s=209.9 gasState=0.06 batch=73.9MB alloc=7.9GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:21:38.292] Served                                   conn=45.118.135.196:53542 method=eth_accounts reqid=1 t=108.181µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:21:43.599] [4/12 Execution] Executed blocks         number=17172574 blk/s=14.3 tx/s=2314.3 Mgas/s=214.3 gasState=0.07 batch=83.6MB alloc=5.6GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:21:48.028] P2P                                      app=caplin peers=33
eth-docker-execution-1  | [WARN] [11-24|09:21:54.812] Served                                   conn=45.118.135.196:58366 method=eth_sendTransaction reqid=0 t=119.385µs err="the method is currently not implemented: eth_sendTransaction"
eth-docker-execution-1  | [INFO] [11-24|09:22:03.617] [4/12 Execution] Executed blocks         number=17172848 blk/s=13.7 tx/s=2344.7 Mgas/s=206.7 gasState=0.08 batch=93.0MB alloc=7.6GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:22:14.960] Served                                   conn=45.118.135.196:33988 method=eth_accounts reqid=1 t=103.516µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:22:15.849] [txpool] stat                            pending=10000 baseFee=19998 queued=30000 alloc=8.8GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:22:23.615] [4/12 Execution] Executed blocks         number=17173132 blk/s=14.2 tx/s=2282.1 Mgas/s=215.1 gasState=0.09 batch=103.2MB alloc=5.2GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:22:43.695] [4/12 Execution] Executed blocks         number=17173420 blk/s=14.3 tx/s=2356.6 Mgas/s=216.9 gasState=0.09 batch=113.6MB alloc=7.2GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:22:48.028] P2P                                      app=caplin peers=34
eth-docker-execution-1  | [WARN] [11-24|09:22:52.345] Served                                   conn=45.118.135.196:50202 method=eth_accounts reqid=1 t=153.141µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:22:56.194] [p2p] GoodPeers                          eth68=181 eth67=172 eth66=7
eth-docker-execution-1  | [INFO] [11-24|09:23:03.600] [4/12 Execution] Executed blocks         number=17173698 blk/s=14.0 tx/s=2262.1 Mgas/s=211.7 gasState=0.10 batch=123.4MB alloc=4.8GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:23:15.879] [txpool] stat                            pending=10000 baseFee=19998 queued=30000 alloc=6.1GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:23:23.614] [4/12 Execution] Executed blocks         number=17173981 blk/s=14.1 tx/s=2246.7 Mgas/s=213.6 gasState=0.11 batch=133.3MB alloc=6.9GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:23:32.609] Served                                   conn=139.162.8.59:41152 method=eth_accounts reqid=0 t=118.001µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [WARN] [11-24|09:23:34.525] Served                                   conn=45.118.135.196:44248 method=eth_accounts reqid=1 t=103.965µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:23:43.657] [4/12 Execution] Executed blocks         number=17174276 blk/s=14.7 tx/s=2398.7 Mgas/s=220.7 gasState=0.12 batch=143.7MB alloc=5.0GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:23:48.029] P2P                                      app=caplin peers=33
eth-docker-execution-1  | [INFO] [11-24|09:24:03.603] [4/12 Execution] Executed blocks         number=17174572 blk/s=14.8 tx/s=2440.5 Mgas/s=225.5 gasState=0.13 batch=154.1MB alloc=6.7GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:24:12.089] Served                                   conn=45.118.135.196:39060 method=eth_accounts reqid=1 t=130.594µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:24:15.848] [txpool] stat                            pending=10000 baseFee=20005 queued=30000 alloc=7.9GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:24:23.640] [4/12 Execution] Executed blocks         number=17174868 blk/s=14.8 tx/s=2516.3 Mgas/s=220.2 gasState=0.13 batch=164.4MB alloc=8.7GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:24:43.513] Served                                   conn=45.118.135.196:39060 method=eth_accounts reqid=1 t=102.821µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:24:43.654] [4/12 Execution] Executed blocks         number=17175142 blk/s=13.7 tx/s=2390.7 Mgas/s=206.8 gasState=0.14 batch=174.2MB alloc=6.2GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:24:48.028] P2P                                      app=caplin peers=33
eth-docker-execution-1  | [INFO] [11-24|09:24:56.194] [p2p] GoodPeers                          eth68=183 eth67=172 eth66=9
eth-docker-execution-1  | [INFO] [11-24|09:25:03.665] [4/12 Execution] Executed blocks         number=17175428 blk/s=14.3 tx/s=2580.8 Mgas/s=215.6 gasState=0.15 batch=184.4MB alloc=8.2GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:25:15.884] [txpool] stat                            pending=10000 baseFee=20006 queued=30000 alloc=4.9GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:25:21.001] Served                                   conn=45.118.135.196:58646 method=eth_accounts reqid=1 t=111.094µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:25:23.613] [4/12 Execution] Executed blocks         number=17175718 blk/s=14.5 tx/s=2354.3 Mgas/s=218.2 gasState=0.16 batch=194.9MB alloc=5.6GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:25:43.612] [4/12 Execution] Executed blocks         number=17176014 blk/s=14.8 tx/s=2339.5 Mgas/s=224.7 gasState=0.16 batch=205.0MB alloc=7.7GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:25:48.028] P2P                                      app=caplin peers=34
eth-docker-execution-1  | [WARN] [11-24|09:25:57.507] Served                                   conn=45.118.135.196:60374 method=eth_accounts reqid=1 t=108.362µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:26:03.610] [4/12 Execution] Executed blocks         number=17176312 blk/s=14.9 tx/s=2282.3 Mgas/s=224.7 gasState=0.17 batch=215.2MB alloc=5.3GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:26:15.853] [txpool] stat                            pending=10000 baseFee=20012 queued=30000 alloc=6.4GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:26:23.631] [4/12 Execution] Executed blocks         number=17176543 blk/s=11.5 tx/s=1685.6 Mgas/s=172.4 gasState=0.18 batch=223.3MB alloc=7.0GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:26:34.294] Served                                   conn=45.118.135.196:52308 method=eth_accounts reqid=1 t=99.27µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:26:43.671] [4/12 Execution] Executed blocks         number=17176833 blk/s=14.5 tx/s=2172.3 Mgas/s=216.3 gasState=0.19 batch=234.0MB alloc=8.9GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:26:48.029] P2P                                      app=caplin peers=33
eth-docker-execution-1  | [WARN] [11-24|09:26:48.724] Served                                   conn=46.114.201.24:24868 method=eth_accounts reqid=1259554562 t=104.004µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:26:56.194] [p2p] GoodPeers                          eth68=186 eth67=173 eth66=7
eth-docker-execution-1  | [INFO] [11-24|09:27:03.625] [4/12 Execution] Executed blocks         number=17177116 blk/s=14.2 tx/s=2130.4 Mgas/s=215.0 gasState=0.19 batch=244.3MB alloc=6.4GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:27:05.690] Served                                   conn=139.162.8.59:41158 method=eth_accounts reqid=0 t=118.697µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [WARN] [11-24|09:27:07.292] Served                                   conn=45.118.135.196:54862 method=eth_sendTransaction reqid=0 t=129.515µs err="the method is currently not implemented: eth_sendTransaction"
eth-docker-execution-1  | [INFO] [11-24|09:27:15.878] [txpool] stat                            pending=10000 baseFee=20012 queued=30000 alloc=7.6GB sys=11.5GB
eth-docker-execution-1  | [WARN] [11-24|09:27:16.592] Served                                   conn=45.118.135.196:57140 method=eth_accounts reqid=1 t=125.799µs err="the method has been deprecated: eth_accounts"
eth-docker-execution-1  | [INFO] [11-24|09:27:23.639] [4/12 Execution] Executed blocks         number=17177408 blk/s=14.6 tx/s=2079.6 Mgas/s=221.5 gasState=0.20 batch=254.9MB alloc=8.4GB sys=11.5GB
eth-docker-execution-1  | [INFO] [11-24|09:27:25.707] Committed State                          gas reached=111994155443 gasTarget=549755813888

Any idea what could it be?

If it's "stuck" at some xxx999, that is often an indication that the snapshots(older blocks) are downloaded and executed (or executing). After this erigon will wait for what to do next, has to be directed by the CL (Nimbus, internalcl, etc.) for what new blocks to fetch/execute. Here "stuck" is not the correct way to describe it, it's more like "progressing" or "waiting". Waiting for blocks to be downloaded, being executed in the background. It can be "stuck" if the CL is removed or is unable to reach the EL (erigon, here). It can also crash due to some reason (Out-of-Memory or a bug) Also, close to the tip, logs may not tell you what blocks were executed, if there were very few blocks executed in the last batch.

Thanks @somnathb1, that means that the syncing still progressing just fine?


ethd cmd exec execution integration print_stages --datadir=/var/lib/erigon --chain=mainnet
INFO[11-24|10:27:24.438] logging to file system                   log dir=/var/lib/erigon/logs file prefix=integration log level=info json=false
INFO[11-24|10:27:31.458] [snapshots] Blocks Stat                  blocks=17000k indices=17000k alloc=2.3GB sys=2.4GB
INFO[11-24|10:27:31.460] [bor snapshots] Blocks Stat              blocks=0k indices=0k alloc=2.3GB sys=2.4GB
Note: prune_at doesn't mean 'all data before were deleted' - it just mean stage.Prune function were run to this block. Because 1 stage may prune multiple data types to different prune distance.

                 stage_at    prune_at
Snapshots            16999999    0
Headers              16999999    0
BorHeimdall              0       0
BlockHashes              16999999    0
Bodies               16999999    0
Senders              16999999    0
Execution            16999999    16999999
Translation              0       0
HashState            16999999    0
IntermediateHashes       16999999    16999999
AccountHistoryIndex          16999999    16999999
StorageHistoryIndex          16999999    16999999
LogIndex             16999999    16999999
CallTraces           16999999    16999999
TxLookup             16999999    16999999
Finish               16999999    0
--
prune distance: --prune.h.older=90000 --prune.t.older=90000 --prune.c.older=90000 --prune.r.before=11052984

blocks.v2: blocks=16999999, segments=16999999, indices=16999999

history.v3: false,  idx steps: 0.00, lastMaxTxNum=0->1, lastBlockInSnap=0

sequence: EthTx=2204862065, NonCanonicalTx=0

in db: first header 16999999, last header 18641033, first body 16999999, last body 18641033
--

It looks like my node has just crashed for the 2nd time today. Anyone have any suggestions?

Nov 24 17:03:12 anon-r2 erigon[3122186]: [INFO] [11-24|17:03:12.147] [4/12 Execution] Executed blocks         number=18542675 blk/s=14.4 tx/s=2383.3 Mgas/s=218.8 gasState=0.16 batch=202.3MB alloc=4.6GB sys=14.6GB
Nov 24 17:03:32 anon-r2 erigon[3122186]: [INFO] [11-24|17:03:32.144] [4/12 Execution] Executed blocks         number=18542968 blk/s=14.7 tx/s=2498.0 Mgas/s=219.4 gasState=0.17 batch=213.1MB alloc=6.3GB sys=14.6GB
Nov 24 17:03:52 anon-r2 erigon[3122186]: [INFO] [11-24|17:03:52.175] [4/12 Execution] Executed blocks         number=18543256 blk/s=14.4 tx/s=2427.0 Mgas/s=216.7 gasState=0.18 batch=223.6MB alloc=4.1GB sys=14.6GB
Nov 24 17:04:01 anon-r2 erigon[3122186]: [INFO] [11-24|17:04:01.988] P2P                                      app=caplin peers=33
Nov 24 17:04:12 anon-r2 erigon[3122186]: [INFO] [11-24|17:04:12.153] [4/12 Execution] Executed blocks         number=18543559 blk/s=15.2 tx/s=2469.5 Mgas/s=229.2 gasState=0.19 batch=234.9MB alloc=5.8GB sys=14.6GB
Nov 24 17:04:32 anon-r2 erigon[3122186]: [INFO] [11-24|17:04:32.115] [4/12 Execution] Executed blocks         number=18543858 blk/s=15.0 tx/s=2442.6 Mgas/s=224.8 gasState=0.19 batch=245.7MB alloc=7.5GB sys=14.6GB
Nov 24 17:04:50 anon-r2 erigon[3122186]: [INFO] [11-24|17:04:50.645] Committed State                          gas reached=111136714349 gasTarget=549755813888
Nov 24 17:05:01 anon-r2 erigon[3122186]: [INFO] [11-24|17:05:01.849] [p2p] GoodPeers                          eth68=32 eth67=32
Nov 24 17:05:01 anon-r2 erigon[3122186]: [INFO] [11-24|17:05:01.988] P2P                                      app=caplin peers=34
Nov 24 17:05:21 anon-r2 erigon[3122186]: [INFO] [11-24|17:05:21.198] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=6.2GB sys=14.6GB
Nov 24 17:05:24 anon-r2 erigon[3122186]: [INFO] [11-24|17:05:24.972] [] ETL [2/2] Loading                     into=PlainState current_prefix=42af2345
Nov 24 17:05:34 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Nov 24 17:05:34 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.
Nov 24 17:05:34 anon-r2 systemd[1]: erigon.service: Consumed 1d 6h 24min 12.637s CPU time.
Nov 24 17:06:19 anon-r2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 3.
Nov 24 17:06:19 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Nov 24 17:06:19 anon-r2 systemd[1]: erigon.service: Consumed 1d 6h 24min 12.637s CPU time.
Nov 24 17:06:19 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Nov 24 17:06:20 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:20.083] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Nov 24 17:06:20 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:20.084] Build info                               git_branch=HEAD git_tag=v2.53.4 git_commit=c4843268b788c0899cdad12d220fea4396afbb56
Nov 24 17:06:20 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:20.084] Starting Erigon on Ethereum mainnet...
Nov 24 17:06:20 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:20.088] Maximum peer count                       ETH=100 total=100
Nov 24 17:06:20 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:20.089] starting HTTP APIs                       APIs=web3,net,eth
Nov 24 17:06:20 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:20.089] torrent verbosity                        level=WRN
Nov 24 17:06:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:22.195] Set global gas cap                       cap=50000000
Nov 24 17:06:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:22.295] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=150mb upload.rate=15mb
Nov 24 17:06:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:22.316] Opening Database                         label=chaindata path=/storage/ethereum/erigon/data/chaindata
Nov 24 17:06:22 anon-r2 erigon[3189645]: mdbx_setup_dxb:15946 filesize mismatch (expect 1850526924800b/225894400p, have 1993519136768b/243349504p)
Nov 24 17:06:31 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:31.287] [db] chaindata                           sizeLimit=3TB pageSize=8192
Nov 24 17:06:31 anon-r2 erigon[3189645]: [INFO] [11-24|17:06:31.293] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: <nil>, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
Nov 24 17:07:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:03.402] Initialising Ethereum protocol           network=1
Nov 24 17:07:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:03.403] Disk storage enabled for ethash DAGs     dir=/storage/ethereum/erigon/data/ethash-dags count=2
Nov 24 17:07:03 anon-r2 erigon[3189645]: 17999999
Nov 24 17:07:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:22.961] Starting private RPC server              on=192.168.2.3:9191
Nov 24 17:07:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:22.961] new subscription to logs established
Nov 24 17:07:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:22.962] [Checkpoint Sync] Requesting beacon state uri=https://mainnet-checkpoint-sync.attestant.io/eth/v2/debug/beacon/states/finalized
Nov 24 17:07:23 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:23.933] [txpool] Started
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.821] [Sentinel] Sentinel started              enr=enr:-Ly4QMg6EO4ATIRmLFtpMiJBaC70l19LaZyGc2PreFAyOq5LLc_B-A5bEz1HK1MDyDhIlpfPwLNvD7UAIQRG7sZ9-DwBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC7pNqWBAAAAP__________gmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQKZi4ClCMBZKqgLSo1naCJjy6hvO_2xNDk81jZ_snG5zohzeW5jbmV0cwCDdGNwgg-hg3VkcIIPoA
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.822] rpc filters: subscribing to Erigon events
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.822] Static peers                             len=0
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.822] new subscription to newHeaders established
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.822] New txs subscriber joined
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.826] HTTP endpoint opened                     url=192.168.2.3:8545 ws=true ws.compression=false grpc=false
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.826] Reading JWT secret                       path=/var/ethereum/jwtsecret
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.829] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=false
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.840] [1/12 Snapshots] Fetching torrent files metadata
Nov 24 17:07:36 anon-r2 erigon[3189645]: [WARN] [11-24|17:07:36.853] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.855] Started P2P networking                   version=67 self=enode://268f193a0b50372c50341b582d4efecf96c68890c7ff802d83508f8071719d3ed641570c3e822b8ce4fbfc6972ddbc77cba79c38f5de1b64c9297f13cd7a6888@127.0.0.1:30304 name=erigon/v2.53.4-c4843268/linux-amd64/go1.20.10
Nov 24 17:07:36 anon-r2 erigon[3189645]: [WARN] [11-24|17:07:36.869] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Nov 24 17:07:36 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:36.873] Started P2P networking                   version=68 self=enode://268f193a0b50372c50341b582d4efecf96c68890c7ff802d83508f8071719d3ed641570c3e822b8ce4fbfc6972ddbc77cba79c38f5de1b64c9297f13cd7a6888@127.0.0.1:30303 name=erigon/v2.53.4-c4843268/linux-amd64/go1.20.10
Nov 24 17:07:39 anon-r2 erigon[3189645]: [INFO] [11-24|17:07:39.687] [snapshots] Blocks Stat                  blocks=18000k indices=18000k alloc=2.6GB sys=5.1GB
Nov 24 17:08:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:08:11.819] Started Ethereum 2.0 Gossip Service      app=caplin
Nov 24 17:08:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:08:11.820] [Caplin] starting clstages loop          app=caplin
Nov 24 17:08:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:08:11.821] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=7835008
Nov 24 17:08:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:08:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7834752 blockNumber=18642570 blk/sec=8.5 mbps/sec=0.0000 peers=11
Nov 24 17:09:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:09:11.820] P2P                                      app=caplin peers=9
Nov 24 17:09:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:09:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7834496 blockNumber=18642316 blk/sec=8.5 mbps/sec=1.7073 peers=9
Nov 24 17:09:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:09:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7834354 blockNumber=18642173 blk/sec=4.7 mbps/sec=0.0000 peers=6
Nov 24 17:10:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:10:03.389] [p2p] GoodPeers                          eth68=3 eth67=1 eth66=1
Nov 24 17:10:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:10:11.820] P2P                                      app=caplin peers=7
Nov 24 17:10:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:10:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7834176 blockNumber=18641999 blk/sec=5.9 mbps/sec=1.4855 peers=7
Nov 24 17:10:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:10:22.964] [txpool] stat                            pending=427 baseFee=0 queued=3244 alloc=3.4GB sys=6.6GB
Nov 24 17:10:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:10:41.822] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7833920 blockNumber=18641747 blk/sec=8.5 mbps/sec=0.0000 peers=26
Nov 24 17:11:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:11:11.820] P2P                                      app=caplin peers=14
Nov 24 17:11:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:11:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7833728 blockNumber=18641555 blk/sec=6.4 mbps/sec=0.0000 peers=14
Nov 24 17:11:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:11:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7833536 blockNumber=18641363 blk/sec=6.4 mbps/sec=0.0000 peers=10
Nov 24 17:12:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:12:11.820] P2P                                      app=caplin peers=7
Nov 24 17:12:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:12:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7833280 blockNumber=18641109 blk/sec=8.5 mbps/sec=0.0000 peers=7
Nov 24 17:12:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:12:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7832960 blockNumber=18640792 blk/sec=10.7 mbps/sec=4.2080 peers=4
Nov 24 17:13:00 anon-r2 erigon[3189645]: [INFO] [11-24|17:13:00.372] [HistoryDownload] Flushed buffer file    name=erigon-sortable-buf-4202018815
Nov 24 17:13:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:13:03.389] [p2p] GoodPeers                          eth68=5 eth67=2
Nov 24 17:13:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:13:11.820] P2P                                      app=caplin peers=14
Nov 24 17:13:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:13:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7832590 blockNumber=18640424 blk/sec=12.3 mbps/sec=3.4742 peers=14
Nov 24 17:13:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:13:22.964] [txpool] stat                            pending=1294 baseFee=0 queued=4513 alloc=4.8GB sys=7.7GB
Nov 24 17:13:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:13:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7832512 blockNumber=18640346 blk/sec=2.6 mbps/sec=0.0000 peers=5
Nov 24 17:14:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:14:11.820] P2P                                      app=caplin peers=3
Nov 24 17:14:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:14:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7832384 blockNumber=18640219 blk/sec=4.3 mbps/sec=2.5186 peers=3
Nov 24 17:14:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:14:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7832320 blockNumber=18640155 blk/sec=2.1 mbps/sec=1.3106 peers=8
Nov 24 17:15:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:15:11.820] P2P                                      app=caplin peers=3
Nov 24 17:15:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:15:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7832000 blockNumber=18639841 blk/sec=10.7 mbps/sec=4.9099 peers=3
Nov 24 17:15:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:15:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7831744 blockNumber=18639588 blk/sec=8.5 mbps/sec=0.0000 peers=17
Nov 24 17:16:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:16:03.389] [p2p] GoodPeers                          eth68=7 eth67=2 eth66=1
Nov 24 17:16:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:16:11.820] P2P                                      app=caplin peers=4
Nov 24 17:16:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:16:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7831488 blockNumber=18639335 blk/sec=8.5 mbps/sec=0.0000 peers=4
Nov 24 17:16:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:16:22.964] [txpool] stat                            pending=1997 baseFee=0 queued=8132 alloc=4.5GB sys=7.9GB
Nov 24 17:16:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:16:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7831232 blockNumber=18639081 blk/sec=8.5 mbps/sec=0.0000 peers=8
Nov 24 17:17:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:17:11.820] P2P                                      app=caplin peers=23
Nov 24 17:17:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:17:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7831040 blockNumber=18638892 blk/sec=6.4 mbps/sec=1.0105 peers=23
Nov 24 17:17:41 anon-r2 erigon[3189645]: [INFO] [11-24|17:17:41.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7830848 blockNumber=18638700 blk/sec=6.4 mbps/sec=0.0000 peers=5
Nov 24 17:18:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:11.820] P2P                                      app=caplin peers=7
Nov 24 17:18:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:11.821] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7830592 blockNumber=18638448 blk/sec=8.5 mbps/sec=0.0000 peers=7
Nov 24 17:18:14 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:14.821] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
Nov 24 17:18:15 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:15.014] [HistoryDownload] Flushed buffer file    name=erigon-sortable-buf-3832445733
Nov 24 17:18:25 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:25.188] [Caplin] Waiting For Peers               app=caplin stage=WaitForPeers have=1 needed=4 retryIn=3s
Nov 24 17:18:28 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:28.188] [Caplin] Waiting For Peers               app=caplin stage=WaitForPeers have=1 needed=4 retryIn=3s
Nov 24 17:18:31 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:31.189] [Caplin] Waiting For Peers               app=caplin stage=WaitForPeers have=3 needed=4 retryIn=3s
Nov 24 17:18:34 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:34.190] [Caplin] Downloading epochs from reqresp app=caplin stage=CatchUpEpochs from=244844 to=244848
Nov 24 17:18:38 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:38.542] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244844
Nov 24 17:18:47 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:47.265] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244845
Nov 24 17:18:57 anon-r2 erigon[3189645]: [INFO] [11-24|17:18:57.210] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244846
Nov 24 17:19:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:19:03.391] [p2p] GoodPeers                          eth68=9 eth67=3
Nov 24 17:19:06 anon-r2 erigon[3189645]: [INFO] [11-24|17:19:06.817] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244847
Nov 24 17:19:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:19:11.820] P2P                                      app=caplin peers=16
Nov 24 17:19:16 anon-r2 erigon[3189645]: [INFO] [11-24|17:19:16.139] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244848
Nov 24 17:19:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:19:22.974] [txpool] stat                            pending=4472 baseFee=0 queued=9824 alloc=4.1GB sys=9.3GB
Nov 24 17:19:25 anon-r2 erigon[3189645]: [INFO] [11-24|17:19:25.378] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=244849
Nov 24 17:20:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:20:11.820] P2P                                      app=caplin peers=19
Nov 24 17:21:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:21:11.820] P2P                                      app=caplin peers=31
Nov 24 17:22:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:22:03.389] [p2p] GoodPeers                          eth68=12 eth67=3
Nov 24 17:22:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:22:11.820] P2P                                      app=caplin peers=23
Nov 24 17:22:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:22:22.964] [txpool] stat                            pending=5499 baseFee=0 queued=11006 alloc=6.3GB sys=9.4GB
Nov 24 17:23:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:23:11.820] P2P                                      app=caplin peers=23
Nov 24 17:24:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:24:11.820] P2P                                      app=caplin peers=31
Nov 24 17:25:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:25:03.389] [p2p] GoodPeers                          eth67=3 eth68=14
Nov 24 17:25:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:25:11.820] P2P                                      app=caplin peers=40
Nov 24 17:25:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:25:22.963] [txpool] stat                            pending=6217 baseFee=0 queued=12112 alloc=3.9GB sys=9.4GB
Nov 24 17:26:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:26:11.820] P2P                                      app=caplin peers=32
Nov 24 17:27:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:27:11.820] P2P                                      app=caplin peers=43
Nov 24 17:28:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:28:03.389] [p2p] GoodPeers                          eth68=15 eth67=3
Nov 24 17:28:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:28:11.820] P2P                                      app=caplin peers=32
Nov 24 17:28:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:28:22.966] [txpool] stat                            pending=6910 baseFee=0 queued=13233 alloc=6.8GB sys=9.4GB
Nov 24 17:29:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:29:11.820] P2P                                      app=caplin peers=35
Nov 24 17:30:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:30:11.820] P2P                                      app=caplin peers=37
Nov 24 17:31:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:31:03.389] [p2p] GoodPeers                          eth67=2 eth68=15
Nov 24 17:31:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:31:11.820] P2P                                      app=caplin peers=24
Nov 24 17:31:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:31:22.972] [txpool] stat                            pending=7662 baseFee=0 queued=14647 alloc=4.1GB sys=9.4GB
Nov 24 17:32:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:32:11.820] P2P                                      app=caplin peers=28
Nov 24 17:33:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:33:11.820] P2P                                      app=caplin peers=37
Nov 24 17:34:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:34:03.389] [p2p] GoodPeers                          eth68=13 eth67=3
Nov 24 17:34:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:34:11.820] P2P                                      app=caplin peers=35
Nov 24 17:34:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:34:22.964] [txpool] stat                            pending=8837 baseFee=0 queued=16369 alloc=4.2GB sys=9.4GB
Nov 24 17:35:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:35:11.821] P2P                                      app=caplin peers=38
Nov 24 17:36:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:36:11.820] P2P                                      app=caplin peers=33
Nov 24 17:37:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:37:03.389] [p2p] GoodPeers                          eth68=15 eth67=4
Nov 24 17:37:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:37:11.820] P2P                                      app=caplin peers=28
Nov 24 17:37:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:37:22.973] [txpool] stat                            pending=9246 baseFee=0 queued=17555 alloc=4.9GB sys=9.4GB
Nov 24 17:38:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:38:11.820] P2P                                      app=caplin peers=29
Nov 24 17:39:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:39:11.820] P2P                                      app=caplin peers=30
Nov 24 17:40:03 anon-r2 erigon[3189645]: [INFO] [11-24|17:40:03.390] [p2p] GoodPeers                          eth68=17 eth67=5
Nov 24 17:40:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:40:11.820] P2P                                      app=caplin peers=37
Nov 24 17:40:22 anon-r2 erigon[3189645]: [INFO] [11-24|17:40:22.963] [txpool] stat                            pending=9676 baseFee=0 queued=18657 alloc=5.9GB sys=9.4GB
Nov 24 17:41:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:41:11.820] P2P                                      app=caplin peers=43
Nov 24 17:42:11 anon-r2 erigon[3189645]: [INFO] [11-24|17:42:11.820] P2P                                      app=caplin peers=37

Since changing to --internalcl, my node keeps crashing and restarting from block 18m. Starting to think I need to restart sync?

Nov 26 09:25:26 anon-r2 erigon[3273783]: [INFO] [11-26|09:25:26.729] [p2p] GoodPeers                          eth68=33 eth67=33
Nov 26 09:25:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:25:28.200] P2P                                      app=caplin peers=17
Nov 26 09:25:30 anon-r2 erigon[3273783]: [INFO] [11-26|09:25:30.071] [4/12 Execution] Executed blocks         number=18648087 blk/s=11.6 tx/s=1625.9 Mgas/s=174.3 gasState=0.05 batch=49.1MB alloc=7.7GB sys=16.8GB
Nov 26 09:25:50 anon-r2 erigon[3273783]: [INFO] [11-26|09:25:50.143] [4/12 Execution] Executed blocks         number=18648311 blk/s=11.2 tx/s=1512.6 Mgas/s=169.4 gasState=0.06 batch=55.2MB alloc=7.4GB sys=16.8GB
Nov 26 09:26:10 anon-r2 erigon[3273783]: [INFO] [11-26|09:26:10.056] [4/12 Execution] Executed blocks         number=18648535 blk/s=11.2 tx/s=1506.2 Mgas/s=170.1 gasState=0.06 batch=61.7MB alloc=6.0GB sys=16.8GB
Nov 26 09:26:25 anon-r2 erigon[3273783]: [INFO] [11-26|09:26:25.651] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=6.9GB sys=16.8GB
Nov 26 09:26:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:26:28.200] P2P                                      app=caplin peers=17
Nov 26 09:26:52 anon-r2 erigon[3273783]: [INFO] [11-26|09:26:52.491] [] ETL [2/2] Loading                     into=PlainState current_prefix=9db9e0e5
Nov 26 09:27:05 anon-r2 erigon[3273783]: [INFO] [11-26|09:27:05.557] Write to db                              progress=0.6M/1.0M current table=Code
Nov 26 09:27:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:27:06.168] [4/12 Execution] Completed on            block=18648650
Nov 26 09:27:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:27:06.177] [4/12 Execution] DONE                    in=19h22m56.145538115s
Nov 26 09:27:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:27:06.179] [5/12 HashState] Promoting plain state   from=17999999 to=18648650
Nov 26 09:27:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:27:06.183] [5/12 HashState] Incremental promotion   from=17999999 to=18648650 codes=true csbucket=AccountChangeSet
Nov 26 09:27:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:27:28.201] P2P                                      app=caplin peers=21
Nov 26 09:27:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:27:36.184] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000112b039
Nov 26 09:28:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:28:06.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000112b93f
Nov 26 09:28:26 anon-r2 erigon[3273783]: [INFO] [11-26|09:28:26.730] [p2p] GoodPeers                          eth67=33 eth68=33
Nov 26 09:28:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:28:28.201] P2P                                      app=caplin peers=23
Nov 26 09:28:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:28:36.187] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000112c847
Nov 26 09:29:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:29:06.189] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000112d94d
Nov 26 09:29:25 anon-r2 erigon[3273783]: [INFO] [11-26|09:29:25.731] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=8.2GB sys=16.8GB
Nov 26 09:29:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:29:28.200] P2P                                      app=caplin peers=26
Nov 26 09:29:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:29:36.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000112ef38
Nov 26 09:30:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:30:06.184] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=0000000001130487
Nov 26 09:30:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:30:28.200] P2P                                      app=caplin peers=29
Nov 26 09:30:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:30:36.189] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=0000000001131828
Nov 26 09:31:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:31:06.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=0000000001132d08
Nov 26 09:31:26 anon-r2 erigon[3273783]: [INFO] [11-26|09:31:26.729] [p2p] GoodPeers                          eth68=33 eth67=33
Nov 26 09:31:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:31:28.200] P2P                                      app=caplin peers=26
Nov 26 09:31:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:31:36.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=00000000011341db
Nov 26 09:32:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:32:06.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000113578b
Nov 26 09:32:25 anon-r2 erigon[3273783]: [INFO] [11-26|09:32:25.689] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.3GB sys=16.8GB
Nov 26 09:32:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:32:28.200] P2P                                      app=caplin peers=27
Nov 26 09:32:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:32:36.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=0000000001136818
Nov 26 09:33:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:33:06.184] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=0000000001137851
Nov 26 09:33:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:33:28.200] P2P                                      app=caplin peers=25
Nov 26 09:33:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:33:36.184] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=0000000001138838
Nov 26 09:34:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:34:06.184] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000113a046
Nov 26 09:34:26 anon-r2 erigon[3273783]: [INFO] [11-26|09:34:26.729] [p2p] GoodPeers                          eth68=33 eth67=33
Nov 26 09:34:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:34:28.200] P2P                                      app=caplin peers=28
Nov 26 09:34:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:34:36.194] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000113b83a
Nov 26 09:35:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:35:06.188] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000113d33a
Nov 26 09:35:25 anon-r2 erigon[3273783]: [INFO] [11-26|09:35:25.653] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=6.2GB sys=16.8GB
Nov 26 09:35:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:35:28.200] P2P                                      app=caplin peers=27
Nov 26 09:35:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:35:36.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000113ebfb
Nov 26 09:36:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:36:06.184] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=00000000011406ee
Nov 26 09:36:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:36:28.200] P2P                                      app=caplin peers=27
Nov 26 09:36:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:36:36.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=00000000011423cd
Nov 26 09:37:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:37:06.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=0000000001143cfb
Nov 26 09:37:26 anon-r2 erigon[3273783]: [INFO] [11-26|09:37:26.729] [p2p] GoodPeers                          eth68=33 eth67=33
Nov 26 09:37:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:37:28.200] P2P                                      app=caplin peers=26
Nov 26 09:37:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:37:36.184] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000114591d
Nov 26 09:38:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:38:06.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=00000000011473eb
Nov 26 09:38:25 anon-r2 erigon[3273783]: [INFO] [11-26|09:38:25.651] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.8GB sys=16.8GB
Nov 26 09:38:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:38:28.200] P2P                                      app=caplin peers=30
Nov 26 09:38:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:38:36.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=00000000011490d5
Nov 26 09:39:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:39:06.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=0000000001149e57
Nov 26 09:39:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:39:28.200] P2P                                      app=caplin peers=27
Nov 26 09:39:36 anon-r2 erigon[3273783]: [INFO] [11-26|09:39:36.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000114c2f7
Nov 26 09:40:06 anon-r2 erigon[3273783]: [INFO] [11-26|09:40:06.185] [5/12 HashState] ETL [1/2] Extracting    from=AccountChangeSet current_prefix=000000000114e36a
Nov 26 09:40:26 anon-r2 erigon[3273783]: [INFO] [11-26|09:40:26.729] [p2p] GoodPeers                          eth68=33 eth67=33
Nov 26 09:40:28 anon-r2 erigon[3273783]: [INFO] [11-26|09:40:28.201] P2P                                      app=caplin peers=27
Nov 26 09:40:38 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Nov 26 09:40:38 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.
Nov 26 09:40:38 anon-r2 systemd[1]: erigon.service: Consumed 4d 19h 22min 45.425s CPU time.
Nov 26 09:41:23 anon-r2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 5.
Nov 26 09:41:23 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Nov 26 09:41:23 anon-r2 systemd[1]: erigon.service: Consumed 4d 19h 22min 45.425s CPU time.
Nov 26 09:41:23 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Nov 26 09:41:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:24.561] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Nov 26 09:41:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:24.562] Build info                               git_branch=HEAD git_tag=v2.53.4 git_commit=c4843268b788c0899cdad12d220fea4396afbb56
Nov 26 09:41:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:24.563] Starting Erigon on Ethereum mainnet...
Nov 26 09:41:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:24.568] Maximum peer count                       ETH=100 total=100
Nov 26 09:41:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:24.568] starting HTTP APIs                       APIs=web3,net,eth
Nov 26 09:41:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:24.569] torrent verbosity                        level=WRN
Nov 26 09:41:26 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:26.675] Set global gas cap                       cap=50000000
Nov 26 09:41:26 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:26.785] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=150mb upload.rate=15mb
Nov 26 09:41:26 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:26.795] Opening Database                         label=chaindata path=/storage/ethereum/erigon/data/chaindata
Nov 26 09:41:26 anon-r2 erigon[3367921]: mdbx_setup_dxb:15946 filesize mismatch (expect 1852825403392b/226174976p, have 2013853122560b/245831680p)
Nov 26 09:41:36 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:36.960] [db] chaindata                           sizeLimit=3TB pageSize=8192
Nov 26 09:41:36 anon-r2 erigon[3367921]: [INFO] [11-26|09:41:36.965] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: <nil>, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
Nov 26 09:42:08 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:08.599] Initialising Ethereum protocol           network=1
Nov 26 09:42:08 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:08.599] Disk storage enabled for ethash DAGs     dir=/storage/ethereum/erigon/data/ethash-dags count=2
Nov 26 09:42:08 anon-r2 erigon[3367921]: 17999999
Nov 26 09:42:30 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:30.226] Starting private RPC server              on=192.168.2.3:9191
Nov 26 09:42:30 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:30.226] new subscription to logs established
Nov 26 09:42:30 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:30.227] [Checkpoint Sync] Requesting beacon state uri=https://sync.invis.tools/eth/v2/debug/beacon/states/finalized
Nov 26 09:42:31 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:31.349] [txpool] Started
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.328] [Sentinel] Sentinel started              enr=enr:-Ly4QPIO6DmULjNChDYkqOQWvcCpzA1q8k0bRKnF_H2E5RofUv-xIJstPMiyP8jaPqF4NotqYgBFgTzAo6TprT064HgBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC7pNqWBAAAAP__________gmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQPSyyr9TXxO3Ivur9IfwGXnQMU-qCb4qBkFWCa1aHCXWIhzeW5jbmV0cwCDdGNwgg-hg3VkcIIPoA
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.328] Static peers                             len=0
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.328] rpc filters: subscribing to Erigon events
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.329] New txs subscriber joined
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.329] new subscription to newHeaders established
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.330] Reading JWT secret                       path=/var/ethereum/jwtsecret
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.332] HTTP endpoint opened                     url=192.168.2.3:8545 ws=true ws.compression=false grpc=false
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.332] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=false
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.348] [1/12 Snapshots] Fetching torrent files metadata
Nov 26 09:42:49 anon-r2 erigon[3367921]: [WARN] [11-26|09:42:49.378] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.381] Started P2P networking                   version=68 self=enode://268f193a0b50372c50341b582d4efecf96c68890c7ff802d83508f8071719d3ed641570c3e822b8ce4fbfc6972ddbc77cba79c38f5de1b64c9297f13cd7a6888@127.0.0.1:30303 name=erigon/v2.53.4-c4843268/linux-amd64/go1.20.10
Nov 26 09:42:49 anon-r2 erigon[3367921]: [WARN] [11-26|09:42:49.387] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Nov 26 09:42:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:49.389] Started P2P networking                   version=67 self=enode://268f193a0b50372c50341b582d4efecf96c68890c7ff802d83508f8071719d3ed641570c3e822b8ce4fbfc6972ddbc77cba79c38f5de1b64c9297f13cd7a6888@127.0.0.1:30304 name=erigon/v2.53.4-c4843268/linux-amd64/go1.20.10
Nov 26 09:42:52 anon-r2 erigon[3367921]: [INFO] [11-26|09:42:52.253] [snapshots] Blocks Stat                  blocks=18000k indices=18000k alloc=2.7GB sys=5.3GB
Nov 26 09:43:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:43:24.149] Started Ethereum 2.0 Gossip Service      app=caplin
Nov 26 09:43:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:43:24.151] [Caplin] starting clstages loop          app=caplin
Nov 26 09:43:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:43:24.151] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=7847200
Nov 26 09:43:54 anon-r2 erigon[3367921]: [INFO] [11-26|09:43:54.151] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7846752 blockNumber=18654473 blk/sec=14.9 mbps/sec=4.5367 peers=12
Nov 26 09:44:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:44:24.150] P2P                                      app=caplin peers=14
Nov 26 09:44:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:44:24.151] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7846112 blockNumber=18653835 blk/sec=21.3 mbps/sec=0.0000 peers=14
Nov 26 09:44:54 anon-r2 erigon[3367921]: [INFO] [11-26|09:44:54.152] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7845665 blockNumber=18653391 blk/sec=14.9 mbps/sec=0.0000 peers=11
Nov 26 09:45:08 anon-r2 erigon[3367921]: [INFO] [11-26|09:45:08.588] [p2p] GoodPeers                          eth68=1 eth67=2
Nov 26 09:45:21 anon-r2 erigon[3367921]: [INFO] [11-26|09:45:21.151] [HistoryDownload] Flushed buffer file    name=erigon-sortable-buf-1202932371
Nov 26 09:45:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:45:24.150] P2P                                      app=caplin peers=6
Nov 26 09:45:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:45:24.152] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7844961 blockNumber=18652694 blk/sec=23.5 mbps/sec=3.7757 peers=6
Nov 26 09:45:30 anon-r2 erigon[3367921]: [INFO] [11-26|09:45:30.228] [txpool] stat                            pending=328 baseFee=0 queued=1648 alloc=4.6GB sys=7.6GB
Nov 26 09:45:54 anon-r2 erigon[3367921]: [INFO] [11-26|09:45:54.151] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7844385 blockNumber=18652128 blk/sec=19.2 mbps/sec=5.1071 peers=5
Nov 26 09:46:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:46:24.150] P2P                                      app=caplin peers=6
Nov 26 09:46:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:46:24.151] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7844032 blockNumber=18651777 blk/sec=11.8 mbps/sec=1.6101 peers=6
Nov 26 09:46:54 anon-r2 erigon[3367921]: [INFO] [11-26|09:46:54.151] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7843809 blockNumber=18651557 blk/sec=7.4 mbps/sec=3.1437 peers=7
Nov 26 09:47:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:47:24.150] P2P                                      app=caplin peers=8
Nov 26 09:47:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:47:24.152] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7843297 blockNumber=18651049 blk/sec=17.1 mbps/sec=4.5432 peers=8
Nov 26 09:47:49 anon-r2 erigon[3367921]: [INFO] [11-26|09:47:49.430] [HistoryDownload] Flushed buffer file    name=erigon-sortable-buf-2695007199
Nov 26 09:47:54 anon-r2 erigon[3367921]: [INFO] [11-26|09:47:54.151] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7842657 blockNumber=18650415 blk/sec=21.3 mbps/sec=0.0000 peers=9
Nov 26 09:48:08 anon-r2 erigon[3367921]: [INFO] [11-26|09:48:08.588] [p2p] GoodPeers                          eth68=2 eth67=3
Nov 26 09:48:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:48:24.150] P2P                                      app=caplin peers=6
Nov 26 09:48:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:48:24.152] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7842209 blockNumber=18649970 blk/sec=14.9 mbps/sec=4.1731 peers=6
Nov 26 09:48:30 anon-r2 erigon[3367921]: [INFO] [11-26|09:48:30.231] [txpool] stat                            pending=548 baseFee=0 queued=2553 alloc=6.5GB sys=8.6GB
Nov 26 09:48:54 anon-r2 erigon[3367921]: [INFO] [11-26|09:48:54.152] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7841761 blockNumber=18649526 blk/sec=14.9 mbps/sec=0.0000 peers=6
Nov 26 09:49:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:49:24.150] P2P                                      app=caplin peers=3
Nov 26 09:49:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:49:24.152] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7841441 blockNumber=18649209 blk/sec=10.7 mbps/sec=0.0000 peers=3
Nov 26 09:49:54 anon-r2 erigon[3367921]: [INFO] [11-26|09:49:54.152] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7841185 blockNumber=18648957 blk/sec=8.5 mbps/sec=2.6274 peers=9
Nov 26 09:50:17 anon-r2 erigon[3367921]: [INFO] [11-26|09:50:17.698] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
Nov 26 09:50:17 anon-r2 erigon[3367921]: [INFO] [11-26|09:50:17.700] [HistoryDownload] Flushed buffer file    name=erigon-sortable-buf-1386947735
Nov 26 09:50:17 anon-r2 erigon[3367921]: [INFO] [11-26|09:50:17.825] [HistoryDownload] Flushed buffer file    name=erigon-sortable-buf-2558945645
Nov 26 09:50:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:50:24.150] P2P                                      app=caplin peers=5
Nov 26 09:50:33 anon-r2 erigon[3367921]: [INFO] [11-26|09:50:33.094] [Caplin] Downloading epochs from reqresp app=caplin stage=CatchUpEpochs from=245225 to=245228
Nov 26 09:50:35 anon-r2 erigon[3367921]: [INFO] [11-26|09:50:35.737] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=245225
Nov 26 09:50:43 anon-r2 erigon[3367921]: [INFO] [11-26|09:50:43.260] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=245226
Nov 26 09:50:51 anon-r2 erigon[3367921]: [INFO] [11-26|09:50:51.593] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=245227
Nov 26 09:51:00 anon-r2 erigon[3367921]: [INFO] [11-26|09:51:00.459] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=245228
Nov 26 09:51:08 anon-r2 erigon[3367921]: [INFO] [11-26|09:51:08.588] [p2p] GoodPeers                          eth68=3 eth67=3
Nov 26 09:51:11 anon-r2 erigon[3367921]: [INFO] [11-26|09:51:11.528] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=245229
Nov 26 09:51:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:51:24.152] P2P                                      app=caplin peers=7
Nov 26 09:51:30 anon-r2 erigon[3367921]: [INFO] [11-26|09:51:30.229] [txpool] stat                            pending=866 baseFee=0 queued=3467 alloc=6.3GB sys=10.0GB
Nov 26 09:52:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:52:24.150] P2P                                      app=caplin peers=15
Nov 26 09:53:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:53:24.150] P2P                                      app=caplin peers=20
Nov 26 09:54:08 anon-r2 erigon[3367921]: [INFO] [11-26|09:54:08.588] [p2p] GoodPeers                          eth66=3 eth68=5 eth67=4
Nov 26 09:54:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:54:24.150] P2P                                      app=caplin peers=29
Nov 26 09:54:30 anon-r2 erigon[3367921]: [INFO] [11-26|09:54:30.234] [txpool] stat                            pending=5859 baseFee=0 queued=7281 alloc=5.8GB sys=10.0GB
Nov 26 09:55:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:55:24.151] P2P                                      app=caplin peers=26
Nov 26 09:56:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:56:24.150] P2P                                      app=caplin peers=28
Nov 26 09:57:08 anon-r2 erigon[3367921]: [INFO] [11-26|09:57:08.588] [p2p] GoodPeers                          eth68=6 eth66=1 eth67=4
Nov 26 09:57:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:57:24.150] P2P                                      app=caplin peers=25
Nov 26 09:57:30 anon-r2 erigon[3367921]: [INFO] [11-26|09:57:30.229] [txpool] stat                            pending=6518 baseFee=0 queued=9859 alloc=6.6GB sys=10.0GB
Nov 26 09:58:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:58:24.150] P2P                                      app=caplin peers=19
Nov 26 09:59:24 anon-r2 erigon[3367921]: [INFO] [11-26|09:59:24.151] P2P                                      app=caplin peers=25
Nov 26 10:00:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:00:08.589] [p2p] GoodPeers                          eth68=8 eth67=5
Nov 26 10:00:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:00:24.150] P2P                                      app=caplin peers=26
Nov 26 10:00:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:00:30.235] [txpool] stat                            pending=6855 baseFee=0 queued=10788 alloc=4.0GB sys=10.0GB
Nov 26 10:01:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:01:24.150] P2P                                      app=caplin peers=29
Nov 26 10:02:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:02:24.150] P2P                                      app=caplin peers=28
Nov 26 10:03:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:03:08.589] [p2p] GoodPeers                          eth68=9 eth67=6
Nov 26 10:03:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:03:24.150] P2P                                      app=caplin peers=26
Nov 26 10:03:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:03:30.235] [txpool] stat                            pending=7436 baseFee=0 queued=11758 alloc=6.5GB sys=10.0GB
Nov 26 10:04:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:04:24.150] P2P                                      app=caplin peers=27
Nov 26 10:05:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:05:24.150] P2P                                      app=caplin peers=26
Nov 26 10:06:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:06:08.589] [p2p] GoodPeers                          eth67=6 eth68=9
Nov 26 10:06:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:06:24.150] P2P                                      app=caplin peers=25
Nov 26 10:06:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:06:30.229] [txpool] stat                            pending=7870 baseFee=0 queued=12579 alloc=7.0GB sys=10.0GB
Nov 26 10:07:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:07:24.150] P2P                                      app=caplin peers=21
Nov 26 10:08:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:08:24.151] P2P                                      app=caplin peers=27
Nov 26 10:09:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:09:08.588] [p2p] GoodPeers                          eth68=13 eth67=7
Nov 26 10:09:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:09:24.150] P2P                                      app=caplin peers=26
Nov 26 10:09:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:09:30.231] [txpool] stat                            pending=9606 baseFee=0 queued=15787 alloc=4.2GB sys=10.0GB
Nov 26 10:10:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:10:24.150] P2P                                      app=caplin peers=30
Nov 26 10:11:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:11:24.150] P2P                                      app=caplin peers=30
Nov 26 10:12:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:12:08.589] [p2p] GoodPeers                          eth68=15 eth67=7
Nov 26 10:12:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:12:24.150] P2P                                      app=caplin peers=28
Nov 26 10:12:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:12:30.229] [txpool] stat                            pending=10000 baseFee=0 queued=16738 alloc=4.9GB sys=10.0GB
Nov 26 10:13:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:13:24.150] P2P                                      app=caplin peers=27
Nov 26 10:14:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:14:24.151] P2P                                      app=caplin peers=38
Nov 26 10:15:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:15:08.589] [p2p] GoodPeers                          eth68=15 eth67=6
Nov 26 10:15:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:15:24.151] P2P                                      app=caplin peers=33
Nov 26 10:15:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:15:30.230] [txpool] stat                            pending=10000 baseFee=0 queued=17608 alloc=5.7GB sys=10.0GB
Nov 26 10:16:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:16:24.150] P2P                                      app=caplin peers=36
Nov 26 10:17:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:17:24.150] P2P                                      app=caplin peers=28
Nov 26 10:18:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:18:08.588] [p2p] GoodPeers                          eth68=17 eth67=9
Nov 26 10:18:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:18:24.150] P2P                                      app=caplin peers=30
Nov 26 10:18:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:18:30.229] [txpool] stat                            pending=10000 baseFee=0 queued=18563 alloc=6.7GB sys=10.0GB
Nov 26 10:19:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:19:24.150] P2P                                      app=caplin peers=30
Nov 26 10:20:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:20:24.151] P2P                                      app=caplin peers=25
Nov 26 10:21:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:21:08.589] [p2p] GoodPeers                          eth68=17 eth67=9
Nov 26 10:21:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:21:24.150] P2P                                      app=caplin peers=32
Nov 26 10:21:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:21:30.229] [txpool] stat                            pending=10000 baseFee=0 queued=19376 alloc=4.0GB sys=10.0GB
Nov 26 10:22:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:22:24.150] P2P                                      app=caplin peers=31
Nov 26 10:23:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:23:24.150] P2P                                      app=caplin peers=35
Nov 26 10:24:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:24:08.589] [p2p] GoodPeers                          eth67=9 eth68=16
Nov 26 10:24:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:24:24.151] P2P                                      app=caplin peers=28
Nov 26 10:24:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:24:30.240] [txpool] stat                            pending=10000 baseFee=0 queued=20168 alloc=4.9GB sys=10.0GB
Nov 26 10:25:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:25:24.150] P2P                                      app=caplin peers=21
Nov 26 10:26:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:26:24.150] P2P                                      app=caplin peers=23
Nov 26 10:27:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:27:08.588] [p2p] GoodPeers                          eth68=17 eth66=2 eth67=10
Nov 26 10:27:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:27:24.150] P2P                                      app=caplin peers=23
Nov 26 10:27:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:27:30.240] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=6.3GB sys=10.0GB
Nov 26 10:28:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:28:24.150] P2P                                      app=caplin peers=23
Nov 26 10:29:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:29:24.151] P2P                                      app=caplin peers=29
Nov 26 10:30:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:30:08.589] [p2p] GoodPeers                          eth68=15 eth66=1 eth67=9
Nov 26 10:30:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:30:24.150] P2P                                      app=caplin peers=28
Nov 26 10:30:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:30:30.230] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=7.3GB sys=10.0GB
Nov 26 10:31:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:31:24.150] P2P                                      app=caplin peers=27
Nov 26 10:32:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:32:24.150] P2P                                      app=caplin peers=30
Nov 26 10:33:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:33:08.589] [p2p] GoodPeers                          eth68=16 eth67=9 eth66=2
Nov 26 10:33:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:33:24.150] P2P                                      app=caplin peers=23
Nov 26 10:33:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:33:30.246] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=4.6GB sys=10.0GB
Nov 26 10:34:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:34:24.150] P2P                                      app=caplin peers=22
Nov 26 10:35:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:35:24.150] P2P                                      app=caplin peers=23
Nov 26 10:36:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:36:08.588] [p2p] GoodPeers                          eth68=17 eth66=2 eth67=10
Nov 26 10:36:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:36:24.150] P2P                                      app=caplin peers=30
Nov 26 10:36:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:36:30.230] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=6.0GB sys=10.0GB
Nov 26 10:37:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:37:24.150] P2P                                      app=caplin peers=26
Nov 26 10:38:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:38:24.151] P2P                                      app=caplin peers=43
Nov 26 10:39:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:39:08.589] [p2p] GoodPeers                          eth66=2 eth68=16 eth67=10
Nov 26 10:39:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:39:24.150] P2P                                      app=caplin peers=34
Nov 26 10:39:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:39:30.230] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=7.0GB sys=10.0GB
Nov 26 10:40:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:40:24.151] P2P                                      app=caplin peers=30
Nov 26 10:41:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:41:24.150] P2P                                      app=caplin peers=24
Nov 26 10:42:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:42:08.589] [p2p] GoodPeers                          eth66=1 eth68=16 eth67=10
Nov 26 10:42:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:42:24.150] P2P                                      app=caplin peers=25
Nov 26 10:42:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:42:30.230] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=4.5GB sys=10.0GB
Nov 26 10:43:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:43:24.150] P2P                                      app=caplin peers=27
Nov 26 10:44:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:44:24.150] P2P                                      app=caplin peers=28
Nov 26 10:45:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:45:08.589] [p2p] GoodPeers                          eth68=17 eth67=11 eth66=2
Nov 26 10:45:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:45:24.151] P2P                                      app=caplin peers=28
Nov 26 10:45:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:45:30.229] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.7GB sys=10.0GB
Nov 26 10:46:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:46:24.150] P2P                                      app=caplin peers=24
Nov 26 10:47:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:47:00.782] [3/12 Senders] Started                   from=17999999 to=18655068
Nov 26 10:47:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:47:24.150] P2P                                      app=caplin peers=24
Nov 26 10:47:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:47:30.784] [3/12 Senders] Recovery                  block_number=18020265 ch=8707/10000
Nov 26 10:48:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:48:00.789] [3/12 Senders] Recovery                  block_number=18039996 ch=9992/10000
Nov 26 10:48:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:48:08.589] [p2p] GoodPeers                          eth68=17 eth66=1 eth67=12
Nov 26 10:48:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:48:24.640] P2P                                      app=caplin peers=25
Nov 26 10:48:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:48:30.243] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=8.0GB sys=12.8GB
Nov 26 10:48:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:48:30.790] [3/12 Senders] Recovery                  block_number=18058982 ch=10000/10000
Nov 26 10:49:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:49:00.785] [3/12 Senders] Recovery                  block_number=18079338 ch=9999/10000
Nov 26 10:49:20 anon-r2 erigon[3367921]: [INFO] [11-26|10:49:20.875] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-3239175528
Nov 26 10:49:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:49:24.150] P2P                                      app=caplin peers=31
Nov 26 10:49:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:49:30.783] [3/12 Senders] Recovery                  block_number=18100740 ch=9979/10000
Nov 26 10:50:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:50:00.784] [3/12 Senders] Recovery                  block_number=18121890 ch=9999/10000
Nov 26 10:50:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:50:24.150] P2P                                      app=caplin peers=26
Nov 26 10:50:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:50:30.788] [3/12 Senders] Recovery                  block_number=18137832 ch=10000/10000
Nov 26 10:51:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:51:00.783] [3/12 Senders] Recovery                  block_number=18159587 ch=10000/10000
Nov 26 10:51:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:51:08.589] [p2p] GoodPeers                          eth68=17 eth67=12 eth66=1
Nov 26 10:51:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:51:24.150] P2P                                      app=caplin peers=30
Nov 26 10:51:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:51:30.263] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=12.6GB sys=14.0GB
Nov 26 10:51:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:51:30.783] [3/12 Senders] Recovery                  block_number=18180402 ch=9996/10000
Nov 26 10:51:35 anon-r2 erigon[3367921]: [INFO] [11-26|10:51:35.524] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-3971228309
Nov 26 10:52:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:52:00.784] [3/12 Senders] Recovery                  block_number=18201646 ch=9996/10000
Nov 26 10:52:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:52:24.150] P2P                                      app=caplin peers=26
Nov 26 10:52:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:52:30.784] [3/12 Senders] Recovery                  block_number=18223208 ch=4064/10000
Nov 26 10:53:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:53:00.784] [3/12 Senders] Recovery                  block_number=18232710 ch=0/10000
Nov 26 10:53:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:53:24.150] P2P                                      app=caplin peers=27
Nov 26 10:53:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:53:30.783] [3/12 Senders] Recovery                  block_number=18237781 ch=0/10000
Nov 26 10:54:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:54:00.783] [3/12 Senders] Recovery                  block_number=18243296 ch=0/10000
Nov 26 10:54:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:54:08.588] [p2p] GoodPeers                          eth68=19 eth67=13 eth66=1
Nov 26 10:54:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:54:24.150] P2P                                      app=caplin peers=28
Nov 26 10:54:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:54:30.245] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.6GB sys=14.0GB
Nov 26 10:54:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:54:30.783] [3/12 Senders] Recovery                  block_number=18248952 ch=0/10000
Nov 26 10:55:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:55:00.783] [3/12 Senders] Recovery                  block_number=18254546 ch=0/10000
Nov 26 10:55:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:55:24.150] P2P                                      app=caplin peers=26
Nov 26 10:55:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:55:30.784] [3/12 Senders] Recovery                  block_number=18259789 ch=0/10000
Nov 26 10:56:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:56:00.784] [3/12 Senders] Recovery                  block_number=18265325 ch=0/10000
Nov 26 10:56:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:56:24.150] P2P                                      app=caplin peers=27
Nov 26 10:56:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:56:30.784] [3/12 Senders] Recovery                  block_number=18270966 ch=0/10000
Nov 26 10:57:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:57:00.783] [3/12 Senders] Recovery                  block_number=18276537 ch=0/10000
Nov 26 10:57:07 anon-r2 erigon[3367921]: [INFO] [11-26|10:57:07.362] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-1596621054
Nov 26 10:57:08 anon-r2 erigon[3367921]: [INFO] [11-26|10:57:08.589] [p2p] GoodPeers                          eth68=19 eth67=13 eth66=3
Nov 26 10:57:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:57:24.150] P2P                                      app=caplin peers=32
Nov 26 10:57:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:57:30.251] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.0GB sys=14.0GB
Nov 26 10:57:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:57:30.784] [3/12 Senders] Recovery                  block_number=18281745 ch=0/10000
Nov 26 10:58:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:58:00.784] [3/12 Senders] Recovery                  block_number=18286861 ch=0/10000
Nov 26 10:58:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:58:24.150] P2P                                      app=caplin peers=29
Nov 26 10:58:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:58:30.783] [3/12 Senders] Recovery                  block_number=18292555 ch=0/10000
Nov 26 10:59:00 anon-r2 erigon[3367921]: [INFO] [11-26|10:59:00.784] [3/12 Senders] Recovery                  block_number=18297846 ch=0/10000
Nov 26 10:59:24 anon-r2 erigon[3367921]: [INFO] [11-26|10:59:24.150] P2P                                      app=caplin peers=29
Nov 26 10:59:30 anon-r2 erigon[3367921]: [INFO] [11-26|10:59:30.784] [3/12 Senders] Recovery                  block_number=18303097 ch=0/10000
Nov 26 11:00:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:00:00.783] [3/12 Senders] Recovery                  block_number=18307953 ch=0/10000
Nov 26 11:00:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:00:08.589] [p2p] GoodPeers                          eth68=19 eth67=14 eth66=3
Nov 26 11:00:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:00:24.150] P2P                                      app=caplin peers=32
Nov 26 11:00:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:00:30.228] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.2GB sys=14.0GB
Nov 26 11:00:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:00:30.784] [3/12 Senders] Recovery                  block_number=18313252 ch=0/10000
Nov 26 11:01:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:01:00.784] [3/12 Senders] Recovery                  block_number=18318505 ch=0/10000
Nov 26 11:01:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:01:24.151] P2P                                      app=caplin peers=33
Nov 26 11:01:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:01:30.784] [3/12 Senders] Recovery                  block_number=18323334 ch=0/10000
Nov 26 11:02:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:02:00.783] [3/12 Senders] Recovery                  block_number=18328338 ch=0/10000
Nov 26 11:02:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:02:24.150] P2P                                      app=caplin peers=39
Nov 26 11:02:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:02:30.783] [3/12 Senders] Recovery                  block_number=18333655 ch=0/10000
Nov 26 11:03:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:03:00.784] [3/12 Senders] Recovery                  block_number=18339260 ch=0/10000
Nov 26 11:03:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:03:08.589] [p2p] GoodPeers                          eth68=19 eth67=13 eth66=2
Nov 26 11:03:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:03:24.150] P2P                                      app=caplin peers=34
Nov 26 11:03:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:03:30.228] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.3GB sys=14.0GB
Nov 26 11:03:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:03:30.783] [3/12 Senders] Recovery                  block_number=18344571 ch=0/10000
Nov 26 11:04:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:04:00.784] [3/12 Senders] Recovery                  block_number=18350280 ch=0/10000
Nov 26 11:04:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:04:24.150] P2P                                      app=caplin peers=33
Nov 26 11:04:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:04:30.783] [3/12 Senders] Recovery                  block_number=18356372 ch=0/10000
Nov 26 11:05:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:05:00.784] [3/12 Senders] Recovery                  block_number=18361856 ch=0/10000
Nov 26 11:05:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:05:24.150] P2P                                      app=caplin peers=36
Nov 26 11:05:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:05:30.783] [3/12 Senders] Recovery                  block_number=18367316 ch=0/10000
Nov 26 11:06:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:06:00.784] [3/12 Senders] Recovery                  block_number=18372642 ch=0/10000
Nov 26 11:06:02 anon-r2 erigon[3367921]: [INFO] [11-26|11:06:02.560] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-1199780069
Nov 26 11:06:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:06:08.589] [p2p] GoodPeers                          eth68=18 eth67=13 eth66=1
Nov 26 11:06:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:06:24.150] P2P                                      app=caplin peers=39
Nov 26 11:06:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:06:30.229] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=4.8GB sys=14.0GB
Nov 26 11:06:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:06:30.784] [3/12 Senders] Recovery                  block_number=18377997 ch=0/10000
Nov 26 11:07:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:07:00.784] [3/12 Senders] Recovery                  block_number=18383522 ch=0/10000
Nov 26 11:07:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:07:24.150] P2P                                      app=caplin peers=39
Nov 26 11:07:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:07:30.783] [3/12 Senders] Recovery                  block_number=18389010 ch=0/10000
Nov 26 11:08:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:08:00.784] [3/12 Senders] Recovery                  block_number=18394438 ch=0/10000
Nov 26 11:08:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:08:24.150] P2P                                      app=caplin peers=42
Nov 26 11:08:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:08:30.783] [3/12 Senders] Recovery                  block_number=18399808 ch=0/10000
Nov 26 11:09:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:09:00.783] [3/12 Senders] Recovery                  block_number=18405117 ch=0/10000
Nov 26 11:09:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:09:08.589] [p2p] GoodPeers                          eth68=19 eth67=14 eth66=1
Nov 26 11:09:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:09:24.150] P2P                                      app=caplin peers=32
Nov 26 11:09:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:09:30.242] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.3GB sys=14.0GB
Nov 26 11:09:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:09:30.784] [3/12 Senders] Recovery                  block_number=18410174 ch=0/10000
Nov 26 11:10:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:10:00.784] [3/12 Senders] Recovery                  block_number=18415300 ch=0/10000
Nov 26 11:10:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:10:24.150] P2P                                      app=caplin peers=27
Nov 26 11:10:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:10:30.784] [3/12 Senders] Recovery                  block_number=18420773 ch=0/10000
Nov 26 11:11:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:11:00.784] [3/12 Senders] Recovery                  block_number=18426281 ch=0/10000
Nov 26 11:11:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:11:24.150] P2P                                      app=caplin peers=41
Nov 26 11:11:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:11:30.783] [3/12 Senders] Recovery                  block_number=18431889 ch=0/10000
Nov 26 11:12:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:12:00.784] [3/12 Senders] Recovery                  block_number=18437348 ch=0/10000
Nov 26 11:12:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:12:08.589] [p2p] GoodPeers                          eth68=19 eth67=15 eth66=2
Nov 26 11:12:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:12:24.150] P2P                                      app=caplin peers=40
Nov 26 11:12:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:12:30.229] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.4GB sys=14.0GB
Nov 26 11:12:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:12:30.784] [3/12 Senders] Recovery                  block_number=18442577 ch=0/10000
Nov 26 11:13:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:13:00.784] [3/12 Senders] Recovery                  block_number=18448030 ch=0/10000
Nov 26 11:13:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:13:24.150] P2P                                      app=caplin peers=35
Nov 26 11:13:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:13:30.783] [3/12 Senders] Recovery                  block_number=18453149 ch=0/10000
Nov 26 11:14:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:14:00.783] [3/12 Senders] Recovery                  block_number=18457823 ch=0/10000
Nov 26 11:14:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:14:24.150] P2P                                      app=caplin peers=31
Nov 26 11:14:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:14:30.783] [3/12 Senders] Recovery                  block_number=18462814 ch=0/10000
Nov 26 11:14:31 anon-r2 erigon[3367921]: [INFO] [11-26|11:14:31.564] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-3294642768
Nov 26 11:15:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:15:00.783] [3/12 Senders] Recovery                  block_number=18468358 ch=0/10000
Nov 26 11:15:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:15:08.588] [p2p] GoodPeers                          eth68=21 eth67=15 eth66=1
Nov 26 11:15:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:15:24.150] P2P                                      app=caplin peers=37
Nov 26 11:15:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:15:30.248] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=4.2GB sys=14.0GB
Nov 26 11:15:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:15:30.783] [3/12 Senders] Recovery                  block_number=18473624 ch=0/10000
Nov 26 11:16:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:16:00.783] [3/12 Senders] Recovery                  block_number=18478852 ch=0/10000
Nov 26 11:16:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:16:24.150] P2P                                      app=caplin peers=26
Nov 26 11:16:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:16:30.783] [3/12 Senders] Recovery                  block_number=18484549 ch=0/10000
Nov 26 11:17:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:17:00.783] [3/12 Senders] Recovery                  block_number=18490090 ch=0/10000
Nov 26 11:17:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:17:24.150] P2P                                      app=caplin peers=27
Nov 26 11:17:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:17:30.784] [3/12 Senders] Recovery                  block_number=18495394 ch=0/10000
Nov 26 11:18:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:18:00.784] [3/12 Senders] Recovery                  block_number=18500094 ch=0/10000
Nov 26 11:18:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:18:08.588] [p2p] GoodPeers                          eth68=22 eth66=2 eth67=16
Nov 26 11:18:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:18:24.150] P2P                                      app=caplin peers=27
Nov 26 11:18:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:18:30.228] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.4GB sys=14.0GB
Nov 26 11:18:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:18:30.783] [3/12 Senders] Recovery                  block_number=18506583 ch=0/10000
Nov 26 11:19:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:19:00.783] [3/12 Senders] Recovery                  block_number=18513337 ch=0/10000
Nov 26 11:19:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:19:24.150] P2P                                      app=caplin peers=26
Nov 26 11:19:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:19:30.784] [3/12 Senders] Recovery                  block_number=18520535 ch=0/10000
Nov 26 11:20:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:20:00.784] [3/12 Senders] Recovery                  block_number=18527224 ch=0/10000
Nov 26 11:20:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:20:24.150] P2P                                      app=caplin peers=28
Nov 26 11:20:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:20:30.784] [3/12 Senders] Recovery                  block_number=18534269 ch=0/10000
Nov 26 11:21:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:21:00.784] [3/12 Senders] Recovery                  block_number=18541508 ch=0/10000
Nov 26 11:21:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:21:08.588] [p2p] GoodPeers                          eth68=24 eth67=16 eth66=1
Nov 26 11:21:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:21:24.150] P2P                                      app=caplin peers=25
Nov 26 11:21:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:21:30.248] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=7.5GB sys=14.0GB
Nov 26 11:21:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:21:30.784] [3/12 Senders] Recovery                  block_number=18548943 ch=0/10000
Nov 26 11:21:33 anon-r2 erigon[3367921]: [INFO] [11-26|11:21:33.325] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-2467882026
Nov 26 11:22:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:22:00.783] [3/12 Senders] Recovery                  block_number=18555393 ch=0/10000
Nov 26 11:22:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:22:24.150] P2P                                      app=caplin peers=28
Nov 26 11:22:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:22:30.784] [3/12 Senders] Recovery                  block_number=18562125 ch=0/10000
Nov 26 11:23:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:23:00.784] [3/12 Senders] Recovery                  block_number=18569567 ch=0/10000
Nov 26 11:23:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:23:24.150] P2P                                      app=caplin peers=33
Nov 26 11:23:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:23:30.783] [3/12 Senders] Recovery                  block_number=18576371 ch=0/10000
Nov 26 11:24:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:24:00.784] [3/12 Senders] Recovery                  block_number=18583424 ch=0/10000
Nov 26 11:24:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:24:08.589] [p2p] GoodPeers                          eth66=3 eth68=24 eth67=16
Nov 26 11:24:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:24:24.150] P2P                                      app=caplin peers=32
Nov 26 11:24:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:24:30.229] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.9GB sys=14.0GB
Nov 26 11:24:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:24:30.784] [3/12 Senders] Recovery                  block_number=18590574 ch=0/10000
Nov 26 11:25:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:25:00.784] [3/12 Senders] Recovery                  block_number=18597364 ch=0/10000
Nov 26 11:25:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:25:24.150] P2P                                      app=caplin peers=32
Nov 26 11:25:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:25:30.784] [3/12 Senders] Recovery                  block_number=18603511 ch=0/10000
Nov 26 11:26:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:26:00.784] [3/12 Senders] Recovery                  block_number=18609383 ch=0/10000
Nov 26 11:26:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:26:24.150] P2P                                      app=caplin peers=32
Nov 26 11:26:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:26:30.783] [3/12 Senders] Recovery                  block_number=18617463 ch=0/10000
Nov 26 11:27:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:27:00.784] [3/12 Senders] Recovery                  block_number=18624540 ch=0/10000
Nov 26 11:27:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:27:08.589] [p2p] GoodPeers                          eth68=22 eth66=1 eth67=15
Nov 26 11:27:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:27:24.150] P2P                                      app=caplin peers=41
Nov 26 11:27:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:27:30.248] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=5.2GB sys=14.0GB
Nov 26 11:27:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:27:30.784] [3/12 Senders] Recovery                  block_number=18632770 ch=0/10000
Nov 26 11:27:41 anon-r2 erigon[3367921]: [INFO] [11-26|11:27:41.360] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-1719530778
Nov 26 11:28:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:28:00.783] [3/12 Senders] Recovery                  block_number=18639113 ch=0/10000
Nov 26 11:28:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:28:24.150] P2P                                      app=caplin peers=34
Nov 26 11:28:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:28:30.784] [3/12 Senders] Recovery                  block_number=18645209 ch=0/10000
Nov 26 11:29:00 anon-r2 erigon[3367921]: [INFO] [11-26|11:29:00.783] [3/12 Senders] Recovery                  block_number=18650650 ch=0/10000
Nov 26 11:29:17 anon-r2 erigon[3367921]: [INFO] [11-26|11:29:17.242] [3/12 Senders] Flushed buffer file       name=erigon-sortable-buf-3141331202
Nov 26 11:29:23 anon-r2 erigon[3367921]: [INFO] [11-26|11:29:23.126] [3/12 Senders] DONE                      in=42m22.344528043s
Nov 26 11:29:23 anon-r2 erigon[3367921]: [INFO] [11-26|11:29:23.128] [4/12 Execution] Blocks execution        from=17999999 to=18655068
Nov 26 11:29:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:29:24.151] P2P                                      app=caplin peers=38
Nov 26 11:29:43 anon-r2 erigon[3367921]: [INFO] [11-26|11:29:43.322] [4/12 Execution] Executed blocks         number=18000070 blk/s=3.5 tx/s=485.6 Mgas/s=51.9 gasState=0.00 batch=1.8MB alloc=5.3GB sys=14.0GB
Nov 26 11:30:03 anon-r2 erigon[3367921]: [INFO] [11-26|11:30:03.264] [4/12 Execution] Executed blocks         number=18000154 blk/s=4.2 tx/s=587.6 Mgas/s=61.6 gasState=0.00 batch=4.3MB alloc=5.9GB sys=14.0GB
Nov 26 11:30:08 anon-r2 erigon[3367921]: [INFO] [11-26|11:30:08.589] [p2p] GoodPeers                          eth68=22 eth67=16 eth66=1
Nov 26 11:30:23 anon-r2 erigon[3367921]: [INFO] [11-26|11:30:23.140] [4/12 Execution] Executed blocks         number=18000259 blk/s=5.3 tx/s=776.4 Mgas/s=79.9 gasState=0.01 batch=7.6MB alloc=6.6GB sys=14.0GB
Nov 26 11:30:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:30:24.151] P2P                                      app=caplin peers=36
Nov 26 11:30:30 anon-r2 erigon[3367921]: [INFO] [11-26|11:30:30.229] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=6.8GB sys=14.0GB
Nov 26 11:30:43 anon-r2 erigon[3367921]: [INFO] [11-26|11:30:43.336] [4/12 Execution] Executed blocks         number=18000368 blk/s=5.4 tx/s=752.6 Mgas/s=81.1 gasState=0.01 batch=10.9MB alloc=7.3GB sys=14.0GB
Nov 26 11:31:03 anon-r2 erigon[3367921]: [INFO] [11-26|11:31:03.263] [4/12 Execution] Executed blocks         number=18000485 blk/s=5.9 tx/s=823.2 Mgas/s=87.5 gasState=0.01 batch=14.6MB alloc=4.4GB sys=14.0GB
Nov 26 11:31:23 anon-r2 erigon[3367921]: [INFO] [11-26|11:31:23.182] [4/12 Execution] Executed blocks         number=18000612 blk/s=6.4 tx/s=904.9 Mgas/s=97.1 gasState=0.02 batch=18.5MB alloc=5.1GB sys=14.0GB
Nov 26 11:31:24 anon-r2 erigon[3367921]: [INFO] [11-26|11:31:24.150] P2P                                      app=caplin peers=36

I don't see any problem in this logs. probably OOM-killer is not happy at some point.

Thanks @AskAlexSharov

Hopefully a system restart will resolve the sync issues. Can't see anything obvious in system logs; total memory usage of the system, has remained below 40% (max is 64GB)

Same issue here.

This problem is mostly caused by erigon-node errors imo, which are usually fixed after node upgrades.

When mem is high: add flag --pprof go tool pprof -inuse_space -png http://127.0.0.1:6060/debug/pprof/heap > mem.png

Had another crash, trying with OOM killer, disabled.

@caedmo - network? (i saw similar thing on mumbai tonight)

@AskAlexSharov mainnet

Just had another OOM killer event

Nov 29 06:49:20 anon-r2 erigon[92876]: [INFO] [11-29|06:49:20.601] [5/12 HashState] Flushed buffer file     name=erigon-sortable-buf-3931622038
Nov 29 06:49:25 anon-r2 erigon[92876]: [INFO] [11-29|06:49:25.943] [p2p] GoodPeers                          eth68=33 eth67=33
Nov 29 06:49:32 anon-r2 erigon[92876]: [INFO] [11-29|06:49:32.078] [5/12 HashState] ETL [1/2] Extracting    from=StorageChangeSet current_prefix=00000000011a4fad8b6a6f5b828f086a2a9d8a3150b958e79e91a82f0000000000000001
Nov 29 06:49:40 anon-r2 systemd[1]: erigon.service: systemd-oomd killed 5 process(es) in this unit.
Nov 29 06:49:40 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Nov 29 06:49:40 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.

This was after a echo 2 > /proc/sys/vm/overcommit_memory

Will keep digging...

in my case it is stuck at block INFO[12-20|09:19:12.105] [txpool] stat INFO[12-20|09:22:12.094] [p2p] GoodPeers INFO[12-20|09:22:12.101] [txpool] stat INFO[12-20|09:25:12.094] [p2p] GoodPeers INFO[12-20|09:25:12.102] [txpool] stat INFO[12-20|09:28:12.094] [p2p] GoodPeers INFO[12-20|09:28:12.102] [txpool] stat INFO[12-20|09:31:12.094] [p2p] GoodPeers INFO[12-20|09:31:12.102] [txpool] stat INFO[12-20|09:34:12.094] [p2p] GoodPeers INFO[12-20|09:34:12.107] [txpool] stat INFO[12-20|09:37:12.094] [p2p] GoodPeers INFO[12-20|09:37:12.125] [txpool] stat INFO[12-20|09:40:12.094] [p2p] GoodPeers INFO[12-20|09:40:12.104] [txpool] stat INFO[12-20|09:43:12.094] [p2p] GoodPeers INFO[12-20|09:43:12.104] [txpool] stat INFO[12-20|09:46:12.094] [p2p] GoodPeers INFO[12-20|09:46:12.104] [txpool] stat INFO[12-20|09:49:12.094] [p2p] GoodPeers INFO[12-20|09:49:12.108] [txpool] stat 18299999, we have tried to upgrade to lastest release 2.55.1, it has been days in the same block, is this normal? how long it will take to process snapshots pending=10000 baseFee=0 queued=30000 alloc=4.0GB sys=5.3GB eth68=100 eth67=93 eth66=7 pending=10000 baseFee=0 queued=30000 alloc=2.8GB sys=5.3GB eth68=100 eth67=93 eth66=7 pending=10000 baseFee=0 queued=30000 alloc=3.9GB sys=5.3GB eth68=100 eth67=93 eth66=7 pending=10000 baseFee=0 queued=30000 alloc=2.8GB sys=5.3GB eth68=100 eth67=93 eth66=7 pending=10000 baseFee=0 queued=30000 alloc=3.9GB sys=5.3GB eth67=92 eth66=7 eth68=100 pending=10000 baseFee=0 queued=30000 alloc=2.9GB sys=5.3GB eth68=100 eth66=8 eth67=91 pending=10000 baseFee=0 queued=30000 alloc=4.0GB sys=5.3GB eth68=100 eth67=92 eth66=8 pending=10000 baseFee=0 queued=30000 alloc=2.8GB sys=5.3GB eth67=92 eth66=8 eth68=100 pending=10000 baseFee=0 queued=30000 alloc=3.9GB sys=5.3GB eth68=100 eth67=92 eth66=8 pending=10000 baseFee=0 queued=30000 alloc=2.8GB sys=5.3GB eth68=100 eth67=92 eth66=8 pending=10000 baseFee=0 queued=30000 alloc=4.2GB sys=5.3GB

I've given up trying to fix this sync, it's stuck in a loop, breaking and restarting from 18000000

Restarting sync from 0

After deleting everything and syncing again, back to the exact same issue.

Service exits during the sync phase from block 18m+

Updating to 2.56.1, and allowing sync to continue

Attempting to run 2.56.1 - need to resolve missing lib;

Aftering make erigon (tried twice) I'm seeing;

erigon: error while loading shared libraries: libsilkworm_capi.so: cannot open shared object file: No such file or directory

@caedmo use ldd erigon to get the linked libs of the erigon bin. You should see something like:

max@nod ~/P/e/b/bin (release/2.56) [1]> ldd erigon
        linux-vdso.so.1 (0x00007ffe48eab000)
        libresolv.so.2 => /usr/lib/libresolv.so.2 (0x00007f69b6940000)
        libsilkworm_capi.so => /home/max/go/pkg/mod/github.com/erigontech/silkworm-go@v0.10.0/lib/linux_x64/libsilkworm_capi.so (0x00007f69b4f36000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f69b4cbb000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f69b4bce000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f69b4ba9000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f69b49c5000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f69b6959000)
        libgmp.so.10 => /usr/lib/libgmp.so.10 (0x00007f69b491f000)

Then make sure the libsilkworm_capi.so shared lib is available in your runtime environment; you may need to use ldconfig to update your linker cache.

try add -tags nosilkworm

in my case it is stuck at block 18299999, we have tried to upgrade to lastest release 2.55.1, it has been days in the same block, is this normal? how long it will take to process snapshots INFO[12-20|09:19:12.105] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=4.0GB sys=5.3GB INFO[12-20|09:22:12.094] [p2p] GoodPeers eth68=100 eth67=93 eth66=7 INFO[12-20|09:22:12.101] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=2.8GB sys=5.3GB INFO[12-20|09:25:12.094] [p2p] GoodPeers eth68=100 eth67=93 eth66=7 INFO[12-20|09:25:12.102] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=3.9GB sys=5.3GB INFO[12-20|09:28:12.094] [p2p] GoodPeers eth68=100 eth67=93 eth66=7 INFO[12-20|09:28:12.102] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=2.8GB sys=5.3GB INFO[12-20|09:31:12.094] [p2p] GoodPeers eth68=100 eth67=93 eth66=7 INFO[12-20|09:31:12.102] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=3.9GB sys=5.3GB INFO[12-20|09:34:12.094] [p2p] GoodPeers eth67=92 eth66=7 eth68=100 INFO[12-20|09:34:12.107] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=2.9GB sys=5.3GB INFO[12-20|09:37:12.094] [p2p] GoodPeers eth68=100 eth66=8 eth67=91 INFO[12-20|09:37:12.125] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=4.0GB sys=5.3GB INFO[12-20|09:40:12.094] [p2p] GoodPeers eth68=100 eth67=92 eth66=8 INFO[12-20|09:40:12.104] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=2.8GB sys=5.3GB INFO[12-20|09:43:12.094] [p2p] GoodPeers eth67=92 eth66=8 eth68=100 INFO[12-20|09:43:12.104] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=3.9GB sys=5.3GB INFO[12-20|09:46:12.094] [p2p] GoodPeers eth68=100 eth67=92 eth66=8 INFO[12-20|09:46:12.104] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=2.8GB sys=5.3GB INFO[12-20|09:49:12.094] [p2p] GoodPeers eth68=100 eth67=92 eth66=8 INFO[12-20|09:49:12.108] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=4.2GB sys=5.3GB

Hello, is there any solution? I seem to have the same problem

Attempting to run 2.56.1 - need to resolve missing lib;

Aftering make erigon (tried twice) I'm seeing;
erigon: error while loading shared libraries: libsilkworm_capi.so: cannot open shared object file: No such file or directory

@caedmo , please try this solution: https://github.com/ledgerwatch/erigon/pull/8985#issuecomment-1880541041

it copies both erigon binary and a proper version of libsilkworm_capi.so

CC @maxmilton , @AskAlexSharov , @yperbasis

Thanks @battlmonstr

Erigon now running again.

Thanks @battlmonstr

Erigon now running again.

Hoping to hear good news from you, I am stuck on 18,300,027, raising bad forkchoice warning again and again, the block number does not increase, so I am looking for a stable version and plan to re-sync.

I am stuck here as well since weeks (at block 18299999). I updated to the latest version 2.57.0 (2 days old) and still the same problem.

I'm past my original 17999999 block height sync issue, and can now also see up to 18299999 (likely due to snapshot update)

However, I'm now dealing with a different sync issue, spanning the last 10 days, where the node is attempting to backfill, fails to insert, and restarts the process (log at bottom).

I've now stopped the node for now, as it just isn't able to move forward - debating whether I try syncing one last time from 0 again :(

On Jan 16th ( a couple days after this started), I decided to restart Erigon service, along with Nimbus (just in case)

Saw this while Erigon was stopping for a restart;

Jan 16 18:54:12 anon-r2 erigon[491288]: [WARN] [01-16|18:54:12.236] [Beacon Gossip] Fatal error receiving gossip err=EOF

...pruned...

Jan 16 18:55:24 anon-r2 erigon[491288]: [WARN] [01-16|18:55:24.962] Found error while processing packet      err="db closed"
Jan 16 18:55:25 anon-r2 erigon[491288]: [INFO] [01-16|18:55:25.424] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8181508 blockNumber=18985830 blk/sec=0.0 mbps/sec=0.0000 peers=3 snapshots=0 reconnected=false
Jan 16 18:55:26 anon-r2 erigon[491288]: [WARN] [01-16|18:55:26.497] Found error while processing packet      err="db closed"
Jan 16 18:55:28 anon-r2 erigon[491288]: [WARN] [01-16|18:55:28.367] Found error while processing packet      err="db closed"
Jan 16 18:55:30 anon-r2 erigon[491288]: [WARN] [01-16|18:55:30.159] Found error while processing packet      err="db closed"
Jan 16 18:55:32 anon-r2 erigon[491288]: [WARN] [01-16|18:55:32.021] Found error while processing packet      err="db closed"
Jan 16 18:55:33 anon-r2 erigon[491288]: [WARN] [01-16|18:55:33.368] Found error while processing packet      err="db closed"
Jan 16 18:55:34 anon-r2 erigon[491288]: [WARN] [01-16|18:55:34.597] Found error while processing packet      err="db closed"
Jan 16 18:55:35 anon-r2 erigon[491288]: [WARN] [01-16|18:55:35.972] Found error while processing packet      err="db closed"
Jan 16 18:55:37 anon-r2 erigon[491288]: [WARN] [01-16|18:55:37.333] Found error while processing packet      err="db closed"
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: State 'stop-sigterm' timed out. Killing.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 491288 (erigon) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 491289 (erigon) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 491311 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 491312 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 491313 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 491316 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 491726 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 491727 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 492297 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 494682 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507359 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507360 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507362 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507364 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507365 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507367 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507368 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507369 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507370 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507371 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507373 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507375 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507376 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507377 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507378 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507379 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507380 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507381 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507382 (erigon) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507383 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507384 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507385 (n/a) with signal SIGKILL.
Jan 16 18:55:42 anon-r2 systemd[1]: erigon.service: Killing process 507386 (n/a) with signal SIGKILL.
Jan 16 18:55:43 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL

New sync errors (pruned)

Jan 16 17:27:17 anon-r2 erigon[491288]: [INFO] [01-16|17:27:17.800] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8163534 blockNumber=18968016 blk/sec=6.2 mbps/sec=1.1931 peers=3 snapshots=0 reconnected=false
Jan 16 17:27:30 anon-r2 erigon[491288]: [INFO] [01-16|17:27:30.223] Backfilling finished
Jan 16 17:27:32 anon-r2 erigon[491288]: [INFO] [01-16|17:27:32.963] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
Jan 16 17:27:35 anon-r2 erigon[491288]: [WARN] [01-16|17:27:35.281] error executing clstage                  app=caplin stage=DownloadHistoricalBlocks err="loadIntoTable Header: error inserting block during collection: ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"
Jan 16 17:27:35 anon-r2 erigon[491288]: [INFO] [01-16|17:27:35.281] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=8201120

...Downloading History...

Jan 16 18:04:35 anon-r2 erigon[491288]: [INFO] [01-16|18:04:35.281] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8163323 blockNumber=18967807 blk/sec=28.1 mbps/sec=5.8448 peers=7 snapshots=0 reconnected=false
Jan 16 18:04:35 anon-r2 erigon[491288]: [INFO] [01-16|18:04:35.358] Backfilling finished
Jan 16 18:04:35 anon-r2 erigon[491288]: [INFO] [01-16|18:04:35.487] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
Jan 16 18:04:37 anon-r2 erigon[491288]: [WARN] [01-16|18:04:37.872] error executing clstage                  app=caplin stage=DownloadHistoricalBlocks err="loadIntoTable Header: error inserting block during collection: ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"
Jan 16 18:04:37 anon-r2 erigon[491288]: [INFO] [01-16|18:04:37.872] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=8201120

...Downloading History...

Jan 16 18:33:07 anon-r2 erigon[491288]: [INFO] [01-16|18:33:07.873] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8163815 blockNumber=18968293 blk/sec=30.9 mbps/sec=6.6818 peers=4 snapshots=0 reconnected=false
Jan 16 18:33:21 anon-r2 erigon[491288]: [INFO] [01-16|18:33:21.391] Backfilling finished
Jan 16 18:33:23 anon-r2 erigon[491288]: [INFO] [01-16|18:33:23.019] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
Jan 16 18:33:25 anon-r2 erigon[491288]: [WARN] [01-16|18:33:25.423] error executing clstage                  app=caplin stage=DownloadHistoricalBlocks err="loadIntoTable Header: error inserting block during collection: ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"
Jan 16 18:33:25 anon-r2 erigon[491288]: [INFO] [01-16|18:33:25.423] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=8201120

Switching off Erigon, I saw this again;

Jan 25 08:59:06 anon-r2 erigon[722086]: [INFO] [01-25|08:59:06.250] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8183592 blockNumber=18987894 blk/sec=0.0 mbps/sec=0.0000 peers=19 snapshots=0 reconnected=false
Jan 25 08:59:36 anon-r2 erigon[722086]: [INFO] [01-25|08:59:36.250] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8183592 blockNumber=18987894 blk/sec=0.0 mbps/sec=0.0000 peers=25 snapshots=0 reconnected=false
Jan 25 08:59:54 anon-r2 erigon[722086]: [INFO] [01-25|08:59:54.805] [p2p] GoodPeers                          eth68=33 eth67=33
Jan 25 09:00:03 anon-r2 erigon[722086]: [INFO] [01-25|09:00:03.032] P2P                                      app=caplin peers=25
Jan 25 09:00:06 anon-r2 erigon[722086]: [INFO] [01-25|09:00:06.250] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8183560 blockNumber=18987863 blk/sec=1.1 mbps/sec=0.1968 peers=25 snapshots=0 reconnected=false
Jan 25 09:00:10 anon-r2 systemd[1]: Stopping erigon.service - Erigon Client - Mainnet full...
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.083] Got interrupt, shutting down...
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.085] Exiting Engine...
Jan 25 09:00:10 anon-r2 erigon[722086]: [WARN] [01-25|09:00:10.085] [Beacon Gossip] Fatal error receiving gossip err=EOF
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.087] Exiting...
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.107] HTTP endpoint closed                     url=192.168.2.3:8545
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.107] HTTP endpoint closed                     url=192.168.2.3:8546
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.107] RPC server shutting down
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.086] RPC server shutting down
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.109] RPC server shutting down
Jan 25 09:00:10 anon-r2 erigon[722086]: [INFO] [01-25|09:00:10.109] Engine HTTP endpoint close               url=127.0.0.1:8551
Jan 25 09:00:36 anon-r2 erigon[722086]: [INFO] [01-25|09:00:36.251] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8183560 blockNumber=18987863 blk/sec=0.0 mbps/sec=0.0000 peers=23 snapshots=0 reconnected=false
Jan 25 09:01:01 anon-r2 erigon[722086]: [WARN] [01-25|09:01:01.091] Found error while processing packet      err="db closed"
Jan 25 09:01:06 anon-r2 erigon[722086]: [INFO] [01-25|09:01:06.250] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8183560 blockNumber=18987863 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
Jan 25 09:01:07 anon-r2 erigon[722086]: [WARN] [01-25|09:01:07.530] Found error while processing packet      err="db closed"
Jan 25 09:01:36 anon-r2 erigon[722086]: [INFO] [01-25|09:01:36.250] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8183560 blockNumber=18987863 blk/sec=0.0 mbps/sec=0.0000 peers=23 snapshots=0 reconnected=false
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: State 'stop-sigterm' timed out. Killing.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722086 (erigon) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722108 (erigon) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722109 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722110 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722111 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722113 (erigon) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722114 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722155 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 722156 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 724559 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 731296 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 966366 (n/a) with signal SIGKILL.
Jan 25 09:01:40 anon-r2 systemd[1]: erigon.service: Killing process 1186887 (n/a) with signal SIGKILL.
Jan 25 09:01:41 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Jan 25 09:01:41 anon-r2 systemd[1]: erigon.service: Failed with result 'timeout'.
Jan 25 09:01:41 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Jan 25 09:01:41 anon-r2 systemd[1]: erigon.service: Consumed 1w 2d 6h 26min 55.714s CPU time.

I also have issue with erigon version 2.57.0 block syncing stucked at 18799999

@caedmo what is your current setup? (I saw that you tried different combinations)

I think I'm facing the same issue, but I started it from scratch on 2.56.1, always with --internalcl. I don't see it being killed, but the last cycle always repeats between 18.3M - chain tip, with no errors in the log.

for comparison, I see logs until stage 9:

[INFO] [01-27|02:37:15.452] [9/12 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=c02aaa39
[INFO] [01-27|02:37:19.651] P2P                                      app=caplin peers=29
[INFO] [01-27|02:37:22.949] [Caplin] Downloading epochs from reqresp app=caplin stage=CatchUpEpochs from=258871 to=259
111
[INFO] [01-27|02:38:03.411] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=258871
[INFO] [01-27|02:38:19.651] P2P                                      app=caplin peers=23
[INFO] [01-27|02:38:43.903] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=258872
[INFO] [01-27|02:39:00.487] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=258873
[INFO] [01-27|02:39:15.017] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=258874
[INFO] [01-27|02:39:19.651] P2P                                      app=caplin peers=20

then it keeps logging only caplin stuff, then a few hours later I just see a stage 3 log, without any other log indicating the previous cycle finish:

[INFO] [01-27|05:18:19.651] P2P                                      app=caplin peers=21
[INFO] [01-27|05:19:03.043] [p2p] GoodPeers                          eth68=100 eth67=85 eth66=15
[INFO] [01-27|05:19:05.184] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259135
[INFO] [01-27|05:19:19.651] P2P                                      app=caplin peers=21
[INFO] [01-27|05:19:29.701] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259136
[INFO] [01-27|05:20:19.652] P2P                                      app=caplin peers=22
[INFO] [01-27|05:21:06.234] [3/12 Senders] Started                   from=18299999 to=19095793
[INFO] [01-27|05:21:19.651] P2P                                      app=caplin peers=25
[INFO] [01-27|05:21:36.234] [3/12 Senders] Recovery                  block_number=18304536 ch=0/10000
[INFO] [01-27|05:22:03.043] [p2p] GoodPeers                          eth68=100 eth67=85 eth66=15

worth mentioning that during the caplin download, I see some warnings like:

[INFO] [01-27|04:41:27.183] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259076
[INFO] [01-27|04:41:47.954] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259077
[INFO] [01-27|04:42:13.931] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259078
[INFO] [01-27|04:42:19.651] P2P                                      app=caplin peers=17
[INFO] [01-27|04:42:27.908] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259079
[WARN] [01-27|04:42:27.912] fail to process block                    reason="replay block, code: 2" slot=8290528
[WARN] [01-27|04:42:27.913] bad blocks segment received              err="replay block, code: 2"
[INFO] [01-27|04:42:35.986] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259078
[WARN] [01-27|04:42:39.865] bad blocks segment received              err="parent's total difficulty not found with has
h d2b61d67248ea4f5d496e97f1775ee6ee04e5d856a1aa9a7c72443b7448d4b89 and height 19093937: <nil>"
[INFO] [01-27|04:43:03.043] [p2p] GoodPeers                          eth66=15 eth68=100 eth67=85
[INFO] [01-27|04:43:06.515] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259077
[INFO] [01-27|04:43:19.651] P2P                                      app=caplin peers=18
[INFO] [01-27|04:43:27.114] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259078

but I'm not sure it has something to do with this issue because it continues downloading and the sync cycle restarts much later.

My config is

/usr/local/bin/erigon --datadir=/erigon --prune=hrtc  --metrics --metrics.addr 0.0.0.0 --http --ws --nat extip:1.1.1.1 --torrent.download.rate 100mb --torrent.upload.rate 16mb

erigon version 2.57.1 Stucked at 18799999

Do you have any idea, guys?

@wmitsuda

"I see logs until stage 9" - show from this node: curl -s -v http://localhost:6060/debug/pprof/goroutine?debug=1 | head -1000 > goroutine.txt
"without any other log indicating the previous cycle finish" - just added Timings log line for --internalcl case (by https://github.com/ledgerwatch/erigon/pull/9315 )

@ehsanhajian --internalcl

@AskAlexSharov Thanks for your responses. I added --internalcla couple of hours ago but it still tucked.

/usr/local/bin/erigon --datadir=/erigon --prune=hrtc  --metrics --metrics.addr 0.0.0.0 --http --ws --nat extip:1.1.1.1 --torrent.download.rate 100mb --torrent.upload.rate 16mb --internalcl

 Jan 28 05:08:07 ip-172-31-34-9 erigon[27731]: [WARN] [01-28|05:08:07.625] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: writeBody: WriteRawTransactions: txId=2230015338, baseTxId=2230015338, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"
Jan 28 05:08:11 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:11.031] P2P                                      app=caplin peers=5
Jan 28 05:08:12 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:12.165] [NewPayload] Handling new payload        height=19102855 hash=0x9763401202fc16ed169515c1c6dffdeb22c7c8a6a571888e8435d57098e54b2c
Jan 28 05:08:12 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:12.165] [EngineBlockDownloader] Downloading PoS headers... hash=0xe12920aa1a8ff406296ea013bd5dd7bebb56cc3b35b4e7485e1def6f0096dfe7 requestId=0
Jan 28 05:08:24 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:24.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8246039 blockNumber=19049967 blk/sec=17.4 mbps/sec=2.9748 peers=6 snapshots=0 reconnected=false
Jan 28 05:08:54 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:54.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8245490 blockNumber=19049421 blk/sec=18.3 mbps/sec=3.5229 peers=4 snapshots=0 reconnected=false
Jan 28 05:08:58 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:58.470] [downloader] Downloaded PoS Headers      now=19101126 blk/sec=32.000
Jan 28 05:09:11 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:09:11.029] P2P                                      app=caplin peers=9
Jan 28 05:09:24 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:09:24.385] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8245074 blockNumber=19049008 blk/sec=13.9 mbps/sec=2.9294 peers=13 snapshots=0 reconnected=false
Jan 28 05:09:32 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:09:32.925] [downloader] Downloaded PoS Headers      now=19099974 blk/sec=38.400
Jan 28 05:09:54 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:09:54.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8244178 blockNumber=19048115 blk/sec=29.9 mbps/sec=6.6299 peers=6 snapshots=0 reconnected=false
Jan 28 05:10:00 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:00.904] [downloader] Downloaded PoS Headers      now=19099014 blk/sec=32.000
Jan 28 05:10:07 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:07.347] [p2p] GoodPeers                          eth68=65 eth67=96 eth66=4
Jan 28 05:10:09 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:09.890] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=5.6GB sys=10.7GB
Jan 28 05:10:11 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:11.029] P2P                                      app=caplin peers=4
Jan 28 05:10:24 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:24.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8243346 blockNumber=19047288 blk/sec=27.7 mbps/sec=5.5227 peers=7 snapshots=0 reconnected=false
Jan 28 05:10:33 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:33.815] [downloader] Downloaded PoS Headers      now=19097862 blk/sec=38.400
Jan 28 05:10:54 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:54.385] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8242546 blockNumber=19046488 blk/sec=26.7 mbps/sec=5.1576 peers=10 snapshots=0 reconnected=false
Jan 28 05:11:00 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:00.095] [downloader] Downloaded PoS Headers      now=19096902 blk/sec=32.000
Jan 28 05:11:11 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:11.029] P2P                                      app=caplin peers=11
Jan 28 05:11:24 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:24.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8242290 blockNumber=19046232 blk/sec=8.5 mbps/sec=1.5350 peers=10 snapshots=0 reconnected=false
Jan 28 05:11:29 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:29.797] [downloader] Downloaded PoS Headers      now=19095750 blk/sec=38.400
Jan 28 05:11:54 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:54.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8241938 blockNumber=19045881 blk/sec=11.7 mbps/sec=2.4110 peers=7 snapshots=0 reconnected=false
Jan 28 05:11:58 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:58.469] [downloader] Downloaded PoS Headers

@AskAlexSharov here:

goroutine.txt.gz

now I'll cherry pick the commit you mentioned and let it run for a few hours until it completes the sync cycle.

after letting it run for some days and some iteractions, it seems the same, I see logs only until stage 9, then it starts to only log caplin epoch download, then it restarts stage 3.

no error logs, the only odd info is some warn logs about caplin not being able to process some block.

for completeness, here are some logs snippets from last cycle, first ending stage 9:

[INFO] [01-30|15:23:55.038] [9/12 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=a0b86991
[INFO] [01-30|15:24:24.993] [9/12 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=a0b86991
[INFO] [01-30|15:24:26.341] P2P                                      app=caplin peers=29
[INFO] [01-30|15:24:48.384] [Caplin] Downloading epochs from reqresp app=caplin stage=CatchUpEpochs from=259646 to=259
905
[INFO] [01-30|15:24:57.094] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259646
[INFO] [01-30|15:25:26.341] P2P                                      app=caplin peers=23
[INFO] [01-30|15:26:03.490] [p2p] GoodPeers                          eth68=100 eth66=15 eth67=85
[INFO] [01-30|15:26:26.341] P2P                                      app=caplin peers=37
[INFO] [01-30|15:27:03.637] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259647
[INFO] [01-30|15:27:23.452] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259648
[INFO] [01-30|15:27:26.341] P2P                                      app=caplin peers=32
[INFO] [01-30|15:28:03.874] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259649

then restarting stage 3:

[INFO] [01-30|17:55:16.567] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259925
[INFO] [01-30|17:55:23.939] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259926
[INFO] [01-30|17:55:26.341] P2P                                      app=caplin peers=5
[INFO] [01-30|17:55:31.928] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259927
[INFO] [01-30|17:55:44.001] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259928
[INFO] [01-30|17:55:51.796] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259929
[INFO] [01-30|17:56:03.488] [p2p] GoodPeers                          eth67=85 eth68=100 eth66=15
[INFO] [01-30|17:56:26.341] P2P                                      app=caplin peers=3
[INFO] [01-30|17:57:17.160] [3/12 Senders] Started                   from=18299999 to=19120953
[INFO] [01-30|17:57:26.341] P2P                                      app=caplin peers=14
[INFO] [01-30|17:57:47.171] [3/12 Senders] Recovery                  block_number=18304617 ch=0/10000
[INFO] [01-30|17:58:17.171] [3/12 Senders] Recovery                  block_number=18308630 ch=0/10000
[INFO] [01-30|17:58:26.341] P2P                                      app=caplin peers=3
[INFO] [01-30|17:58:47.171] [3/12 Senders] Recovery                  block_number=18313046 ch=0/10000

and there are a couple of caplin warns in between, they are all like this (with different block heights):

[INFO] [01-30|17:26:06.320] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259814
[INFO] [01-30|17:26:26.342] P2P                                      app=caplin peers=9
[INFO] [01-30|17:26:47.750] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815
[INFO] [01-30|17:26:54.587] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259816
[WARN] [01-30|17:26:54.592] fail to process block                    reason="replay block, code: 2" slot=8314112
[WARN] [01-30|17:26:54.592] bad blocks segment received              err="replay block, code: 2"
[INFO] [01-30|17:27:19.243] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815
[WARN] [01-30|17:27:23.398] bad blocks segment received              err="parent's total difficulty not found with has
h 98ad8a938c0c1ce1abe124724c5089419abe33a321909c96282e652b3489d88d and height 19117305: <nil>"
[INFO] [01-30|17:27:26.341] P2P                                      app=caplin peers=9
[INFO] [01-30|17:28:21.654] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259814
[INFO] [01-30|17:28:26.341] P2P                                      app=caplin peers=10
[INFO] [01-30|17:28:31.768] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815

it is 2.57.1 with commit e423c5f523395fc10be9e0373bdb79deea921dd9 cherry-picked on it.

after letting it run for some days and some iteractions, it seems the same, I see logs only until stage 9, then it starts to only log caplin epoch download, then it restarts stage 3.

no error logs, the only odd info is some warn logs about caplin not being able to process some block.

for completeness, here are some logs snippets from last cycle, first ending stage 9:

[INFO] [01-30|15:23:55.038] [9/12 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=a0b86991
[INFO] [01-30|15:24:24.993] [9/12 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=a0b86991
[INFO] [01-30|15:24:26.341] P2P                                      app=caplin peers=29
[INFO] [01-30|15:24:48.384] [Caplin] Downloading epochs from reqresp app=caplin stage=CatchUpEpochs from=259646 to=259
905
[INFO] [01-30|15:24:57.094] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259646
[INFO] [01-30|15:25:26.341] P2P                                      app=caplin peers=23
[INFO] [01-30|15:26:03.490] [p2p] GoodPeers                          eth68=100 eth66=15 eth67=85
[INFO] [01-30|15:26:26.341] P2P                                      app=caplin peers=37
[INFO] [01-30|15:27:03.637] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259647
[INFO] [01-30|15:27:23.452] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259648
[INFO] [01-30|15:27:26.341] P2P                                      app=caplin peers=32
[INFO] [01-30|15:28:03.874] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259649

then restarting stage 3:

[INFO] [01-30|17:55:16.567] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259925
[INFO] [01-30|17:55:23.939] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259926
[INFO] [01-30|17:55:26.341] P2P                                      app=caplin peers=5
[INFO] [01-30|17:55:31.928] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259927
[INFO] [01-30|17:55:44.001] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259928
[INFO] [01-30|17:55:51.796] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259929
[INFO] [01-30|17:56:03.488] [p2p] GoodPeers                          eth67=85 eth68=100 eth66=15
[INFO] [01-30|17:56:26.341] P2P                                      app=caplin peers=3
[INFO] [01-30|17:57:17.160] [3/12 Senders] Started                   from=18299999 to=19120953
[INFO] [01-30|17:57:26.341] P2P                                      app=caplin peers=14
[INFO] [01-30|17:57:47.171] [3/12 Senders] Recovery                  block_number=18304617 ch=0/10000
[INFO] [01-30|17:58:17.171] [3/12 Senders] Recovery                  block_number=18308630 ch=0/10000
[INFO] [01-30|17:58:26.341] P2P                                      app=caplin peers=3
[INFO] [01-30|17:58:47.171] [3/12 Senders] Recovery                  block_number=18313046 ch=0/10000

and there are a couple of caplin warns in between, they are all like this (with different block heights):

[INFO] [01-30|17:26:06.320] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259814
[INFO] [01-30|17:26:26.342] P2P                                      app=caplin peers=9
[INFO] [01-30|17:26:47.750] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815
[INFO] [01-30|17:26:54.587] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259816
[WARN] [01-30|17:26:54.592] fail to process block                    reason="replay block, code: 2" slot=8314112
[WARN] [01-30|17:26:54.592] bad blocks segment received              err="replay block, code: 2"
[INFO] [01-30|17:27:19.243] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815
[WARN] [01-30|17:27:23.398] bad blocks segment received              err="parent's total difficulty not found with has
h 98ad8a938c0c1ce1abe124724c5089419abe33a321909c96282e652b3489d88d and height 19117305: <nil>"
[INFO] [01-30|17:27:26.341] P2P                                      app=caplin peers=9
[INFO] [01-30|17:28:21.654] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259814
[INFO] [01-30|17:28:26.341] P2P                                      app=caplin peers=10
[INFO] [01-30|17:28:31.768] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815

@Giulio2002 could you take a look please?

after letting it run for some days and some iteractions, it seems the same, I see logs only until stage 9, then it starts to only log caplin epoch download, then it restarts stage 3.

no error logs, the only odd info is some warn logs about caplin not being able to process some block.

for completeness, here are some logs snippets from last cycle, first ending stage 9:

[INFO] [01-30|15:23:55.038] [9/12 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=a0b86991
[INFO] [01-30|15:24:24.993] [9/12 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=a0b86991
[INFO] [01-30|15:24:26.341] P2P                                      app=caplin peers=29
[INFO] [01-30|15:24:48.384] [Caplin] Downloading epochs from reqresp app=caplin stage=CatchUpEpochs from=259646 to=259
905
[INFO] [01-30|15:24:57.094] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259646
[INFO] [01-30|15:25:26.341] P2P                                      app=caplin peers=23
[INFO] [01-30|15:26:03.490] [p2p] GoodPeers                          eth68=100 eth66=15 eth67=85
[INFO] [01-30|15:26:26.341] P2P                                      app=caplin peers=37
[INFO] [01-30|15:27:03.637] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259647
[INFO] [01-30|15:27:23.452] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259648
[INFO] [01-30|15:27:26.341] P2P                                      app=caplin peers=32
[INFO] [01-30|15:28:03.874] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259649

then restarting stage 3:

[INFO] [01-30|17:55:16.567] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259925
[INFO] [01-30|17:55:23.939] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259926
[INFO] [01-30|17:55:26.341] P2P                                      app=caplin peers=5
[INFO] [01-30|17:55:31.928] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259927
[INFO] [01-30|17:55:44.001] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259928
[INFO] [01-30|17:55:51.796] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259929
[INFO] [01-30|17:56:03.488] [p2p] GoodPeers                          eth67=85 eth68=100 eth66=15
[INFO] [01-30|17:56:26.341] P2P                                      app=caplin peers=3
[INFO] [01-30|17:57:17.160] [3/12 Senders] Started                   from=18299999 to=19120953
[INFO] [01-30|17:57:26.341] P2P                                      app=caplin peers=14
[INFO] [01-30|17:57:47.171] [3/12 Senders] Recovery                  block_number=18304617 ch=0/10000
[INFO] [01-30|17:58:17.171] [3/12 Senders] Recovery                  block_number=18308630 ch=0/10000
[INFO] [01-30|17:58:26.341] P2P                                      app=caplin peers=3
[INFO] [01-30|17:58:47.171] [3/12 Senders] Recovery                  block_number=18313046 ch=0/10000

and there are a couple of caplin warns in between, they are all like this (with different block heights):

[INFO] [01-30|17:26:06.320] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259814
[INFO] [01-30|17:26:26.342] P2P                                      app=caplin peers=9
[INFO] [01-30|17:26:47.750] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815
[INFO] [01-30|17:26:54.587] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259816
[WARN] [01-30|17:26:54.592] fail to process block                    reason="replay block, code: 2" slot=8314112
[WARN] [01-30|17:26:54.592] bad blocks segment received              err="replay block, code: 2"
[INFO] [01-30|17:27:19.243] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815
[WARN] [01-30|17:27:23.398] bad blocks segment received              err="parent's total difficulty not found with has
h 98ad8a938c0c1ce1abe124724c5089419abe33a321909c96282e652b3489d88d and height 19117305: <nil>"
[INFO] [01-30|17:27:26.341] P2P                                      app=caplin peers=9
[INFO] [01-30|17:28:21.654] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259814
[INFO] [01-30|17:28:26.341] P2P                                      app=caplin peers=10
[INFO] [01-30|17:28:31.768] [Caplin] Epoch downloaded                app=caplin stage=CatchUpEpochs epoch=259815

Yeah, it is syncing up; it is 2 days behind the chain tip. Speeding that download up will be one of my priorities sometimes in the future.

@AskAlexSharov Thanks for your responses. I added --internalcla couple of hours ago but it still tucked.

/usr/local/bin/erigon --datadir=/erigon --prune=hrtc  --metrics --metrics.addr 0.0.0.0 --http --ws --nat extip:1.1.1.1 --torrent.download.rate 100mb --torrent.upload.rate 16mb --internalcl

Jan 28 05:08:07 ip-172-31-34-9 erigon[27731]: [WARN] [01-28|05:08:07.625] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: writeBody: WriteRawTransactions: txId=2230015338, baseTxId=2230015338, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"
Jan 28 05:08:11 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:11.031] P2P                                      app=caplin peers=5
Jan 28 05:08:12 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:12.165] [NewPayload] Handling new payload        height=19102855 hash=0x9763401202fc16ed169515c1c6dffdeb22c7c8a6a571888e8435d57098e54b2c
Jan 28 05:08:12 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:12.165] [EngineBlockDownloader] Downloading PoS headers... hash=0xe12920aa1a8ff406296ea013bd5dd7bebb56cc3b35b4e7485e1def6f0096dfe7 requestId=0
Jan 28 05:08:24 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:24.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8246039 blockNumber=19049967 blk/sec=17.4 mbps/sec=2.9748 peers=6 snapshots=0 reconnected=false
Jan 28 05:08:54 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:54.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8245490 blockNumber=19049421 blk/sec=18.3 mbps/sec=3.5229 peers=4 snapshots=0 reconnected=false
Jan 28 05:08:58 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:08:58.470] [downloader] Downloaded PoS Headers      now=19101126 blk/sec=32.000
Jan 28 05:09:11 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:09:11.029] P2P                                      app=caplin peers=9
Jan 28 05:09:24 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:09:24.385] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8245074 blockNumber=19049008 blk/sec=13.9 mbps/sec=2.9294 peers=13 snapshots=0 reconnected=false
Jan 28 05:09:32 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:09:32.925] [downloader] Downloaded PoS Headers      now=19099974 blk/sec=38.400
Jan 28 05:09:54 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:09:54.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8244178 blockNumber=19048115 blk/sec=29.9 mbps/sec=6.6299 peers=6 snapshots=0 reconnected=false
Jan 28 05:10:00 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:00.904] [downloader] Downloaded PoS Headers      now=19099014 blk/sec=32.000
Jan 28 05:10:07 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:07.347] [p2p] GoodPeers                          eth68=65 eth67=96 eth66=4
Jan 28 05:10:09 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:09.890] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=5.6GB sys=10.7GB
Jan 28 05:10:11 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:11.029] P2P                                      app=caplin peers=4
Jan 28 05:10:24 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:24.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8243346 blockNumber=19047288 blk/sec=27.7 mbps/sec=5.5227 peers=7 snapshots=0 reconnected=false
Jan 28 05:10:33 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:33.815] [downloader] Downloaded PoS Headers      now=19097862 blk/sec=38.400
Jan 28 05:10:54 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:10:54.385] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8242546 blockNumber=19046488 blk/sec=26.7 mbps/sec=5.1576 peers=10 snapshots=0 reconnected=false
Jan 28 05:11:00 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:00.095] [downloader] Downloaded PoS Headers      now=19096902 blk/sec=32.000
Jan 28 05:11:11 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:11.029] P2P                                      app=caplin peers=11
Jan 28 05:11:24 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:24.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8242290 blockNumber=19046232 blk/sec=8.5 mbps/sec=1.5350 peers=10 snapshots=0 reconnected=false
Jan 28 05:11:29 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:29.797] [downloader] Downloaded PoS Headers      now=19095750 blk/sec=38.400
Jan 28 05:11:54 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:54.384] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8241938 blockNumber=19045881 blk/sec=11.7 mbps/sec=2.4110 peers=7 snapshots=0 reconnected=false
Jan 28 05:11:58 ip-172-31-34-9 erigon[27731]: [INFO] [01-28|05:11:58.469] [downloader] Downloaded PoS Headers

This looks like it is progressing, also you should disconnect the external cl you have --internalcl allows you to not use the external consensus layer

Running Erigon 2.56.1 and recently updated to latest Nimbus (my beacon chain setup) 24.1.2 from 23.10.1

I am now seeing a much better Erigon log output

I'm currently still see seeing block height 18299999 on GQL endpoint (my guess would be the head of the snapshot). This is likely due to the lag in block bodies, to be downloaded, which has been going for a couple days now.

My payload height has remained current, with the network (great news!).

Block bodies aren't, at this time - and are slowly trailing behind, every so often. Over the last 24 hours, I've kept track of the disparity from payload height, and downloading block bodies.

I'll update here, when I finally reach that ultimate goal of GQL serving latest blocks.

The block disparity for me (every 6 hours) of the last 24;

2024-01-31 - 1500z - 61000 blocks

2024-01-31 - 2100z - 46000 blocks

2024-02-01 - 0300z - 13700 blocks

2024-02-01 - 0900z - 14000 blocks

2024-02-01 - 1500z - 27000 blocks

What I am seeing on the logs;

Feb 01 15:28:09 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:09.890] [txpool] stat                            pending=1518 baseFee=0 queued=30000 alloc=6.5GB sys=9.9GB
Feb 01 15:28:12 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:12.402] [NewPayload] Handling new payload        height=19134468 hash=0x1c77de4fe54f17299f418755b64e768e367ad0bb2fa1e66d8c9062c484d2752a
Feb 01 15:28:18 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:18.796] P2P                                      app=caplin peers=28
Feb 01 15:28:24 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:24.976] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8286316 blockNumber=19089761 blk/sec=0.0 mbps/sec=0.0000 peers=28 snapshots=0 reconnected=false
Feb 01 15:28:26 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:26.115] [NewPayload] Handling new payload        height=19134469 hash=0x4a8f93b55f81c34dda4035cb26b87c80badb5aceff477fc06fdb78f930605d4a
Feb 01 15:28:32 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:32.698] [EngineBlockDownloader] Downloading block bodies block_num=19118258 delivery/sec=7.6MB wasted/sec=1.0MB remaining=15975 delivered=150766 cache=50.9MB alloc=7.6GB sys=9.9GB
Feb 01 15:28:36 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:36.512] [NewPayload] Handling new payload        height=19134470 hash=0xd3b4ab0bce8c7c4487b71e55a9cb26ef05ed5ece01e4da61f3490704f66c213e
Feb 01 15:28:48 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:48.715] [NewPayload] Handling new payload        height=19134471 hash=0x15d1deb882888644281e85683db88d0053e0ba9a456758f39caecb3c6d57345c
Feb 01 15:28:54 anon-r2 erigon[2035426]: [INFO] [02-01|15:28:54.976] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8286252 blockNumber=19089701 blk/sec=2.1 mbps/sec=0.4088 peers=27 snapshots=0 reconnected=false
Feb 01 15:29:00 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:00.755] [NewPayload] Handling new payload        height=19134472 hash=0x8424d193d3473e942d60a55aadecbe79babf17055aa8e8fb5ef8cf4c565449ef
Feb 01 15:29:02 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:02.713] [EngineBlockDownloader] Downloading block bodies block_num=19119645 delivery/sec=7.1MB wasted/sec=951.9KB remaining=14588 delivered=152140 cache=43.7MB alloc=5.2GB sys=9.9GB
Feb 01 15:29:12 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:12.426] [NewPayload] Handling new payload        height=19134473 hash=0x6d18b6b56edf9d157fa87ec1db884d17614e1d0039f86366027f8ec6e6d4a39e
Feb 01 15:29:18 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:18.788] P2P                                      app=caplin peers=24
Feb 01 15:29:24 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:24.484] [NewPayload] Handling new payload        height=19134474 hash=0xd3f00946bfc4615431fd206bb52d459150a89348abd65b2d944715b0f218b2d5
Feb 01 15:29:24 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:24.976] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8286252 blockNumber=19089701 blk/sec=0.0 mbps/sec=0.0000 peers=22 snapshots=0 reconnected=false
Feb 01 15:29:32 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:32.698] [EngineBlockDownloader] Downloading block bodies block_num=19121222 delivery/sec=7.9MB wasted/sec=942.0KB remaining=13011 delivered=153745 cache=49.2MB alloc=6.7GB sys=9.9GB
Feb 01 15:29:36 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:36.381] [NewPayload] Handling new payload        height=19134475 hash=0xfd935f04d5fc7d9071d0cb7d8497053c0b4025da4419d7c983e30a742a1056f6
Feb 01 15:29:48 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:48.799] [NewPayload] Handling new payload        height=19134476 hash=0xcbd8c246bab8a7db1f74a14db4bb9a56e6c45bb2854a0df095938852f751b9d8
Feb 01 15:29:54 anon-r2 erigon[2035426]: [INFO] [02-01|15:29:54.983] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8286252 blockNumber=19089701 blk/sec=0.0 mbps/sec=0.0000 peers=20 snapshots=0 reconnected=false
Feb 01 15:30:00 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:00.973] [NewPayload] Handling new payload        height=19134477 hash=0xa016de1852997d1d126053b70aa58cc5656916c1abaa251973209113f0cf37d3
Feb 01 15:30:02 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:02.697] [EngineBlockDownloader] Downloading block bodies block_num=19123010 delivery/sec=9.1MB wasted/sec=1012.4KB remaining=11223 delivered=155370 cache=23.0MB alloc=4.4GB sys=9.9GB
Feb 01 15:30:13 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:13.525] [NewPayload] Handling new payload        height=19134478 hash=0xa3cf2d04f6674640266373f8d59b8a2ebdc08139e562511febd4dd9f6d5a78ca
Feb 01 15:30:18 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:18.788] P2P                                      app=caplin peers=23
Feb 01 15:30:24 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:24.376] [NewPayload] Handling new payload        height=19134479 hash=0xf6800ea51e87b753f13c1d905b4997781a200dff63e5da579c7c7f9c8a635efe
Feb 01 15:30:24 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:24.976] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8286252 blockNumber=19089701 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
Feb 01 15:30:32 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:32.698] [EngineBlockDownloader] Downloading block bodies block_num=19124479 delivery/sec=7.7MB wasted/sec=1.8MB remaining=9754 delivered=156900 cache=36.3MB alloc=6.1GB sys=9.9GB
Feb 01 15:30:36 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:36.562] [NewPayload] Handling new payload        height=19134480 hash=0xd9db011c1eacd52a7e7f7d70d51ace62bcdc2872dea0905af6adbae153f234cd
Feb 01 15:30:45 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:45.774] [p2p] GoodPeers                          eth68=33 eth67=24
Feb 01 15:30:48 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:48.600] [NewPayload] Handling new payload        height=19134481 hash=0x66c0a1852c5c19b868525694e7ca342ebe2c72249a7d6ec41cd63e3e4f3bcefa
Feb 01 15:30:54 anon-r2 erigon[2035426]: [INFO] [02-01|15:30:54.976] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8286252 blockNumber=19089701 blk/sec=0.0 mbps/sec=0.0000 peers=19 snapshots=0 reconnected=false
Feb 01 15:31:00 anon-r2 erigon[2035426]: [INFO] [02-01|15:31:00.340] [NewPayload] Handling new payload        height=19134482 hash=0xbb3d5028d543542795514ac94e8d88ed89085460e55b8a4b75f866a6581e94ce
Feb 01 15:31:02 anon-r2 erigon[2035426]: [INFO] [02-01|15:31:02.720] [EngineBlockDownloader] Writing block bodies block_num=19125416 remaining=8818 alloc=7.5GB sys=9.9GB

Seems like I am almost there - hope it helps some of you!

If it helps anything, I'm also including all warnings (collated) over the last 24 hours;

Jan 30 19:12:25 anon-r2 erigon[1882581]: [WARN] [01-30|19:12:25.141] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 30 20:08:08 anon-r2 erigon[1882581]: [WARN] [01-30|20:08:08.424] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 30 20:12:07 anon-r2 erigon[1882581]: [WARN] [01-30|20:12:07.117] error executing clstage                  app=caplin stage=DownloadHistoricalBlocks err="loadIntoTable Header: error inserting block during collection: ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, md>
Jan 30 21:03:18 anon-r2 erigon[1882581]: [WARN] [01-30|21:03:18.505] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 30 22:01:37 anon-r2 erigon[1882581]: [WARN] [01-30|22:01:37.094] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 30 22:47:01 anon-r2 erigon[1901784]: [WARN] [01-30|22:47:01.769] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 30 22:47:01 anon-r2 erigon[1901784]: [WARN] [01-30|22:47:01.776] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 30 23:43:09 anon-r2 erigon[1901784]: [WARN] [01-30|23:43:09.804] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 00:59:05 anon-r2 erigon[1901784]: [WARN] [01-31|00:59:05.311] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 02:01:58 anon-r2 erigon[1901784]: [WARN] [01-31|02:01:58.717] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 02:56:18 anon-r2 erigon[1901784]: [WARN] [01-31|02:56:18.808] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 03:37:22 anon-r2 erigon[1918214]: [WARN] [01-31|03:37:22.226] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 31 03:37:22 anon-r2 erigon[1918214]: [WARN] [01-31|03:37:22.233] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 31 03:39:06 anon-r2 erigon[1918214]: [WARN] [01-31|03:39:06.528] [downloader] Timeout                     requestId=0 peerID=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Jan 31 03:39:06 anon-r2 erigon[1918214]: [WARN] [01-31|03:39:06.532] [EngineBlockDownloader] Header download did not yield success
Jan 31 03:40:42 anon-r2 erigon[1918214]: [WARN] [01-31|03:40:42.641] [downloader] Timeout                     requestId=0 peerID=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Jan 31 03:40:42 anon-r2 erigon[1918214]: [WARN] [01-31|03:40:42.650] [EngineBlockDownloader] Header download did not yield success
Jan 31 04:47:38 anon-r2 erigon[1918214]: [WARN] [01-31|04:47:38.030] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 05:48:17 anon-r2 erigon[1918214]: [WARN] [01-31|05:48:17.797] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 06:24:53 anon-r2 erigon[1918214]: [WARN] [01-31|06:24:53.766] error executing clstage                  app=caplin stage=DownloadHistoricalBlocks err="loadIntoTable Header: error inserting block during collection: ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, md>
Jan 31 06:51:39 anon-r2 erigon[1918214]: [WARN] [01-31|06:51:39.202] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 07:25:16 anon-r2 erigon[1930164]: [WARN] [01-31|07:25:16.073] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 31 07:25:16 anon-r2 erigon[1930164]: [WARN] [01-31|07:25:16.080] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 31 09:07:46 anon-r2 erigon[1930164]: [WARN] [01-31|09:07:46.773] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 10:41:20 anon-r2 erigon[1930164]: [WARN] [01-31|10:41:20.995] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 11:53:14 anon-r2 erigon[1930164]: [WARN] [01-31|11:53:14.836] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 12:42:51 anon-r2 erigon[1930164]: [WARN] [01-31|12:42:51.558] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 13:30:29 anon-r2 erigon[1930164]: [WARN] [01-31|13:30:29.699] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 14:21:46 anon-r2 erigon[1953370]: [WARN] [01-31|14:21:46.007] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 31 14:21:46 anon-r2 erigon[1953370]: [WARN] [01-31|14:21:46.014] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 31 15:28:11 anon-r2 erigon[1953370]: [WARN] [01-31|15:28:11.010] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 17:21:28 anon-r2 erigon[1953370]: [WARN] [01-31|17:21:28.162] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 18:28:48 anon-r2 erigon[1953370]: [WARN] [01-31|18:28:48.391] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 19:33:27 anon-r2 erigon[1953370]: [WARN] [01-31|19:33:27.081] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 20:08:05 anon-r2 erigon[1973793]: [WARN] [01-31|20:08:05.324] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 31 20:08:05 anon-r2 erigon[1973793]: [WARN] [01-31|20:08:05.333] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Jan 31 20:09:40 anon-r2 erigon[1973793]: [WARN] [01-31|20:09:40.424] [downloader] Timeout                     requestId=0 peerID=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Jan 31 20:09:40 anon-r2 erigon[1973793]: [WARN] [01-31|20:09:40.429] [EngineBlockDownloader] Header download did not yield success
Jan 31 21:09:31 anon-r2 erigon[1973793]: [WARN] [01-31|21:09:31.686] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 22:07:11 anon-r2 erigon[1973793]: [WARN] [01-31|22:07:11.616] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 23:09:43 anon-r2 erigon[1973793]: [WARN] [01-31|23:09:43.067] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Jan 31 23:20:54 anon-r2 erigon[1973793]: [WARN] [01-31|23:20:54.882] error executing clstage                  app=caplin stage=DownloadHistoricalBlocks err="loadIntoTable Header: error inserting block during collection: ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, md>
Feb 01 00:14:48 anon-r2 erigon[1973793]: [WARN] [02-01|00:14:48.449] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 00:50:20 anon-r2 erigon[1990484]: [WARN] [02-01|00:50:20.803] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 01 00:50:20 anon-r2 erigon[1990484]: [WARN] [02-01|00:50:20.816] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 01 01:49:42 anon-r2 erigon[1990484]: [WARN] [02-01|01:49:42.343] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 02:19:46 anon-r2 erigon[1990484]: [WARN] [02-01|02:19:46.894] [EngineBlockDownloader] Could not download bodies err=stopped
Feb 01 02:19:46 anon-r2 erigon[1990484]: [WARN] [02-01|02:19:46.894] [Beacon Gossip] Fatal error receiving gossip err=EOF
Feb 01 02:19:47 anon-r2 erigon[1990484]: [WARN] [02-01|02:19:47.544] Found error while processing packet      err="db closed"
Feb 01 02:20:39 anon-r2 erigon[1995892]: [WARN] [02-01|02:20:39.685] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 01 02:20:39 anon-r2 erigon[1995892]: [WARN] [02-01|02:20:39.688] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 01 02:22:18 anon-r2 erigon[1995892]: [WARN] [02-01|02:22:18.532] [downloader] Timeout                     requestId=0 peerID=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Feb 01 02:22:18 anon-r2 erigon[1995892]: [WARN] [02-01|02:22:18.542] [EngineBlockDownloader] Header download did not yield success
Feb 01 03:31:02 anon-r2 erigon[1995892]: [WARN] [02-01|03:31:02.932] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 04:21:55 anon-r2 erigon[1995892]: [WARN] [02-01|04:21:55.596] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 05:20:03 anon-r2 erigon[1995892]: [WARN] [02-01|05:20:03.278] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 06:20:01 anon-r2 erigon[1995892]: [WARN] [02-01|06:20:01.580] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 07:03:20 anon-r2 erigon[2012722]: [WARN] [02-01|07:03:20.077] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 01 07:03:20 anon-r2 erigon[2012722]: [WARN] [02-01|07:03:20.085] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 01 07:04:53 anon-r2 erigon[2012722]: [WARN] [02-01|07:04:53.455] [downloader] Timeout                     requestId=0 peerID=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Feb 01 07:04:53 anon-r2 erigon[2012722]: [WARN] [02-01|07:04:53.459] [EngineBlockDownloader] Header download did not yield success
Feb 01 07:06:30 anon-r2 erigon[2012722]: [WARN] [02-01|07:06:30.552] [downloader] Timeout                     requestId=0 peerID=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Feb 01 07:06:30 anon-r2 erigon[2012722]: [WARN] [02-01|07:06:30.562] [EngineBlockDownloader] Header download did not yield success
Feb 01 07:08:07 anon-r2 erigon[2012722]: [WARN] [02-01|07:08:07.018] [downloader] Timeout                     requestId=0 peerID=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Feb 01 07:08:07 anon-r2 erigon[2012722]: [WARN] [02-01|07:08:07.018] [EngineBlockDownloader] Header download did not yield success
Feb 01 08:56:29 anon-r2 erigon[2012722]: [WARN] [02-01|08:56:29.073] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 10:15:05 anon-r2 erigon[2012722]: [WARN] [02-01|10:15:05.445] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 11:31:15 anon-r2 erigon[2012722]: [WARN] [02-01|11:31:15.800] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 12:54:53 anon-r2 erigon[2012722]: [WARN] [02-01|12:54:53.915] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cur>
Feb 01 13:34:39 anon-r2 erigon[2035426]: [WARN] [02-01|13:34:39.820] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 01 13:34:39 anon-r2 erigon[2035426]: [WARN] [02-01|13:34:39.829] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"

Erigon service has been 9/KILL'd twice in the last 24 hours. Sync is back to 18299999

Feb 02 15:47:23 anon-r2 erigon[2120332]: [INFO] [02-02|15:47:23.498] [txpool] stat                            pending=4344 baseFee=0 queued=30000 alloc=6.7GB sys=10.3GB
Feb 02 15:47:24 anon-r2 erigon[2120332]: [INFO] [02-02|15:47:24.761] [NewPayload] Handling new payload        height=19141696 hash=0x50d2758ddd0ccecff772e8249514488388db29f90ee1bdb39d4abeb506db1cb7
Feb 02 15:47:28 anon-r2 erigon[2120332]: [INFO] [02-02|15:47:28.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8245049 blockNumber=19048983 blk/sec=8.5 mbps/sec=1.7254 peers=11 snapshots=0 reconnected=false
Feb 02 15:47:36 anon-r2 erigon[2120332]: [INFO] [02-02|15:47:36.282] [EngineBlockDownloader] Downloading block bodies block_num=19075858 delivery/sec=8.7MB wasted/sec=1.0MB remaining=65662 delivered=108326 cache=47.9MB alloc=8.0GB sys=10.3GB
Feb 02 15:47:36 anon-r2 erigon[2120332]: [INFO] [02-02|15:47:36.684] [NewPayload] Handling new payload        height=19141697 hash=0x48d9069bf3dfdf60dec28ad0d5e89ffdf02eb3657dc3dcdca10423c4c70d0c80
Feb 02 15:47:48 anon-r2 erigon[2120332]: [INFO] [02-02|15:47:48.846] [NewPayload] Handling new payload        height=19141698 hash=0x0b2c06d6cd8cc2541af3f1f2c25db39e818af827423793f020bdf0519349779a
Feb 02 15:47:58 anon-r2 erigon[2120332]: [INFO] [02-02|15:47:58.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8244601 blockNumber=19048537 blk/sec=14.9 mbps/sec=3.3341 peers=5 snapshots=0 reconnected=false
Feb 02 15:48:00 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:00.644] [NewPayload] Handling new payload        height=19141699 hash=0xc8141e6c30e608e74db4076383e7b7d2fed993eddec47118906af2585cbd0089
Feb 02 15:48:06 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:06.281] [EngineBlockDownloader] Downloading block bodies block_num=19076932 delivery/sec=7.2MB wasted/sec=2.0MB remaining=64559 delivered=109579 cache=72.1MB alloc=4.6GB sys=10.3GB
Feb 02 15:48:08 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:08.521] P2P                                      app=caplin peers=4
Feb 02 15:48:13 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:13.099] [NewPayload] Handling new payload        height=19141700 hash=0xf360ee26004ba5002813ec7c5bb68fd79cba041cb356fb3229a5addbc68e11a8
Feb 02 15:48:25 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:25.186] [NewPayload] Handling new payload        height=19141701 hash=0xa77b53f07fdbf1131e86a525f9797689471a0c08725d9c74c4640b305ca40565
Feb 02 15:48:28 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:28.738] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8243865 blockNumber=19047805 blk/sec=24.5 mbps/sec=5.3326 peers=2 snapshots=0 reconnected=false
Feb 02 15:48:36 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:36.283] [EngineBlockDownloader] Downloading block bodies block_num=19078308 delivery/sec=6.1MB wasted/sec=1013.0KB remaining=63183 delivered=110718 cache=27.7MB alloc=6.1GB sys=10.3GB
Feb 02 15:48:37 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:37.682] [NewPayload] Handling new payload        height=19141702 hash=0x560766d9c9c51a202e9be61ef0cdcbc2257e28729d5789fb3b327fcb4059a823
Feb 02 15:48:50 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:50.173] [NewPayload] Handling new payload        height=19141703 hash=0xf18f20508fcb10433cc966483e5e253158980eac658f6fd4b48df09f783d5ba5
Feb 02 15:48:58 anon-r2 erigon[2120332]: [INFO] [02-02|15:48:58.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8243097 blockNumber=19047039 blk/sec=25.6 mbps/sec=4.9433 peers=10 snapshots=0 reconnected=false
Feb 02 15:49:01 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:01.012] [NewPayload] Handling new payload        height=19141704 hash=0x087c5af2e240fa41d06216d249fbc03bc97b1cb08654455fb1d3ab91b76beb5d
Feb 02 15:49:06 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:06.282] [EngineBlockDownloader] Downloading block bodies block_num=19079564 delivery/sec=6.4MB wasted/sec=1.4MB remaining=61927 delivered=112003 cache=36.0MB alloc=7.0GB sys=10.3GB
Feb 02 15:49:08 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:08.522] P2P                                      app=caplin peers=0
Feb 02 15:49:13 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:13.313] [NewPayload] Handling new payload        height=19141705 hash=0x227d34ec7d45b8bd41fd0dac4f1f7bf248461544b235957242684c471d74ceb0
Feb 02 15:49:24 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:24.979] [NewPayload] Handling new payload        height=19141706 hash=0x62a14a7e5de731f513d619dfd76417a3a62077d5bb3209217966bfba8f35acfd
Feb 02 15:49:28 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:28.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8242713 blockNumber=19046655 blk/sec=12.8 mbps/sec=2.5175 peers=5 snapshots=0 reconnected=false
Feb 02 15:49:36 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:36.284] [EngineBlockDownloader] Downloading block bodies block_num=19080861 delivery/sec=5.6MB wasted/sec=1.8MB remaining=60630 delivered=113186 cache=18.1MB alloc=6.4GB sys=10.3GB
Feb 02 15:49:36 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:36.888] [NewPayload] Handling new payload        height=19141707 hash=0xa258e22915d9eeba488810e8c090e68170e0580a7cde34b34ce4a2633eef67a8
Feb 02 15:49:48 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:48.341] [NewPayload] Handling new payload        height=19141708 hash=0x8032535e51b98b672c35c70eca043408c433f6ec46a54bc6b5b4734fbb22e5a8
Feb 02 15:49:58 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:58.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8242297 blockNumber=19046239 blk/sec=13.9 mbps/sec=2.4993 peers=2 snapshots=0 reconnected=false
Feb 02 15:49:59 anon-r2 erigon[2120332]: [INFO] [02-02|15:49:59.095] [p2p] GoodPeers                          eth68=32 eth67=33
Feb 02 15:50:01 anon-r2 erigon[2120332]: [INFO] [02-02|15:50:01.247] [NewPayload] Handling new payload        height=19141709 hash=0x602065e3c52cf08d73fc43c14eb2526e79c81272b934ddf66f61535e6e6cc942
Feb 02 15:50:06 anon-r2 erigon[2120332]: [INFO] [02-02|15:50:06.281] [EngineBlockDownloader] Downloading block bodies block_num=19082181 delivery/sec=6.8MB wasted/sec=942.1KB remaining=59322 delivered=114568 cache=27.5MB alloc=5.9GB sys=10.3GB
Feb 02 15:50:08 anon-r2 erigon[2120332]: [INFO] [02-02|15:50:08.522] P2P                                      app=caplin peers=6
Feb 02 15:50:12 anon-r2 erigon[2120332]: [INFO] [02-02|15:50:12.736] [NewPayload] Handling new payload        height=19141710 hash=0xbaaf4ba7741d8971823b89ba80931920ad8319ea9aebc017390e60b30463e319
Feb 02 15:50:23 anon-r2 erigon[2120332]: [INFO] [02-02|15:50:23.496] [txpool] stat                            pending=4402 baseFee=0 queued=30000 alloc=4.7GB sys=10.3GB
Feb 02 15:50:28 anon-r2 erigon[2120332]: [INFO] [02-02|15:50:28.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8241913 blockNumber=19045856 blk/sec=12.8 mbps/sec=2.5913 peers=3 snapshots=0 reconnected=false
Feb 02 15:50:36 anon-r2 erigon[2120332]: [INFO] [02-02|15:50:36.284] [EngineBlockDownloader] Downloading block bodies block_num=19083613 delivery/sec=6.2MB wasted/sec=1001.2KB remaining=57878 delivered=115930 cache=17.4MB alloc=5.8GB sys=10.3GB
Feb 02 15:50:58 anon-r2 erigon[2120332]: [INFO] [02-02|15:50:58.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8241593 blockNumber=19045538 blk/sec=10.7 mbps/sec=2.1794 peers=5 snapshots=0 reconnected=false
Feb 02 15:51:06 anon-r2 erigon[2120332]: [INFO] [02-02|15:51:06.281] [EngineBlockDownloader] Downloading block bodies block_num=19084841 delivery/sec=6.8MB wasted/sec=1.3MB remaining=56650 delivered=117378 cache=47.5MB alloc=4.3GB sys=10.3GB
Feb 02 15:51:08 anon-r2 erigon[2120332]: [INFO] [02-02|15:51:08.522] P2P                                      app=caplin peers=2
Feb 02 15:51:28 anon-r2 erigon[2120332]: [INFO] [02-02|15:51:28.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8241177 blockNumber=19045126 blk/sec=13.9 mbps/sec=2.4199 peers=8 snapshots=0 reconnected=false
Feb 02 15:51:36 anon-r2 erigon[2120332]: [INFO] [02-02|15:51:36.282] [EngineBlockDownloader] Downloading block bodies block_num=19086523 delivery/sec=9.6MB wasted/sec=1.5MB remaining=54979 delivered=119191 cache=76.7MB alloc=4.6GB sys=10.3GB
Feb 02 15:51:58 anon-r2 erigon[2120332]: [INFO] [02-02|15:51:58.735] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8240857 blockNumber=19044806 blk/sec=10.7 mbps/sec=2.1508 peers=2 snapshots=0 reconnected=false
Feb 02 15:52:05 anon-r2 erigon[2120332]: [INFO] [02-02|15:52:05.649] [NewPayload] Handling new payload        height=19141711 hash=0x417257c7208a1f81259694129f2cb0750a995e601bd9736c160113bd68634bc9
Feb 02 15:52:06 anon-r2 erigon[2120332]: [INFO] [02-02|15:52:06.098] [NewPayload] Handling new payload        height=19141712 hash=0x7c498f24bbac06563aa6aea7b9df839e2ac197c9adb2d0c569630b31f7d6e03f
Feb 02 15:52:06 anon-r2 erigon[2120332]: [INFO] [02-02|15:52:06.282] [EngineBlockDownloader] Downloading block bodies block_num=19088157 delivery/sec=7.4MB wasted/sec=1.7MB remaining=53334 delivered=120656 cache=43.5MB alloc=7.4GB sys=10.3GB
Feb 02 15:52:06 anon-r2 erigon[2120332]: [INFO] [02-02|15:52:06.580] [NewPayload] Handling new payload        height=19141713 hash=0x640e34db12cbe3cbcd743024ed68094f6c2895e5a479381c75a3c95a824f415a
Feb 02 15:52:08 anon-r2 erigon[2120332]: [INFO] [02-02|15:52:08.343] [NewPayload] Handling new payload        height=19141714 hash=0x7e8d86dfaba75c240a5b39a068b90f9d978e8014b438a4e3cf2fde67a52297c7
Feb 02 15:52:08 anon-r2 erigon[2120332]: [INFO] [02-02|15:52:08.522] P2P                                      app=caplin peers=3
Feb 02 15:52:08 anon-r2 erigon[2120332]: [INFO] [02-02|15:52:08.832] [NewPayload] Handling new payload        height=19141715 hash=0x48de07d836dbe202e2ec44b2ecfc716da36a46769fd87600362743e1afbe5085
Feb 02 15:52:09 anon-r2 erigon[2120332]: [INFO] [02-02|15:52:09.899] [NewPayload] Handling new payload        height=19141716 hash=0x3400c2eb18fd90ee438a51a550941ad2ddd5d7bf93d6e3353d72af267a36d373
Feb 02 15:52:14 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Feb 02 15:52:14 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.
Feb 02 15:52:14 anon-r2 systemd[1]: erigon.service: Consumed 7h 20.487s CPU time.
Feb 02 15:52:59 anon-r2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 4.
Feb 02 15:52:59 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Feb 02 15:52:59 anon-r2 systemd[1]: erigon.service: Consumed 7h 20.487s CPU time.
Feb 02 15:52:59 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Feb 02 15:53:00 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:00.353] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Feb 02 15:53:00 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:00.353] Build info                               git_branch=HEAD git_tag=v2.56.1 git_commit=d079008d5901117410e2d471a591748b16720966
Feb 02 15:53:00 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:00.353] Starting Erigon on Ethereum mainnet...
Feb 02 15:53:00 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:00.359] Maximum peer count                       ETH=100 total=100
Feb 02 15:53:00 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:00.359] starting HTTP APIs                       APIs=web3,net,eth
Feb 02 15:53:00 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:00.359] torrent verbosity                        level=WRN
Feb 02 15:53:02 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:02.464] Set global gas cap                       cap=50000000
Feb 02 15:53:02 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:02.577] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=150mb upload.rate=15mb
Feb 02 15:53:04 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:04.198] Opening Database                         label=chaindata path=/storage/ethereum/erigon/data/chaindata
Feb 02 15:53:04 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:04.203] [db] open                                lable=chaindata sizeLimit=12TB pageSize=8192
Feb 02 15:53:04 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:04.211] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000>
Feb 02 15:53:37 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:37.863] Initialising Ethereum protocol           network=1
Feb 02 15:53:37 anon-r2 erigon[2149161]: [INFO] [02-02|15:53:37.864] Disk storage enabled for ethash DAGs     dir=/storage/ethereum/erigon/data/ethash-dags count=2
Feb 02 15:54:01 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:01.941] Starting private RPC server              on=192.168.2.3:9191
Feb 02 15:54:01 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:01.941] new subscription to logs established
Feb 02 15:54:01 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:01.944] [Checkpoint Sync] Requesting beacon state uri=https://mainnet-checkpoint-sync.attestant.io/eth/v2/debug/beacon/states/finalized
Feb 02 15:54:02 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:02.043] [txpool] Started
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.979] [Sentinel] Sentinel started              enr=enr:-Ly4QK5A-BI22SUVMYWreoYBVCsptwEy7jz0ofs8jjkkdU2tFsnSakIFukZsK5uu3cc2rVv9uIG95yXO0QF2gMw8vnQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC7pNqWBAAAAP__________>
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.979] Static peers                             len=0
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.979] rpc filters: subscribing to Erigon events
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.980] New txs subscriber joined
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.980] new subscription to newHeaders established
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.981] JsonRpc endpoint opened                  ws=true ws.compression=false grpc=false websocket.url=192.168.2.3:8546 http.url=192.168.2.3:8545
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.981] Reading JWT secret                       path=/var/ethereum/jwtsecret
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.982] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=false
Feb 02 15:54:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:05.991] [1/12 Snapshots] Fetching torrent files metadata
Feb 02 15:54:06 anon-r2 erigon[2149161]: [WARN] [02-02|15:54:06.014] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 02 15:54:06 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:06.018] Started P2P networking                   version=67 self=enode://4fcb1da8d3e727376d2901100abb06db5730d8024c64e120ae25c539fd8dca7c30957c7e28c8aebedc75bf4771e8b1426f8711e5e75c9d50292a9eb354338ddd>
Feb 02 15:54:06 anon-r2 erigon[2149161]: [WARN] [02-02|15:54:06.028] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Feb 02 15:54:06 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:06.034] Started P2P networking                   version=68 self=enode://4fcb1da8d3e727376d2901100abb06db5730d8024c64e120ae25c539fd8dca7c30957c7e28c8aebedc75bf4771e8b1426f8711e5e75c9d50292a9eb354338ddd>
Feb 02 15:54:08 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:08.684] [snapshots:download] Blocks Stat         blocks=18300k indices=18300k alloc=3.3GB sys=5.6GB
Feb 02 15:54:08 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:08.688] [snapshots] Prune Blocks                 to=18209000 limit=100
Feb 02 15:54:11 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:11.596] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0xb2470b489e928b152c2824c531133fedb379c4d170f0f8d8106ebefeda12a245 requestId=0
Feb 02 15:54:45 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:45.148] Started Ethereum 2.0 Gossip Service      app=caplin
Feb 02 15:54:51 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:51.486] [Caplin] starting clstages loop          app=caplin
Feb 02 15:54:51 anon-r2 erigon[2149161]: [INFO] [02-02|15:54:51.486] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=8338656
Feb 02 15:55:21 anon-r2 erigon[2149161]: [INFO] [02-02|15:55:21.486] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8338192 blockNumber=19141156 blk/sec=15.5 mbps/sec=2.7020 peers=7 snapshots=0 reconnected=false
Feb 02 15:55:45 anon-r2 erigon[2149161]: [INFO] [02-02|15:55:45.149] P2P                                      app=caplin peers=6
Feb 02 15:55:51 anon-r2 erigon[2149161]: [INFO] [02-02|15:55:51.486] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8337889 blockNumber=19140858 blk/sec=10.1 mbps/sec=1.8761 peers=4 snapshots=0 reconnected=false
Feb 02 15:56:05 anon-r2 erigon[2149161]: [INFO] [02-02|15:56:05.980] [downloader] Downloaded PoS Headers      now=19083936 blk/sec=1081.600
Feb 02 15:56:21 anon-r2 erigon[2149161]: [INFO] [02-02|15:56:21.487] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8337473 blockNumber=19140448 blk/sec=13.9 mbps/sec=2.4221 peers=7 snapshots=0 reconnected=false
Feb 02 15:56:35 anon-r2 erigon[2149161]: [INFO] [02-02|15:56:35.979] [downloader] Downloaded PoS Headers      now=19051872 blk/sec=1068.800
Feb 02 15:56:37 anon-r2 erigon[2149161]: [INFO] [02-02|15:56:37.852] [p2p] GoodPeers                          eth68=3 eth67=2
Feb 02 15:56:45 anon-r2 erigon[2149161]: [INFO] [02-02|15:56:45.148] P2P                                      app=caplin peers=11
Feb 02 15:56:51 anon-r2 erigon[2149161]: [INFO] [02-02|15:56:51.486] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8336897 blockNumber=19139877 blk/sec=19.2 mbps/sec=3.3840 peers=15 snapshots=0 reconnected=false
Feb 02 15:57:01 anon-r2 erigon[2149161]: [INFO] [02-02|15:57:01.942] [txpool] stat                            pending=47 baseFee=0 queued=2216 alloc=3.7GB sys=7.5GB
Feb 02 15:57:06 anon-r2 erigon[2149161]: [INFO] [02-02|15:57:06.018] [downloader] Downloaded PoS Headers      now=19019424 blk/sec=1081.600
Feb 02 15:57:21 anon-r2 erigon[2149161]: [INFO] [02-02|15:57:21.486] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8336474 blockNumber=19139459 blk/sec=14.1 mbps/sec=2.5667 peers=15 snapshots=0 reconnected=false
Feb 02 15:57:35 anon-r2 erigon[2149161]: [INFO] [02-02|15:57:35.979] [downloader] Downloaded PoS Headers      now=18987936 blk/sec=1049.600
Feb 02 15:57:45 anon-r2 erigon[2149161]: [INFO] [02-02|15:57:45.149] P2P                                      app=caplin peers=6
Feb 02 15:57:51 anon-r2 erigon[2149161]: [INFO] [02-02|15:57:51.487] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8335969 blockNumber=19138967 blk/sec=16.8 mbps/sec=3.0846 peers=3 snapshots=0 reconnected=false
Feb 02 15:57:59 anon-r2 erigon[2149161]: [INFO] [02-02|15:57:59.662] [EngineBlockDownloader] Processing bodies... from=18967799 to=19141728
Feb 02 15:58:00 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:00.858] [NewPayload] Handling new payload        height=19141749 hash=0x58a06ff3cb676cafecf36a2025414a2a4384216dde2f6d29b3719bf90fc25d06
Feb 02 15:58:12 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:12.542] [NewPayload] Handling new payload        height=19141750 hash=0x8c7d41fda0b9be4a7aab11f7390d5087e578d3da19bbbb93be6a9204fb675360
Feb 02 15:58:21 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:21.487] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8335461 blockNumber=19138462 blk/sec=16.9 mbps/sec=2.7959 peers=19 snapshots=0 reconnected=false
Feb 02 15:58:24 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:24.903] [NewPayload] Handling new payload        height=19141751 hash=0x16cd0e1126fc2217f6e1a9ef63f9ea59af28fab26b318e0439cb0e02f3d5cd5d
Feb 02 15:58:29 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:29.680] [EngineBlockDownloader] Downloading block bodies block_num=18968719 delivery/sec=6.0MB wasted/sec=143.3KB remaining=173022 delivered=1052 cache=21.3MB alloc=6.8GB sys=7.8GB
Feb 02 15:58:36 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:36.469] [NewPayload] Handling new payload        height=19141752 hash=0x49e0b86f602696fbb613ee2d04a82aab9a31713a286efb465c0d72005979b601
Feb 02 15:58:45 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:45.149] P2P                                      app=caplin peers=12
Feb 02 15:58:48 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:48.987] [NewPayload] Handling new payload        height=19141753 hash=0x21079f070343a7733dae95aae74e2e483d2b7efd3596e5dbabc439d9718023d2
Feb 02 15:58:51 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:51.486] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8334978 blockNumber=19137984 blk/sec=16.1 mbps/sec=2.6672 peers=22 snapshots=0 reconnected=false
Feb 02 15:58:59 anon-r2 erigon[2149161]: [INFO] [02-02|15:58:59.663] [EngineBlockDownloader] Downloading block bodies block_num=18969867 delivery/sec=7.9MB wasted/sec=627.7KB remaining=171868 delivered=2396 cache=55.8MB alloc=4.7GB sys=7.8GB
Feb 02 15:59:01 anon-r2 erigon[2149161]: [INFO] [02-02|15:59:01.126] [NewPayload] Handling new payload        height=19141754 hash=0x455345cf270c059e09427b9f557951eba112a57617424b35b2b0843e5962a8fd
Feb 02 15:59:13 anon-r2 erigon[2149161]: [INFO] [02-02|15:59:13.626] [NewPayload] Handling new payload        height=19141755 hash=0xe8aa929efe93ac3484b88836ee101c6f2e3271e94dc485902137726d21f554d6

erigontech / erigon