search

erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.13k stars 1.11k forks source link

[Erigon 2.56.0] Caplin restarts DownloadHistoricalBlocks and doesnt save state #9170

Open AndrewMohawk opened 9 months ago

AndrewMohawk commented 9 months ago

System information

Erigon version: ./erigon --version

OS & Version: Ubuntu server 20.04

Commit hash:

Erigon Command (with flags/config): https://github.com/ledgerwatch/erigon/issues/9046#issuecomment-1874304946 Current docker-compose:

# Connections: erigon -> (sentries, downloader), rpcdaemon -> (erigon, txpool), txpool -> erigon
version: '2.2'

# Basic erigon's service
x-erigon-service: &default-erigon-service
  image: thorax/erigon:${TAG:-latest}
  pid: service:erigon # Use erigon's PID namespace. It's required to open Erigon's DB from another process (RPCDaemon local-mode)
  volumes_from: [ erigon ]
  restart: unless-stopped
  mem_swappiness: 0
  user: 1000:1000
services:
  erigon:
    image: thorax/erigon:${TAG:-latest}
    build:
      args:
        UID: 1000
        GID: 1000
      context: .
    command: |
      --private.api.addr=0.0.0.0:9090 --internalcl
      --sentry.api.addr=sentry:9091 --downloader.api.addr=downloader:9093 --txpool.disable
      --metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --pprof --pprof.addr=0.0.0.0 --pprof.port=6061
      --nat "extip:<mylocalip>"
      --authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwt.hex --datadir=/home/erigon/.local/share/erigon
      --http.api "eth,erigon,ots,web3,net,debug,trace,txpool"
      --verbosity 4
    ports: [ "4000:4000/udp", "6060:6060","6061:6061","9090:9090","7777:7777","4001:4001","8551:8551","40517:40517"]

    volumes:
      # It's ok to mount sub-dirs of "datadir" to different drives
      - "/mnt/nvme/erigon-data/:/home/erigon/.local/share/erigon"
    restart: unless-stopped
    mem_swappiness: 0
    user: 1000:1000

  sentry:
    <<: *default-erigon-service
    entrypoint: sentry
    command: --sentry.api.addr=0.0.0.0:9091 --datadir=/home/erigon/.local/share/erigon
    ports: [ "30303:30303/tcp", "30303:30303/udp","30304:30304/tcp", "30304:30304/udp", "9091:9091" ]

  downloader:
    <<: *default-erigon-service
    entrypoint: downloader
    command: --downloader.api.addr=0.0.0.0:9093 --datadir=/home/erigon/.local/share/erigon  --torrent.download.rate=5000mb  --torrent.maxpeers 300 --verbosity 4 --torrent.verbosity=4 --torrent.port 42069
    ports: [ "42069:42069/tcp", "42069:42069/udp", "9093:9093"  ]

  txpool:
    <<: *default-erigon-service
    entrypoint: txpool
    command: --private.api.addr=erigon:9090 --txpool.api.addr=0.0.0.0:9094 --sentry.api.addr=sentry:9091 --datadir=/home/erigon/.local/share/erigon
    ports: [ "9094:9094"  ]

  rpcdaemon:
    <<: *default-erigon-service
    entrypoint: rpcdaemon
    command: |
      --http.addr=0.0.0.0 --http.vhosts=any --http.corsdomain=* --ws
      --private.api.addr=erigon:9090 --txpool.api.addr=txpool:9094 --datadir=/home/erigon/.local/share/erigon
    ports: [ "8545:8545" ]

  prometheus:
    image: prom/prometheus:v2.47.2
    user: 1000:1000 # Uses erigon user from Dockerfile
    command: --log.level=warn --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=150d --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles

    volumes:
      - "/mnt/nvme/erigon-data/cmd/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml"
      - "/mnt/nvme/erigon-data/erigon-prometheus:/prometheus"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.2.1
    user: "472:0" # required for grafana version >= 7.3
    ports: [ "3000:3000" ]
    volumes:
      - "/mnt/nvme/erigon-data/cmd/prometheus/grafana.ini:/etc/grafana/grafana.ini"
      - "/mnt/nvme/erigon-data/cmd/prometheus/datasources:/etc/grafana/provisioning/datasources"
      - "/mnt/nvme/erigon-data/cmd/prometheus/dashboards:/etc/grafana/provisioning/dashboards"
      - "/mnt/nvme/erigon-data/erigon-grafana:/var/lib/grafana"
    restart: unless-stopped

Consensus Layer: --internalcl

Consensus Layer Command (with flags/config):

Chain/Network: mainnet

Expected behaviour

Caplin saves state for DownloadHistoricalBlocks stage

Actual behaviour

During the "DownloadHistoricalBlocks" stage if caplin is interrupted, restarting the service will start again even after multiple days :'(

Steps to reproduce the behaviour

Stop erigon Restart

Backtrace

[INFO] [01-09|02:26:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8041963 blockNumber=18847942 blk/sec=22.6 mbps/sec=4.8600 peers=6 snapshots=0 reconnected=false
[INFO] [01-09|02:27:13.056] P2P                                      app=caplin peers=6
[INFO] [01-09|02:27:21.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8041317 blockNumber=18847300 blk/sec=21.5 mbps/sec=4.3568 peers=4 snapshots=0 reconnected=false
[INFO] [01-09|02:27:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8040651 blockNumber=18846644 blk/sec=22.2 mbps/sec=4.2335 peers=6 snapshots=0 reconnected=false
[INFO] [01-09|02:28:13.055] P2P                                      app=caplin peers=5
[INFO] [01-09|02:28:21.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8039787 blockNumber=18845792 blk/sec=28.8 mbps/sec=5.4138 peers=4 snapshots=0 reconnected=false
[INFO] [01-09|02:28:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8038763 blockNumber=18844775 blk/sec=34.1 mbps/sec=6.3248 peers=8 snapshots=0 reconnected=false
[INFO] [01-09|02:29:13.055] P2P                                      app=caplin peers=5
[INFO] [01-09|02:29:21.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8037771 blockNumber=18843794 blk/sec=33.1 mbps/sec=5.7389 peers=9 snapshots=0 reconnected=false
[INFO] [01-09|02:29:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8036971 blockNumber=18843000 blk/sec=26.7 mbps/sec=4.8400 peers=7 snapshots=0 reconnected=false
[INFO] [01-09|02:30:13.055] P2P                                      app=caplin peers=3
[INFO] [01-09|02:30:21.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8035916 blockNumber=18841957 blk/sec=35.2 mbps/sec=6.7064 peers=7 snapshots=0 reconnected=false
[INFO] [01-09|02:30:51.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8034700 blockNumber=18840751 blk/sec=40.5 mbps/sec=8.1204 peers=6 snapshots=0 reconnected=false
[INFO] [01-09|02:31:13.056] P2P                                      app=caplin peers=2
[INFO] [01-09|02:31:21.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8033825 blockNumber=18839885 blk/sec=29.2 mbps/sec=5.9488 peers=9 snapshots=0 reconnected=false
[INFO] [01-09|02:31:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8033100 blockNumber=18839167 blk/sec=24.2 mbps/sec=4.6647 peers=4 snapshots=0 reconnected=false
[INFO] [01-09|02:32:13.056] P2P                                      app=caplin peers=10
[INFO] [01-09|02:32:21.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8032364 blockNumber=18838438 blk/sec=24.5 mbps/sec=4.3391 peers=6 snapshots=0 reconnected=false
[INFO] [01-09|02:32:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8032012 blockNumber=18838089 blk/sec=11.7 mbps/sec=1.9874 peers=3 snapshots=0 reconnected=false
[INFO] [01-09|02:33:13.056] P2P                                      app=caplin peers=5
[INFO] [01-09|02:33:21.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8031244 blockNumber=18837326 blk/sec=25.6 mbps/sec=4.2630 peers=4 snapshots=0 reconnected=false
[INFO] [01-09|02:33:51.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8030508 blockNumber=18836594 blk/sec=24.5 mbps/sec=4.1443 peers=3 snapshots=0 reconnected=false
[INFO] [01-09|02:34:13.055] P2P                                      app=caplin peers=5
[INFO] [01-09|02:34:21.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8029356 blockNumber=18835449 blk/sec=38.4 mbps/sec=6.4211 peers=6 snapshots=0 reconnected=false
[INFO] [01-09|02:34:51.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8027980 blockNumber=18834082 blk/sec=45.9 mbps/sec=8.4402 peers=5 snapshots=0 reconnected=false
[INFO] [01-09|02:35:13.056] P2P                                      app=caplin peers=2
[INFO] [01-09|02:35:21.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8027597 blockNumber=18833701 blk/sec=12.8 mbps/sec=2.3753 peers=2 snapshots=0 reconnected=false
[INFO] [01-09|02:35:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8027117 blockNumber=18833228 blk/sec=16.0 mbps/sec=3.0002 peers=5 snapshots=0 reconnected=false
[INFO] [01-09|02:36:13.056] P2P                                      app=caplin peers=5
[INFO] [01-09|02:36:21.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8026925 blockNumber=18833040 blk/sec=6.4 mbps/sec=1.2206 peers=5 snapshots=0 reconnected=false
[INFO] [01-09|02:36:51.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8026797 blockNumber=18832915 blk/sec=4.3 mbps/sec=0.7744 peers=2 snapshots=0 reconnected=false
[INFO] [01-09|02:37:13.056] P2P                                      app=caplin peers=4
[INFO] [01-09|02:37:21.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8026669 blockNumber=18832789 blk/sec=4.3 mbps/sec=0.7820 peers=5 snapshots=0 reconnected=false
[INFO] [01-09|02:37:51.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8026249 blockNumber=18832370 blk/sec=14.0 mbps/sec=2.6405 peers=9 snapshots=0 reconnected=false
[INFO] [01-09|02:38:13.055] P2P                                      app=caplin peers=9
[INFO] [01-09|02:38:21.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8025054 blockNumber=18831188 blk/sec=39.8 mbps/sec=6.7539 peers=9 snapshots=0 reconnected=false
[INFO] [01-09|02:38:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8024229 blockNumber=18830372 blk/sec=27.5 mbps/sec=6.3786 peers=3 snapshots=0 reconnected=false
[INFO] [01-09|02:39:13.055] P2P                                      app=caplin peers=3
[INFO] [01-09|02:39:21.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8022831 blockNumber=18828991 blk/sec=46.6 mbps/sec=10.9799 peers=3 snapshots=0 reconnected=false
[INFO] [01-09|02:39:51.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8022319 blockNumber=18828487 blk/sec=17.1 mbps/sec=2.9919 peers=5 snapshots=0 reconnected=false
[INFO] [01-09|02:40:13.055] P2P                                      app=caplin peers=9
[INFO] [01-09|02:40:21.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8021775 blockNumber=18827951 blk/sec=18.1 mbps/sec=3.3315 peers=3 snapshots=0 reconnected=false
[INFO] [01-09|02:40:51.931] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8021551 blockNumber=18827727 blk/sec=7.5 mbps/sec=1.3844 peers=3 snapshots=0 reconnected=false
[INFO] [01-09|02:41:13.055] P2P                                      app=caplin peers=9
[INFO] [01-09|02:41:21.932] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8020847 blockNumber=18827029 blk/sec=23.5 mbps/sec=4.3340 peers=5 snapshots=0 reconnected=false

System is Xeon Gold 6138 20C/40T 196 gb DDR4 2666 WD_BLACK 4TB SN850X NVMe Internal Gaming SSD Solid State Drive - Gen4 PCIe, M.2 2280 Ubuntu server 20.04 LTS

Giulio2002 commented 9 months ago

I think I have an easssyyyy solution for this, thanks for the report :)

AndrewMohawk commented 9 months ago

@Giulio2002 any idea of what I could do for now, I'm currently syncing these very slowly and worried something might happen and it will have to start all over again ;_; 

[INFO] [01-17|06:26:50.589] P2P                                      app=caplin peers=30
[INFO] [01-17|06:27:29.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6346678 blockNumber=17169277 blk/sec=1.1 mbps/sec=0.1136 peers=27 snapshots=0 reconnected=false
[INFO] [01-17|06:27:50.589] P2P                                      app=caplin peers=27
[INFO] [01-17|06:27:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6346678 blockNumber=17169277 blk/sec=0.0 mbps/sec=0.0000 peers=32 snapshots=0 reconnected=false
[INFO] [01-17|06:28:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6346678 blockNumber=17169277 blk/sec=0.0 mbps/sec=0.0000 peers=40 snapshots=0 reconnected=false
[INFO] [01-17|06:28:50.597] P2P                                      app=caplin peers=31
[INFO] [01-17|06:28:59.028] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6346678 blockNumber=17169277 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
benjamindamm commented 9 months ago

Same issue I am experiencing. No solution so far? Seems like there are no peers for the latest data.

Giulio2002 commented 9 months ago

It does remember it now, the no peer issue fix will be on the next release

AndrewMohawk commented 9 months ago

@Giulio2002 do you have any idea how long this will take? Mine has been on this one stage for 10 days, what does it need to get to to be completed? Is there a way I can save the state right now and then upgrade erigon and then restart from here?

[INFO] [01-24|03:41:50.589] P2P                                      app=caplin peers=26
[INFO] [01-24|03:41:59.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=1.1 mbps/sec=0.1317 peers=26 snapshots=0 reconnected=false
[INFO] [01-24|03:42:29.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=29 snapshots=0 reconnected=false
[INFO] [01-24|03:42:50.591] P2P                                      app=caplin peers=27
[INFO] [01-24|03:42:59.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
[INFO] [01-24|03:43:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=26 snapshots=0 reconnected=false
[INFO] [01-24|03:43:50.589] P2P                                      app=caplin peers=26
[INFO] [01-24|03:43:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=28 snapshots=0 reconnected=false
[INFO] [01-24|03:44:29.028] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=24 snapshots=0 reconnected=false
[INFO] [01-24|03:44:50.589] P2P                                      app=caplin peers=25
[INFO] [01-24|03:44:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=25 snapshots=0 reconnected=false
[INFO] [01-24|03:45:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=26 snapshots=0 reconnected=false
[INFO] [01-24|03:45:50.590] P2P                                      app=caplin peers=28
[INFO] [01-24|03:45:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
[INFO] [01-24|03:46:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
[INFO] [01-24|03:46:50.589] P2P                                      app=caplin peers=26
[INFO] [01-24|03:46:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
[INFO] [01-24|03:47:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=26 snapshots=0 reconnected=false
[INFO] [01-24|03:47:50.589] P2P                                      app=caplin peers=30
[INFO] [01-24|03:47:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=25 snapshots=0 reconnected=false
[INFO] [01-24|03:48:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=25 snapshots=0 reconnected=false
[INFO] [01-24|03:48:50.589] P2P                                      app=caplin peers=27
[INFO] [01-24|03:48:59.028] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
[INFO] [01-24|03:49:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
[INFO] [01-24|03:49:50.589] P2P                                      app=caplin peers=29
[INFO] [01-24|03:49:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=29 snapshots=0 reconnected=false
[INFO] [01-24|03:50:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=28 snapshots=0 reconnected=false
[INFO] [01-24|03:50:50.590] P2P                                      app=caplin peers=30
[INFO] [01-24|03:50:59.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=30 snapshots=0 reconnected=false
[INFO] [01-24|03:51:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
[INFO] [01-24|03:51:50.590] P2P                                      app=caplin peers=30
[INFO] [01-24|03:51:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=30 snapshots=0 reconnected=false
[INFO] [01-24|03:52:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=35 snapshots=0 reconnected=false
[INFO] [01-24|03:52:50.590] P2P                                      app=caplin peers=37
[INFO] [01-24|03:52:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
[INFO] [01-24|03:53:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
[INFO] [01-24|03:53:50.589] P2P                                      app=caplin peers=30
[INFO] [01-24|03:53:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
[INFO] [01-24|03:54:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=29 snapshots=0 reconnected=false
[INFO] [01-24|03:54:50.600] P2P                                      app=caplin peers=32
[INFO] [01-24|03:54:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=31 snapshots=0 reconnected=false
[INFO] [01-24|03:55:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=28 snapshots=0 reconnected=false
[INFO] [01-24|03:55:50.589] P2P                                      app=caplin peers=26
[INFO] [01-24|03:55:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
[INFO] [01-24|03:56:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=29 snapshots=0 reconnected=false
[INFO] [01-24|03:56:50.589] P2P                                      app=caplin peers=29
[INFO] [01-24|03:56:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
[INFO] [01-24|03:57:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
[INFO] [01-24|03:57:50.589] P2P                                      app=caplin peers=26
[INFO] [01-24|03:57:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=28 snapshots=0 reconnected=false
[INFO] [01-24|03:58:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=22 snapshots=0 reconnected=false
[INFO] [01-24|03:58:50.590] P2P                                      app=caplin peers=22
[INFO] [01-24|03:58:59.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=24 snapshots=0 reconnected=false
[INFO] [01-24|03:59:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=24 snapshots=0 reconnected=false
[INFO] [01-24|03:59:50.589] P2P                                      app=caplin peers=33
[INFO] [01-24|03:59:59.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=24 snapshots=0 reconnected=false
[INFO] [01-24|04:00:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=28 snapshots=0 reconnected=false
[INFO] [01-24|04:00:50.590] P2P                                      app=caplin peers=27
[INFO] [01-24|04:00:59.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=28 snapshots=0 reconnected=false
[INFO] [01-24|04:01:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
[INFO] [01-24|04:01:50.589] P2P                                      app=caplin peers=28
[INFO] [01-24|04:01:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=30 snapshots=0 reconnected=false
[INFO] [01-24|04:02:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=27 snapshots=0 reconnected=false
[INFO] [01-24|04:02:50.589] P2P                                      app=caplin peers=27
[INFO] [01-24|04:02:59.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=28 snapshots=0 reconnected=false
[INFO] [01-24|04:03:29.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=26 snapshots=0 reconnected=false
[INFO] [01-24|04:03:50.589] P2P                                      app=caplin peers=24
[INFO] [01-24|04:03:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=25 snapshots=0 reconnected=false
[INFO] [01-24|04:04:29.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=25 snapshots=0 reconnected=false
[INFO] [01-24|04:04:50.589] P2P                                      app=caplin peers=21
[INFO] [01-24|04:04:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=22 snapshots=0 reconnected=false
[INFO] [01-24|04:05:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=22 snapshots=0 reconnected=false
[INFO] [01-24|04:05:50.590] P2P                                      app=caplin peers=22
[INFO] [01-24|04:05:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=21 snapshots=0 reconnected=false
[INFO] [01-24|04:06:29.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294727 blockNumber=17117917 blk/sec=0.0 mbps/sec=0.0000 peers=19 snapshots=0 reconnected=false
[INFO] [01-24|04:06:50.590] P2P                                      app=caplin peers=19
[INFO] [01-24|04:06:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294695 blockNumber=17117885 blk/sec=1.1 mbps/sec=0.1415 peers=20 snapshots=0 reconnected=false
[INFO] [01-24|04:07:29.027] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294695 blockNumber=17117885 blk/sec=0.0 mbps/sec=0.0000 peers=21 snapshots=0 reconnected=false
[INFO] [01-24|04:07:50.590] P2P                                      app=caplin peers=32
[INFO] [01-24|04:07:59.026] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=6294695 blockNumber=17117885 blk/sec=0.0 mbps/sec=0.0000 peers=22 snapshots=0 reconnected=false

Eth syncing returns false and blocknumber hasnt moved in week and a half or so :( 

» curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' http://<rpc>:8545                                                                 
{"jsonrpc":"2.0","id":1,"result":"0x1173c5f"}

» curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' http://<rpc>:8545
{"jsonrpc":"2.0","id":1,"result":false}

System is Xeon Gold 6138 20C/40T 196 gb DDR4 2666 WD_BLACK 4TB SN850X NVMe Internal Gaming SSD Solid State Drive - Gen4 PCIe, M.2 2280 Ubuntu server 20.04 LTS

jasonyic commented 2 months ago

can this issue be reopened? we are still experiencing the same issue in v2.60.4. when node restarted. it always try to re-download historical blocks

[INFO] [08-11|23:27:50.099] [Caplin] starting clstages loop          app=caplin
[INFO] [08-11|23:27:50.099] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=9716128
[INFO] [08-11|23:27:53.994] [Antiquary]: Stopping Caplin to process historical indicies from=9699999 to=9699999
[INFO] [08-11|23:27:54.006] [Antiquary]: Restarting Caplin
[INFO] [08-11|23:28:20.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9714560 blockNumber=20508480 blk/sec=52.3 snapshots=9699999
[INFO] [08-11|23:28:47.518] P2P                                      app=caplin peers=59
[INFO] [08-11|23:28:50.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9712240 blockNumber=20508480 blk/sec=77.3 snapshots=9699999
[INFO] [08-11|23:29:20.099] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9709603 blockNumber=20508480 blk/sec=87.9 snapshots=9699999
[INFO] [08-11|23:29:39.804] [p2p] GoodPeers                          eth68=1
[INFO] [08-11|23:29:39.950] [mem] memory stats                       Rss=13.5GB Size=0B Pss=13.5GB SharedClean=3.1MB SharedDirty=0B PrivateClean=6.2GB PrivateDirty=7.3GB Referenced=13.5GB Anonymous=7.3GB Swap=0B alloc=7.1GB sys=7.5GB
[INFO] [08-11|23:29:47.518] P2P                                      app=caplin peers=64
[INFO] [08-11|23:29:50.099] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9707027 blockNumber=20508480 blk/sec=85.9 snapshots=9699999
[INFO] [08-11|23:30:20.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9704324 blockNumber=20508480 blk/sec=90.1 snapshots=9699999
[INFO] [08-11|23:30:47.517] P2P                                      app=caplin peers=61
[INFO] [08-11|23:30:50.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9701449 blockNumber=20508480 blk/sec=95.8 snapshots=9699999
[INFO] [08-11|23:31:20.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9698537 blockNumber=20508480 blk/sec=97.1 snapshots=9699999
[INFO] [08-11|23:31:47.517] P2P                                      app=caplin peers=61
[INFO] [08-11|23:31:50.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9695550 blockNumber=20508480 blk/sec=99.6 snapshots=9699999
[INFO] [08-11|23:32:20.099] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9692508 blockNumber=20508480 blk/sec=101.4 snapshots=9699999
[INFO] [08-11|23:32:39.804] [p2p] GoodPeers                          eth68=1
[INFO] [08-11|23:32:39.989] [mem] memory stats                       Rss=13.0GB Size=0B Pss=13.0GB SharedClean=9.0MB SharedDirty=0B PrivateClean=6.5GB PrivateDirty=6.5GB Referenced=13.0GB Anonymous=6.5GB Swap=0B alloc=5.2GB sys=7.6GB
[INFO] [08-11|23:32:47.517] P2P                                      app=caplin peers=64
[INFO] [08-11|23:32:50.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9689515 blockNumber=20508480 blk/sec=99.8 snapshots=9699999
[INFO] [08-11|23:33:20.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9686339 blockNumber=20508480 blk/sec=105.9 snapshots=9699999
[INFO] [08-11|23:33:47.518] P2P                                      app=caplin peers=65
[INFO] [08-11|23:33:50.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9683123 blockNumber=20508480 blk/sec=107.2 snapshots=9699999
[INFO] [08-11|23:34:20.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9679923 blockNumber=20508480 blk/sec=106.7 snapshots=9699999
[INFO] [08-11|23:34:47.517] P2P                                      app=caplin peers=60
[INFO] [08-11|23:34:50.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9676646 blockNumber=20508480 blk/sec=109.2 snapshots=9699999
[INFO] [08-11|23:35:20.100] Node is still syncing... downloading past blocks app=caplin stage=DownloadHistoricalBlocks slot=9673312 blockNumber=20508480 blk/sec=111.1 snapshots=9699999
[INFO] [08-11|23:35:39.805] [p2p] GoodPeers                          eth68=1
Giulio2002 commented 2 months ago

Ok, i think this is a different thing, I will investigate. my only question here is: what is the progress of the Erigon in previous run? did it manage to sync up in previous runs? progress is saved in bulk only at the end but not during download (this system will be improved in next major release of Erigon). But if the some block execution happened, this should not happen. you can also ls <datadir>/snapshots - that also will give me a better picture

jasonyic commented 2 months ago

@Giulio2002 thanks for re-opening this issue. the node was not fully synced in the last run and still has ~100 blocks leeway. here is the ls /data/snapshots. let me know if you need any further info. appreciate.

FYI: the node is running internal CL with the following command options:

erigon --datadir=/data --chain=mainnet --http --http.port=8545 --http.addr=0.0.0.0 --http.api=admin,eth,web3,net,debug,trace,txpool --http.corsdomain=* --http.vhosts=* --nat=extip:$(NAT_IP) --port=$(NAT_PORT) --metrics --metrics.addr=0.0.0.0 --metrics.port=9545 --config=/data/config.toml --db.size.limit=8TB --internalcl=true --maxpeers=100 --caplin.archive=true --beacon.api=beacon,builder,config,debug,events,node,validator --beacon.api.addr=0.0.0.0 --beacon.api.port=5051 --beacon.api.read.timeout=15 --txpool.disable=true --http.trace=false --snapshots=false

Screenshot 2024-08-12 at 1 29 20 PM

jasonyic commented 2 months ago

after remove - --caplin.archive=true and node did not re-download historical blocks for caplin. but will this node run as a full node and missing historical data since the point which I disabled the archive flag for caplin?