search

allada / bsc-archive-snapshot

Free public Binance Smart Chain (BSC) Archive Snapshot
Apache License 2.0
112 stars 25 forks source link

Sync slow #41

Closed naviocean closed 1 year ago

naviocean commented 1 year ago

Hi @allada, It takes 2 days but can not catchup to the latest blocks. Do you have any suggestions?

erigon_1      | [INFO] [11-12|16:08:04.971] [7/16 Execution] Executed blocks         number=22978706 blk/s=2.1 tx/s=234.1 Mgas/s=30.1 gasState=0.15 batch=192.5MB alloc=3.1GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:08:23.074] [7/16 Execution] Executed blocks         number=22978728 blk/s=1.2 tx/s=180.5 Mgas/s=21.8 gasState=0.15 batch=193.4MB alloc=3.3GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:08:43.412] [7/16 Execution] Executed blocks         number=22978764 blk/s=1.8 tx/s=212.0 Mgas/s=25.4 gasState=0.15 batch=194.6MB alloc=3.5GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:09:03.094] [7/16 Execution] Executed blocks         number=22978795 blk/s=1.6 tx/s=176.7 Mgas/s=25.3 gasState=0.15 batch=195.9MB alloc=3.7GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:09:23.249] [7/16 Execution] Executed blocks         number=22978847 blk/s=2.6 tx/s=280.3 Mgas/s=34.6 gasState=0.15 batch=197.5MB alloc=4.0GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:09:43.442] [7/16 Execution] Executed blocks         number=22978896 blk/s=2.4 tx/s=265.1 Mgas/s=33.1 gasState=0.15 batch=199.0MB alloc=4.3GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:10:03.030] [7/16 Execution] Executed blocks         number=22978949 blk/s=2.7 tx/s=277.3 Mgas/s=32.5 gasState=0.15 batch=200.6MB alloc=3.1GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:10:04.196] [p2p] GoodPeers                          eth66=91
erigon_1      | [INFO] [11-12|16:10:23.159] [7/16 Execution] Executed blocks         number=22979002 blk/s=2.6 tx/s=264.7 Mgas/s=32.4 gasState=0.15 batch=202.3MB alloc=3.4GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:10:43.320] [7/16 Execution] Executed blocks         number=22979040 blk/s=1.9 tx/s=226.1 Mgas/s=28.0 gasState=0.15 batch=203.6MB alloc=3.8GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:11:02.982] [7/16 Execution] Executed blocks         number=22979069 blk/s=1.5 tx/s=181.8 Mgas/s=23.9 gasState=0.15 batch=204.8MB alloc=4.0GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:11:23.067] [7/16 Execution] Executed blocks         number=22979108 blk/s=1.9 tx/s=197.1 Mgas/s=28.2 gasState=0.16 batch=206.2MB alloc=4.2GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:11:44.505] [7/16 Execution] Executed blocks         number=22979147 blk/s=1.8 tx/s=181.0 Mgas/s=27.1 gasState=0.16 batch=207.7MB alloc=4.5GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:12:04.196] [p2p] GoodPeers                          eth66=91
erigon_1      | [INFO] [11-12|16:12:05.067] [7/16 Execution] Executed blocks         number=22979189 blk/s=2.0 tx/s=214.0 Mgas/s=29.2 gasState=0.16 batch=209.2MB alloc=3.3GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:12:23.257] [7/16 Execution] Executed blocks         number=22979226 blk/s=2.0 tx/s=257.4 Mgas/s=31.6 gasState=0.16 batch=210.5MB alloc=3.5GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:12:46.150] [7/16 Execution] Executed blocks         number=22979273 blk/s=2.1 tx/s=248.1 Mgas/s=33.1 gasState=0.16 batch=212.3MB alloc=3.8GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:13:03.174] [7/16 Execution] Executed blocks         number=22979312 blk/s=2.3 tx/s=268.0 Mgas/s=34.5 gasState=0.16 batch=213.7MB alloc=4.0GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:13:23.197] [7/16 Execution] Executed blocks         number=22979355 blk/s=2.1 tx/s=308.4 Mgas/s=37.8 gasState=0.16 batch=215.2MB alloc=4.4GB sys=8.1GB
erigon_1      | [INFO] [11-12|16:13:43.064] [7/16 Execution] Executed blocks         number=22979395 blk/s=2.0 tx/s=228.8 Mgas/s=31.5 gasState=0.16 batch=216.8MB alloc=4.6GB sys=8.1GB
allada commented 1 year ago

It looks like you need to run it on better hardware. More memory and faster disk latency. SSDs can barely keep up, so use NVMSs instead.

naviocean commented 1 year ago

@allada I'm using SSD with 7500 IOPS, 16. Do I need to update?

allada commented 1 year ago

Probably... Check your disk IO, it's probably maxed reading. Don't use EBS or other non ephemeral storage, the latency is too high.

You can also try using EXT4 instead of zfs, this sometimes gives better performance, but you should be getting closer to 8 to 14 blocks per second.

naviocean commented 1 year ago

@allada I've terminated all and try again. Could you please confirm the size of data and snapshots are correct?

Screen Shot 2022-11-13 at 17 37 53

And how long to pass the snapshots indexing progress? the size of /temp is increasing.

erigon_1  | [INFO] [11-13|10:36:11.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h1m20s alloc=4.8GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:36:31.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h1m40s alloc=6.2GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:36:38.150] [p2p] GoodPeers                          eth66=90
erigon_1  | [INFO] [11-13|10:36:51.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h2m0s alloc=7.6GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:36:52.773] [index] write                            file=v1-012500-013000-transactions.idx
erigon_1  | [INFO] [11-13|10:37:11.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h2m20s alloc=8.3GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:37:31.190] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h2m40s alloc=9.1GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:37:51.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h3m0s alloc=5.7GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:38:11.198] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h3m20s alloc=7.0GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:38:31.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h3m40s alloc=9.0GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:38:38.150] [p2p] GoodPeers                          eth66=90
erigon_1  | [INFO] [11-13|10:38:51.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h4m0s alloc=6.9GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:39:11.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h4m20s alloc=9.0GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:39:31.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h4m40s alloc=5.9GB sys=11.9GB

I'm using the docker-compose for running Erigon

version: '3.9'

x-logging: &logging
  logging:
    driver: "json-file"
    options:
      max-size: "2m"
      max-file: "10"

services:
  erigon:
    image: thorax/erigon:v2.29.0
    command: |
      erigon --chain bsc
      --snapshots=true
      --db.pagesize=16k
      --datadir=/home/erigon/.local/share/erigon
      --metrics --metrics.addr=0.0.0.0 --metrics.port=6060
      --private.api.addr=0.0.0.0:9090
      --pprof --pprof.addr=0.0.0.0 --pprof.port=6061
      --torrent.port=42069 --torrent.upload.rate=15mb --torrent.download.rate=1024mb
      --authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwt.hex
      --txpool.disable
    volumes:
      - /erigon/data/mainnet:/home/erigon/.local/share/erigon
    ports:
      - 30303:30303/tcp
      - 30303:30303/udp
      - 30304:30304/tcp
      - 30304:30304/udp
      - 9090:9090
      - 8551:8551
    restart: unless-stopped
    mem_swappiness: 0
    <<: *logging
allada commented 1 year ago

Indexing usually takes a few hours. If you change the mtime of the .idx files it may allow you to skip this stage. The sizes look appropriate. The temp folder is used to store ETL data.

naviocean commented 1 year ago

@allada I faced a slow execution again

erigon_1      | [INFO] [11-13|17:51:24.477] [7/16 Execution] Executed blocks         number=22899922 blk/s=0.2 tx/s=25.7 Mgas/s=4.1 gasState=0.01 batch=15.2MB alloc=2.3GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:51:41.857] [7/16 Execution] Executed blocks         number=22899931 blk/s=0.5 tx/s=52.9 Mgas/s=6.5 gasState=0.01 batch=15.5MB alloc=2.3GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:52:01.290] [7/16 Execution] Executed blocks         number=22899938 blk/s=0.4 tx/s=69.9 Mgas/s=7.3 gasState=0.01 batch=15.8MB alloc=2.4GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:52:38.024] [7/16 Execution] Executed blocks         number=22899943 blk/s=0.1 tx/s=16.3 Mgas/s=3.6 gasState=0.01 batch=16.2MB alloc=2.5GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:52:42.239] [7/16 Execution] Executed blocks         number=22899946 blk/s=0.7 tx/s=106.5 Mgas/s=10.0 gasState=0.01 batch=16.3MB alloc=2.5GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:52:46.913] [p2p] GoodPeers                          eth66=24
erigon_1      | [INFO] [11-13|17:53:01.571] [7/16 Execution] Executed blocks         number=22899954 blk/s=0.4 tx/s=62.1 Mgas/s=6.6 gasState=0.01 batch=16.6MB alloc=2.2GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:53:50.678] [7/16 Execution] Executed blocks         number=22899964 blk/s=0.2 tx/s=29.2 Mgas/s=4.4 gasState=0.01 batch=17.1MB alloc=2.4GB sys=3.8GB

I've updated my SSD to 16000 IOPS and Max throughput 500, But execution speed is too low. Do you have any suggestion or tools to check the main of problem? I'm using Azure service instead of AWS

allada commented 1 year ago

You need to use an ephemeral disk to capitalize of the low latency.

You can also try creating many disks and raid them together (you'll need to then re-copy the data on the raid). This might help some.

naviocean commented 1 year ago

@allada thank you so much. I've set raid for 4 disks and the issue is solved. The speed of execution now is reached to 20. :)