search

node-real / bsc-erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
122 stars 43 forks source link

Chapel failed to sync snapshot on v1.2.15 #495

Closed insider89 closed 2 months ago

insider89 commented 3 months ago

System information

Erigon version: 1.2.15

OS & Version: Linux(docker container)

Commit hash:

Erigon Command (with flags/config):

- --chain=chapel
        - --snapshots=true
        - --txpool.nolocals
        - --db.pagesize=16k
        - --private.api.addr=0.0.0.0:9090
        - --nat=extip:IP
        - --authrpc.vhosts=*
        - --authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwtsecret
        - --authrpc.addr=0.0.0.0
        - --datadir=/home/erigon/.local/share/erigon
        - --internalcl
        - --db.size.limit=15TB
        - --metrics
        - --port=30603
        - --p2p.allowed-ports=30603
        - --p2p.allowed-ports=30604
        - --p2p.allowed-ports=30605
        - --p2p.allowed-ports=30606
        - --metrics.addr=0.0.0.0
        - --metrics.port=6060
        - --torrent.download.rate=1000mb
        - --http.api=admin,net,eth,erigon,web3,net,debug,trace,txpool,engine,ots
        - --http.addr=0.0.0.0
        - --http.vhosts=*
        - --http.corsdomain=*
        - --rpc.batch.limit=1000
        - --bodies.cache=5G
        - --ws
        - --db.read.concurrency=1024
        - --rpc.batch.concurrency=64
        - --bootnodes=enode://7708d7a409ec4f47c0b0e6ee4939a7357fd7f8e23fe43b4c33090627bbdef65b9260a0836ded71ace68e842b8d1271a6c71fbbf200c0193ca423fe01431516b4@65.109.25.36:30623,enode://715b7d96f34fe295178017a7d7484434beb5e9c4fd97b15fb11ea20571a4ee0da9b0d377f3132c21ec2e91feaeafbdad5120a616d0caa567c3129d81b22ba593@67.235.115.91:30643,enode://6f805ff1e3eca98deb52d01460954d849c4439fa359679d6c43477546bd592304bce709811b4bc1a662169fe5004ee6f508d51fac529cebac78c7826a29a524f@65.109.25.36:30603,enode://a22aa97d2016613a407f2adc2a5fbb34800beae2f2cf10d9cc3802a58bac97109371cc8454d90b225d801b0abe5e90e5ad307487373329b14c232134146b3de0@67.235.115.91:30613,enode://0637d1e62026e0c8685b1db0ca1c767c78c95c3fab64abc468d1a64b12ca4b530b46b8f80c915aec96f74f7ffc5999e8ad6d1484476f420f0c10e3d42361914b@52.199.214.252:30311,enode://330d768f6de90e7825f0ea6fe59611ce9d50712e73547306846a9304663f9912bf1611037f7f90f21606242ded7fb476c7285cb7cd792836b8c0c5ef0365855c@18.181.52.189:30311,enode://df1e8eb59e42cad3c4551b2a53e31a7e55a2fdde1287babd1e94b0836550b489ba16c40932e4dacb16cba346bd442c432265a299c4aca63ee7bb0f832b9f45eb@52.51.80.128:30311,enode://0bd566a7fd136ecd19414a601bfdc530d5de161e3014033951dd603e72b1a8959eb5b70b06c87a5a75cbf45e4055c387d2a842bd6b1bd8b5041b3a61bab615cf@34.242.33.165:30311,enode://ecd664250ca19b1074dcfbfb48576a487cc18d052064222a363adacd2650f8e08fb3db9de7a7aecb48afa410eaeb3285e92e516ead01fb62598553aed91ee15e@3.209.122.123:30311,enode://665cf77ca26a8421cfe61a52ac312958308d4912e78ce8e0f61d6902e4494d4cc38f9b0dd1b23a427a7a5734e27e5d9729231426b06bb9c73b56a142f83f6b68@52.72.123.113:30311
        - --staticpeers=enode://7708d7a409ec4f47c0b0e6ee4939a7357fd7f8e23fe43b4c33090627bbdef65b9260a0836ded71ace68e842b8d1271a6c71fbbf200c0193ca423fe01431516b4@65.109.25.36:30623,enode://715b7d96f34fe295178017a7d7484434beb5e9c4fd97b15fb11ea20571a4ee0da9b0d377f3132c21ec2e91feaeafbdad5120a616d0caa567c3129d81b22ba593@67.235.115.91:30643,enode://6f805ff1e3eca98deb52d01460954d849c4439fa359679d6c43477546bd592304bce709811b4bc1a662169fe5004ee6f508d51fac529cebac78c7826a29a524f@65.109.25.36:30603,enode://a22aa97d2016613a407f2adc2a5fbb34800beae2f2cf10d9cc3802a58bac97109371cc8454d90b225d801b0abe5e90e5ad307487373329b14c232134146b3de0@67.235.115.91:30613,enode://0637d1e62026e0c8685b1db0ca1c767c78c95c3fab64abc468d1a64b12ca4b530b46b8f80c915aec96f74f7ffc5999e8ad6d1484476f420f0c10e3d42361914b@52.199.214.252:30311,enode://330d768f6de90e7825f0ea6fe59611ce9d50712e73547306846a9304663f9912bf1611037f7f90f21606242ded7fb476c7285cb7cd792836b8c0c5ef0365855c@18.181.52.189:30311,enode://df1e8eb59e42cad3c4551b2a53e31a7e55a2fdde1287babd1e94b0836550b489ba16c40932e4dacb16cba346bd442c432265a299c4aca63ee7bb0f832b9f45eb@52.51.80.128:30311,enode://0bd566a7fd136ecd19414a601bfdc530d5de161e3014033951dd603e72b1a8959eb5b70b06c87a5a75cbf45e4055c387d2a842bd6b1bd8b5041b3a61bab615cf@34.242.33.165:30311,enode://ecd664250ca19b1074dcfbfb48576a487cc18d052064222a363adacd2650f8e08fb3db9de7a7aecb48afa410eaeb3285e92e516ead01fb62598553aed91ee15e@3.209.122.123:30311,enode://665cf77ca26a8421cfe61a52ac312958308d4912e78ce8e0f61d6902e4494d4cc38f9b0dd1b23a427a7a5734e27e5d9729231426b06bb9c73b56a142f83f6b68@52.72.123.113:30311
        - --log.dir.path=/home/erigon/.local/share/erigon/logs

Consensus Layer:

Consensus Layer Command (with flags/config):

Chain/Network: chapel

Expected behaviour

Snapshot stage successfully sync.

Actual behaviour

Failed to sync snapshots stage:

[INFO] [09-02|08:56:43.743] logging to file system                   log dir=/home/erigon/.local/share/erigon/logs file prefix=erigon log level=info json=false
[INFO] [09-02|08:56:43.743] Enabling metrics export to prometheus    path=http://0.0.0.0:6060/debug/metrics/prometheus
[INFO] [09-02|08:56:43.743] Build info                               git_branch= git_tag= git_commit=
[INFO] [09-02|08:56:43.743] Starting Erigon on Chapel testnet... 
[INFO] [09-02|08:56:43.744] Maximum peer count                       ETH=100 total=100
[INFO] [09-02|08:56:43.744] starting HTTP APIs                       port=8545 APIs=admin,net,eth,erigon,web3,net,debug,trace,txpool,engine,ots
[INFO] [09-02|08:56:43.744] torrent verbosity                        level=WRN
[INFO] [09-02|08:56:43.745] [torrent] Public IP                      ip=65.109.25.36
[INFO] [09-02|08:56:43.746] Set global gas cap                       cap=50000000
[INFO] [09-02|08:56:43.749] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=1000mb upload.rate=4mb
[INFO] [09-02|08:56:43.749] Opening Database                         label=chaindata path=/home/erigon/.local/share/erigon/chaindata
[INFO] [09-02|08:56:43.752] [db] open                                label=chaindata sizeLimit=15TB pageSize=16384
[INFO] [09-02|08:56:43.767] Initialised chain configuration          config="{ChainID: 97 Ramanujan: 1010000, Niels: 1014369, MirrorSync: 5582500, Bruno: 13837000, Euler: 19203503, Gibbs: 22800220, Nano: 23482428, Moran: 23603940, Planck: 28196022, Luban: 29295050, Plato: 29861024, Hertz: 31103030, Hertzfix: 35682300, ShanghaiTime: 1702972800, KeplerTime 1702972800, FeynmanTime 1710136800, FeynmanFixTime 1711342800, CancunTime 1713330442, HaberTime 1716962820, HaberFixTime 1719986788, Bohr 1724116996, Engine: parlia}" genesis=0x6d3c66c5357ec91d5c43af47e234a939b22557cbb552dc45bebbceeed90fbe34
[INFO] [09-02|08:56:43.780] Initialising Ethereum protocol           network=97
[INFO] [09-02|08:56:43.780] Opening Database                         label=parlia path=/home/erigon/.local/share/erigon/parlia
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(a729dc789ec227c1-f9ba8ef44f3e435a) is MATCH: rollback NOT needed, steady-sync NEEDED
[INFO] [09-02|08:56:43.784] Opening Database                         label=blob path=/home/erigon/.local/share/erigon/blobs/blob
[INFO] [09-02|08:56:44.259] Starting private RPC server              on=0.0.0.0:9090
[INFO] [09-02|08:56:44.259] new subscription to logs established 
[INFO] [09-02|08:56:44.259] rpc filters: subscribing to Erigon events 
[INFO] [09-02|08:56:44.259] [txpool] Started 
[INFO] [09-02|08:56:44.259] New txs subscriber joined 
[INFO] [09-02|08:56:44.259] new subscription to newHeaders established 
[INFO] [09-02|08:56:44.260] Reading JWT secret                       path=/home/erigon/.local/share/erigon/jwtsecret
[INFO] [09-02|08:56:44.260] HTTP endpoint opened for Engine API      url=:8551 ws=true ws.compression=true
[INFO] [09-02|08:56:44.260] JsonRpc endpoint opened                  ws=true ws.compression=true grpc=false websocket.url=:8546 http.url=:8545
[INFO] [09-02|08:56:44.291] Started P2P networking                   version=68 self=enode://6f805ff1e3eca98deb52d01460954d849c4439fa359679d6c43477546bd592304bce709811b4bc1a662169fe5004ee6f508d51fac529cebac78c7826a29a524f@65.109.25.36:30603 name=erigon/v1.2.15/linux-amd64/go1.21.5
[INFO] [09-02|08:56:44.296] [1/15 Snapshots] Requesting downloads 
[INFO] [09-02|08:56:44.367] [snapshots] no metadata yet              files=35 list=v1-040100-040200-bscblobsidecars.seg,v1-041900-042000-bscblobsidecars.seg,v1-042000-042100-bscblobsidecars.seg,v1-040900-041000-bscblobsidecars.seg,v1-042200-042300-bscblobsidecars.seg,...
[INFO] [09-02|08:57:03.780] [snapshots] no metadata yet              files=35 list=v1-042900-043000-bscblobsidecars.seg,v1-040200-040300-bscblobsidecars.seg,v1-041700-041800-bscblobsidecars.seg,v1-042700-042800-bscblobsidecars.seg,v1-040100-040200-bscblobsidecars.seg,...
[INFO] [09-02|08:57:04.368] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=20s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=70.6MB sys=136.0MB
[INFO] [09-02|08:57:23.781] [snapshots] no metadata yet              files=35 list=v1-040300-040400-bscblobsidecars.seg,v1-041600-041700-bscblobsidecars.seg,v1-042400-042500-bscblobsidecars.seg,v1-042500-042600-bscblobsidecars.seg,v1-042800-042900-bscblobsidecars.seg,...
[INFO] [09-02|08:57:24.368] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=40s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=87.0MB sys=136.0MB
[INFO] [09-02|08:57:43.781] [snapshots] no metadata yet              files=35 list=v1-040600-040700-bscblobsidecars.seg,v1-040800-040900-bscblobsidecars.seg,v1-040900-041000-bscblobsidecars.seg,v1-042200-042300-bscblobsidecars.seg,v1-042300-042400-bscblobsidecars.seg,...
[INFO] [09-02|08:57:44.367] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=1m0s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=111.8MB sys=144.3MB
[INFO] [09-02|08:58:03.780] [snapshots] no metadata yet              files=35 list=v1-040400-040500-bscblobsidecars.seg,v1-040700-040800-bscblobsidecars.seg,v1-042100-042200-bscblobsidecars.seg,v1-039600-039700-bscblobsidecars.seg,v1-039700-039800-bscblobsidecars.seg,...
[INFO] [09-02|08:58:04.368] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=1m20s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=79.9MB sys=149.4MB
[INFO] [09-02|08:58:23.780] [snapshots] no metadata yet              files=35 list=v1-040100-040200-bscblobsidecars.seg,v1-041900-042000-bscblobsidecars.seg,v1-042000-042100-bscblobsidecars.seg,v1-042300-042400-bscblobsidecars.seg,v1-040600-040700-bscblobsidecars.seg,...
[INFO] [09-02|08:58:24.367] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=1m40s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=99.8MB sys=149.4MB
[INFO] [09-02|08:58:43.780] [snapshots] no metadata yet              files=35 list=v1-041900-042000-bscblobsidecars.seg,v1-042000-042100-bscblobsidecars.seg,v1-040100-040200-bscblobsidecars.seg,v1-040600-040700-bscblobsidecars.seg,v1-040800-040900-bscblobsidecars.seg,...
[INFO] [09-02|08:58:44.368] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=2m0s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=115.1MB sys=149.4MB
[INFO] [09-02|08:59:03.780] [snapshots] no metadata yet              files=35 list=v1-040100-040200-bscblobsidecars.seg,v1-041900-042000-bscblobsidecars.seg,v1-042000-042100-bscblobsidecars.seg,v1-040900-041000-bscblobsidecars.seg,v1-042200-042300-bscblobsidecars.seg,...
[INFO] [09-02|08:59:04.368] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=2m20s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=79.5MB sys=153.4MB
[INFO] [09-02|08:59:23.781] [snapshots] no metadata yet              files=35 list=v1-040200-040300-bscblobsidecars.seg,v1-041700-041800-bscblobsidecars.seg,v1-042700-042800-bscblobsidecars.seg,v1-042900-043000-bscblobsidecars.seg,v1-040100-040200-bscblobsidecars.seg,...
[INFO] [09-02|08:59:24.367] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=2m40s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=105.9MB sys=157.4MB
[INFO] [09-02|08:59:43.780] [snapshots] no metadata yet              files=35 list=v1-040300-040400-bscblobsidecars.seg,v1-041600-041700-bscblobsidecars.seg,v1-042500-042600-bscblobsidecars.seg,v1-042800-042900-bscblobsidecars.seg,v1-039900-040000-bscblobsidecars.seg,...
[INFO] [09-02|08:59:43.780] [p2p] GoodPeers                          eth68=5
[INFO] [09-02|08:59:43.840] [mem] memory stats                       Rss=680.7MB Size=0B Pss=680.5MB SharedClean=432.0KB SharedDirty=0B PrivateClean=510.6MB PrivateDirty=169.7MB Referenced=679.2MB Anonymous=169.3MB Swap=0B alloc=120.5MB sys=157.4MB
[INFO] [09-02|08:59:44.259] [txpool] stat                            pending=0 baseFee=0 queued=259 alloc=120.7MB sys=157.4MB
[INFO] [09-02|08:59:44.367] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=3m0s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=120.8MB sys=157.4MB
[INFO] [09-02|09:00:03.780] [snapshots] no metadata yet              files=35 list=v1-040400-040500-bscblobsidecars.seg,v1-040700-040800-bscblobsidecars.seg,v1-042100-042200-bscblobsidecars.seg,v1-041800-041900-bscblobsidecars.seg,v1-039600-039700-bscblobsidecars.seg,...
[INFO] [09-02|09:00:04.367] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=3m20s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=88.3MB sys=161.5MB

After downgrade to 1.2.14, all works fine.

Steps to reproduce the behaviour

Run the bsc-erigon with chapel network.

blxdyx commented 3 months ago

Try again.

insider89 commented 3 months ago

@blxdyx with v1.2.15 I have the same result. I see you merged some fix, can you make docker image for it, please?

blxdyx commented 3 months ago

Temp fix: add --no-downloader flag. Show logs, i don't think need a new docker image. Because the erigon code don't need change. Try remove the data/snapshots/preverified.toml.

insider89 commented 3 months ago

@blxdyx I don't have preverified.toml file in the snapshots directory. But when I add --no-downloader everything works fine. I tried to remove this flag after, but have issue again. Should I run just run with this flag constantly?

blxdyx commented 3 months ago

@blxdyx I don't have preverified.toml file in the snapshots directory. But when I add --no-downloader everything works fine. I tried to remove this flag after, but have issue again. Should I run just run with this flag constantly?

Yes, it's ok run with this flag. What i want to make sure theno metadata yet is still about bscblobsidecars.seg?

insider89 commented 3 months ago

I don't see such message in the logs now:

[INFO] [09-03|07:36:07.184] RPC Daemon notified of new headers       from=43542790 to=43542791 amount=1 hash=0x7ba778cc4170bf83fedb39031205677e4a0c013d34a452bcd257bf2a820cd267 header sending=8.876µs log sending=201ns
[INFO] [09-03|07:36:07.184] Timings (slower than 50ms)               Headers=2.686s
[INFO] [09-03|07:36:07.184] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.2GB
[INFO] [09-03|07:36:07.188] [2/15 Headers] Waiting for headers...    from=43542791
[INFO] [09-03|07:36:10.071] [2/15 Headers] Processed                 highest=43542792 age=0 headers=1 in=2.884 blk/sec=0
[INFO] [09-03|07:36:10.078] [7/15 Execution] Completed on            block=43542792
[INFO] [09-03|07:36:10.090] RPC Daemon notified of new headers       from=43542791 to=43542792 amount=1 hash=0x4305435c3a7af68bf8ae9dbfb4ddfbe35f551280236a359db7a47482b19b8948 header sending=12.092µs log sending=231ns
[INFO] [09-03|07:36:10.090] Timings (slower than 50ms)               Headers=2.883s
[INFO] [09-03|07:36:10.090] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.2GB
[INFO] [09-03|07:36:10.094] [2/15 Headers] Waiting for headers...    from=43542792
[INFO] [09-03|07:36:13.075] [2/15 Headers] Processed                 highest=43542793 age=0 headers=1 in=2.981 blk/sec=0
[INFO] [09-03|07:36:13.078] [7/15 Execution] Completed on            block=43542793
[INFO] [09-03|07:36:13.089] RPC Daemon notified of new headers       from=43542792 to=43542793 amount=1 hash=0x8db7dcbd4ebaf028eb236cf79cfda1f7c1b50daaf046b67f94d34724c7228b07 header sending=10.058µs log sending=271ns
[INFO] [09-03|07:36:13.089] Timings (slower than 50ms)               Headers=2.981s
[INFO] [09-03|07:36:13.089] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:13.092] [2/15 Headers] Waiting for headers...    from=43542793
[INFO] [09-03|07:36:16.078] [2/15 Headers] Processed                 highest=43542794 age=0 headers=1 in=2.986 blk/sec=0
[INFO] [09-03|07:36:16.082] [7/15 Execution] Completed on            block=43542794
[INFO] [09-03|07:36:16.094] RPC Daemon notified of new headers       from=43542793 to=43542794 amount=1 hash=0xc2a2e8e3853f349fa2c0ce51c6f3ae0c134c6f03ba0a567884db35895e1a6e57 header sending=8.245µs log sending=230ns
[INFO] [09-03|07:36:16.094] Timings (slower than 50ms)               Headers=2.986s
[INFO] [09-03|07:36:16.094] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:16.099] [2/15 Headers] Waiting for headers...    from=43542794
[INFO] [09-03|07:36:19.122] [2/15 Headers] Processed                 highest=43542795 age=0 headers=1 in=3.023 blk/sec=0
[INFO] [09-03|07:36:19.129] [7/15 Execution] Completed on            block=43542795
[INFO] [09-03|07:36:19.145] RPC Daemon notified of new headers       from=43542794 to=43542795 amount=1 hash=0x30738d5cd284cada823f170eb72c7e842a57a2de8afd8a692cfa83acfa7ea3d8 header sending=8.435µs log sending=191ns
[INFO] [09-03|07:36:19.145] Timings (slower than 50ms)               Headers=3.023s
[INFO] [09-03|07:36:19.145] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:19.149] [2/15 Headers] Waiting for headers...    from=43542795
[INFO] [09-03|07:36:22.297] [2/15 Headers] Processed                 highest=43542796 age=0 headers=1 in=3.148 blk/sec=0
[INFO] [09-03|07:36:22.302] [7/15 Execution] Completed on            block=43542796
[INFO] [09-03|07:36:22.318] RPC Daemon notified of new headers       from=43542795 to=43542796 amount=1 hash=0x71103d0cb648a490f2c0ae23d58d21c02ebd4c5d8ca0dde534a991899ced255b header sending=8.446µs log sending=240ns
[INFO] [09-03|07:36:22.318] Timings (slower than 50ms)               Headers=3.148s
[INFO] [09-03|07:36:22.318] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:22.322] [2/15 Headers] Waiting for headers...    from=43542796
[INFO] [09-03|07:36:25.299] [2/15 Headers] Processed                 highest=43542797 age=0 headers=1 in=2.977 blk/sec=0
[INFO] [09-03|07:36:25.309] [7/15 Execution] Completed on            block=43542797
[INFO] [09-03|07:36:25.326] RPC Daemon notified of new headers       from=43542796 to=43542797 amount=1 hash=0x0e7ff736865eca8e56b769b5951098b7f1e15023365b0854af8e345b2267d7ed header sending=12.183µs log sending=180ns
[INFO] [09-03|07:36:25.326] Timings (slower than 50ms)               Headers=2.977s
[INFO] [09-03|07:36:25.326] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:25.333] [2/15 Headers] Waiting for headers...    from=43542797
[INFO] [09-03|07:36:28.142] [2/15 Headers] Processed                 highest=43542798 age=0 headers=1 in=2.809 blk/sec=0
[INFO] [09-03|07:36:28.143] [7/15 Execution] Completed on            block=43542798
[INFO] [09-03|07:36:28.145] RPC Daemon notified of new headers       from=43542797 to=43542798 amount=1 hash=0x2b40a602107345acdda0feedec39d2d8179ea2d353951eb252556349d80f64bf header sending=15.729µs log sending=340ns
[INFO] [09-03|07:36:28.145] Timings (slower than 50ms)               Headers=2.809s
[INFO] [09-03|07:36:28.145] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:28.147] [2/15 Headers] Waiting for headers...    from=43542798
[INFO] [09-03|07:36:31.178] [2/15 Headers] Processed                 highest=43542799 age=0 headers=1 in=3.031 blk/sec=0
[INFO] [09-03|07:36:31.180] [7/15 Execution] Completed on            block=43542799
[INFO] [09-03|07:36:31.186] RPC Daemon notified of new headers       from=43542798 to=43542799 amount=1 hash=0xbfd2c806c1953bad030b6d7f475677f2d98bb76e5883a95884fce76277b71257 header sending=8.786µs log sending=211ns
[INFO] [09-03|07:36:31.186] Timings (slower than 50ms)               Headers=3.03s
[INFO] [09-03|07:36:31.186] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:31.188] [2/15 Headers] Waiting for headers...    from=43542799
[INFO] [09-03|07:36:34.786] [2/15 Headers] Processed                 highest=43542800 age=0 headers=1 in=3.598 blk/sec=0
[INFO] [09-03|07:36:34.794] [7/15 Execution] Completed on            block=43542800
[INFO] [09-03|07:36:34.802] RPC Daemon notified of new headers       from=43542799 to=43542800 amount=1 hash=0x76abac174811eb1554dd52ea83c17aad8608bcd8e5baa59c9851faf655dcae5c header sending=11.251µs log sending=230ns
[INFO] [09-03|07:36:34.802] Timings (slower than 50ms)               Headers=3.597s
[INFO] [09-03|07:36:34.802] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:34.806] [2/15 Headers] Waiting for headers...    from=43542800
[INFO] [09-03|07:36:37.171] [2/15 Headers] Processed                 highest=43542801 age=0 headers=1 in=2.365 blk/sec=0
[INFO] [09-03|07:36:37.175] [7/15 Execution] Completed on            block=43542801
[INFO] [09-03|07:36:37.184] RPC Daemon notified of new headers       from=43542800 to=43542801 amount=1 hash=0x0b0df5cc6c5463e2fd456b23d05bab847717e4893cdb8c194159c3af2f976c7b header sending=11.641µs log sending=271ns
[INFO] [09-03|07:36:37.184] Timings (slower than 50ms)               Headers=2.365s
[INFO] [09-03|07:36:37.184] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:37.187] [2/15 Headers] Waiting for headers...    from=43542801
[INFO] [09-03|07:36:40.372] [2/15 Headers] Processed                 highest=43542802 age=0 headers=1 in=3.185 blk/sec=0
[INFO] [09-03|07:36:40.377] [7/15 Execution] Completed on            block=43542802
[INFO] [09-03|07:36:40.388] RPC Daemon notified of new headers       from=43542801 to=43542802 amount=1 hash=0x4fde6654d477407433de1a0cca976d8ebd4da23ebe73cf03bb473c992b4f0758 header sending=7.614µs log sending=231ns
[INFO] [09-03|07:36:40.388] Timings (slower than 50ms)               Headers=3.184s
[INFO] [09-03|07:36:40.388] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:40.391] [2/15 Headers] Waiting for headers...    from=43542802
[INFO] [09-03|07:36:43.366] [2/15 Headers] Processed                 highest=43542803 age=0 headers=1 in=2.975 blk/sec=0
[INFO] [09-03|07:36:43.369] [7/15 Execution] Completed on            block=43542803
[INFO] [09-03|07:36:43.380] RPC Daemon notified of new headers       from=43542802 to=43542803 amount=1 hash=0x3cb6207fa48578983dfc0f9191446a0dbb5b817d59138d586a7cb15ae72bea27 header sending=13.364µs log sending=191ns
[INFO] [09-03|07:36:43.380] Timings (slower than 50ms)               Headers=2.974s
[INFO] [09-03|07:36:43.380] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.2GB
[INFO] [09-03|07:36:43.383] [2/15 Headers] Waiting for headers...    from=43542803
[INFO] [09-03|07:36:46.165] [2/15 Headers] Processed                 highest=43542804 age=0 headers=1 in=2.782 blk/sec=0
[INFO] [09-03|07:36:46.170] [7/15 Execution] Completed on            block=43542804
[INFO] [09-03|07:36:46.188] RPC Daemon notified of new headers       from=43542803 to=43542804 amount=1 hash=0x57cbbfae458fa604ae8b91483512803fcd87ab4ea29d2b763172a450ebb98082 header sending=12.002µs log sending=170ns
[INFO] [09-03|07:36:46.188] Timings (slower than 50ms)               Headers=2.782s
[INFO] [09-03|07:36:46.188] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.2GB
[INFO] [09-03|07:36:46.195] [2/15 Headers] Waiting for headers...    from=43542804
[INFO] [09-03|07:36:49.199] [2/15 Headers] Processed                 highest=43542805 age=0 headers=1 in=3.004 blk/sec=0
[INFO] [09-03|07:36:49.205] [7/15 Execution] Completed on            block=43542805
[INFO] [09-03|07:36:49.215] RPC Daemon notified of new headers       from=43542804 to=43542805 amount=1 hash=0x0ce41a12d0744d890be41619104cb4acf7d01aceb66fe4b5e2a7616710b4c32e header sending=7.895µs log sending=491ns
[INFO] [09-03|07:36:49.215] Timings (slower than 50ms)               Headers=3.003s
[INFO] [09-03|07:36:49.215] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:49.218] [2/15 Headers] Waiting for headers...    from=43542805
[INFO] [09-03|07:36:52.357] [2/15 Headers] Processed                 highest=43542806 age=0 headers=1 in=3.139 blk/sec=0
[INFO] [09-03|07:36:52.364] [7/15 Execution] Completed on            block=43542806
[INFO] [09-03|07:36:52.378] RPC Daemon notified of new headers       from=43542805 to=43542806 amount=1 hash=0x94ecab259f2bdd2b19ddaf67e863cc4b126dda6c09ff782cf5199e7d9d3340c5 header sending=9.667µs log sending=250ns
[INFO] [09-03|07:36:52.379] Timings (slower than 50ms)               Headers=3.138s
[INFO] [09-03|07:36:52.379] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:52.384] [2/15 Headers] Waiting for headers...    from=43542806
[INFO] [09-03|07:36:55.230] [2/15 Headers] Processed                 highest=43542807 age=0 headers=1 in=2.846 blk/sec=0
[INFO] [09-03|07:36:55.232] [7/15 Execution] Completed on            block=43542807
[INFO] [09-03|07:36:55.237] RPC Daemon notified of new headers       from=43542806 to=43542807 amount=1 hash=0x90a53d92b7f53e5ed1c7cd7cda3f0a24dedad3753b3e6015459d0002d181561a header sending=7.824µs log sending=170ns
[INFO] [09-03|07:36:55.237] Timings (slower than 50ms)               Headers=2.846s
[INFO] [09-03|07:36:55.237] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:55.240] [2/15 Headers] Waiting for headers...    from=43542807
[INFO] [09-03|07:36:58.263] [2/15 Headers] Processed                 highest=43542808 age=0 headers=1 in=3.023 blk/sec=0
[INFO] [09-03|07:36:58.271] [7/15 Execution] Completed on            block=43542808
[INFO] [09-03|07:36:58.291] RPC Daemon notified of new headers       from=43542807 to=43542808 amount=1 hash=0xd099089acc0c1f134a5d34d7ae666db9a6c1fee9991dddd3fd26af5b85bf11af header sending=9.778µs log sending=200ns
[INFO] [09-03|07:36:58.291] Timings (slower than 50ms)               Headers=3.023s
[INFO] [09-03|07:36:58.291] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:36:58.298] [2/15 Headers] Waiting for headers...    from=43542808
[INFO] [09-03|07:37:01.208] [2/15 Headers] Processed                 highest=43542809 age=0 headers=1 in=2.910 blk/sec=0
[INFO] [09-03|07:37:01.210] [7/15 Execution] Completed on            block=43542809
[INFO] [09-03|07:37:01.217] RPC Daemon notified of new headers       from=43542808 to=43542809 amount=1 hash=0x99cadf508096bc7bed5e53590b535e3764d42b6cd39d6ad994e275de179578b9 header sending=7.254µs log sending=190ns
[INFO] [09-03|07:37:01.217] Timings (slower than 50ms)               Headers=2.91s
[INFO] [09-03|07:37:01.217] Tables                                   PlainState=43.1GB AccountChangeSet=30.8GB StorageChangeSet=92.2GB BlockTransaction=105.4GB TransactionLog=201.8GB FreeList=1.5MB ReclaimableSpace=6.1GB
[INFO] [09-03|07:37:01.219] [2/15 Headers] Waiting for headers...    from=43542809
blxdyx commented 3 months ago

Does it with --no-downloader flag?

insider89 commented 3 months ago

Yes, with --no-downloader flag

insider89 commented 3 months ago

Without flag:

[INFO] [09-04|09:22:49.634] Initialising Ethereum protocol           network=97
[INFO] [09-04|09:22:49.635] Opening Database                         label=parlia path=/home/erigon/.local/share/erigon/parlia
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(a729dc789ec227c1-f9ba8ef44f3e435a) is MATCH: rollback NOT needed, steady-sync NEEDED
[INFO] [09-04|09:22:49.638] Opening Database                         label=blob path=/home/erigon/.local/share/erigon/blobs/blob
[INFO] [09-04|09:22:50.125] Starting private RPC server              on=0.0.0.0:9090
[INFO] [09-04|09:22:50.125] new subscription to logs established 
[INFO] [09-04|09:22:50.125] rpc filters: subscribing to Erigon events 
[INFO] [09-04|09:22:50.125] New txs subscriber joined 
[INFO] [09-04|09:22:50.125] [txpool] Started 
[INFO] [09-04|09:22:50.125] new subscription to newHeaders established 
[INFO] [09-04|09:22:50.126] Reading JWT secret                       path=/home/erigon/.local/share/erigon/jwtsecret
[INFO] [09-04|09:22:50.126] HTTP endpoint opened for Engine API      url=:8551 ws=true ws.compression=true
[INFO] [09-04|09:22:50.126] JsonRpc endpoint opened                  ws=true ws.compression=true grpc=false websocket.url=:8546 http.url=:8545
[INFO] [09-04|09:22:50.155] Started P2P networking                   version=68 self=enode://6f805ff1e3eca98deb52d01460954d849c4439fa359679d6c43477546bd592304bce709811b4bc1a662169fe5004ee6f508d51fac529cebac78c7826a29a524f@65.109.25.36:30603 name=erigon/v1.2.15/linux-amd64/go1.21.5
[INFO] [09-04|09:22:50.161] [1/15 Snapshots] Requesting downloads 
[INFO] [09-04|09:22:50.202] [snapshots] no metadata yet              files=35 list=v1-039800-039900-bscblobsidecars.seg,v1-040600-040700-bscblobsidecars.seg,v1-041500-041600-bscblobsidecars.seg,v1-041700-041800-bscblobsidecars.seg,v1-042000-042100-bscblobsidecars.seg,...
[INFO] [09-04|09:23:09.635] [snapshots] no metadata yet              files=35 list=v1-040600-040700-bscblobsidecars.seg,v1-041500-041600-bscblobsidecars.seg,v1-041700-041800-bscblobsidecars.seg,v1-042000-042100-bscblobsidecars.seg,v1-039600-039700-bscblobsidecars.seg,...
[INFO] [09-04|09:23:10.202] [1/15 Snapshots] download                progress="99.90% 83.0MB/83.0MB" time-left=999hrs:99m total-time=20s download=0B/s upload=0B/s peers=0 files=37 metadata=2/37 connections=0 alloc=81.3MB sys=131.3MB
blxdyx commented 2 months ago

I found the route cause, you can run with --no-downloader flag. BTW, we suggest try with the erigon3.

insider89 commented 2 months ago

Thank you. I've tried erigon3 for polygon and ethereum chains. It still has some bugs, which I observed: high IO usage(already fixed) and slow startup(cannot use liveness/readiness probes). I'll probably try it for chapel as well.