search

0xPolygonHermez / cdk-erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
30 stars 24 forks source link

"[9/16 IntermediateHashes] SMT regenerate progress" Takes A Lot of Time #804

Closed andreclaro closed 1 month ago

andreclaro commented 1 month ago

Description

I have noticed this during X Layer testnet upgrade, where we had to start the node from a snapshot (1 day old)...

Today, I'm working on the mainnet upgrade and we also had to restart the node from a snapshot (2 days old), so node needs to download old blocks.

The node was started at 10:27:13 UTC and it still at [9/16 IntermediateHashes] SMT regenerate progress: (58%) now (12:15:30)

Active: active (running) since Wed 2024-07-17 10:27:13 UTC; 1h 47min ago
[INFO] [07-17|12:15:30.659] [9/16 IntermediateHashes] SMT regenerate progress: (58%)

I don't see any bootleneck (CPU, Memory or disk)...

System information

Cdk-erigon version: v1.1.5.8

cdk-erigon --version
[cdk-erigon-lib] timestamp 2024-03-12:16:34
cdk-erigon version 2.43.0-dev-81c4c6c2

OS & Version: Ubuntu

Node Config: https://github.com/okx/Deploy/blob/main/mainnet/setup-erigon-rpc.md

Steps to reproduce the behaviour

Follow https://github.com/okx/Deploy/blob/main/mainnet/setup-erigon-rpc.md

Full Node Logs

Jul 17 10:27:13 m-xlayer-01 systemd[1]: Started "zkEVM CDK-Erigon service".
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [cdk-erigon-lib] timestamp 2024-03-12:16:34
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.370] logging to file system                   log dir=/var/lib/xlayer/data/erigon/logs file prefix=erigon log level=info json=false
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.370] Starting metrics server                  addr=http://0.0.0.0:6060/debug/metrics/prometheus
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.388] logging to file system                   log dir=/var/lib/xlayer/data/erigon/logs file prefix=cdk-erigon log level=info json=false
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.391] Build info                               git_branch=HEAD git_tag=tags/v1.1.5.8 git_commit=81c4c6c2ceed97c647657007f3b5cd771fd62fd4
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.391] Poseidon hashing                         Accelerated=true
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.391] Starting Erigon on                       devnet=xlayer-mainnet
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.392] Maximum peer count                       ETH=100 total=100
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.392] starting HTTP APIs                       APIs=eth,debug,net,trace,web3,erigon,zkevm
Jul 17 10:27:13 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:13.392] torrent verbosity                        level=WRN
Jul 17 10:27:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:15.494] Set global gas cap                       cap=50000000
Jul 17 10:27:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:15.496] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
Jul 17 10:27:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:15.496] Opening Database                         label=chaindata path=/var/lib/xlayer/data/erigon/chaindata
Jul 17 10:27:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:15.542] Initialised chain configuration          config="{ChainID: 196, Homestead: 0, DAO: 0, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 0, Muir Glacier: 0, Berlin: 0, London: 18446744073709551615, Arrow Glacier: 9999999999999999999999999999999999999999999999999, Gray Glacier: 9999999999999999999999999999999999999999999999999, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 18446744073709551615, Cancun: 18446744073709551615, Prague: 18446744073709551615, Engine: ethash}" genesis=0x11f32f605beb94a1acb783cb3b6da6d7975461ce3addf441e7ad60c2ec95e88f
Jul 17 10:27:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:15.542] Effective                                prune_flags= snapshot_flags= history.v3=false
Jul 17 10:27:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:15.549] Initialising Ethereum protocol           network=196
Jul 17 10:27:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:15.549] Disk storage enabled for ethash DAGs     dir=/var/lib/xlayer/data/erigon/ethash-dags count=2
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.052] Starting private RPC server              on=localhost:9091
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.052] new subscription to logs established
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.054] Starting datastream client...
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.072] [txpool] Started
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.292] Datastream client initialized...
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.294] rpc filters: subscribing to Erigon events
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.294] New txs subscriber joined
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.295] new subscription to newHeaders established
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.295] Reading JWT secret                       path=/var/lib/xlayer/jwt.hex
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.295] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.296] HTTP endpoint opened                     url=[::]:8545 ws=true ws.compression=true grpc=false
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.300] Started P2P networking                   version=67 self=enode://f2a5174c2a94380ae2671a229bc1c1780c9c6a8f27822297a0ad54c1594d92f6ef8568dcb805266e0eb8938759bf252ab4ff23ebb125c6cd3643d199138ace4b@127.0.0.1:0 name=erigon/v2.43.0-dev-81c4c6c2/linux-amd64/go1.19
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.304] Started P2P networking                   version=68 self=enode://f2a5174c2a94380ae2671a229bc1c1780c9c6a8f27822297a0ad54c1594d92f6ef8568dcb805266e0eb8938759bf252ab4ff23ebb125c6cd3643d199138ace4b@127.0.0.1:0 name=erigon/v2.43.0-dev-81c4c6c2/linux-amd64/go1.19
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.305] [1/16 L1Syncer] Starting L1 sync stage
Jul 17 10:27:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:21.306] Starting L1 syncer thread
Jul 17 10:27:22 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:22.515] [1/16 L1Syncer] Saving L1 syncer progress latestCheckedBlock=20325568 newVerificationsCount=47 newSequencesCount=47
Jul 17 10:27:22 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:22.515] [1/16 L1Syncer]                          highestVerificationBatchNo=74689
Jul 17 10:27:22 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:22.525] [1/16 L1Syncer] Finished L1 sync stage
Jul 17 10:27:22 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:22.525] [2/16 L1InfoTree] Starting L1 Info Tree stage
Jul 17 10:27:22 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:22.529] Starting L1 syncer thread
Jul 17 10:27:27 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:27.817] [2/16 L1InfoTree] Info tree updates      count=118
Jul 17 10:27:27 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:27.844] [2/16 L1InfoTree] Finished L1 Info Tree stage
Jul 17 10:27:27 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:27.844] [3/16 Batches] Starting batches stage
Jul 17 10:27:27 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:27.847] [3/16 Batches] Starting stream           startBlock=3069191
Jul 17 10:27:27 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:27.847] [3/16 Batches] Reading blocks from the datastream.
Jul 17 10:27:27 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:27.848] [3/16 Batches] Started downloading L2Blocks routine
Jul 17 10:27:37 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:37.848] [3/16 Batches] Downloaded blocks from datastream progress: 2674
Jul 17 10:27:47 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:47.848] [3/16 Batches] Downloaded blocks from datastream progress: 7664
Jul 17 10:27:57 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:27:57.848] [3/16 Batches] Downloaded blocks from datastream progress: 17289
Jul 17 10:28:07 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:07.848] [3/16 Batches] Downloaded blocks from datastream progress: 31290
Jul 17 10:28:17 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:17.831] [3/16 Batches] No new blocks in 1555 miliseconds. Ending the stage. lastBlockHeight=3129197
Jul 17 10:28:17 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:17.832] [3/16 Batches] Total blocks read: 60006
Jul 17 10:28:17 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:17.832] [3/16 Batches] Finished writing blocks   blocksWritten=60006 elapsed=49.984234086s
Jul 17 10:28:17 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:17.832] [3/16 Batches] Saving stage progress     lastBlockHeight=3129197
Jul 17 10:28:17 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:17.848] [3/16 Batches] Downloaded blocks from datastream progress: 60006
Jul 17 10:28:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=159.1MB sys=285.7MB
Jul 17 10:28:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:25.664] [3/16 Batches] Finished Batches stage
Jul 17 10:28:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:25.664] [4/16 BlockHashes] Etl transform started
Jul 17 10:28:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:28:55.665] [4/16 BlockHashes] ETL [1/2] Extracting  from=Header current_prefix=00000000001636dfae0ac6f03c597063c9421836b3a331304f23cdcf66b6c207f793395c6891ea6d
Jul 17 10:29:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:29:15.550] [p2p] GoodPeers
Jul 17 10:29:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:29:21.052] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=350.1MB sys=696.5MB
Jul 17 10:29:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:29:25.664] [4/16 BlockHashes] ETL [1/2] Extracting  from=Header current_prefix=000000000025263eab6667939ffcf40b13cfb1b070bf53f157f51b46a411be1aff46d24bd1ba6287
Jul 17 10:29:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:29:55.664] [4/16 BlockHashes] ETL [1/2] Extracting  from=Header current_prefix=00000000002d8c4882f0f77736f0f6f2dcbee32715c44a1468f88740bcf2ce05b536d1e3570d8bb1
Jul 17 10:30:12 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:12.832] [4/16 BlockHashes] Etl transform ended
Jul 17 10:30:12 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:12.832] [4/16 BlockHashes] DONE                  in=1m47.16830796s
Jul 17 10:30:12 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:12.832] [5/16 Senders] Started                   from=3069191 to=3129197
Jul 17 10:30:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:15.662] [5/16 Senders] Finished
Jul 17 10:30:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:15.663] [6/16 Execution] Short circuit           batch=74689 block=3127612
Jul 17 10:30:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:15.663] [6/16 Execution] Blocks execution        from=3069191 to=3127612
Jul 17 10:30:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:21.052] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=140.7MB sys=825.6MB
Jul 17 10:30:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:35.663] [6/16 Execution] Executed blocks         number=3073382 %=7.174 blk/s=209.5 tx/s=344.7 Mgas/s=643.6 gasState%=0.12 batch=2.7MB alloc=118.3MB sys=826.1MB
Jul 17 10:30:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:30:55.663] [6/16 Execution] Executed blocks         number=3078728 %=16.325 blk/s=267.3 tx/s=464.4 Mgas/s=822.2 gasState%=0.31 batch=6.7MB alloc=161.3MB sys=826.3MB
Jul 17 10:31:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:31:15.549] [p2p] GoodPeers
Jul 17 10:31:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:31:15.664] [6/16 Execution] Executed blocks         number=3087922 %=32.062 blk/s=459.7 tx/s=413.2 Mgas/s=1417.4 gasState%=0.47 batch=11.4MB alloc=191.5MB sys=827.1MB
Jul 17 10:31:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:31:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=112.4MB sys=827.1MB
Jul 17 10:31:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:31:35.666] [6/16 Execution] Executed blocks         number=3095038 %=44.243 blk/s=355.7 tx/s=479.0 Mgas/s=1099.7 gasState%=0.62 batch=15.3MB alloc=196.0MB sys=827.1MB
Jul 17 10:31:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:31:55.663] [6/16 Execution] Executed blocks         number=3101706 %=55.656 blk/s=333.4 tx/s=509.9 Mgas/s=1033.1 gasState%=0.81 batch=19.6MB alloc=150.1MB sys=827.1MB
Jul 17 10:32:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:32:15.663] [6/16 Execution] Executed blocks         number=3108975 %=68.099 blk/s=363.5 tx/s=496.0 Mgas/s=1128.7 gasState%=0.96 batch=23.7MB alloc=195.9MB sys=827.1MB
Jul 17 10:32:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:32:21.052] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=202.1MB sys=827.1MB
Jul 17 10:32:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:32:35.664] [6/16 Execution] Executed blocks         number=3118421 %=84.268 blk/s=472.3 tx/s=451.2 Mgas/s=1470.6 gasState%=1.10 batch=28.3MB alloc=183.1MB sys=827.1MB
Jul 17 10:32:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:32:55.663] [6/16 Execution] Executed blocks         number=3127008 %=98.966 blk/s=429.4 tx/s=529.6 Mgas/s=1340.8 gasState%=1.23 batch=32.2MB alloc=158.0MB sys=827.1MB
Jul 17 10:33:04 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:33:04.795] [6/16 Execution] Commiting DB transaction... block=3127612
Jul 17 10:33:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:33:15.549] [p2p] GoodPeers
Jul 17 10:33:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:33:15.664] [6/16 Execution] Executed blocks         number=3127612 %=100.000 blk/s=30.2 tx/s=40.5 Mgas/s=94.4 gasState%=1.24 batch=0B alloc=147.0MB sys=827.1MB
Jul 17 10:33:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:33:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=122.1MB sys=827.1MB
Jul 17 10:33:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:33:35.663] [6/16 Execution] Executed blocks         number=3127612 %=100.000 blk/s=0.0 tx/s=0.0 Mgas/s=0.0 gasState%=1.24 batch=0B alloc=94.2MB sys=827.1MB
Jul 17 10:33:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:33:55.663] [6/16 Execution] Executed blocks         number=3127612 %=100.000 blk/s=0.0 tx/s=0.0 Mgas/s=0.0 gasState%=1.24 batch=0B alloc=139.5MB sys=827.1MB
Jul 17 10:34:14 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:14.518] [6/16 Execution] Completed on            block=3127612
Jul 17 10:34:14 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:14.519] [6/16 Execution] DONE                    in=3m58.856294514s
Jul 17 10:34:14 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:14.519] [7/16 CumulativeIndex] Started
Jul 17 10:34:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:15.439] [7/16 CumulativeIndex] Finished
Jul 17 10:34:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:15.439] [8/16 HashState] Started
Jul 17 10:34:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:15.439] [8/16 HashState] Promoting plain state   from=3069191 to=3127612
Jul 17 10:34:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:15.440] [8/16 HashState] Incremental promotion   from=3069191 to=3127612 codes=true csbucket=AccountChangeSet
Jul 17 10:34:16 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:16.124] [8/16 HashState] Incremental promotion   from=3069191 to=3127612 codes=false csbucket=AccountChangeSet
Jul 17 10:34:16 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:16.880] [8/16 HashState] Incremental promotion   from=3069191 to=3127612 codes=false csbucket=StorageChangeSet
Jul 17 10:34:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:34:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=232.3MB sys=827.3MB
Jul 17 10:35:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:35:15.550] [p2p] GoodPeers
Jul 17 10:35:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:35:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=166.4MB sys=827.3MB
Jul 17 10:35:46 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:35:46.775] [8/16 HashState] Finished
Jul 17 10:35:46 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:35:46.775] [8/16 HashState] DONE                    in=1m31.336003664s
Jul 17 10:35:46 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:35:46.776] [9/16 IntermediateHashes] Generating intermediate hashes from=3069191 to=3127612
Jul 17 10:35:46 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:35:46.776] [9/16 IntermediateHashes] SMT not using mapmutation
Jul 17 10:35:46 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:35:46.776] [9/16 IntermediateHashes] Regeneration trie hashes started
Jul 17 10:35:46 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:35:46.776] [9/16 IntermediateHashes] Collecting account data...
Jul 17 10:36:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:36:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (2%)
Jul 17 10:36:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:36:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (5%)
Jul 17 10:36:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:36:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=657.4MB sys=832.1MB
Jul 17 10:36:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:36:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (8%)
Jul 17 10:36:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:36:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (10%)
Jul 17 10:36:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:36:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (12%)
Jul 17 10:36:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:36:55.958] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:37:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:37:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:37:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:37:15.550] [p2p] GoodPeers
Jul 17 10:37:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:37:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:37:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:37:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=996.2MB sys=1.6GB
Jul 17 10:37:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:37:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:37:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:37:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:37:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:37:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:37:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:37:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:38:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:38:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:38:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:38:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:38:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:38:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=989.1MB sys=1.6GB
Jul 17 10:38:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:38:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:38:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:38:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:38:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:38:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:38:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:38:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:39:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:39:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:39:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:39:15.549] [p2p] GoodPeers
Jul 17 10:39:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:39:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:39:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:39:21.054] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=1.5GB sys=1.7GB
Jul 17 10:39:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:39:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:39:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:39:35.958] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:39:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:39:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:39:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:39:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:40:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:40:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:40:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:40:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:40:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:40:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=875.7MB sys=2.0GB
Jul 17 10:40:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:40:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:40:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:40:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:40:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:40:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:40:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:40:55.963] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:41:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:41:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:41:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:41:15.552] [p2p] GoodPeers
Jul 17 10:41:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:41:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:41:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:41:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=837.8MB sys=2.0GB
Jul 17 10:41:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:41:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:41:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:41:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (13%)
Jul 17 10:41:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:41:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (15%)
Jul 17 10:41:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:41:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (15%)
Jul 17 10:42:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:42:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (15%)
Jul 17 10:42:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:42:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (15%)
Jul 17 10:42:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:42:21.054] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=284.6MB sys=2.1GB
Jul 17 10:42:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:42:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (15%)
Jul 17 10:42:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:42:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (16%)
Jul 17 10:42:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:42:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (16%)
Jul 17 10:42:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:42:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (17%)
Jul 17 10:43:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:43:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (17%)
Jul 17 10:43:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:43:15.550] [p2p] GoodPeers
Jul 17 10:43:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:43:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (17%)
Jul 17 10:43:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:43:21.052] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=291.5MB sys=2.1GB
Jul 17 10:43:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:43:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (17%)
Jul 17 10:43:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:43:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (20%)
Jul 17 10:43:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:43:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:43:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:43:55.965] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:44:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:44:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:44:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:44:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:44:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:44:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=700.8MB sys=2.1GB
Jul 17 10:44:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:44:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:44:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:44:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:44:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:44:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:44:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:44:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:45:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:45:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:45:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:45:15.549] [p2p] GoodPeers
Jul 17 10:45:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:45:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:45:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:45:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=1001.1MB sys=2.1GB
Jul 17 10:45:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:45:25.959] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:45:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:45:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (22%)
Jul 17 10:45:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:45:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (23%)
Jul 17 10:45:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:45:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (23%)
Jul 17 10:46:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:46:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (23%)
Jul 17 10:46:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:46:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (23%)
Jul 17 10:46:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:46:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=520.4MB sys=2.1GB
Jul 17 10:46:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:46:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (23%)
Jul 17 10:46:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:46:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (23%)
Jul 17 10:46:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:46:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (24%)
Jul 17 10:46:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:46:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (24%)
Jul 17 10:47:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:47:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (24%)
Jul 17 10:47:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:47:15.550] [p2p] GoodPeers
Jul 17 10:47:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:47:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (25%)
Jul 17 10:47:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:47:21.052] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=437.8MB sys=2.1GB
Jul 17 10:47:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:47:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (25%)
Jul 17 10:47:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:47:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (25%)
Jul 17 10:47:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:47:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (25%)
Jul 17 10:47:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:47:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (26%)
Jul 17 10:48:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:48:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (26%)
Jul 17 10:48:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:48:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (26%)
Jul 17 10:48:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:48:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=589.9MB sys=2.1GB
Jul 17 10:48:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:48:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (27%)
Jul 17 10:48:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:48:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (27%)
Jul 17 10:48:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:48:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (27%)
Jul 17 10:48:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:48:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (27%)
Jul 17 10:49:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:49:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (27%)
Jul 17 10:49:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:49:15.550] [p2p] GoodPeers
Jul 17 10:49:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:49:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (27%)
Jul 17 10:49:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:49:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=487.5MB sys=2.1GB
Jul 17 10:49:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:49:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (27%)
Jul 17 10:49:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:49:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (29%)
Jul 17 10:49:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:49:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (29%)
Jul 17 10:49:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:49:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (29%)
Jul 17 10:50:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:50:05.958] [9/16 IntermediateHashes] SMT regenerate progress: (29%)
Jul 17 10:50:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:50:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (29%)
Jul 17 10:50:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:50:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=638.5MB sys=2.1GB
Jul 17 10:50:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:50:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (29%)
Jul 17 10:50:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:50:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (30%)
Jul 17 10:50:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:50:45.958] [9/16 IntermediateHashes] SMT regenerate progress: (30%)
Jul 17 10:50:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:50:55.958] [9/16 IntermediateHashes] SMT regenerate progress: (30%)
Jul 17 10:51:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:51:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (30%)
Jul 17 10:51:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:51:15.550] [p2p] GoodPeers
Jul 17 10:51:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:51:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (30%)
Jul 17 10:51:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:51:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=502.8MB sys=2.1GB
Jul 17 10:51:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:51:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (31%)
Jul 17 10:51:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:51:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (31%)
Jul 17 10:51:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:51:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (32%)
Jul 17 10:51:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:51:55.958] [9/16 IntermediateHashes] SMT regenerate progress: (32%)
Jul 17 10:52:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:52:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (32%)
Jul 17 10:52:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:52:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (32%)
Jul 17 10:52:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:52:21.052] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=438.7MB sys=2.4GB
Jul 17 10:52:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:52:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (34%)
Jul 17 10:52:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:52:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (34%)
Jul 17 10:52:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:52:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (34%)
Jul 17 10:52:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:52:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (34%)
Jul 17 10:53:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:53:05.958] [9/16 IntermediateHashes] SMT regenerate progress: (34%)
Jul 17 10:53:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:53:15.550] [p2p] GoodPeers
Jul 17 10:53:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:53:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (36%)
Jul 17 10:53:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:53:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=528.8MB sys=2.4GB
Jul 17 10:53:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:53:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (36%)
Jul 17 10:53:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:53:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (36%)
Jul 17 10:53:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:53:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (36%)
Jul 17 10:53:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:53:55.958] [9/16 IntermediateHashes] SMT regenerate progress: (36%)
Jul 17 10:54:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:54:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (36%)
Jul 17 10:54:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:54:15.958] [9/16 IntermediateHashes] SMT regenerate progress: (37%)
Jul 17 10:54:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:54:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=498.8MB sys=2.4GB
Jul 17 10:54:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:54:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (37%)
Jul 17 10:54:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:54:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (37%)
Jul 17 10:54:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:54:45.958] [9/16 IntermediateHashes] SMT regenerate progress: (38%)
Jul 17 10:54:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:54:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (39%)
Jul 17 10:55:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:55:05.958] [9/16 IntermediateHashes] SMT regenerate progress: (39%)
Jul 17 10:55:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:55:15.550] [p2p] GoodPeers
Jul 17 10:55:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:55:15.958] [9/16 IntermediateHashes] SMT regenerate progress: (39%)
Jul 17 10:55:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:55:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=428.0MB sys=2.4GB
Jul 17 10:55:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:55:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (39%)
Jul 17 10:55:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:55:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (39%)
Jul 17 10:55:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:55:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (40%)
Jul 17 10:55:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:55:55.958] [9/16 IntermediateHashes] SMT regenerate progress: (41%)
Jul 17 10:56:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:56:05.958] [9/16 IntermediateHashes] SMT regenerate progress: (41%)
Jul 17 10:56:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:56:15.958] [9/16 IntermediateHashes] SMT regenerate progress: (41%)
Jul 17 10:56:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:56:21.086] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=827.0MB sys=2.4GB
Jul 17 10:56:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:56:25.957] [9/16 IntermediateHashes] SMT regenerate progress: (41%)
Jul 17 10:56:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:56:35.958] [9/16 IntermediateHashes] SMT regenerate progress: (41%)
Jul 17 10:56:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:56:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (42%)
Jul 17 10:56:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:56:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (42%)
Jul 17 10:57:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:57:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (42%)
Jul 17 10:57:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:57:15.550] [p2p] GoodPeers
Jul 17 10:57:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:57:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (43%)
Jul 17 10:57:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:57:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=723.6MB sys=2.4GB
Jul 17 10:57:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:57:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (43%)
Jul 17 10:57:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:57:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (43%)
Jul 17 10:57:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:57:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (43%)
Jul 17 10:57:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:57:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (44%)
Jul 17 10:58:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:58:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (44%)
Jul 17 10:58:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:58:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (45%)
Jul 17 10:58:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:58:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=839.4MB sys=2.4GB
Jul 17 10:58:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:58:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (45%)
Jul 17 10:58:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:58:35.957] [9/16 IntermediateHashes] SMT regenerate progress: (45%)
Jul 17 10:58:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:58:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (45%)
Jul 17 10:58:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:58:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (47%)
Jul 17 10:59:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:59:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (47%)
Jul 17 10:59:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:59:15.550] [p2p] GoodPeers
Jul 17 10:59:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:59:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (47%)
Jul 17 10:59:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:59:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=659.9MB sys=2.4GB
Jul 17 10:59:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:59:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (47%)
Jul 17 10:59:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:59:35.958] [9/16 IntermediateHashes] SMT regenerate progress: (47%)
Jul 17 10:59:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:59:45.957] [9/16 IntermediateHashes] SMT regenerate progress: (47%)
Jul 17 10:59:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|10:59:55.957] [9/16 IntermediateHashes] SMT regenerate progress: (49%)
Jul 17 11:00:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:00:05.958] [9/16 IntermediateHashes] SMT regenerate progress: (49%)
Jul 17 11:00:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:00:15.957] [9/16 IntermediateHashes] SMT regenerate progress: (49%)
Jul 17 11:00:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:00:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=704.6MB sys=2.4GB
Jul 17 11:00:25 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:00:25.958] [9/16 IntermediateHashes] SMT regenerate progress: (49%)
Jul 17 11:00:35 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:00:35.958] [9/16 IntermediateHashes] SMT regenerate progress: (49%)
Jul 17 11:00:45 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:00:45.958] [9/16 IntermediateHashes] SMT regenerate progress: (49%)
Jul 17 11:00:55 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:00:55.958] [9/16 IntermediateHashes] SMT regenerate progress: (49%)
Jul 17 11:01:05 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:05.957] [9/16 IntermediateHashes] SMT regenerate progress: (49%)
Jul 17 11:01:11 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:11.675] [9/16 IntermediateHashes] Collecting account data finished in 25m24.896504406s
Jul 17 11:01:11 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:11.677] [9/16 IntermediateHashes] Building temp binary tree started
Jul 17 11:01:11 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:11.679] [9/16 IntermediateHashes] Total values to insert: 11861293
Jul 17 11:01:11 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:11.680] [9/16 IntermediateHashes] Sorting keys...
Jul 17 11:01:15 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:15.549] [p2p] GoodPeers
Jul 17 11:01:20 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:20.657] [9/16 IntermediateHashes] Keys sorted in 8.977137867s
Jul 17 11:01:21 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:21.053] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=1.5GB sys=2.9GB
Jul 17 11:01:30 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:30.660] [9/16 IntermediateHashes] SMT regenerate progress: (50%)
Jul 17 11:01:40 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:01:40.659] [9/16 IntermediateHashes] SMT regenerate progress: (50%)
...
Jul 17 11:17:40 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:17:40.660] [9/16 IntermediateHashes] SMT regenerate progress: (51%)
...
...
Jul 17 11:52:00 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|11:52:00.659] [9/16 IntermediateHashes] SMT regenerate progress: (55%)
...
...
...
Jul 17 12:17:50 m-xlayer-01 cdk-erigon[2183529]: [INFO] [07-17|12:17:50.660] [9/16 IntermediateHashes] SMT regenerate progress: (58%)
andreclaro commented 1 month ago

image

image

iostat nvme1n1
Linux 6.2.0-1012-aws (m-xlayer-01)  07/17/24    _x86_64_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          31.60    0.01    2.53   14.24    0.00   51.61

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
nvme1n1         846.55     49378.88     56419.74         0.00 115888644449 132413039148          0
andreclaro commented 1 month ago

it was quicker when I switched to a bigger instance type, from m6a.xlarger to m6a.4xlarger (64 Gb of memory)... probably due to the memory requirements of this step 🤔