search

ledgerwatch / erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
3.03k stars 1.05k forks source link

Node does not get in sync and is always behind a couple hundred blocks #10141

Open tnet-ag opened 2 months ago

tnet-ag commented 2 months ago

I am trying to install erigon on a bare metal kubernetes cluster

The node is in sync now always a couple hundred blocks behind

web3.is_connected() is always false but I can use the rpc endpoint for example web3.eth.get_block('latest')['number'] (web3py)

As I can see it in the logs it always tries to start the snapshot process again(?)

There are no pod crashes/restarts.

How to solve/debug this?

ENV Description:

Image: thorax/erigon:v2.59.3

The pod resource Request/Limits (==) 64 GB Ram 12 Cores

The Node start command:

erigon --datadir=/data --http=false --private.api.addr=127.0.0.1:9090 --authrpc.jwtsecret=/data/jwt.hex --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --ws --metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --chain=mainnet --log.console.verbosity=info --nat=extip:*** --port=31000 --http.api=engine,eth,erigon,web3,net,debug,trace,txpool --torrent.download.rate=10000mb --torrent.upload.rate=16mb --internalcl I have also tried to start with --snapshots=false --snap.stop -> no effect

The logs (on of the loops)

[INFO] [04-30|04:10:05.177] [11/12 TxLookup] ETL [2/2] Loading       into=BlockTransactionLookup current_prefix=a493be41
[INFO] [04-30|04:10:16.663] [p2p] GoodPeers                          eth67=13 eth68=30
[INFO] [04-30|04:10:21.814] [mem] memory stats                       Rss=54.0MB Size=0B Pss=53.9MB SharedClean=226.6KB SharedDirty=4B PrivateClean=46.5MB PrivateDirty=7.3MB Referenced=52.3MB Anonymous=7.3MB Swap=0B alloc=5.7GB sys=14.4GB
[INFO] [04-30|04:10:27.623] RPC Daemon notified of new headers       from=19765529 to=19765609 amount=80 hash=0x455fdfb3947eb59b58fa5fe47efe3f16a503ec3f36bbba854e228351c5c36878 header sending=182.104µs log sending=539ns
[INFO] [04-30|04:10:27.623] head updated                             hash=0x455fdfb3947eb59b58fa5fe47efe3f16a503ec3f36bbba854e228351c5c36878 number=19765609
[INFO] [04-30|04:10:34.006] [txpool] stat                            pending=3273 baseFee=15279 queued=12280 alloc=5.8GB sys=14.4GB
[INFO] [04-30|04:11:03.653] P2P                                      app=caplin peers=67
[INFO] [04-30|04:12:03.653] P2P                                      app=caplin peers=64
[INFO] [04-30|04:13:03.654] P2P                                      app=caplin peers=64
[INFO] [04-30|04:13:16.663] [p2p] GoodPeers                          eth68=30 eth67=13
[INFO] [04-30|04:13:21.143] [mem] memory stats                       Rss=55.2MB Size=0B Pss=55.1MB SharedClean=226.6KB SharedDirty=4B PrivateClean=45.3MB PrivateDirty=9.7MB Referenced=53.8MB Anonymous=9.7MB Swap=0B alloc=7.5GB sys=14.4GB
[INFO] [04-30|04:13:33.993] [txpool] stat                            pending=4398 baseFee=15614 queued=12717 alloc=8.2GB sys=14.4GB
[INFO] [04-30|04:14:03.653] P2P                                      app=caplin peers=62
[INFO] [04-30|04:15:03.654] P2P                                      app=caplin peers=60
[INFO] [04-30|04:16:03.653] P2P                                      app=caplin peers=65
[INFO] [04-30|04:16:16.663] [p2p] GoodPeers                          eth68=30 eth67=13
[INFO] [04-30|04:16:21.386] [mem] memory stats                       Rss=56.3MB Size=0B Pss=56.2MB SharedClean=226.6KB SharedDirty=0B PrivateClean=44.0MB PrivateDirty=12.0MB Referenced=54.9MB Anonymous=12.1MB Swap=0B alloc=6.1GB sys=14.4GB
[INFO] [04-30|04:16:33.992] [txpool] stat                            pending=5341 baseFee=16160 queued=13063 alloc=6.7GB sys=14.4GB
[INFO] [04-30|04:17:03.653] P2P                                      app=caplin peers=52
[INFO] [04-30|04:18:03.653] P2P                                      app=caplin peers=59
[INFO] [04-30|04:19:03.653] P2P                                      app=caplin peers=55
[INFO] [04-30|04:19:16.663] [p2p] GoodPeers                          eth68=31 eth67=14
[INFO] [04-30|04:19:21.697] [mem] memory stats                       Rss=55.6MB Size=0B Pss=55.5MB SharedClean=226.6KB SharedDirty=4B PrivateClean=43.3MB PrivateDirty=12.1MB Referenced=54.3MB Anonymous=12.2MB Swap=0B alloc=6.8GB sys=14.4GB
[INFO] [04-30|04:19:33.993] [txpool] stat                            pending=6269 baseFee=16549 queued=13276 alloc=7.2GB sys=14.4GB
[INFO] [04-30|04:20:03.653] P2P                                      app=caplin peers=65
[INFO] [04-30|04:21:03.653] P2P                                      app=caplin peers=68
[INFO] [04-30|04:21:09.628] [3/12 Senders] Started                   from=19765609 to=19765701
[INFO] [04-30|04:21:09.976] [4/12 Execution] Blocks execution        from=19765609 to=19765701
[INFO] [04-30|04:21:40.191] [4/12 Execution] Executed blocks         number=19765667 blk/s=1.9 tx/s=323.9 Mgas/s=28.7 gasState=0.00 batch=1.6MB alloc=7.3GB sys=14.4GB
[INFO] [04-30|04:22:03.653] P2P                                      app=caplin peers=69
[INFO] [04-30|04:22:08.607] [4/12 Execution] Completed on            block=19765701
[INFO] [04-30|04:22:08.607] [5/12 HashState] Promoting plain state   from=19765609 to=19765701
[INFO] [04-30|04:22:08.607] [5/12 HashState] Incremental promotion   from=19765609 to=19765701 codes=true csbucket=AccountChangeSet
[INFO] [04-30|04:22:11.848] [5/12 HashState] Incremental promotion   from=19765609 to=19765701 codes=false csbucket=AccountChangeSet
[INFO] [04-30|04:22:16.663] [p2p] GoodPeers                          eth68=31 eth67=14
[INFO] [04-30|04:22:21.064] [mem] memory stats                       Rss=55.3MB Size=0B Pss=55.2MB SharedClean=226.6KB SharedDirty=4B PrivateClean=42.5MB PrivateDirty=12.6MB Referenced=54.0MB Anonymous=12.3MB Swap=0B alloc=8.0GB sys=14.4GB
[INFO] [04-30|04:22:33.996] [txpool] stat                            pending=7222 baseFee=17004 queued=13457 alloc=8.1GB sys=14.4GB
[INFO] [04-30|04:22:42.086] [5/12 HashState] ETL [2/2] Loading       into=HashedAccount current_prefix=cb4aa26c
[INFO] [04-30|04:22:46.560] [5/12 HashState] Incremental promotion   from=19765609 to=19765701 codes=false csbucket=StorageChangeSet
[INFO] [04-30|04:23:03.653] P2P                                      app=caplin peers=65
[INFO] [04-30|04:23:12.703] [5/12 HashState] DONE                    in=1m4.096516608s
[INFO] [04-30|04:23:12.703] [6/12 IntermediateHashes] Generating intermediate hashes from=19765609 to=19765701
[INFO] [04-30|04:23:43.147] [6/12 IntermediateHashes] Calculating Merkle root current key=625fb4db
[INFO] [04-30|04:24:03.654] P2P                                      app=caplin peers=62
[INFO] [04-30|04:24:13.132] [6/12 IntermediateHashes] Calculating Merkle root current key=d531de8c
[INFO] [04-30|04:24:25.124] [6/12 IntermediateHashes] DONE           in=1m12.420163731s
[INFO] [04-30|04:24:25.603] [7/12 CallTraces] Pruned call trace intermediate table from=19675609 to=19675700
[INFO] [04-30|04:24:55.630] [7/12 CallTraces] ETL [2/2] Loading      into=CallFromIndex current_prefix=e96edd48
[INFO] [04-30|04:25:03.654] P2P                                      app=caplin peers=64
[INFO] [04-30|04:25:16.663] [p2p] GoodPeers                          eth68=31 eth67=14
[INFO] [04-30|04:25:21.457] [mem] memory stats                       Rss=51.9MB Size=0B Pss=51.8MB SharedClean=226.6KB SharedDirty=4B PrivateClean=43.8MB PrivateDirty=7.8MB Referenced=50.5MB Anonymous=7.5MB Swap=0B alloc=5.1GB sys=14.4GB
[INFO] [04-30|04:25:28.642] [7/12 CallTraces] ETL [2/2] Loading      into=CallToIndex current_prefix=d357880c
[INFO] [04-30|04:25:33.993] [txpool] stat                            pending=8058 baseFee=17451 queued=14425 alloc=5.2GB sys=14.4GB
[INFO] [04-30|04:25:36.160] [7/12 CallTraces] DONE                   in=1m11.035922429s
[INFO] [04-30|04:26:03.653] P2P                                      app=caplin peers=60
[INFO] [04-30|04:26:06.327] [8/12 AccountHistoryIndex] ETL [2/2] Loading into=AccountHistory current_prefix=8e153f00
[INFO] [04-30|04:26:54.866] [9/12 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=b8901acb
[INFO] [04-30|04:27:03.653] P2P                                      app=caplin peers=61
[INFO] [04-30|04:27:43.613] [10/12 LogIndex] ETL [2/2] Loading       into=LogTopicIndex current_prefix=c5475653
[INFO] [04-30|04:28:03.653] P2P                                      app=caplin peers=61
[INFO] [04-30|04:28:15.935] [11/12 TxLookup] ETL [2/2] Loading       into=BlockTransactionLookup current_prefix=a6a9574a
[INFO] [04-30|04:28:16.662] [p2p] GoodPeers                          eth68=31 eth67=14
[INFO] [04-30|04:28:21.645] [mem] memory stats                       Rss=53.0MB Size=0B Pss=52.9MB SharedClean=226.6KB SharedDirty=0B PrivateClean=45.3MB PrivateDirty=7.4MB Referenced=51.7MB Anonymous=7.3MB Swap=0B alloc=5.7GB sys=14.4GB
[INFO] [04-30|04:28:33.995] [txpool] stat                            pending=8923 baseFee=17896 queued=14574 alloc=5.8GB sys=14.4GB
[INFO] [04-30|04:28:37.032] RPC Daemon notified of new headers       from=19765609 to=19765701 amount=92 hash=0x1de0c220c4dc021bb91e98d8bde7479c97f69580b2dcd459f26f32e9f843f1b0 header sending=198.107µs log sending=666ns
[INFO] [04-30|04:28:37.032] head updated                             hash=0x1de0c220c4dc021bb91e98d8bde7479c97f69580b2dcd459f26f32e9f843f1b0 number=19765701
[INFO] [04-30|04:29:03.653] P2P                                      app=caplin peers=64
[INFO] [04-30|04:30:03.653] P2P                                      app=caplin peers=63
[INFO] [04-30|04:31:03.653] P2P                                      app=caplin peers=64
[INFO] [04-30|04:31:16.663] [p2p] GoodPeers                          eth68=31 eth67=14
[INFO] [04-30|04:31:21.328] [mem] memory stats                       Rss=54.8MB Size=0B Pss=54.6MB SharedClean=226.6KB SharedDirty=0B PrivateClean=44.6MB PrivateDirty=10.0MB Referenced=53.4MB Anonymous=10.0MB Swap=0B alloc=6.6GB sys=14.4GB
[INFO] [04-30|04:31:33.992] [txpool] stat                            pending=4457 baseFee=15965 queued=13678 alloc=7.3GB sys=14.4GB
[INFO] [04-30|04:32:03.653] P2P                                      app=caplin peers=64
[INFO] [04-30|04:33:03.653] P2P                                      app=caplin peers=59
[INFO] [04-30|04:34:03.653] P2P                                      app=caplin peers=54
[INFO] [04-30|04:34:16.662] [p2p] GoodPeers                          eth68=31 eth67=14
[INFO] [04-30|04:34:21.691] [mem] memory stats                       Rss=55.2MB Size=0B Pss=55.1MB SharedClean=226.6KB SharedDirty=0B PrivateClean=42.8MB PrivateDirty=12.2MB Referenced=53.8MB Anonymous=12.2MB Swap=0B alloc=6.7GB sys=14.4GB
[INFO] [04-30|04:34:33.993] [txpool] stat                            pending=5802 baseFee=16196 queued=13983 alloc=7.4GB sys=14.4GB
[INFO] [04-30|04:35:03.653] P2P                                      app=caplin peers=53
[INFO] [04-30|04:36:03.653] P2P                                      app=caplin peers=66
[INFO] [04-30|04:37:03.653] P2P                                      app=caplin peers=54
[INFO] [04-30|04:37:16.663] [p2p] GoodPeers                          eth67=14 eth68=31
[INFO] [04-30|04:37:21.579] [mem] memory stats                       Rss=55.0MB Size=0B Pss=54.9MB SharedClean=226.6KB SharedDirty=4B PrivateClean=42.4MB PrivateDirty=12.4MB Referenced=53.6MB Anonymous=12.4MB Swap=0B alloc=9.5GB sys=14.4GB
[INFO] [04-30|04:37:34.022] [txpool] stat                            pending=7127 baseFee=16267 queued=14181 alloc=6.5GB sys=14.4GB
[INFO] [04-30|04:38:03.654] P2P                                      app=caplin peers=62
[INFO] [04-30|04:39:03.653] P2P                                      app=caplin peers=57
[INFO] [04-30|04:40:03.653] P2P                                      app=caplin peers=65
[INFO] [04-30|04:40:16.663] [p2p] GoodPeers                          eth68=31 eth67=14
[INFO] [04-30|04:40:21.426] [mem] memory stats                       Rss=54.0MB Size=0B Pss=53.9MB SharedClean=226.6KB SharedDirty=4B PrivateClean=41.2MB PrivateDirty=12.7MB Referenced=52.7MB Anonymous=12.7MB Swap=0B alloc=9.2GB sys=14.7GB
[INFO] [04-30|04:40:34.003] [txpool] stat                            pending=8250 baseFee=16392 queued=14337 alloc=9.4GB sys=14.7GB
[INFO] [04-30|04:40:58.027] [3/12 Senders] Started                   from=19765701 to=19765800
[INFO] [04-30|04:40:58.376] [4/12 Execution] Blocks execution        from=19765701 to=19765800
[INFO] [04-30|04:41:03.654] P2P                                      app=caplin peers=57
AskAlexSharov commented 2 months ago

--http=false - you disabled RPC

tnet-ag commented 2 months ago

@AskAlexSharov oh yea was not aware of that comes from the Helm chart i used as a base (forked https://artifacthub.io/packages/helm/ethereum-helm-charts/erigon)

But RPC is working isn't http.api=engine,eth,erigon,web3,net,debug,trace,txpool overriding the previous --http=false ??

I will remove that and try if it solves the problem

AskAlexSharov commented 2 months ago

i don't know. you can try by 

 curl -X POST -H "dbg: true" -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "method": "eth_getBlockByNumber", "params": ["0x4C4B40", true], "id":1}' localhost:8545
tnet-ag commented 2 months ago

@AskAlexSharov have checked that

--http=false is correct in that context

the helm charts starts an extra container erigon-rpcd with the rpc deamon running.

of course the http.api=engine,eth,erigon,web3,net,debug,trace,txpool parameter has no effect then on the base node container, will move that as parameter to the RPC Deamon pod and try again

the original command to start the rpcdaemon is

rpcdaemon --datadir=/data --private.api.addr=127.0.0.1:9090 --txpool.api.addr=127.0.0.1:9090 --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --ws --metrics --metrics.addr=0.0.0.0 --metrics.port=6061

This is why the rpc deamon was working.

Good to clarify that.

But the original problem that the node is always behind, should not be affected/caused by that.

AskAlexSharov commented 1 month ago

about erigon - remove: --http.api, --ws

about rpcd: change --private.api.addr=127.0.0.1:9090 to erigon's IP - likely it's in another container and not visible by 127.0.0.1

see our docker-compose.yml as an example of separated start.

show rpcd logs

tnet-ag commented 1 month ago

Changed the RPC Deamon and now web3.is_connected() returns True.

But the Node is still always behind a reference Erigon Node running on Dappnode on another machine in the same Network.

Wrote a test script to visualize that a bit:

while True:

    print('--------')

    web3 = Web3(Web3.HTTPProvider(reference_node_uris))
    is_connected_reference_node =  web3.is_connected()

    reference_block_number = web3.eth.get_block('latest')['number']

    print('REFERENCE', is_connected_reference_node, reference_block_number)

    # check nodes
    for node in nodes:
        web3 = Web3(Web3.HTTPProvider(node['uri']))
        is_connected = web3.is_connected()

        latest_block_number = web3.eth.get_block('latest')['number']

        diff_to_latest = reference_block_number - latest_block_number

        print(node['node_id']  ,is_connected, latest_block_number, 'diff', diff_to_latest)

    time.sleep(60)
--------
REFERENCE True 19768321
erigon-0 True 19767020 diff 1301
--------
REFERENCE True 19768326
erigon-0 True 19767909 diff 417
--------
REFERENCE True 19768331
erigon-0 True 19767909 diff 422
--------
REFERENCE True 19768336
erigon-0 True 19767909 diff 427
--------
REFERENCE True 19768341
erigon-0 True 19767909 diff 432
--------
REFERENCE True 19768346
erigon-0 True 19767909 diff 437
--------
REFERENCE True 19768351
erigon-0 True 19767909 diff 442
--------
REFERENCE True 19768356
erigon-0 True 19767909 diff 447
--------
REFERENCE True 19768361
erigon-0 True 19767909 diff 452
--------
REFERENCE True 19768366
erigon-0 True 19767909 diff 457
--------
REFERENCE True 19768371
erigon-0 True 19767909 diff 462
--------
REFERENCE True 19768376
erigon-0 True 19767909 diff 467
--------
REFERENCE True 19768381
erigon-0 True 19767909 diff 472
--------
REFERENCE True 19768386
erigon-0 True 19767909 diff 477
--------
REFERENCE True 19768391
erigon-0 True 19767909 diff 482
--------
REFERENCE True 19768396
erigon-0 True 19767909 diff 487
--------
REFERENCE True 19768401
erigon-0 True 19767909 diff 492
--------
REFERENCE True 19768406
erigon-0 True 19767909 diff 497
--------
REFERENCE True 19768411
erigon-0 True 19767909 diff 502
--------
REFERENCE True 19768416
erigon-0 True 19767909 diff 507
--------
REFERENCE True 19768421
erigon-0 True 19767909 diff 512
--------
REFERENCE True 19768426
erigon-0 True 19767909 diff 517
--------
REFERENCE True 19768431
erigon-0 True 19767909 diff 522
--------
REFERENCE True 19768436
erigon-0 True 19767909 diff 527
--------
REFERENCE True 19768441
erigon-0 True 19767909 diff 532
--------
REFERENCE True 19768446
erigon-0 True 19767909 diff 537
--------
REFERENCE True 19768451
erigon-0 True 19767909 diff 542
--------
REFERENCE True 19768456
erigon-0 True 19767909 diff 547
--------
REFERENCE True 19768461
erigon-0 True 19767909 diff 552
--------
REFERENCE True 19768466
erigon-0 True 19767909 diff 557
--------
REFERENCE True 19768471
erigon-0 True 19767909 diff 562
--------
REFERENCE True 19768476
erigon-0 True 19767909 diff 567
--------
REFERENCE True 19768481
erigon-0 True 19767909 diff 572
--------
REFERENCE True 19768486
erigon-0 True 19767909 diff 577
--------
REFERENCE True 19768491
erigon-0 True 19767909 diff 582
--------
REFERENCE True 19768496
erigon-0 True 19767909 diff 587
--------
REFERENCE True 19768501
erigon-0 True 19767909 diff 592
--------
REFERENCE True 19768506
erigon-0 True 19767909 diff 597
--------
REFERENCE True 19768511
erigon-0 True 19767909 diff 602
--------
REFERENCE True 19768516
erigon-0 True 19767909 diff 607
--------
REFERENCE True 19768521
erigon-0 True 19767909 diff 612
--------
REFERENCE True 19768526
erigon-0 True 19767909 diff 617
--------
REFERENCE True 19768531
erigon-0 True 19767909 diff 622
--------
REFERENCE True 19768536
erigon-0 True 19767909 diff 627
--------
REFERENCE True 19768541
erigon-0 True 19767909 diff 632
--------
REFERENCE True 19768546
erigon-0 True 19767909 diff 637
--------
REFERENCE True 19768551
erigon-0 True 19767909 diff 642
--------
REFERENCE True 19768557
erigon-0 True 19767909 diff 648
--------
REFERENCE True 19768561
erigon-0 True 19767909 diff 652
--------
REFERENCE True 19768566
erigon-0 True 19768383 diff 183
--------
REFERENCE True 19768571
erigon-0 True 19768383 diff 188
--------
REFERENCE True 19768577
erigon-0 True 19768383 diff 194
--------
REFERENCE True 19768581
erigon-0 True 19768383 diff 198
--------
REFERENCE True 19768586
erigon-0 True 19768383 diff 203
--------
REFERENCE True 19768590
erigon-0 True 19768383 diff 207
--------
REFERENCE True 19768595
erigon-0 True 19768383 diff 212
--------
REFERENCE True 19768600
erigon-0 True 19768383 diff 217
--------
REFERENCE True 19768606
erigon-0 True 19768383 diff 223
--------
REFERENCE True 19768610
erigon-0 True 19768383 diff 227
--------
REFERENCE True 19768616
erigon-0 True 19768383 diff 233
--------
REFERENCE True 19768620
erigon-0 True 19768383 diff 237
--------
REFERENCE True 19768626
erigon-0 True 19768383 diff 243
--------
REFERENCE True 19768630
erigon-0 True 19768383 diff 247
--------
REFERENCE True 19768636
erigon-0 True 19768383 diff 253
--------
REFERENCE True 19768641
erigon-0 True 19768383 diff 258
--------
REFERENCE True 19768645
erigon-0 True 19768383 diff 262
--------
REFERENCE True 19768650
erigon-0 True 19768383 diff 267
--------
REFERENCE True 19768655
erigon-0 True 19768383 diff 272
--------
REFERENCE True 19768660
erigon-0 True 19768383 diff 277
--------
REFERENCE True 19768666
erigon-0 True 19768383 diff 283
--------
REFERENCE True 19768670
erigon-0 True 19768383 diff 287
--------
REFERENCE True 19768675
erigon-0 True 19768383 diff 292
--------
REFERENCE True 19768680
erigon-0 True 19768383 diff 297
--------
REFERENCE True 19768685
erigon-0 True 19768383 diff 302
--------
REFERENCE True 19768691
erigon-0 True 19768383 diff 308
--------
REFERENCE True 19768695
erigon-0 True 19768383 diff 312
--------
REFERENCE True 19768700
erigon-0 True 19768383 diff 317
--------
REFERENCE True 19768705
erigon-0 True 19768383 diff 322
--------
REFERENCE True 19768711
erigon-0 True 19768383 diff 328
--------
REFERENCE True 19768715
erigon-0 True 19768627 diff 88
--------
REFERENCE True 19768721
erigon-0 True 19768627 diff 94
--------
REFERENCE True 19768726
erigon-0 True 19768627 diff 99
--------
REFERENCE True 19768731
erigon-0 True 19768627 diff 104
--------
REFERENCE True 19768736
erigon-0 True 19768627 diff 109
--------
REFERENCE True 19768740
erigon-0 True 19768627 diff 113
--------
REFERENCE True 19768745
erigon-0 True 19768627 diff 118
--------
REFERENCE True 19768751
erigon-0 True 19768627 diff 124
--------
REFERENCE True 19768756
erigon-0 True 19768627 diff 129
--------
REFERENCE True 19768760
erigon-0 True 19768627 diff 133
--------
REFERENCE True 19768766
erigon-0 True 19768627 diff 139
--------
REFERENCE True 19768771
erigon-0 True 19768627 diff 144
--------
REFERENCE True 19768776
erigon-0 True 19768627 diff 149
--------
REFERENCE True 19768781
erigon-0 True 19768627 diff 154
--------
REFERENCE True 19768786
erigon-0 True 19768627 diff 159
--------
REFERENCE True 19768791
erigon-0 True 19768627 diff 164
--------
REFERENCE True 19768796
erigon-0 True 19768627 diff 169
--------
REFERENCE True 19768801
erigon-0 True 19768751 diff 50
--------
REFERENCE True 19768805
erigon-0 True 19768751 diff 54
--------
REFERENCE True 19768811
erigon-0 True 19768751 diff 60
--------
REFERENCE True 19768816
erigon-0 True 19768751 diff 65
--------
REFERENCE True 19768821
erigon-0 True 19768751 diff 70
--------
REFERENCE True 19768826
erigon-0 True 19768751 diff 75
--------
REFERENCE True 19768831
erigon-0 True 19768751 diff 80
--------
REFERENCE True 19768836
erigon-0 True 19768751 diff 85
--------
REFERENCE True 19768841
erigon-0 True 19768751 diff 90
--------
REFERENCE True 19768846
erigon-0 True 19768751 diff 95
--------
REFERENCE True 19768850
erigon-0 True 19768751 diff 99
--------
REFERENCE True 19768855
erigon-0 True 19768751 diff 104
--------
REFERENCE True 19768861
erigon-0 True 19768751 diff 110
--------
REFERENCE True 19768866
erigon-0 True 19768751 diff 115
--------
REFERENCE True 19768870
erigon-0 True 19768751 diff 119
--------
REFERENCE True 19768876
erigon-0 True 19768751 diff 125
--------
REFERENCE True 19768881
erigon-0 True 19768751 diff 130
--------
REFERENCE True 19768886
erigon-0 True 19768751 diff 135
--------
REFERENCE True 19768891
erigon-0 True 19768751 diff 140
--------
REFERENCE True 19768896
erigon-0 True 19768751 diff 145
--------
REFERENCE True 19768901
erigon-0 True 19768751 diff 150
--------
REFERENCE True 19768906
erigon-0 True 19768751 diff 155
--------
REFERENCE True 19768911
erigon-0 True 19768751 diff 160
--------
REFERENCE True 19768916
erigon-0 True 19768751 diff 165
--------
REFERENCE True 19768921
erigon-0 True 19768751 diff 170
--------
REFERENCE True 19768925
erigon-0 True 19768867 diff 58
--------
REFERENCE True 19768930
erigon-0 True 19768867 diff 63
--------
REFERENCE True 19768935
erigon-0 True 19768867 diff 68
--------
REFERENCE True 19768940
erigon-0 True 19768867 diff 73
--------
REFERENCE True 19768945
erigon-0 True 19768867 diff 78
--------
REFERENCE True 19768950
erigon-0 True 19768867 diff 83
--------
REFERENCE True 19768955
erigon-0 True 19768867 diff 88
--------
REFERENCE True 19768960
erigon-0 True 19768867 diff 93
--------
REFERENCE True 19768965
erigon-0 True 19768867 diff 98
--------
REFERENCE True 19768970
erigon-0 True 19768867 diff 103
--------
REFERENCE True 19768975
erigon-0 True 19768867 diff 108
--------
REFERENCE True 19768980
erigon-0 True 19768867 diff 113
--------
REFERENCE True 19768985
erigon-0 True 19768867 diff 118
--------
REFERENCE True 19768990
erigon-0 True 19768867 diff 123
--------
REFERENCE True 19768995
erigon-0 True 19768867 diff 128
--------
REFERENCE True 19769000
erigon-0 True 19768867 diff 133
--------
REFERENCE True 19769005
erigon-0 True 19768867 diff 138
--------
REFERENCE True 19769010
erigon-0 True 19768867 diff 143
--------
REFERENCE True 19769015
erigon-0 True 19768867 diff 148
--------
REFERENCE True 19769020
erigon-0 True 19768867 diff 153
--------
REFERENCE True 19769025
erigon-0 True 19768867 diff 158
--------
REFERENCE True 19769030
erigon-0 True 19768867 diff 163
--------
REFERENCE True 19769034
erigon-0 True 19768986 diff 48
--------
REFERENCE True 19769040
erigon-0 True 19768986 diff 54
--------
REFERENCE True 19769045
erigon-0 True 19768986 diff 59
--------
REFERENCE True 19769050
erigon-0 True 19768986 diff 64
--------
REFERENCE True 19769055
erigon-0 True 19768986 diff 69
--------
REFERENCE True 19769060
erigon-0 True 19768986 diff 74
--------
REFERENCE True 19769065
erigon-0 True 19768986 diff 79
--------
REFERENCE True 19769070
erigon-0 True 19768986 diff 84
--------
REFERENCE True 19769075
erigon-0 True 19768986 diff 89
--------
REFERENCE True 19769080
erigon-0 True 19768986 diff 94
--------
REFERENCE True 19769085
erigon-0 True 19768986 diff 99
--------
REFERENCE True 19769090
erigon-0 True 19768986 diff 104
--------
REFERENCE True 19769095
erigon-0 True 19768986 diff 109
--------
REFERENCE True 19769100
erigon-0 True 19768986 diff 114
--------
REFERENCE True 19769105
erigon-0 True 19768986 diff 119
--------
REFERENCE True 19769110
erigon-0 True 19768986 diff 124
--------
REFERENCE True 19769115
erigon-0 True 19768986 diff 129
--------
REFERENCE True 19769120
erigon-0 True 19768986 diff 134
--------
REFERENCE True 19769124
erigon-0 True 19768986 diff 138
--------
REFERENCE True 19769129
erigon-0 True 19768986 diff 143
--------
REFERENCE True 19769134
erigon-0 True 19768986 diff 148
--------
REFERENCE True 19769139
erigon-0 True 19768986 diff 153
--------
REFERENCE True 19769144
erigon-0 True 19768986 diff 158
--------
REFERENCE True 19769149
erigon-0 True 19768986 diff 163
--------
REFERENCE True 19769154
erigon-0 True 19768986 diff 168
--------
REFERENCE True 19769159
erigon-0 True 19768986 diff 173
--------
REFERENCE True 19769164
erigon-0 True 19769119 diff 45
--------
REFERENCE True 19769168
erigon-0 True 19769119 diff 49
--------
REFERENCE True 19769173
erigon-0 True 19769119 diff 54
--------
REFERENCE True 19769178
erigon-0 True 19769119 diff 59
--------
REFERENCE True 19769183
erigon-0 True 19769119 diff 64
--------
REFERENCE True 19769188
erigon-0 True 19769119 diff 69
--------
REFERENCE True 19769193
erigon-0 True 19769119 diff 74
--------
REFERENCE True 19769198
erigon-0 True 19769119 diff 79
--------
REFERENCE True 19769203
erigon-0 True 19769119 diff 84
--------
REFERENCE True 19769208
erigon-0 True 19769119 diff 89
--------
REFERENCE True 19769213
erigon-0 True 19769119 diff 94
--------
REFERENCE True 19769218
erigon-0 True 19769119 diff 99
--------
REFERENCE True 19769223
erigon-0 True 19769119 diff 104
--------
REFERENCE True 19769228
erigon-0 True 19769119 diff 109
--------
REFERENCE True 19769233
erigon-0 True 19769119 diff 114
--------
REFERENCE True 19769238
erigon-0 True 19769119 diff 119
--------
REFERENCE True 19769243
erigon-0 True 19769119 diff 124
--------
REFERENCE True 19769248
erigon-0 True 19769208 diff 40
--------
REFERENCE True 19769253
erigon-0 True 19769208 diff 45
--------
REFERENCE True 19769258
erigon-0 True 19769208 diff 50
--------
REFERENCE True 19769263
erigon-0 True 19769208 diff 55
--------
REFERENCE True 19769268
erigon-0 True 19769208 diff 60
--------
REFERENCE True 19769273
erigon-0 True 19769208 diff 65
--------
REFERENCE True 19769278
erigon-0 True 19769208 diff 70
--------
REFERENCE True 19769283
erigon-0 True 19769208 diff 75
--------
REFERENCE True 19769288
erigon-0 True 19769208 diff 80
--------
REFERENCE True 19769293
erigon-0 True 19769208 diff 85
--------
REFERENCE True 19769298
erigon-0 True 19769208 diff 90
--------
REFERENCE True 19769303
erigon-0 True 19769208 diff 95
--------
REFERENCE True 19769308
erigon-0 True 19769208 diff 100
--------
REFERENCE True 19769313
erigon-0 True 19769208 diff 105
--------
REFERENCE True 19769318
erigon-0 True 19769208 diff 110
--------
REFERENCE True 19769323
erigon-0 True 19769208 diff 115
--------
REFERENCE True 19769328
erigon-0 True 19769208 diff 120
--------
REFERENCE True 19769333
erigon-0 True 19769208 diff 125
--------
REFERENCE True 19769338
erigon-0 True 19769208 diff 130
--------
REFERENCE True 19769343
erigon-0 True 19769208 diff 135
--------
REFERENCE True 19769348
erigon-0 True 19769309 diff 39
--------
REFERENCE True 19769353
erigon-0 True 19769309 diff 44
--------
REFERENCE True 19769358
erigon-0 True 19769309 diff 49
--------
REFERENCE True 19769363
erigon-0 True 19769309 diff 54
--------
REFERENCE True 19769368
erigon-0 True 19769309 diff 59
--------
REFERENCE True 19769373
erigon-0 True 19769309 diff 64
--------
REFERENCE True 19769378
erigon-0 True 19769309 diff 69
--------
REFERENCE True 19769383
erigon-0 True 19769309 diff 74
--------
REFERENCE True 19769388
erigon-0 True 19769309 diff 79
--------
REFERENCE True 19769393
erigon-0 True 19769309 diff 84
--------
REFERENCE True 19769398
erigon-0 True 19769309 diff 89
--------
REFERENCE True 19769403
erigon-0 True 19769309 diff 94
--------
REFERENCE True 19769408
erigon-0 True 19769309 diff 99
--------
REFERENCE True 19769413
erigon-0 True 19769309 diff 104
--------
REFERENCE True 19769418
erigon-0 True 19769309 diff 109
--------
REFERENCE True 19769423
erigon-0 True 19769309 diff 114
--------
REFERENCE True 19769428
erigon-0 True 19769309 diff 119
--------
REFERENCE True 19769433
erigon-0 True 19769309 diff 124
--------
REFERENCE True 19769438
erigon-0 True 19769309 diff 129
--------
REFERENCE True 19769443
erigon-0 True 19769309 diff 134
--------
REFERENCE True 19769448
erigon-0 True 19769309 diff 139
--------
REFERENCE True 19769453
erigon-0 True 19769309 diff 144
--------
REFERENCE True 19769458
erigon-0 True 19769417 diff 41
--------
REFERENCE True 19769463
erigon-0 True 19769417 diff 46
--------
REFERENCE True 19769468
erigon-0 True 19769417 diff 51
--------
REFERENCE True 19769473
erigon-0 True 19769417 diff 56
--------
REFERENCE True 19769478
erigon-0 True 19769417 diff 61
--------
REFERENCE True 19769483
erigon-0 True 19769417 diff 66
--------
REFERENCE True 19769488
erigon-0 True 19769417 diff 71
--------
REFERENCE True 19769493
erigon-0 True 19769473 diff 20
--------
REFERENCE True 19769498
erigon-0 True 19769473 diff 25
--------
REFERENCE True 19769503
erigon-0 True 19769473 diff 30
--------
REFERENCE True 19769508
erigon-0 True 19769473 diff 35
--------
REFERENCE True 19769513
erigon-0 True 19769473 diff 40
--------
REFERENCE True 19769518
erigon-0 True 19769473 diff 45
--------
REFERENCE True 19769523
erigon-0 True 19769473 diff 50
--------
REFERENCE True 19769528
erigon-0 True 19769473 diff 55
--------
REFERENCE True 19769533
erigon-0 True 19769473 diff 60
--------
REFERENCE True 19769538
erigon-0 True 19769473 diff 65

Where can I read more about the Sync stages of Erigon (1 to 12) PLUS the "production mode"( I mean the mode where the Blocks are not comming from Snapshots anymore.)?

Is there a status endpoint or so which tells me in which mode Erigon currently is?

The Node had also more Resources now to run the test above:

resources: # Erigon Node
  limits:
    cpu: 20000m
    memory: 96Gi
  requests:
    cpu: 20000m
    memory: 96Gi

resourcesRPCDaemon:
  limits:
    cpu: 4000m
    memory: 16Gi
  requests:
    cpu: 4000m
    memory: 16Gi

May I have to test/improve the IO, Disk etc?

The PVC (longhorn) with Erigons Datadir is using a single SSD in a HPE Smart Array RAID Setup.

tnet-ag commented 1 month ago

(Closed unintentionally)