search

erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.12k stars 1.11k forks source link

Erigon not progressing #9646

Open PanosChtz opened 7 months ago

PanosChtz commented 7 months ago

System information

Erigon version: ./erigon --version erigon version 2.58.1

OS & Version: Windows/Linux/OSX Ubuntu 22.04

Erigon Command (with flags/config): erigon --chain=mainnet --datadir=/mnt/sdd1/erigon --authrpc.jwtsecret=/var/lib/jwtsecret/jwt.hex --private.api.addr="127.0.0.1:9098" --port=30309 --http.api=web3,eth,net,debug,trace,erigon,txpool,parity,engine,admin --authrpc.port 8552 --http.port 8545 --authrpc.addr localhost --authrpc.vhosts localhost --nat none

Consensus Layer: Nimbus

Consensus Layer Command (with flags/config): nimbus_beacon_node --network=mainnet --data-dir=/mnt/sdb1/nimbus --web3-url=http://127.0.0.1:8551 --jwt-secret=/var/lib/jwtsecret/jwt.hex

Chain/Network: Main

Expected behaviour

Sync

Actual behaviour

stuck at block 

>>> archive_node_w3.eth.get_block_number()
19142631

Backtrace

Mar 08 17:47:14 ubnt-srv systemd[1]: Started Erigon Execution Client (Mainnet).
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.286] logging to file system                   log dir=/mnt/sdd1/erigon/logs file prefix=erigon log level=info json=false
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.286] Build info                               git_branch= git_tag= git_commit=
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.286] Starting Erigon on Ethereum mainnet...
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.287] Maximum peer count                       ETH=100 total=100
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.288] starting HTTP APIs                       port=8545 APIs=web3,eth,net,debug,trace,erigon,txpool,parity,engine,admin
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.288] torrent verbosity                        level=WRN
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.289] Set global gas cap                       cap=50000000
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.356] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.357] Opening Database                         label=chaindata path=/mnt/sdd1/erigon/chaindata
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.359] [db] open                                lable=chaindata sizeLimit=12TB pageSize=8192
Mar 08 17:47:14 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:14.360] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: 1710338135, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
Mar 08 17:47:27 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:27.411] Initialising Ethereum protocol           network=1
Mar 08 17:47:27 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:27.411] Disk storage enabled for ethash DAGs     dir=/mnt/sdd1/erigon/ethash-dags count=2
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.080] Starting private RPC server              on=127.0.0.1:9098
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.081] new subscription to logs established
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.083] rpc filters: subscribing to Erigon events
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.086] [txpool] Started
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.086] New txs subscriber joined
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.086] new subscription to newHeaders established
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.087] Reading JWT secret                       path=/var/lib/jwtsecret/jwt.hex
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.088] HTTP endpoint opened for Engine API      url=127.0.0.1:8552 ws=true ws.compression=true
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.088] JsonRpc endpoint opened                  ws=false ws.compression=true grpc=false http.url=127.0.0.1:8545
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.093] Started P2P networking                   version=67 self=enode://c6b822d3f96f062067c4c35b9c1b98c0e70c0e55125d41d052085322c992a7adc3178ad6025af1863035612203908346b009d324e411e0ae319c46afaabc2ec8@127.0.0.1:30303 name=erigon/v2.58.1/linux-amd64/go1.21.6
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.096] Started P2P networking                   version=68 self=enode://c6b822d3f96f062067c4c35b9c1b98c0e70c0e55125d41d052085322c992a7adc3178ad6025af1863035612203908346b009d324e411e0ae319c46afaabc2ec8@127.0.0.1:30309 name=erigon/v2.58.1/linux-amd64/go1.21.6
Mar 08 17:47:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:28.097] [1/12 Snapshots] Requesting downloads
Mar 08 17:47:35 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:35.098] [snapshots:download] Blocks Stat         blocks=19052k indices=19052k alloc=4.7GB sys=4.9GB
Mar 08 17:47:35 ubnt-srv erigon[2878374]: [INFO] [03-08|17:47:35.100] [snapshots] Prune Blocks                 to=19052000 limit=100
Mar 08 17:50:27 ubnt-srv erigon[2878374]: [INFO] [03-08|17:50:27.411] [p2p] GoodPeers                          eth68=2 eth67=1
Mar 08 17:50:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:50:28.087] [txpool] stat                            pending=0 baseFee=0 queued=1245 alloc=2.4GB sys=4.9GB
Mar 08 17:53:27 ubnt-srv erigon[2878374]: [INFO] [03-08|17:53:27.410] [p2p] GoodPeers                          eth67=1 eth68=4
Mar 08 17:53:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:53:28.087] [txpool] stat                            pending=0 baseFee=0 queued=2341 alloc=2.5GB sys=4.9GB
Mar 08 17:56:27 ubnt-srv erigon[2878374]: [INFO] [03-08|17:56:27.411] [p2p] GoodPeers                          eth66=1 eth68=6 eth67=2
Mar 08 17:56:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:56:28.086] [txpool] stat                            pending=0 baseFee=0 queued=7226 alloc=2.5GB sys=4.9GB
Mar 08 17:59:27 ubnt-srv erigon[2878374]: [INFO] [03-08|17:59:27.410] [p2p] GoodPeers                          eth68=6 eth67=3
Mar 08 17:59:28 ubnt-srv erigon[2878374]: [INFO] [03-08|17:59:28.086] [txpool] stat                            pending=0 baseFee=0 queued=10952 alloc=2.6GB sys=4.9GB

same "GoodPeers + stat log messages continue for ever (waited 1 month)

I want only an archival node for experimental purposes (no staking) but I am in a university network.

AskAlexSharov commented 6 months ago

try increase logs verbosity: --log.console.verbosity=4 try check nimbus log, maybe it can't call erigon

AskAlexSharov commented 6 months ago

seems missconfigured: --authrpc.port 8552 and --web3-url=http://127.0.0.1:8551/

PanosChtz commented 6 months ago

Since my setup is just for experimental purposes and no staking, I restarted erigon a couple of days ago using the internal CL. So basically I stopped nimbus and added --internalcl to my command. The logs are as follows, does this mean that it is progressing?

Mar 11 15:35:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:35:43.872] P2P                                      app=caplin peers=15
Mar 11 15:35:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:35:51.593] [p2p] GoodPeers                          eth68=20 eth67=8
Mar 11 15:35:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:35:54.317] [txpool] stat                            pending=5942 baseFee=6233 queued=5234 alloc=7.6GB sys=15.6GB
Mar 11 15:36:33 ubnt-srv erigon[3952280]: [INFO] [03-11|15:36:33.232] [3/12 Senders] Started                   from=19412717 to=19412788
Mar 11 15:36:33 ubnt-srv erigon[3952280]: [INFO] [03-11|15:36:33.882] [4/12 Execution] Blocks execution        from=19412717 to=19412788
Mar 11 15:36:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:36:43.872] P2P                                      app=caplin peers=17
Mar 11 15:37:04 ubnt-srv erigon[3952280]: [INFO] [03-11|15:37:04.105] [4/12 Execution] Executed blocks         number=19412763 blk/s=1.5 tx/s=274.2 Mgas/s=22.5 gasState=0.00 batch=1.2MB alloc=6.3GB sys=15.6GB
Mar 11 15:37:37 ubnt-srv erigon[3952280]: [INFO] [03-11|15:37:37.270] [4/12 Execution] Completed on            block=19412788
Mar 11 15:37:37 ubnt-srv erigon[3952280]: [INFO] [03-11|15:37:37.275] [4/12 Execution] DONE                    in=1m3.393695783s
Mar 11 15:37:37 ubnt-srv erigon[3952280]: [INFO] [03-11|15:37:37.276] [5/12 HashState] Promoting plain state   from=19412717 to=19412788
Mar 11 15:37:37 ubnt-srv erigon[3952280]: [INFO] [03-11|15:37:37.277] [5/12 HashState] Incremental promotion   from=19412717 to=19412788 codes=true csbucket=AccountChangeSet
Mar 11 15:37:41 ubnt-srv erigon[3952280]: [INFO] [03-11|15:37:41.758] [5/12 HashState] Incremental promotion   from=19412717 to=19412788 codes=false csbucket=AccountChangeSet
Mar 11 15:37:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:37:43.873] P2P                                      app=caplin peers=10
Mar 11 15:37:50 ubnt-srv erigon[3952280]: [INFO] [03-11|15:37:50.891] [5/12 HashState] Incremental promotion   from=19412717 to=19412788 codes=false csbucket=StorageChangeSet
Mar 11 15:38:04 ubnt-srv erigon[3952280]: [INFO] [03-11|15:38:04.713] [6/12 IntermediateHashes] Generating intermediate hashes from=19412717 to=19412788
Mar 11 15:38:35 ubnt-srv erigon[3952280]: [INFO] [03-11|15:38:35.554] [6/12 IntermediateHashes] Calculating Merkle root current key=b9c7d8de
Mar 11 15:38:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:38:43.876] P2P                                      app=caplin peers=22
Mar 11 15:38:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:38:51.593] [p2p] GoodPeers                          eth66=1 eth67=7 eth68=20
Mar 11 15:38:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:38:54.317] [txpool] stat                            pending=6905 baseFee=6686 queued=5177 alloc=8.2GB sys=15.6GB
Mar 11 15:39:04 ubnt-srv erigon[3952280]: [INFO] [03-11|15:39:04.490] [7/12 CallTraces] Pruned call trace intermediate table from=19322717 to=19322787
Mar 11 15:39:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:39:43.878] P2P                                      app=caplin peers=27
Mar 11 15:39:57 ubnt-srv erigon[3952280]: [INFO] [03-11|15:39:57.859] RPC Daemon notified of new headers       from=19412717 to=19412788 hash=0xa9a4976dbdb1877ddfd3a4a0b2a6a13da0bd4cc4a2c22e9e0173b157c2041102 header sending=4.981989ms log sending=283ns
Mar 11 15:39:57 ubnt-srv erigon[3952280]: [INFO] [03-11|15:39:57.861] head updated                             hash=0xa9a4976dbdb1877ddfd3a4a0b2a6a13da0bd4cc4a2c22e9e0173b157c2041102 number=19412788
Mar 11 15:39:57 ubnt-srv erigon[3952280]: [INFO] [03-11|15:39:57.862] [snapshots] Prune Blocks                 to=19050000 limit=100
Mar 11 15:39:57 ubnt-srv erigon[3952280]: [WARN] [03-11|15:39:57.885] [snapshots] retire blocks                err="DumpBlocks: DumpBodies: header missed in db: block_num=19050000,  hash=01be0f86a10ea554774cf6a5d8c8e4ad33fbe87eb1e5b8951bced2611f079db9"
Mar 11 15:40:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:40:43.872] P2P                                      app=caplin peers=24
Mar 11 15:41:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:41:43.873] P2P                                      app=caplin peers=23
Mar 11 15:41:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:41:51.593] [p2p] GoodPeers                          eth68=21 eth67=7
Mar 11 15:41:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:41:54.317] [txpool] stat                            pending=2667 baseFee=5355 queued=4794 alloc=5.5GB sys=15.6GB
Mar 11 15:42:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:42:43.873] P2P                                      app=caplin peers=23
Mar 11 15:43:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:43:43.873] P2P                                      app=caplin peers=20
Mar 11 15:44:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:44:43.873] P2P                                      app=caplin peers=24
Mar 11 15:44:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:44:51.594] [p2p] GoodPeers                          eth68=22 eth67=7
Mar 11 15:44:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:44:54.317] [txpool] stat                            pending=3986 baseFee=5411 queued=5005 alloc=4.7GB sys=15.6GB
Mar 11 15:45:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:45:43.872] P2P                                      app=caplin peers=26
Mar 11 15:46:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:46:43.872] P2P                                      app=caplin peers=23
Mar 11 15:47:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:47:43.882] P2P                                      app=caplin peers=14
Mar 11 15:47:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:47:51.593] [p2p] GoodPeers                          eth67=7 eth68=24
Mar 11 15:47:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:47:54.332] [txpool] stat                            pending=5275 baseFee=5470 queued=5166 alloc=7.7GB sys=15.6GB
Mar 11 15:48:34 ubnt-srv erigon[3952280]: [INFO] [03-11|15:48:34.476] [3/12 Senders] Started                   from=19412788 to=19412847
Mar 11 15:48:35 ubnt-srv erigon[3952280]: [INFO] [03-11|15:48:35.257] [4/12 Execution] Blocks execution        from=19412788 to=19412847
Mar 11 15:48:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:48:43.876] P2P                                      app=caplin peers=10
Mar 11 15:49:05 ubnt-srv erigon[3952280]: [INFO] [03-11|15:49:05.338] [4/12 Execution] Executed blocks         number=19412809 blk/s=0.7 tx/s=125.1 Mgas/s=10.9 gasState=0.00 batch=663.2KB alloc=6.4GB sys=15.6GB
Mar 11 15:49:40 ubnt-srv erigon[3952280]: [INFO] [03-11|15:49:40.776] [4/12 Execution] Completed on            block=19412847
Mar 11 15:49:40 ubnt-srv erigon[3952280]: [INFO] [03-11|15:49:40.781] [4/12 Execution] DONE                    in=1m5.546423848s
Mar 11 15:49:40 ubnt-srv erigon[3952280]: [INFO] [03-11|15:49:40.784] [5/12 HashState] Promoting plain state   from=19412788 to=19412847
Mar 11 15:49:40 ubnt-srv erigon[3952280]: [INFO] [03-11|15:49:40.784] [5/12 HashState] Incremental promotion   from=19412788 to=19412847 codes=true csbucket=AccountChangeSet
Mar 11 15:49:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:49:43.857] [5/12 HashState] Incremental promotion   from=19412788 to=19412847 codes=false csbucket=AccountChangeSet
Mar 11 15:49:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:49:43.884] P2P                                      app=caplin peers=12
Mar 11 15:49:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:49:51.554] [5/12 HashState] Incremental promotion   from=19412788 to=19412847 codes=false csbucket=StorageChangeSet
Mar 11 15:50:01 ubnt-srv erigon[3952280]: [INFO] [03-11|15:50:01.763] [6/12 IntermediateHashes] Generating intermediate hashes from=19412788 to=19412847
Mar 11 15:50:32 ubnt-srv erigon[3952280]: [INFO] [03-11|15:50:32.265] [6/12 IntermediateHashes] Calculating Merkle root current key=ced84a96
Mar 11 15:50:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:50:43.880] P2P                                      app=caplin peers=13
Mar 11 15:50:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:50:51.594] [p2p] GoodPeers                          eth68=24 eth67=9
Mar 11 15:50:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:50:54.324] [txpool] stat                            pending=6507 baseFee=5527 queued=5420 alloc=8.0GB sys=15.6GB
Mar 11 15:50:59 ubnt-srv erigon[3952280]: [INFO] [03-11|15:50:59.655] [7/12 CallTraces] Pruned call trace intermediate table from=19322788 to=19322846
Mar 11 15:51:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:51:43.875] P2P                                      app=caplin peers=17
Mar 11 15:51:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:51:51.802] RPC Daemon notified of new headers       from=19412788 to=19412847 hash=0xb7a8aece8b4cb3ccbce92b18b8347e30f2bece4c1cddec14391e5b7f47e47b1a header sending=2.910802ms log sending=205ns
Mar 11 15:51:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:51:51.806] head updated                             hash=0xb7a8aece8b4cb3ccbce92b18b8347e30f2bece4c1cddec14391e5b7f47e47b1a number=19412847
Mar 11 15:51:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:51:51.809] [snapshots] Prune Blocks                 to=19050000 limit=100
Mar 11 15:51:51 ubnt-srv erigon[3952280]: [WARN] [03-11|15:51:51.840] [snapshots] retire blocks                err="DumpBlocks: DumpBodies: header missed in db: block_num=19050000,  hash=01be0f86a10ea554774cf6a5d8c8e4ad33fbe87eb1e5b8951bced2611f079db9"
Mar 11 15:52:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:52:43.873] P2P                                      app=caplin peers=19
Mar 11 15:53:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:53:43.875] P2P                                      app=caplin peers=21
Mar 11 15:53:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:53:51.594] [p2p] GoodPeers                          eth68=24 eth67=7
Mar 11 15:53:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:53:54.321] [txpool] stat                            pending=2328 baseFee=5874 queued=4778 alloc=9.2GB sys=15.6GB
Mar 11 15:54:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:54:43.885] P2P                                      app=caplin peers=12
Mar 11 15:55:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:55:43.874] P2P                                      app=caplin peers=18
Mar 11 15:56:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:56:43.873] P2P                                      app=caplin peers=23
Mar 11 15:56:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:56:51.593] [p2p] GoodPeers                          eth68=27 eth67=7
Mar 11 15:56:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:56:54.317] [txpool] stat                            pending=3551 baseFee=6240 queued=5058 alloc=6.9GB sys=15.6GB
Mar 11 15:57:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:57:43.874] P2P                                      app=caplin peers=18
Mar 11 15:58:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:58:43.874] P2P                                      app=caplin peers=19
Mar 11 15:58:57 ubnt-srv erigon[3952280]: [INFO] [03-11|15:58:57.793] [3/12 Senders] Started                   from=19412847 to=19412899
Mar 11 15:58:58 ubnt-srv erigon[3952280]: [INFO] [03-11|15:58:58.312] [4/12 Execution] Blocks execution        from=19412847 to=19412899
Mar 11 15:59:29 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:29.189] [4/12 Execution] Executed blocks         number=19412872 blk/s=0.8 tx/s=142.6 Mgas/s=12.1 gasState=0.00 batch=798.8KB alloc=10.9GB sys=15.6GB
Mar 11 15:59:43 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:43.873] P2P                                      app=caplin peers=23
Mar 11 15:59:48 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:48.833] [4/12 Execution] Completed on            block=19412899
Mar 11 15:59:48 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:48.837] [5/12 HashState] Promoting plain state   from=19412847 to=19412899
Mar 11 15:59:48 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:48.837] [5/12 HashState] Incremental promotion   from=19412847 to=19412899 codes=true csbucket=AccountChangeSet
Mar 11 15:59:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:51.457] [5/12 HashState] Incremental promotion   from=19412847 to=19412899 codes=false csbucket=AccountChangeSet
Mar 11 15:59:51 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:51.593] [p2p] GoodPeers                          eth68=27 eth67=7
Mar 11 15:59:54 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:54.326] [txpool] stat                            pending=4770 baseFee=6450 queued=5326 alloc=5.1GB sys=15.6GB
Mar 11 15:59:56 ubnt-srv erigon[3952280]: [INFO] [03-11|15:59:56.714] [5/12 HashState] Incremental promotion   from=19412847 to=19412899 codes=false csbucket=StorageChangeSet
Mar 11 16:00:03 ubnt-srv erigon[3952280]: [INFO] [03-11|16:00:03.650] [6/12 IntermediateHashes] Generating intermediate hashes from=19412847 to=19412899
Mar 11 16:00:34 ubnt-srv erigon[3952280]: [INFO] [03-11|16:00:34.818] [7/12 CallTraces] Pruned call trace intermediate table from=19322847 to=19322898
Mar 11 16:00:43 ubnt-srv erigon[3952280]: [INFO] [03-11|16:00:43.874] P2P                                      app=caplin peers=19
Mar 11 16:01:08 ubnt-srv erigon[3952280]: [INFO] [03-11|16:01:07.954] RPC Daemon notified of new headers       from=19412847 to=19412899 hash=0x4644b95d9789f1f8b21c604c94b041e0ca655252200f963b461562a7e6f2f44b header sending=1.749095ms log sending=259ns
Mar 11 16:01:08 ubnt-srv erigon[3952280]: [INFO] [03-11|16:01:08.003] head updated                             hash=0x4644b95d9789f1f8b21c604c94b041e0ca655252200f963b461562a7e6f2f44b number=19412899
Mar 11 16:01:08 ubnt-srv erigon[3952280]: [INFO] [03-11|16:01:08.018] [snapshots] Prune Blocks                 to=19050000 limit=100
Mar 11 16:01:08 ubnt-srv erigon[3952280]: [WARN] [03-11|16:01:08.149] [snapshots] retire blocks                err="DumpBlocks: DumpBodies: header missed in db: block_num=19050000,  hash=01be0f86a10ea554774cf6a5d8c8e4ad33fbe87eb1e5b8951bced2611f079db9"
Mar 11 16:01:43 ubnt-srv erigon[3952280]: [INFO] [03-11|16:01:43.875] P2P                                      app=caplin peers=23
Mar 11 16:02:43 ubnt-srv erigon[3952280]: [INFO] [03-11|16:02:43.872] P2P                                      app=caplin peers=16
Mar 11 16:02:51 ubnt-srv erigon[3952280]: [INFO] [03-11|16:02:51.594] [p2p] GoodPeers                          eth68=28 eth67=8
Mar 11 16:02:54 ubnt-srv erigon[3952280]: [INFO] [03-11|16:02:54.317] [txpool] stat                            pending=2889 baseFee=6046 queued=4976 alloc=5.8GB sys=15.6GB
Mar 11 16:03:43 ubnt-srv erigon[3952280]: [INFO] [03-11|16:03:43.873] P2P                                      app=caplin peers=17
Mar 11 16:04:43 ubnt-srv erigon[3952280]: [INFO] [03-11|16:04:43.873] P2P                                      app=caplin peers=21
Mar 11 16:05:43 ubnt-srv erigon[3952280]: [INFO] [03-11|16:05:43.874] P2P                                      app=caplin peers=26
Mar 11 16:05:51 ubnt-srv erigon[3952280]: [INFO] [03-11|16:05:51.594] [p2p] GoodPeers                          eth68=28 eth67=7
Mar 11 16:05:54 ubnt-srv erigon[3952280]: [INFO] [03-11|16:05:54.319] [txpool] stat                            pending=4241 baseFee=6324 queued=5210 alloc=5.2GB sys=15.6GB
Mar 11 16:06:43 ubnt-srv erigon[3952280]: [INFO] [03-11|16:06:43.872] P2P                                      app=caplin peers=22
AskAlexSharov commented 6 months ago

you are progressing fine:

[7/12 CallTraces] Pruned call trace intermediate table from=19322788 to=19322846
[7/12 CallTraces] Pruned call trace intermediate table from=19322847 to=19322898
PanosChtz commented 6 months ago

So my current configuration is

erigon --chain=mainnet --datadir=/mnt/sdd1/erigon --authrpc.jwtsecret=/var/lib/jwtsecret/jwt.hex --internalcl --private.api.addr="127.0.0.1:9098" --http.api=web3,eth,net,debug,trace,erigon,txpool,parity,engine,admin --authrpc.port 8552 --http.port 8545 --authrpc.addr localhost --authrpc.vhosts localhost

However in python when I do

from web3 import HTTPProvider, Web3
archive_node_w3 = Web3(HTTPProvider('http://127.0.0.1:8545'))
archive_node_w3.eth.get_block_number()

I'm always getting a block that is about 40-50 blocks behind. Is it maybe similar issue to #9658 ? Here are my logs

Mar 15 17:22:04 ubnt-srv erigon[17472]: [INFO] [03-15|17:22:04.439] [4/12 Execution] Blocks execution        from=19441713 to=19441802
Mar 15 17:22:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:22:18.101] P2P                                      app=caplin peers=8
Mar 15 17:22:34 ubnt-srv erigon[17472]: [INFO] [03-15|17:22:34.554] [4/12 Execution] Executed blocks         number=19441725 blk/s=0.4 tx/s=75.7 Mgas/s=6.0 gasState=0.00 batch=466.1KB alloc=4.7GB sys=10.0GB
Mar 15 17:23:04 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:04.783] [4/12 Execution] Executed blocks         number=19441800 blk/s=2.5 tx/s=472.7 Mgas/s=37.4 gasState=0.00 batch=2.4MB alloc=5.3GB sys=10.0GB
Mar 15 17:23:09 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:09.300] [4/12 Execution] Completed on            block=19441802
Mar 15 17:23:09 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:09.301] [4/12 Execution] DONE                    in=1m4.86828728s
Mar 15 17:23:09 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:09.302] [5/12 HashState] Promoting plain state   from=19441713 to=19441802
Mar 15 17:23:09 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:09.302] [5/12 HashState] Incremental promotion   from=19441713 to=19441802 codes=true csbucket=AccountChangeSet
Mar 15 17:23:14 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:14.137] [5/12 HashState] Incremental promotion   from=19441713 to=19441802 codes=false csbucket=AccountChangeSet
Mar 15 17:23:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:18.097] P2P                                      app=caplin peers=19
Mar 15 17:23:23 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:23.786] [5/12 HashState] Incremental promotion   from=19441713 to=19441802 codes=false csbucket=StorageChangeSet
Mar 15 17:23:39 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:39.069] [p2p] GoodPeers                          eth68=4 eth67=1
Mar 15 17:23:40 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:40.090] [6/12 IntermediateHashes] Generating intermediate hashes from=19441713 to=19441802
Mar 15 17:23:48 ubnt-srv erigon[17472]: [INFO] [03-15|17:23:48.600] [txpool] stat                            pending=0 baseFee=0 queued=989 alloc=5.8GB sys=10.0GB
Mar 15 17:24:10 ubnt-srv erigon[17472]: [INFO] [03-15|17:24:10.471] [6/12 IntermediateHashes] Calculating Merkle root current key=909e5152
Mar 15 17:24:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:24:18.100] P2P                                      app=caplin peers=26
Mar 15 17:24:42 ubnt-srv erigon[17472]: [INFO] [03-15|17:24:42.925] [6/12 IntermediateHashes] DONE           in=1m2.836656758s
Mar 15 17:24:43 ubnt-srv erigon[17472]: [INFO] [03-15|17:24:43.140] [7/12 CallTraces] Pruned call trace intermediate table from=19351713 to=19351801
Mar 15 17:25:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:25:18.096] P2P                                      app=caplin peers=29
Mar 15 17:25:40 ubnt-srv erigon[17472]: [INFO] [03-15|17:25:40.520] RPC Daemon notified of new headers       from=19441713 to=19441802 hash=0x45dc2f0fe466a86d5af4ca2ddba48bc8c7f32efe9639e96f538e8f01868a604a header sending=5.60626ms log sending=248ns
Mar 15 17:25:40 ubnt-srv erigon[17472]: [INFO] [03-15|17:25:40.529] head updated                             hash=0x45dc2f0fe466a86d5af4ca2ddba48bc8c7f32efe9639e96f538e8f01868a604a number=19441802
Mar 15 17:25:40 ubnt-srv erigon[17472]: [INFO] [03-15|17:25:40.532] [snapshots] Prune Blocks                 to=19050000 limit=100
Mar 15 17:25:40 ubnt-srv erigon[17472]: [WARN] [03-15|17:25:40.662] [snapshots] retire blocks                err="DumpBlocks: DumpBodies: header missed in db: block_num=19050000,  hash=01be0f86a10ea554774cf6a5d8c8e4ad33fbe87eb1e5b8951bced2611f079db9"
Mar 15 17:26:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:26:18.096] P2P                                      app=caplin peers=20
Mar 15 17:26:39 ubnt-srv erigon[17472]: [INFO] [03-15|17:26:39.068] [p2p] GoodPeers                          eth68=5 eth67=1
Mar 15 17:26:48 ubnt-srv erigon[17472]: [INFO] [03-15|17:26:48.606] [txpool] stat                            pending=1007 baseFee=176 queued=1572 alloc=4.4GB sys=10.7GB
Mar 15 17:27:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:27:18.096] P2P                                      app=caplin peers=28
Mar 15 17:28:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:28:18.097] P2P                                      app=caplin peers=24
Mar 15 17:29:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:29:18.097] P2P                                      app=caplin peers=41
Mar 15 17:29:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:29:18.708] Adding light client update               period=1054
Mar 15 17:29:39 ubnt-srv erigon[17472]: [INFO] [03-15|17:29:39.068] [p2p] GoodPeers                          eth68=6 eth67=3
Mar 15 17:29:48 ubnt-srv erigon[17472]: [INFO] [03-15|17:29:48.600] [txpool] stat                            pending=2219 baseFee=435 queued=1795 alloc=5.7GB sys=12.0GB
Mar 15 17:30:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:30:18.096] P2P                                      app=caplin peers=28
Mar 15 17:31:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:31:18.097] P2P                                      app=caplin peers=26
Mar 15 17:32:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:32:18.099] P2P                                      app=caplin peers=32
Mar 15 17:32:20 ubnt-srv erigon[17472]: [INFO] [03-15|17:32:20.595] [3/12 Senders] Started                   from=19441802 to=19441853
Mar 15 17:32:21 ubnt-srv erigon[17472]: [INFO] [03-15|17:32:21.252] [4/12 Execution] Blocks execution        from=19441802 to=19441853
Mar 15 17:32:39 ubnt-srv erigon[17472]: [INFO] [03-15|17:32:39.069] [p2p] GoodPeers                          eth67=2 eth68=7
Mar 15 17:32:48 ubnt-srv erigon[17472]: [INFO] [03-15|17:32:48.606] [txpool] stat                            pending=3924 baseFee=710 queued=1980 alloc=10.9GB sys=14.7GB
Mar 15 17:32:53 ubnt-srv erigon[17472]: [INFO] [03-15|17:32:53.751] [4/12 Execution] Executed blocks         number=19441821 blk/s=0.6 tx/s=111.5 Mgas/s=8.9 gasState=0.00 batch=688.8KB alloc=11.0GB sys=14.7GB
Mar 15 17:33:18 ubnt-srv erigon[17472]: [INFO] [03-15|17:33:18.097] P2P                                      app=caplin peers=32
Mar 15 17:33:30 ubnt-srv erigon[17472]: [INFO] [03-15|17:33:30.735] [4/12 Execution] Completed on            block=19441853
Mar 15 17:33:30 ubnt-srv erigon[17472]: [INFO] [03-15|17:33:30.739] [4/12 Execution] DONE                    in=1m9.487619982s
Mar 15 17:33:30 ubnt-srv erigon[17472]: [INFO] [03-15|17:33:30.743] [5/12 HashState] Promoting plain state   from=19441802 to=19441853
Mar 15 17:33:30 ubnt-srv erigon[17472]: [INFO] [03-15|17:33:30.743] [5/12 HashState] Incremental promotion   from=19441802 to=19441853 codes=true csbucket=AccountChangeSet
Mar 15 17:33:34 ubnt-srv erigon[17472]: [INFO] [03-15|17:33:34.078] [5/12 HashState] Incremental promotion   from=19441802 to=19441853 codes=false csbucket=AccountChangeSet
Mar 15 17:33:40 ubnt-srv erigon[17472]: [INFO] [03-15|17:33:40.750] [5/12 HashState] Incremental promotion   from=19441802 to=19441853 codes=false csbucket=StorageChangeSet
Mar 15 17:33:52 ubnt-srv erigon[17472]: [INFO] [03-15|17:33:52.265] [6/12 IntermediateHashes] Generating intermediate hashes from=19441802 to=19441853
PanosChtz commented 6 months ago

The internalcl was the issue for the lag, now using an external consensus client and I'm in sync. However I am still getting some errors which repeat for ever and won't go away:

Mar 20 18:31:03 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:03.960] head updated                             hash=0xcdbafb9381eff6a02f6752c43d7e1735095db06eacf9f7664b2497acf247be50 number=19477708
Mar 20 18:31:03 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:03.960] [snapshots] Prune Blocks                 to=19050000 limit=100
Mar 20 18:31:03 ubnt-srv erigon[22192]: [WARN] [03-20|18:31:03.965] [snapshots] retire blocks                err="DumpBlocks: DumpBodies: header missed in db: block_num=19050000,  hash=01be0f86a10ea554774cf6a5d8c8e4ad33fbe87eb1e5b8951bced2611f079db9"
Mar 20 18:31:12 ubnt-srv erigon[22192]: [WARN] [03-20|18:31:12.675] [txpool] flush: sender address not found by ID senderID=678927
Mar 20 18:31:12 ubnt-srv erigon[22192]: [WARN] [03-20|18:31:12.678] [txpool] flush: sender address not found by ID senderID=269726
Mar 20 18:31:12 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:12.761] [NewPayload] Handling new payload        height=19477709 hash=0x1221aa0d79e437750f74af6432efd0d5d2ef780dbf2b0f58cc590790298b00f5
Mar 20 18:31:15 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:15.308] [updateForkchoice] Fork choice update: flushing in-memory state (built by previous newPayload)
Mar 20 18:31:16 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:16.603] RPC Daemon notified of new headers       from=19477708 to=19477709 hash=0x1221aa0d79e437750f74af6432efd0d5d2ef780dbf2b0f58cc590790298b00f5 header sending=26.319µs log sending=239ns
Mar 20 18:31:16 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:16.603] head updated                             hash=0x1221aa0d79e437750f74af6432efd0d5d2ef780dbf2b0f58cc590790298b00f5 number=19477709
Mar 20 18:31:16 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:16.603] [snapshots] Prune Blocks                 to=19050000 limit=100
Mar 20 18:31:16 ubnt-srv erigon[22192]: [WARN] [03-20|18:31:16.604] [snapshots] retire blocks                err="DumpBlocks: DumpBodies: header missed in db: block_num=19050000,  hash=01be0f86a10ea554774cf6a5d8c8e4ad33fbe87eb1e5b8951bced2611f079db9"
Mar 20 18:31:24 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:24.979] [NewPayload] Handling new payload        height=19477710 hash=0x31ec93a91a9d78acf7cc153b83c265160b8e52efe4e50daf0bb928ae741d8fa3
Mar 20 18:31:26 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:26.733] [updateForkchoice] Fork choice update: flushing in-memory state (built by previous newPayload)
Mar 20 18:31:27 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:27.653] RPC Daemon notified of new headers       from=19477709 to=19477710 hash=0x31ec93a91a9d78acf7cc153b83c265160b8e52efe4e50daf0bb928ae741d8fa3 header sending=22.476µs log sending=182ns
Mar 20 18:31:27 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:27.653] head updated                             hash=0x31ec93a91a9d78acf7cc153b83c265160b8e52efe4e50daf0bb928ae741d8fa3 number=19477710
Mar 20 18:31:27 ubnt-srv erigon[22192]: [INFO] [03-20|18:31:27.655] [snapshots] Prune Blocks                 to=19050000 limit=100
Mar 20 18:31:27 ubnt-srv erigon[22192]: [WARN] [03-20|18:31:27.656] [snapshots] retire blocks                err="DumpBlocks: DumpBodies: header missed in db: block_num=19050000,  hash=01be0f86a10ea554774cf6a5d8c8e4ad33fbe87eb1e5b8951bced2611f079db9"
Mar 20 18:31:28 ubnt-srv erigon[22192]: [WARN] [03-20|18:31:28.435] [txpool] flush: sender address not found by ID senderID=678927
Mar 20 18:31:28 ubnt-srv erigon[22192]: [WARN] [03-20|18:31:28.437] [txpool] flush: sender address not found by ID senderID=269726

Any solution for these?