search

erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.12k stars 1.11k forks source link

Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position" #9571

Closed caedmo closed 6 months ago

caedmo commented 7 months ago

Hi all,

My previous ticket was closed (https://github.com/ledgerwatch/erigon/issues/8796). I am still having a problem with Erigon not providing the latest block on the GraphQL endpoint. Highest value is 18299999

Request

{
  block {
    number
  }
}

Response

{
  "data": {
    "block": {
      "number": 18299999
    }
  }
}

I would really appreciate some suggestions? Do I need to start syncing from scratch again? Am I configuring a full-node correctly?

My node has been looping this error for the last 3 weeks, it can't seem to resolve this.

[WARN] [03-02|19:58:23.372] error executing clstage                  app=caplin stage=DownloadHistoricalBlocks err="loadIntoTable Header: error inserting block during collection: ethereumExecutionModule.InsertBlocks: writeBody: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"

Erigon reached tip, then threw an error on insertion Feb 03 11:36:22 anon-r2 erigon[2250351]: [WARN] [02-03|11:36:22.320] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position" Has now restarted sync from 18299999 Also had another OOM a couple hours ago - nothing specific in the logs as to why, and it seems the node continued that time round, instead of restarting sync from snapshot tip

Feb 03 11:36:08 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:08.232] [EngineBlockDownloader] Downloading block bodies block_num=19146830 delivery/sec=6.9MB wasted/sec=1.3MB remaining=393 delivered=179304 cache=39.9MB alloc=5.9GB sys=9.3GB
Feb 03 11:36:13 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:13.152] [NewPayload] Handling new payload        height=19147573 hash=0x1583e47659d4868855a6b956cecb8d4e56f739223768257e3413ce8584106449
Feb 03 11:36:21 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:21.887] [EngineBlockDownloader] Processed        highest=19147224
Feb 03 11:36:21 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:21.890] Beginning downloaded blocks insertion
Feb 03 11:36:22 anon-r2 erigon[2250351]: [WARN] [02-03|11:36:22.320] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"
Feb 03 11:36:22 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:22.883] P2P                                      app=caplin peers=6
Feb 03 11:36:24 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:24.902] [NewPayload] Handling new payload        height=19147574 hash=0x4e98257dfc7a466d22f889306e9a6f32f675ec5736c486b8b4d20d9a8ffced55
Feb 03 11:36:24 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:24.903] [EngineBlockDownloader] Downloading PoS headers... height=unknown hash=0x1583e47659d4868855a6b956cecb8d4e56f739223768257e3413ce8584106449 requestId=0
Feb 03 11:36:29 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:29.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8272014 blockNumber=19075577 blk/sec=6.4 mbps/sec=1.3418 peers=4 snapshots=0 reconnected=false
Feb 03 11:36:59 anon-r2 erigon[2250351]: [INFO] [02-03|11:36:59.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8271256 blockNumber=19074827 blk/sec=25.3 mbps/sec=5.3596 peers=6 snapshots=0 reconnected=false
Feb 03 11:37:01 anon-r2 erigon[2250351]: [INFO] [02-03|11:37:01.910] [p2p] GoodPeers                          eth68=17 eth67=28
Feb 03 11:37:13 anon-r2 erigon[2250351]: [INFO] [02-03|11:37:13.402] [downloader] Downloaded PoS Headers      now=19141429 blk/sec=140.800
Feb 03 11:37:22 anon-r2 erigon[2250351]: [INFO] [02-03|11:37:22.885] P2P                                      app=caplin peers=7
Feb 03 11:37:29 anon-r2 erigon[2250351]: [INFO] [02-03|11:37:29.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8270382 blockNumber=19073965 blk/sec=29.1 mbps/sec=5.3092 peers=6 snapshots=0 reconnected=false
Feb 03 11:37:38 anon-r2 erigon[2250351]: [INFO] [02-03|11:37:38.696] [txpool] stat                            pending=1297 baseFee=0 queued=30000 alloc=6.0GB sys=9.3GB
Feb 03 11:37:43 anon-r2 erigon[2250351]: [INFO] [02-03|11:37:43.109] [downloader] Downloaded PoS Headers      now=19136821 blk/sec=153.600
Feb 03 11:37:59 anon-r2 erigon[2250351]: [INFO] [02-03|11:37:59.175] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8269998 blockNumber=19073581 blk/sec=12.8 mbps/sec=2.6003 peers=3 snapshots=0 reconnected=false
Feb 03 11:38:13 anon-r2 erigon[2250351]: [INFO] [02-03|11:38:13.425] [downloader] Downloaded PoS Headers      now=19131253 blk/sec=185.600
Feb 03 11:38:22 anon-r2 erigon[2250351]: [INFO] [02-03|11:38:22.882] P2P                                      app=caplin peers=4
Feb 03 11:38:29 anon-r2 erigon[2250351]: [INFO] [02-03|11:38:29.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8269774 blockNumber=19073360 blk/sec=7.5 mbps/sec=1.6564 peers=11 snapshots=0 reconnected=false
Feb 03 11:38:43 anon-r2 erigon[2250351]: [INFO] [02-03|11:38:43.423] [downloader] Downloaded PoS Headers      now=19124917 blk/sec=211.200
Feb 03 11:38:59 anon-r2 erigon[2250351]: [INFO] [02-03|11:38:59.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8269582 blockNumber=19073169 blk/sec=6.4 mbps/sec=1.3525 peers=3 snapshots=0 reconnected=false
Feb 03 11:39:12 anon-r2 erigon[2250351]: [INFO] [02-03|11:39:12.990] [downloader] Downloaded PoS Headers      now=19116853 blk/sec=268.800
Feb 03 11:39:22 anon-r2 erigon[2250351]: [INFO] [02-03|11:39:22.882] P2P                                      app=caplin peers=4
Feb 03 11:39:29 anon-r2 erigon[2250351]: [INFO] [02-03|11:39:29.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8269326 blockNumber=19072915 blk/sec=8.5 mbps/sec=1.8904 peers=8 snapshots=0 reconnected=false
Feb 03 11:39:43 anon-r2 erigon[2250351]: [INFO] [02-03|11:39:43.116] [downloader] Downloaded PoS Headers      now=19108789 blk/sec=268.800
Feb 03 11:39:59 anon-r2 erigon[2250351]: [INFO] [02-03|11:39:59.175] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8269198 blockNumber=19072788 blk/sec=4.3 mbps/sec=0.8474 peers=9 snapshots=0 reconnected=false
Feb 03 11:40:01 anon-r2 erigon[2250351]: [INFO] [02-03|11:40:01.909] [p2p] GoodPeers                          eth68=17 eth67=27 eth66=2
Feb 03 11:40:13 anon-r2 erigon[2250351]: [INFO] [02-03|11:40:13.339] [downloader] Downloaded PoS Headers      now=19098805 blk/sec=332.800
Feb 03 11:40:22 anon-r2 erigon[2250351]: [INFO] [02-03|11:40:22.883] P2P                                      app=caplin peers=9
Feb 03 11:40:29 anon-r2 erigon[2250351]: [INFO] [02-03|11:40:29.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8268398 blockNumber=19072000 blk/sec=26.7 mbps/sec=5.2127 peers=9 snapshots=0 reconnected=false
Feb 03 11:40:38 anon-r2 erigon[2250351]: [INFO] [02-03|11:40:38.698] [txpool] stat                            pending=1335 baseFee=0 queued=30000 alloc=5.4GB sys=9.3GB
Feb 03 11:40:42 anon-r2 erigon[2250351]: [INFO] [02-03|11:40:42.990] [downloader] Downloaded PoS Headers      now=19086325 blk/sec=416.000
Feb 03 11:40:59 anon-r2 erigon[2250351]: [INFO] [02-03|11:40:59.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8267598 blockNumber=19071211 blk/sec=26.7 mbps/sec=4.4458 peers=9 snapshots=0 reconnected=false
Feb 03 11:41:13 anon-r2 erigon[2250351]: [INFO] [02-03|11:41:13.124] [downloader] Downloaded PoS Headers      now=19070581 blk/sec=524.800
Feb 03 11:41:22 anon-r2 erigon[2250351]: [INFO] [02-03|11:41:22.882] P2P                                      app=caplin peers=9
Feb 03 11:41:29 anon-r2 erigon[2250351]: [INFO] [02-03|11:41:29.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8266670 blockNumber=19070293 blk/sec=30.9 mbps/sec=6.0149 peers=9 snapshots=0 reconnected=false
Feb 03 11:41:43 anon-r2 erigon[2250351]: [INFO] [02-03|11:41:43.167] [downloader] Downloaded PoS Headers      now=19047349 blk/sec=774.400
Feb 03 11:41:59 anon-r2 erigon[2250351]: [INFO] [02-03|11:41:59.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8266030 blockNumber=19069660 blk/sec=21.3 mbps/sec=4.3020 peers=5 snapshots=0 reconnected=false
Feb 03 11:42:13 anon-r2 erigon[2250351]: [INFO] [02-03|11:42:13.066] [downloader] Downloaded PoS Headers      now=19021429 blk/sec=864.000
Feb 03 11:42:22 anon-r2 erigon[2250351]: [INFO] [02-03|11:42:22.882] P2P                                      app=caplin peers=4
Feb 03 11:42:29 anon-r2 erigon[2250351]: [INFO] [02-03|11:42:29.175] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8265350 blockNumber=19068983 blk/sec=22.7 mbps/sec=4.5990 peers=12 snapshots=0 reconnected=false
Feb 03 11:42:43 anon-r2 erigon[2250351]: [INFO] [02-03|11:42:43.022] [downloader] Downloaded PoS Headers      now=18983413 blk/sec=1267.200
Feb 03 11:42:58 anon-r2 erigon[2250351]: [INFO] [02-03|11:42:58.798] [EngineBlockDownloader] Processing bodies... from=18967799 to=19147573
Feb 03 11:42:59 anon-r2 erigon[2250351]: [INFO] [02-03|11:42:59.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8264750 blockNumber=19068387 blk/sec=20.0 mbps/sec=4.2126 peers=6 snapshots=0 reconnected=false
Feb 03 11:43:00 anon-r2 erigon[2250351]: [INFO] [02-03|11:43:00.537] [NewPayload] Handling new payload        height=19147607 hash=0x4b68b9eb0159b75396b21c58cc30baeea088794fdd53822737b131c116e532a8
Feb 03 11:43:01 anon-r2 erigon[2250351]: [INFO] [02-03|11:43:01.909] [p2p] GoodPeers                          eth68=18 eth67=27 eth66=1
Feb 03 11:43:13 anon-r2 erigon[2250351]: [INFO] [02-03|11:43:13.013] [NewPayload] Handling new payload        height=19147608 hash=0x9a067e4382f1682a5b5df55dfdc6f15b96418dc1de69a636baee27aeb982b72c
Feb 03 11:43:22 anon-r2 erigon[2250351]: [INFO] [02-03|11:43:22.883] P2P                                      app=caplin peers=5
Feb 03 11:43:24 anon-r2 erigon[2250351]: [INFO] [02-03|11:43:24.625] [NewPayload] Handling new payload        height=19147609 hash=0x04b8c817f373f76e4642499e64fd51b10fc481e0f037c39e711204cf2f079370
Feb 03 11:43:28 anon-r2 erigon[2250351]: [INFO] [02-03|11:43:28.800] [EngineBlockDownloader] Downloading block bodies block_num=18968346 delivery/sec=4.3MB wasted/sec=1.0MB remaining=179226 delivered=784 cache=38.2MB alloc=6.8GB sys=9.6GB
Feb 03 11:43:29 anon-r2 erigon[2250351]: [INFO] [02-03|11:43:29.176] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8264289 blockNumber=19067938 blk/sec=15.4 mbps/sec=3.1864 peers=6 snapshots=0 reconnected=false
Feb 03 11:43:36 anon-r2 erigon[2250351]: [INFO] [02-03|11:43:36.481] [NewPayload] Handling new payload        height=19147610 hash=0x71d848a468e01731ab1e386231d22573b1ee792719463ad631854a198dd74fae

Hi, You are running internalcl while forcing the connection to an externalcl, please turn off the external consensus layer

External consensus is offline. Unfortunately, saw another error when attempting to insert blocks.

Feb 07 07:03:16 anon-r2 erigon[2658140]: [INFO] [02-07|07:03:16.088] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=4.8GB sys=9.8GB
Feb 07 07:03:43 anon-r2 erigon[2658140]: [INFO] [02-07|07:03:43.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8170393 blockNumber=18974813 blk/sec=8.5 mbps/sec=2.0696 peers=1 snapshots=0 reconnected=false
Feb 07 07:04:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:04:02.120] P2P                                      app=caplin peers=3
Feb 07 07:04:13 anon-r2 erigon[2658140]: [INFO] [02-07|07:04:13.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8170233 blockNumber=18974654 blk/sec=5.3 mbps/sec=1.2442 peers=0 snapshots=0 reconnected=false
Feb 07 07:04:43 anon-r2 erigon[2658140]: [INFO] [02-07|07:04:43.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8169721 blockNumber=18974144 blk/sec=17.1 mbps/sec=2.8601 peers=2 snapshots=0 reconnected=false
Feb 07 07:05:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:05:02.120] P2P                                      app=caplin peers=2
Feb 07 07:05:13 anon-r2 erigon[2658140]: [INFO] [02-07|07:05:13.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8169337 blockNumber=18973767 blk/sec=12.8 mbps/sec=1.9928 peers=1 snapshots=0 reconnected=false
Feb 07 07:05:43 anon-r2 erigon[2658140]: [INFO] [02-07|07:05:43.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8168377 blockNumber=18972817 blk/sec=32.0 mbps/sec=5.8099 peers=4 snapshots=0 reconnected=false
Feb 07 07:06:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:06:02.120] P2P                                      app=caplin peers=2
Feb 07 07:06:12 anon-r2 erigon[2658140]: [INFO] [02-07|07:06:12.799] [p2p] GoodPeers                          eth68=33 eth67=33
Feb 07 07:06:13 anon-r2 erigon[2658140]: [INFO] [02-07|07:06:13.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8168217 blockNumber=18972657 blk/sec=5.3 mbps/sec=0.8737 peers=2 snapshots=0 reconnected=false
Feb 07 07:06:16 anon-r2 erigon[2658140]: [INFO] [02-07|07:06:16.087] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=5.4GB sys=9.8GB
Feb 07 07:06:43 anon-r2 erigon[2658140]: [INFO] [02-07|07:06:43.136] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8167466 blockNumber=18971909 blk/sec=25.0 mbps/sec=4.8855 peers=6 snapshots=0 reconnected=false
Feb 07 07:07:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:07:02.120] P2P                                      app=caplin peers=1
Feb 07 07:07:13 anon-r2 erigon[2658140]: [INFO] [02-07|07:07:13.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8166709 blockNumber=18971160 blk/sec=25.2 mbps/sec=4.6081 peers=1 snapshots=0 reconnected=false
Feb 07 07:07:43 anon-r2 erigon[2658140]: [INFO] [02-07|07:07:43.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8165625 blockNumber=18970088 blk/sec=36.1 mbps/sec=7.1318 peers=3 snapshots=0 reconnected=false
Feb 07 07:08:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:08:02.120] P2P                                      app=caplin peers=7
Feb 07 07:08:13 anon-r2 erigon[2658140]: [INFO] [02-07|07:08:13.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8164697 blockNumber=18969166 blk/sec=30.9 mbps/sec=6.6207 peers=5 snapshots=0 reconnected=false
Feb 07 07:08:43 anon-r2 erigon[2658140]: [INFO] [02-07|07:08:43.135] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8163898 blockNumber=18968376 blk/sec=26.6 mbps/sec=5.8291 peers=8 snapshots=0 reconnected=false
Feb 07 07:09:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:09:02.120] P2P                                      app=caplin peers=4
Feb 07 07:09:05 anon-r2 erigon[2658140]: [INFO] [02-07|07:09:05.043] Backfilling finished
Feb 07 07:09:08 anon-r2 erigon[2658140]: [INFO] [02-07|07:09:08.946] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
Feb 07 07:09:12 anon-r2 erigon[2658140]: [WARN] [02-07|07:09:12.580] error executing clstage                  app=caplin stage=DownloadHistoricalBlocks err="loadIntoTable Header: error inserting block during collection: ethereumExecutionModule.InsertBlocks: writeBody: Wr>
Feb 07 07:09:12 anon-r2 erigon[2658140]: [INFO] [02-07|07:09:12.580] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=8369504
Feb 07 07:09:12 anon-r2 erigon[2658140]: [INFO] [02-07|07:09:12.799] [p2p] GoodPeers                          eth68=33 eth67=33
Feb 07 07:09:16 anon-r2 erigon[2658140]: [INFO] [02-07|07:09:16.087] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=5.2GB sys=9.8GB
Feb 07 07:09:42 anon-r2 erigon[2658140]: [INFO] [02-07|07:09:42.580] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8368289 blockNumber=19170926 blk/sec=40.5 mbps/sec=6.5715 peers=1 snapshots=0 reconnected=false
Feb 07 07:10:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:10:02.120] P2P                                      app=caplin peers=5
Feb 07 07:10:12 anon-r2 erigon[2658140]: [INFO] [02-07|07:10:12.580] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8367385 blockNumber=19170028 blk/sec=30.1 mbps/sec=4.7982 peers=6 snapshots=0 reconnected=false
Feb 07 07:10:42 anon-r2 erigon[2658140]: [INFO] [02-07|07:10:42.580] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8366935 blockNumber=19169585 blk/sec=15.0 mbps/sec=2.6324 peers=5 snapshots=0 reconnected=false
Feb 07 07:11:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:11:02.120] P2P                                      app=caplin peers=5
Feb 07 07:11:12 anon-r2 erigon[2658140]: [INFO] [02-07|07:11:12.580] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8366366 blockNumber=19169032 blk/sec=19.0 mbps/sec=3.3211 peers=7 snapshots=0 reconnected=false
Feb 07 07:11:42 anon-r2 erigon[2658140]: [INFO] [02-07|07:11:42.580] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8365870 blockNumber=19168540 blk/sec=16.5 mbps/sec=2.8865 peers=10 snapshots=0 reconnected=false
Feb 07 07:12:02 anon-r2 erigon[2658140]: [INFO] [02-07|07:12:02.120] P2P                                      app=caplin peers=11
Feb 07 07:12:12 anon-r2 erigon[2658140]: [INFO] [02-07|07:12:12.580] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8365379 blockNumber=19168052 blk/sec=16.4 mbps/sec=2.7750 peers=13 snapshots=0 reconnected=false
Feb 07 07:12:12 anon-r2 erigon[2658140]: [INFO] [02-07|07:12:12.799] [p2p] GoodPeers                          eth68=33 eth67=32
Feb 07 07:12:16 anon-r2 erigon[2658140]: [INFO] [02-07|07:12:16.088] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=6.8GB sys=9.8GB

Service

# cat /etc/systemd/system/erigon.service
[Unit]
Description=Erigon Client - Mainnet full
After=network.target
Wants=network.target
[Service]
User=erigon
Group=erigon
Type=simple
Restart=always
RestartSec=45
ExecStart=/usr/local/bin/erigon \
        --chain="mainnet" \
        --datadir="/storage/ethereum/erigon/data" \
        --snapshots="true" \
        --internalcl \
        --authrpc.jwtsecret="/var/ethereum/jwtsecret" \
        --torrent.download.rate="150mb" --torrent.upload.rate="15mb" \
        --http="true" --http.api="web3,net,eth" --http.addr="192.168.2.3" --http.vhosts="*" \
        --ws="true" \
        --ws.compression="false" \
#        --ws.api="web3,net,eth" \
#        --ws.addr="192.168.2.3" \
        --graphql="true" \
        --private.api.addr="192.168.2.3:9191" \
        --authrpc.addr="127.0.0.1" --authrpc.port="8551" --authrpc.vhosts="127.0.0.1"
[Install]
WantedBy=default.target
AskAlexSharov commented 7 months ago

blocks will become visible after batch execution (after pass all stages once)

caedmo commented 7 months ago

blocks will become visible after batch execution (after pass all stages once)

Yes, this seems to be caused by the recurring issue of my node failing to insert headers/bodies for the past 4 weeks.

Perhaps this issue was fixed in a more recent version of Erigon? I'll try updating to latest Erigon this afternoon - just curious if anyone has run into this issue, and had luck trying/configuring things in a different way?

Feb 03 11:36:22 anon-r2 erigon[2250351]: [WARN] [02-03|11:36:22.320] [EngineBlockDownloader] Could not insert headers and bodies err="ethereumExecutionModule.InsertBlocks: could not insert: WriteRawTransactions: txId=2153968712, baseTxId=2153968712, label: chaindata, bucket: BlockTransaction, mdbx_cursor_put: MDBX_EKEYMISMATCH: The given key value is mismatched to the current cursor position"
AskAlexSharov commented 7 months ago

integration stage_headers --reset

caedmo commented 7 months ago

After running the reset command

# ./build/bin/integration stage_headers --reset --datadir="/storage/ethereum/erigon/data" --chain mainnet

INFO[03-08|10:24:37.288] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=integration log level=info json=false
INFO[03-08|10:24:37.291] [db] open                                lable=chaindata sizeLimit=12TB pageSize=8192
INFO[03-08|10:24:47.429] [snapshots:all] Blocks Stat              blocks=18300k indices=18300k alloc=2.3GB sys=2.4GB
INFO[03-08|10:24:47.435] [bor snapshots:all] Blocks Stat          blocks=0k indices=0k alloc=2.3GB sys=2.4GB
INFO[03-08|10:24:47.780] Disk storage enabled for ethash DAGs     dir=/root/.local/share/erigon-ethash count=2
INFO[03-08|10:25:10.588] TruncateBlocks                           block=18003731
INFO[03-08|10:25:30.536] TruncateBlocks                           block=18004345
INFO[03-08|10:25:50.532] TruncateBlocks                           block=18012213
INFO[03-08|10:26:10.535] TruncateBlocks                           block=18020519
INFO[03-08|10:26:30.532] TruncateBlocks                           block=18028525
INFO[03-08|10:26:50.536] TruncateBlocks                           block=18036913
INFO[03-08|10:27:10.532] TruncateBlocks                           block=18045319
INFO[03-08|10:27:30.533] TruncateBlocks                           block=18054132
INFO[03-08|10:27:50.533] TruncateBlocks                           block=18062642
INFO[03-08|10:28:10.533] TruncateBlocks                           block=18071532
INFO[03-08|10:28:30.532] TruncateBlocks                           block=18079671
INFO[03-08|10:28:50.532] TruncateBlocks                           block=18088090
INFO[03-08|10:29:10.536] TruncateBlocks                           block=18096576
INFO[03-08|10:29:30.536] TruncateBlocks                           block=18105532
INFO[03-08|10:29:50.533] TruncateBlocks                           block=18114031
INFO[03-08|10:30:10.532] TruncateBlocks                           block=18122302
INFO[03-08|10:30:30.533] TruncateBlocks                           block=18129793
INFO[03-08|10:30:50.533] TruncateBlocks                           block=18138195
INFO[03-08|10:31:10.534] TruncateBlocks                           block=18146763
INFO[03-08|10:31:30.533] TruncateBlocks                           block=18155934
INFO[03-08|10:31:50.534] TruncateBlocks                           block=18164633
INFO[03-08|10:32:10.533] TruncateBlocks                           block=18172974
INFO[03-08|10:32:30.532] TruncateBlocks                           block=18181285
INFO[03-08|10:32:50.534] TruncateBlocks                           block=18189414
INFO[03-08|10:33:10.536] TruncateBlocks                           block=18198530
INFO[03-08|10:33:30.532] TruncateBlocks                           block=18207630
INFO[03-08|10:33:50.532] TruncateBlocks                           block=18215963
INFO[03-08|10:34:10.532] TruncateBlocks                           block=18224436
INFO[03-08|10:34:30.532] TruncateBlocks                           block=18232810
INFO[03-08|10:34:50.533] TruncateBlocks                           block=18241370
INFO[03-08|10:35:10.535] TruncateBlocks                           block=18249821
INFO[03-08|10:35:30.533] TruncateBlocks                           block=18258654
INFO[03-08|10:35:50.534] TruncateBlocks                           block=18266821
INFO[03-08|10:36:10.533] TruncateBlocks                           block=18275563
INFO[03-08|10:36:30.533] TruncateBlocks                           block=18284197
INFO[03-08|10:36:50.533] TruncateBlocks                           block=18292772
INFO[03-08|10:37:10.532] TruncateBlocks                           block=18301069
INFO[03-08|10:37:30.535] TruncateBlocks                           block=18309301
INFO[03-08|10:37:50.535] TruncateBlocks                           block=18317531
INFO[03-08|10:38:10.534] TruncateBlocks                           block=18325925
INFO[03-08|10:38:30.535] TruncateBlocks                           block=18334229
INFO[03-08|10:38:50.541] TruncateBlocks                           block=18342502
INFO[03-08|10:39:10.532] TruncateBlocks                           block=18351013
INFO[03-08|10:39:30.534] TruncateBlocks                           block=18359948
INFO[03-08|10:39:50.532] TruncateBlocks                           block=18368303
INFO[03-08|10:40:10.535] TruncateBlocks                           block=18376722
INFO[03-08|10:40:30.535] TruncateBlocks                           block=18384898
INFO[03-08|10:40:50.532] TruncateBlocks                           block=18392878
INFO[03-08|10:41:10.536] TruncateBlocks                           block=18400979
INFO[03-08|10:41:30.533] TruncateBlocks                           block=18408943
INFO[03-08|10:41:50.536] TruncateBlocks                           block=18416630
INFO[03-08|10:42:10.533] TruncateBlocks                           block=18424148
INFO[03-08|10:42:30.531] TruncateBlocks                           block=18431647
INFO[03-08|10:42:50.535] TruncateBlocks                           block=18439149
INFO[03-08|10:43:10.535] TruncateBlocks                           block=18446799
INFO[03-08|10:43:30.534] TruncateBlocks                           block=18454420
INFO[03-08|10:43:50.532] TruncateBlocks                           block=18461698
INFO[03-08|10:44:10.537] TruncateBlocks                           block=18468717
INFO[03-08|10:44:30.532] TruncateBlocks                           block=18476157
INFO[03-08|10:44:50.533] TruncateBlocks                           block=18484053
INFO[03-08|10:45:10.537] TruncateBlocks                           block=18491942
INFO[03-08|10:45:30.533] TruncateBlocks                           block=18499483
INFO[03-08|10:45:50.532] TruncateBlocks                           block=18507141
INFO[03-08|10:46:10.534] TruncateBlocks                           block=18514962
INFO[03-08|10:46:30.535] TruncateBlocks                           block=18522968
INFO[03-08|10:46:50.535] TruncateBlocks                           block=18530873
INFO[03-08|10:47:10.534] TruncateBlocks                           block=18538853
INFO[03-08|10:47:30.533] TruncateBlocks                           block=18546953
INFO[03-08|10:47:50.534] TruncateBlocks                           block=18554771
INFO[03-08|10:48:10.532] TruncateBlocks                           block=18562521
INFO[03-08|10:48:30.535] TruncateBlocks                           block=18570425
INFO[03-08|10:48:50.534] TruncateBlocks                           block=18578342
INFO[03-08|10:49:10.534] TruncateBlocks                           block=18586258
INFO[03-08|10:49:30.533] TruncateBlocks                           block=18594043
INFO[03-08|10:49:50.534] TruncateBlocks                           block=18601722
INFO[03-08|10:50:10.534] TruncateBlocks                           block=18609722
INFO[03-08|10:50:30.534] TruncateBlocks                           block=18617773
INFO[03-08|10:50:50.534] TruncateBlocks                           block=18625893
INFO[03-08|10:51:10.533] TruncateBlocks                           block=18633789
INFO[03-08|10:51:30.532] TruncateBlocks                           block=18641732
INFO[03-08|10:51:50.535] TruncateBlocks                           block=18649658
INFO[03-08|10:52:10.533] TruncateBlocks                           block=18657558
INFO[03-08|10:52:30.536] TruncateBlocks                           block=18665309
INFO[03-08|10:52:50.533] TruncateBlocks                           block=18673021
INFO[03-08|10:53:10.536] TruncateBlocks                           block=18680880
INFO[03-08|10:53:30.532] TruncateBlocks                           block=18688689
INFO[03-08|10:53:50.534] TruncateBlocks                           block=18696391
INFO[03-08|10:54:10.535] TruncateBlocks                           block=18703818
INFO[03-08|10:54:30.534] TruncateBlocks                           block=18711851
INFO[03-08|10:54:50.532] TruncateBlocks                           block=18719809
INFO[03-08|10:55:10.533] TruncateBlocks                           block=18727531
INFO[03-08|10:55:30.533] TruncateBlocks                           block=18735542
INFO[03-08|10:55:50.537] TruncateBlocks                           block=18743322
INFO[03-08|10:56:10.533] TruncateBlocks                           block=18751109
INFO[03-08|10:56:30.533] TruncateBlocks                           block=18759020
INFO[03-08|10:56:50.536] TruncateBlocks                           block=18766918
INFO[03-08|10:57:10.536] TruncateBlocks                           block=18774866
INFO[03-08|10:57:30.537] TruncateBlocks                           block=18782671
INFO[03-08|10:57:50.534] TruncateBlocks                           block=18790410
INFO[03-08|10:58:10.534] TruncateBlocks                           block=18798306
INFO[03-08|10:58:30.535] TruncateBlocks                           block=18806459
INFO[03-08|10:58:50.532] TruncateBlocks                           block=18814259
INFO[03-08|10:59:10.532] TruncateBlocks                           block=18822457
INFO[03-08|10:59:30.535] TruncateBlocks                           block=18830277
INFO[03-08|10:59:50.533] TruncateBlocks                           block=18838040
INFO[03-08|11:00:10.533] TruncateBlocks                           block=18845929
INFO[03-08|11:00:30.533] TruncateBlocks                           block=18854169
INFO[03-08|11:00:50.535] TruncateBlocks                           block=18862360
INFO[03-08|11:01:10.534] TruncateBlocks                           block=18870400
INFO[03-08|11:01:30.533] TruncateBlocks                           block=18878391
INFO[03-08|11:01:50.533] TruncateBlocks                           block=18886490
INFO[03-08|11:02:10.532] TruncateBlocks                           block=18894664
INFO[03-08|11:02:30.533] TruncateBlocks                           block=18902833
INFO[03-08|11:02:50.534] TruncateBlocks                           block=18911121
INFO[03-08|11:03:10.532] TruncateBlocks                           block=18919462
INFO[03-08|11:03:30.538] TruncateBlocks                           block=18927545
INFO[03-08|11:03:50.533] TruncateBlocks                           block=18935622
INFO[03-08|11:04:10.530] TruncateBlocks                           block=18943650
INFO[03-08|11:04:30.533] TruncateBlocks                           block=18951737
INFO[03-08|11:04:50.533] TruncateBlocks                           block=18959825
INFO[03-08|11:07:03.117] Clear                                    table=NonCanonicalTransaction
INFO[03-08|11:07:03.122] Clear                                    table=BlockTransaction
INFO[03-08|11:07:03.208] Clear                                    table=MaxTxNum
INFO[03-08|11:07:33.210] [fillind_db_from_snapshots] Total difficulty index: 2400k/18299k
INFO[03-08|11:08:03.210] [fillind_db_from_snapshots] Total difficulty index: 4785k/18299k
INFO[03-08|11:08:33.215] [fillind_db_from_snapshots] Total difficulty index: 6063k/18299k
INFO[03-08|11:09:03.210] [fillind_db_from_snapshots] Total difficulty index: 7298k/18299k
INFO[03-08|11:09:33.209] [fillind_db_from_snapshots] Total difficulty index: 10203k/18299k
INFO[03-08|11:10:03.210] [fillind_db_from_snapshots] Total difficulty index: 13181k/18299k
INFO[03-08|11:10:33.210] [fillind_db_from_snapshots] Total difficulty index: 15444k/18299k
INFO[03-08|11:11:03.210] [fillind_db_from_snapshots] Total difficulty index: 18159k/18299k

Started Erigon to allow syncing again - we progressed a little further than before - then saw an OOM-kill event as it was inserting data. The node is now attempting to sync from the HEAD again;

Mar 09 01:06:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:06:53.288] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=7.4GB sys=9.5GB
Mar 09 01:07:23 anon-r2 erigon[1791491]: [INFO] [03-09|01:07:23.125] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7494407 blockNumber=18304840 blk/sec=28.8 mbps/sec=6.4811 peers=1 snapshots=0 reconnected=false
Mar 09 01:07:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:07:46.543] P2P                                      app=caplin peers=1
Mar 09 01:07:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:07:53.125] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7493607 blockNumber=18304044 blk/sec=26.7 mbps/sec=6.0536 peers=5 snapshots=0 reconnected=false
Mar 09 01:08:23 anon-r2 erigon[1791491]: [INFO] [03-09|01:08:23.125] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7492623 blockNumber=18303067 blk/sec=32.8 mbps/sec=6.1461 peers=6 snapshots=0 reconnected=false
Mar 09 01:08:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:08:46.543] P2P                                      app=caplin peers=9
Mar 09 01:08:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:08:53.125] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7491751 blockNumber=18302200 blk/sec=29.1 mbps/sec=5.4131 peers=4 snapshots=0 reconnected=false
Mar 09 01:09:23 anon-r2 erigon[1791491]: [INFO] [03-09|01:09:23.125] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7490860 blockNumber=18301314 blk/sec=29.7 mbps/sec=5.4810 peers=1 snapshots=0 reconnected=false
Mar 09 01:09:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:09:46.543] P2P                                      app=caplin peers=10
Mar 09 01:09:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:09:53.124] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=7490027 blockNumber=18300486 blk/sec=27.8 mbps/sec=5.7607 peers=2 snapshots=0 reconnected=false
Mar 09 01:09:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:09:53.245] [p2p] GoodPeers                          eth67=32 eth66=1 eth68=33
Mar 09 01:09:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:09:53.289] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=6.2GB sys=9.5GB
Mar 09 01:10:09 anon-r2 erigon[1791491]: [INFO] [03-09|01:10:09.419] Backfilling finished
Mar 09 01:10:12 anon-r2 erigon[1791491]: [INFO] [03-09|01:10:12.112] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
Mar 09 01:10:16 anon-r2 erigon[1791491]: [INFO] [03-09|01:10:16.002] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18300000
Mar 09 01:10:40 anon-r2 erigon[1791491]: [INFO] [03-09|01:10:40.191] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18310000
Mar 09 01:10:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:10:46.543] P2P                                      app=caplin peers=4
Mar 09 01:11:04 anon-r2 erigon[1791491]: [INFO] [03-09|01:11:04.440] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18320000
Mar 09 01:11:29 anon-r2 erigon[1791491]: [INFO] [03-09|01:11:29.593] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18330000
Mar 09 01:11:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:11:46.545] P2P                                      app=caplin peers=7
Mar 09 01:11:52 anon-r2 erigon[1791491]: [INFO] [03-09|01:11:52.036] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18340000
Mar 09 01:12:14 anon-r2 erigon[1791491]: [INFO] [03-09|01:12:14.176] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18350000
Mar 09 01:12:35 anon-r2 erigon[1791491]: [INFO] [03-09|01:12:35.482] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18360000
Mar 09 01:12:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:12:46.544] P2P                                      app=caplin peers=7
Mar 09 01:12:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:12:53.245] [p2p] GoodPeers                          eth68=33 eth67=32 eth66=1
Mar 09 01:12:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:12:53.289] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=6.2GB sys=9.6GB
Mar 09 01:13:01 anon-r2 erigon[1791491]: [INFO] [03-09|01:13:01.068] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18370000
Mar 09 01:13:27 anon-r2 erigon[1791491]: [INFO] [03-09|01:13:27.625] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18380000
Mar 09 01:13:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:13:46.543] P2P                                      app=caplin peers=6
Mar 09 01:13:52 anon-r2 erigon[1791491]: [INFO] [03-09|01:13:52.824] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18390000
Mar 09 01:14:18 anon-r2 erigon[1791491]: [INFO] [03-09|01:14:18.412] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18400000
Mar 09 01:14:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:14:46.543] P2P                                      app=caplin peers=14
Mar 09 01:14:49 anon-r2 erigon[1791491]: [INFO] [03-09|01:14:49.942] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18410000
Mar 09 01:15:17 anon-r2 erigon[1791491]: [INFO] [03-09|01:15:17.003] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18420000
Mar 09 01:15:44 anon-r2 erigon[1791491]: [INFO] [03-09|01:15:44.563] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18430000
Mar 09 01:15:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:15:46.542] P2P                                      app=caplin peers=16
Mar 09 01:15:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:15:53.245] [p2p] GoodPeers                          eth66=1 eth67=32 eth68=33
Mar 09 01:15:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:15:53.289] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=7.6GB sys=9.6GB
Mar 09 01:16:07 anon-r2 erigon[1791491]: [INFO] [03-09|01:16:07.993] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18440000
Mar 09 01:16:37 anon-r2 erigon[1791491]: [INFO] [03-09|01:16:37.491] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18450000
Mar 09 01:16:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:16:46.543] P2P                                      app=caplin peers=19
Mar 09 01:17:04 anon-r2 erigon[1791491]: [INFO] [03-09|01:17:04.123] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18460000
Mar 09 01:17:30 anon-r2 erigon[1791491]: [INFO] [03-09|01:17:30.773] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18470000
Mar 09 01:17:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:17:46.543] P2P                                      app=caplin peers=26
Mar 09 01:17:56 anon-r2 erigon[1791491]: [INFO] [03-09|01:17:56.539] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18480000
Mar 09 01:18:22 anon-r2 erigon[1791491]: [INFO] [03-09|01:18:22.563] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18490000
Mar 09 01:18:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:18:46.542] P2P                                      app=caplin peers=24
Mar 09 01:18:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:18:53.094] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18500000
Mar 09 01:18:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:18:53.248] [p2p] GoodPeers                          eth68=33 eth67=32 eth66=1
Mar 09 01:18:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:18:53.290] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=6.8GB sys=9.6GB
Mar 09 01:19:23 anon-r2 erigon[1791491]: [INFO] [03-09|01:19:23.319] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18510000
Mar 09 01:19:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:19:46.542] P2P                                      app=caplin peers=24
Mar 09 01:19:51 anon-r2 erigon[1791491]: [INFO] [03-09|01:19:51.513] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18520000
Mar 09 01:20:27 anon-r2 erigon[1791491]: [INFO] [03-09|01:20:27.450] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18530000
Mar 09 01:20:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:20:46.549] P2P                                      app=caplin peers=17
Mar 09 01:20:55 anon-r2 erigon[1791491]: [INFO] [03-09|01:20:55.780] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18540000
Mar 09 01:21:24 anon-r2 erigon[1791491]: [INFO] [03-09|01:21:24.007] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18550000
Mar 09 01:21:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:21:46.542] P2P                                      app=caplin peers=24
Mar 09 01:21:52 anon-r2 erigon[1791491]: [INFO] [03-09|01:21:52.839] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18560000
Mar 09 01:21:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:21:53.245] [p2p] GoodPeers                          eth68=33 eth67=32 eth66=1
Mar 09 01:21:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:21:53.289] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=7.0GB sys=9.6GB
Mar 09 01:22:20 anon-r2 erigon[1791491]: [INFO] [03-09|01:22:20.658] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18570000
Mar 09 01:22:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:22:46.543] P2P                                      app=caplin peers=21
Mar 09 01:22:48 anon-r2 erigon[1791491]: [INFO] [03-09|01:22:48.979] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18580000
Mar 09 01:23:22 anon-r2 erigon[1791491]: [INFO] [03-09|01:23:22.313] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18590000
Mar 09 01:23:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:23:46.543] P2P                                      app=caplin peers=19
Mar 09 01:23:49 anon-r2 erigon[1791491]: [INFO] [03-09|01:23:49.716] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18600000
Mar 09 01:24:22 anon-r2 erigon[1791491]: [INFO] [03-09|01:24:22.308] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18610000
Mar 09 01:24:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:24:46.543] P2P                                      app=caplin peers=22
Mar 09 01:24:52 anon-r2 erigon[1791491]: [INFO] [03-09|01:24:52.356] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18620000
Mar 09 01:24:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:24:53.245] [p2p] GoodPeers                          eth67=32 eth68=33 eth66=1
Mar 09 01:24:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:24:53.289] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=7.0GB sys=9.6GB
Mar 09 01:25:20 anon-r2 erigon[1791491]: [INFO] [03-09|01:25:20.373] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18630000
Mar 09 01:25:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:25:46.542] P2P                                      app=caplin peers=17
Mar 09 01:25:54 anon-r2 erigon[1791491]: [INFO] [03-09|01:25:54.363] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18640000
Mar 09 01:26:25 anon-r2 erigon[1791491]: [INFO] [03-09|01:26:25.309] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18650000
Mar 09 01:26:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:26:46.542] P2P                                      app=caplin peers=22
Mar 09 01:27:06 anon-r2 erigon[1791491]: [INFO] [03-09|01:27:06.693] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18660000
Mar 09 01:27:41 anon-r2 erigon[1791491]: [INFO] [03-09|01:27:41.806] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18670000
Mar 09 01:27:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:27:46.542] P2P                                      app=caplin peers=21
Mar 09 01:27:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:27:53.245] [p2p] GoodPeers                          eth68=33 eth67=32 eth66=1
Mar 09 01:27:53 anon-r2 erigon[1791491]: [INFO] [03-09|01:27:53.288] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=4.4GB sys=9.6GB
Mar 09 01:28:17 anon-r2 erigon[1791491]: [INFO] [03-09|01:28:17.474] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18680000
Mar 09 01:28:43 anon-r2 erigon[1791491]: [INFO] [03-09|01:28:43.756] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18690000
Mar 09 01:28:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:28:46.543] P2P                                      app=caplin peers=18
Mar 09 01:29:14 anon-r2 erigon[1791491]: [INFO] [03-09|01:29:14.735] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18700000
Mar 09 01:29:46 anon-r2 erigon[1791491]: [INFO] [03-09|01:29:46.542] P2P                                      app=caplin peers=21
Mar 09 01:29:49 anon-r2 erigon[1791491]: [INFO] [03-09|01:29:49.750] Inserting execution payload              app=caplin stage=DownloadHistoricalBlocks blockNumber=18710000
Mar 09 01:30:06 anon-r2 systemd[1]: erigon.service: systemd-oomd killed 3 process(es) in this unit.
Mar 09 01:30:06 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Mar 09 01:30:06 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.
Mar 09 01:30:06 anon-r2 systemd[1]: erigon.service: Consumed 17h 27min 39.626s CPU time.
Mar 09 01:30:51 anon-r2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 1.
Mar 09 01:30:51 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Mar 09 01:30:51 anon-r2 systemd[1]: erigon.service: Consumed 17h 27min 39.626s CPU time.
Mar 09 01:30:51 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Mar 09 01:30:52 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:52.540] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Mar 09 01:30:52 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:52.542] Build info                               git_branch=HEAD git_tag=v2.57.3 git_commit=705814b1ab72f3f25bef5230bd0c4dbaa0bc1f2a
Mar 09 01:30:52 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:52.542] Starting Erigon on Ethereum mainnet...
Mar 09 01:30:52 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:52.547] Maximum peer count                       ETH=100 total=100
Mar 09 01:30:52 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:52.550] starting HTTP APIs                       port=8545 APIs=web3,net,eth
Mar 09 01:30:52 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:52.550] torrent verbosity                        level=WRN
Mar 09 01:30:54 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:54.658] Set global gas cap                       cap=50000000
Mar 09 01:30:54 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:54.770] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=150mb upload.rate=15mb
Mar 09 01:30:56 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:56.111] Opening Database                         label=chaindata path=/storage/ethereum/erigon/data/chaindata
Mar 09 01:30:56 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:56.113] [db] open                                lable=chaindata sizeLimit=12TB pageSize=8192
Mar 09 01:30:56 anon-r2 erigon[1856377]: [INFO] [03-09|01:30:56.121] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: <nil>, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
Mar 09 01:31:28 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:28.490] Initialising Ethereum protocol           network=1
Mar 09 01:31:28 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:28.490] Disk storage enabled for ethash DAGs     dir=/storage/ethereum/erigon/data/ethash-dags count=2
Mar 09 01:31:33 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:33.467] Starting private RPC server              on=192.168.2.3:9191
Mar 09 01:31:33 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:33.467] new subscription to logs established
Mar 09 01:31:33 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:33.470] [Checkpoint Sync] Requesting beacon state uri=https://sync.invis.tools/eth/v2/debug/beacon/states/finalized
Mar 09 01:31:33 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:33.605] [txpool] Started
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.102] Static peers                             len=0
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.102] [Sentinel] Sentinel started              enr=enr:-Ly4QLmLa7VhmYnz9-8z00EBP0Ec_NEaP4GRPQpOfApoeiGiC2kPKCLUfSa376A20pXd1vvMLZTrluPzB94SLXy8_VkBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpC7pNqWBAAAAP__________gmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQJV5wr0cmIMHqmJK5ne_EJteYAXUVJYR8Nlvz0W3BtM7ohzeW5jbmV0cwCDdGNwgg-hg3VkcIIPoA
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.103] rpc filters: subscribing to Erigon events
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.103] new subscription to newHeaders established
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.103] New txs subscriber joined
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.107] JsonRpc endpoint opened                  ws=true ws.compression=false grpc=false websocket.url=192.168.2.3:8546 http.url=192.168.2.3:8545
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.107] Reading JWT secret                       path=/var/ethereum/jwtsecret
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.107] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=false
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.117] [1/12 Snapshots] Fetching torrent files metadata
Mar 09 01:31:38 anon-r2 erigon[1856377]: [WARN] [03-09|01:31:38.141] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.146] Started P2P networking                   version=68 self=enode://4fcb1da8d3e727376d2901100abb06db5730d8024c64e120ae25c539fd8dca7c30957c7e28c8aebedc75bf4771e8b1426f8711e5e75c9d50292a9eb354338ddd@127.0.0.1:30303 name=erigon/v2.57.3-705814b1/linux-amd64/go1.20.10
Mar 09 01:31:38 anon-r2 erigon[1856377]: [WARN] [03-09|01:31:38.149] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
Mar 09 01:31:38 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:38.154] Started P2P networking                   version=67 self=enode://4fcb1da8d3e727376d2901100abb06db5730d8024c64e120ae25c539fd8dca7c30957c7e28c8aebedc75bf4771e8b1426f8711e5e75c9d50292a9eb354338ddd@127.0.0.1:30304 name=erigon/v2.57.3-705814b1/linux-amd64/go1.20.10
Mar 09 01:31:40 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:40.964] [snapshots:download] Blocks Stat         blocks=18300k indices=18300k alloc=2.8GB sys=5.5GB
Mar 09 01:31:40 anon-r2 erigon[1856377]: [INFO] [03-09|01:31:40.969] [snapshots] Prune Blocks                 to=18209000 limit=100
Mar 09 01:32:21 anon-r2 erigon[1856377]: [INFO] [03-09|01:32:21.883] Started Ethereum 2.0 Gossip Service      app=caplin
Mar 09 01:32:28 anon-r2 erigon[1856377]: [INFO] [03-09|01:32:28.368] [Caplin] starting clstages loop          app=caplin
Mar 09 01:32:28 anon-r2 erigon[1856377]: [INFO] [03-09|01:32:28.368] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=8593568
Mar 09 01:32:58 anon-r2 erigon[1856377]: [INFO] [03-09|01:32:58.369] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8593105 blockNumber=19393733 blk/sec=15.4 mbps/sec=2.6728 peers=18 snapshots=0 reconnected=false
Mar 09 01:33:21 anon-r2 erigon[1856377]: [INFO] [03-09|01:33:21.884] P2P                                      app=caplin peers=15
Mar 09 01:33:28 anon-r2 erigon[1856377]: [INFO] [03-09|01:33:28.368] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8592610 blockNumber=19393263 blk/sec=16.5 mbps/sec=2.7466 peers=16 snapshots=0 reconnected=false
Mar 09 01:33:58 anon-r2 erigon[1856377]: [INFO] [03-09|01:33:58.369] Downloading History                      app=caplin stage=DownloadHistoricalBlocks slot=8592067 blockNumber=19392763 blk/sec=18.1 mbps/sec=3.0215 peers=15 snapshots=0 reconnected=false
caedmo commented 7 months ago

Some good news; the node has now nudged ahead of 18299999 to 19399481, however it's not been keeping up with the chain head.

Updated Erigon to 2.58.2; now awaiting sync:

Mar 11 17:14:35 anon-r2 erigon[2150511]: [INFO] [03-11|17:14:35.805] [Caplin] starting clstages loop          app=caplin
Mar 11 17:14:35 anon-r2 erigon[2150511]: [INFO] [03-11|17:14:35.966] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=8612672
Mar 11 17:14:45 anon-r2 erigon[2150511]: [INFO] [03-11|17:14:45.399] [1/12 Snapshots] download                progress="88.07% 510.5GB/579.6GB" time-left=11hrs:19m total-time=1m0s download=1.7MB/s upload=0B/s peers=4 files=218 metadata=218/218 connections=9 alloc=4.9GB sys=5.5GB
caedmo commented 7 months ago

I've now disabled systemd-oomd as it keeps killing Erigon at seemingly random points of the sync process.

Once the sync is back on track, and keeping up to date with the chain head, I'll re-enable systemd-oomd, test again.

I.e.

Mar 11 20:50:12 anon-r2 erigon[2164511]: [INFO] [03-11|20:50:12.399] [p2p] GoodPeers                          eth68=16 eth67=8
Mar 11 20:50:13 anon-r2 erigon[2164511]: [INFO] [03-11|20:50:13.401] [4/12 Execution] Executed blocks         number=19413116 blk/s=12.3 tx/s=2225.1 Mgas/s=187.3 gasState=0.38 batch=170.2MB alloc=5.2GB sys=11.6GB
Mar 11 20:50:15 anon-r2 erigon[2164511]: [INFO] [03-11|20:50:15.068] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=5.3GB sys=11.6GB
Mar 11 20:50:43 anon-r2 erigon[2164511]: [INFO] [03-11|20:50:43.404] [4/12 Execution] Executed blocks         number=19413460 blk/s=11.5 tx/s=2111.3 Mgas/s=172.8 gasState=0.39 batch=173.7MB alloc=7.2GB sys=11.6GB
Mar 11 20:51:04 anon-r2 erigon[2164511]: [INFO] [03-11|20:51:04.037] P2P                                      app=caplin peers=39
Mar 11 20:51:13 anon-r2 erigon[2164511]: [INFO] [03-11|20:51:13.404] [4/12 Execution] Executed blocks         number=19413832 blk/s=12.4 tx/s=2219.3 Mgas/s=187.5 gasState=0.40 batch=177.9MB alloc=7.6GB sys=11.6GB
Mar 11 20:51:47 anon-r2 erigon[2164511]: [INFO] [03-11|20:51:47.033] [] Flushed buffer file                   name=erigon-sortable-buf-1597020331
Mar 11 20:51:47 anon-r2 erigon[2164511]: [INFO] [03-11|20:51:47.229] [] Flushed buffer file                   name=erigon-sortable-buf-355019751
Mar 11 20:51:50 anon-r2 erigon[2164511]: [INFO] [03-11|20:51:50.069] [] Flushed buffer file                   name=erigon-sortable-buf-2222841804
Mar 11 20:52:04 anon-r2 erigon[2164511]: [INFO] [03-11|20:52:04.037] P2P                                      app=caplin peers=37
Mar 11 20:52:17 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Mar 11 20:52:17 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.
Mar 11 20:52:17 anon-r2 systemd[1]: erigon.service: Consumed 1h 53min 34.161s CPU time.
Mar 11 20:53:02 anon-r2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 4.
Mar 11 20:53:02 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Mar 11 20:53:02 anon-r2 systemd[1]: erigon.service: Consumed 1h 53min 34.161s CPU time.
Mar 11 20:53:02 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Mar 11 20:53:03 anon-r2 erigon[2167188]: [INFO] [03-11|20:53:03.188] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Mar 11 20:53:03 anon-r2 erigon[2167188]: [INFO] [03-11|20:53:03.188] Build info                               git_branch=HEAD git_tag=v2.58.2 git_commit=125509e483326d77eb810fdc33a64d30ed98e18f

and

Mar 11 18:57:25 anon-r2 erigon[2150511]: [INFO] [03-11|18:57:25.399] [1/12 Snapshots] download                progress="99.88% 578.9GB/579.6GB" time-left=0hrs:0m total-time=1h43m40s download=21.0MB/s upload=0B/s peers=10 files=218 metadata=218/218 connections=11 alloc=6.6GB sys=9.0GB
Mar 11 18:57:28 anon-r2 erigon[2150511]: [INFO] [03-11|18:57:28.836] P2P                                      app=caplin peers=41
Mar 11 18:57:45 anon-r2 erigon[2150511]: [INFO] [03-11|18:57:45.399] [1/12 Snapshots] download                progress="99.95% 579.3GB/579.6GB" time-left=0hrs:0m total-time=1h44m0s download=20.9MB/s upload=0B/s peers=10 files=218 metadata=218/218 connections=10 alloc=4.5GB sys=9.0GB
Mar 11 18:58:05 anon-r2 erigon[2150511]: [INFO] [03-11|18:58:05.398] [1/12 Snapshots] download finished       time=1h44m20.000354324s
Mar 11 18:58:10 anon-r2 erigon[2150511]: [INFO] [03-11|18:58:10.354] [snapshots:download] Blocks Stat         blocks=18300k indices=18300k alloc=8.0GB sys=9.0GB
Mar 11 18:58:10 anon-r2 erigon[2150511]: [INFO] [03-11|18:58:10.354] [snapshots:missed-idx] Blocks Stat       blocks=18300k indices=18300k alloc=8.0GB sys=9.0GB
Mar 11 18:58:26 anon-r2 systemd[1]: erigon.service: systemd-oomd killed 12 process(es) in this unit.
Mar 11 18:58:26 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Mar 11 18:58:26 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.
Mar 11 18:58:26 anon-r2 systemd[1]: erigon.service: Consumed 1h 51min 21.595s CPU time.
Mar 11 18:59:11 anon-r2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 1.
Mar 11 18:59:11 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Mar 11 18:59:11 anon-r2 systemd[1]: erigon.service: Consumed 1h 51min 21.595s CPU time.
Mar 11 18:59:11 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Mar 11 18:59:12 anon-r2 erigon[2158850]: [INFO] [03-11|18:59:12.451] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Mar 11 18:59:12 anon-r2 erigon[2158850]: [INFO] [03-11|18:59:12.451] Build info                               git_branch=HEAD git_tag=v2.58.2 git_commit=125509e483326d77eb810fdc33a64d30ed98e18f
AskAlexSharov commented 7 months ago

you can add GOMEMLIMIT=8GiB

caedmo commented 7 months ago

Thanks Alex

Re-running with Environment="GOMEMLIMIT=16GiB" on Erigon service file.

Re-enabled OOM killer, as Erigon was still being killed by something else.

Will update.

caedmo commented 6 months ago

Node appears to be syncing, and stable now. Thanks @AskAlexSharov

Quick question on the block syncing process; is there any way to achieve the same sort of individual block sequencing, seen in Geth? Currently I am seeing chunks of blocks every 10 minutes, which will mean delays for bots interfacing with GraphQL. Is it related to the snapshot feature, that is enabled by default? Thanks!

Just incase it's helpful; after having added Environment="GOMEMLIMIT=16GiB", I did still notice a few more OOM kill events.

Mar 12 11:52:38 anon-r2 erigon[2228265]: [INFO] [03-12|11:52:38.429] [p2p] GoodPeers                          eth68=33 eth67=20 eth66=1
Mar 12 11:52:40 anon-r2 erigon[2228265]: [INFO] [03-12|11:52:40.855] [txpool] stat                            pending=3463 baseFee=2079 queued=2953 alloc=6.3GB sys=16.2GB
Mar 12 11:53:18 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:18.893] [3/12 Senders] Started                   from=19418789 to=19418823
Mar 12 11:53:19 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:19.010] [4/12 Execution] Blocks execution        from=19418789 to=19418823
Mar 12 11:53:21 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:21.680] [4/12 Execution] Completed on            block=19418823
Mar 12 11:53:21 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:21.681] [5/12 HashState] Promoting plain state   from=19418789 to=19418823
Mar 12 11:53:21 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:21.681] [5/12 HashState] Incremental promotion   from=19418789 to=19418823 codes=true csbucket=AccountChangeSet
Mar 12 11:53:21 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:21.920] [5/12 HashState] Incremental promotion   from=19418789 to=19418823 codes=false csbucket=AccountChangeSet
Mar 12 11:53:22 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:22.397] [5/12 HashState] Incremental promotion   from=19418789 to=19418823 codes=false csbucket=StorageChangeSet
Mar 12 11:53:22 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:22.844] [6/12 IntermediateHashes] Generating intermediate hashes from=19418789 to=19418823
Mar 12 11:53:25 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:25.144] [7/12 CallTraces] Pruned call trace intermediate table from=19328789 to=19328822
Mar 12 11:53:28 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:28.014] P2P                                      app=caplin peers=38
Mar 12 11:53:34 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:34.664] RPC Daemon notified of new headers       from=19418789 to=19418823 hash=0x11cd0c796f8b229d47ed1cd60c586486345a03fd49b580fccb14858ab82ca0f3 header sending=53.593µs log sending=320ns
Mar 12 11:53:34 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:34.664] head updated                             hash=0x11cd0c796f8b229d47ed1cd60c586486345a03fd49b580fccb14858ab82ca0f3 number=19418823
Mar 12 11:53:35 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:35.005] [snapshots] Prune Blocks                 to=19000000 limit=100
Mar 12 11:53:42 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:42.460] [4/12 Execution] Completed on            block=19418825
Mar 12 11:53:43 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:43.646] RPC Daemon notified of new headers       from=19418823 to=19418825 hash=0xfb9185990bfbe1e71506dcf3e82e77394ed44fb1fe2ed6d8f3d5382ca6d66ef5 header sending=25.023µs log sending=440ns
Mar 12 11:53:43 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:43.646] head updated                             hash=0xfb9185990bfbe1e71506dcf3e82e77394ed44fb1fe2ed6d8f3d5382ca6d66ef5 number=19418825
Mar 12 11:53:44 anon-r2 erigon[2228265]: [INFO] [03-12|11:53:44.059] [snapshots] Prune Blocks                 to=19000000 limit=100
Mar 12 11:54:28 anon-r2 erigon[2228265]: [INFO] [03-12|11:54:28.014] P2P                                      app=caplin peers=21
Mar 12 11:55:28 anon-r2 erigon[2228265]: [INFO] [03-12|11:55:28.013] P2P                                      app=caplin peers=18
Mar 12 11:55:38 anon-r2 erigon[2228265]: [INFO] [03-12|11:55:38.428] [p2p] GoodPeers                          eth67=20 eth68=33 eth66=1
Mar 12 11:55:40 anon-r2 erigon[2228265]: [INFO] [03-12|11:55:40.856] [txpool] stat                            pending=1256 baseFee=1824 queued=2643 alloc=6.1GB sys=16.2GB
Mar 12 11:56:28 anon-r2 erigon[2228265]: [INFO] [03-12|11:56:28.013] P2P                                      app=caplin peers=22
Mar 12 11:57:28 anon-r2 erigon[2228265]: [INFO] [03-12|11:57:28.014] P2P                                      app=caplin peers=30
Mar 12 11:58:28 anon-r2 erigon[2228265]: [INFO] [03-12|11:58:28.014] P2P                                      app=caplin peers=33
Mar 12 11:58:38 anon-r2 erigon[2228265]: [INFO] [03-12|11:58:38.428] [p2p] GoodPeers                          eth67=19 eth68=33 eth66=1
Mar 12 11:58:40 anon-r2 erigon[2228265]: [INFO] [03-12|11:58:40.879] [txpool] stat                            pending=2706 baseFee=1922 queued=2740 alloc=10.3GB sys=16.2GB
Mar 12 11:59:28 anon-r2 erigon[2228265]: [INFO] [03-12|11:59:28.013] P2P                                      app=caplin peers=29
Mar 12 12:00:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:00:28.014] P2P                                      app=caplin peers=26
Mar 12 12:01:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:01:28.014] P2P                                      app=caplin peers=23
Mar 12 12:01:38 anon-r2 erigon[2228265]: [INFO] [03-12|12:01:38.429] [p2p] GoodPeers                          eth68=33 eth67=19 eth66=1
Mar 12 12:01:40 anon-r2 erigon[2228265]: [INFO] [03-12|12:01:40.857] [txpool] stat                            pending=4384 baseFee=2188 queued=2901 alloc=8.4GB sys=16.2GB
Mar 12 12:02:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:02:28.013] P2P                                      app=caplin peers=25
Mar 12 12:03:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:03:28.014] P2P                                      app=caplin peers=31
Mar 12 12:04:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:04:28.013] P2P                                      app=caplin peers=28
Mar 12 12:04:38 anon-r2 erigon[2228265]: [INFO] [03-12|12:04:38.428] [p2p] GoodPeers                          eth68=33 eth67=19 eth66=1
Mar 12 12:04:40 anon-r2 erigon[2228265]: [INFO] [03-12|12:04:40.855] [txpool] stat                            pending=5594 baseFee=2351 queued=3027 alloc=8.3GB sys=16.2GB
Mar 12 12:05:08 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:08.083] [3/12 Senders] Started                   from=19418825 to=19418882
Mar 12 12:05:08 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:08.237] [4/12 Execution] Blocks execution        from=19418825 to=19418882
Mar 12 12:05:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:28.013] P2P                                      app=caplin peers=19
Mar 12 12:05:44 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:44.470] [4/12 Execution] Executed blocks         number=19418873 blk/s=1.3 tx/s=232.9 Mgas/s=20.3 gasState=0.00 batch=1.4MB alloc=5.7GB sys=16.2GB
Mar 12 12:05:57 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:57.364] [4/12 Execution] Completed on            block=19418882
Mar 12 12:05:57 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:57.365] [5/12 HashState] Promoting plain state   from=19418825 to=19418882
Mar 12 12:05:57 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:57.365] [5/12 HashState] Incremental promotion   from=19418825 to=19418882 codes=true csbucket=AccountChangeSet
Mar 12 12:05:57 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:57.780] [5/12 HashState] Incremental promotion   from=19418825 to=19418882 codes=false csbucket=AccountChangeSet
Mar 12 12:05:58 anon-r2 erigon[2228265]: [INFO] [03-12|12:05:58.871] [5/12 HashState] Incremental promotion   from=19418825 to=19418882 codes=false csbucket=StorageChangeSet
Mar 12 12:06:00 anon-r2 erigon[2228265]: [INFO] [03-12|12:06:00.528] [6/12 IntermediateHashes] Generating intermediate hashes from=19418825 to=19418882
Mar 12 12:06:06 anon-r2 erigon[2228265]: [INFO] [03-12|12:06:06.300] [7/12 CallTraces] Pruned call trace intermediate table from=19328825 to=19328881
Mar 12 12:06:20 anon-r2 erigon[2228265]: [INFO] [03-12|12:06:20.464] RPC Daemon notified of new headers       from=19418825 to=19418882 hash=0x6fc5bd40c60e276e90ffb3daf6ada791122111e353b1553adc28005ab342e104 header sending=90.712µs log sending=255ns
Mar 12 12:06:20 anon-r2 erigon[2228265]: [INFO] [03-12|12:06:20.464] head updated                             hash=0x6fc5bd40c60e276e90ffb3daf6ada791122111e353b1553adc28005ab342e104 number=19418882
Mar 12 12:06:20 anon-r2 erigon[2228265]: [INFO] [03-12|12:06:20.766] [snapshots] Prune Blocks                 to=19000000 limit=100
Mar 12 12:06:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:06:28.013] P2P                                      app=caplin peers=16
Mar 12 12:07:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:07:28.013] P2P                                      app=caplin peers=20
Mar 12 12:07:38 anon-r2 erigon[2228265]: [INFO] [03-12|12:07:38.429] [p2p] GoodPeers                          eth68=33 eth67=19 eth66=1
Mar 12 12:07:40 anon-r2 erigon[2228265]: [INFO] [03-12|12:07:40.857] [txpool] stat                            pending=1630 baseFee=1745 queued=2772 alloc=8.9GB sys=16.2GB
Mar 12 12:08:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:08:28.014] P2P                                      app=caplin peers=15
Mar 12 12:09:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:09:28.014] P2P                                      app=caplin peers=13
Mar 12 12:10:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:10:28.013] P2P                                      app=caplin peers=22
Mar 12 12:10:38 anon-r2 erigon[2228265]: [INFO] [03-12|12:10:38.428] [p2p] GoodPeers                          eth66=1 eth68=33 eth67=21
Mar 12 12:10:40 anon-r2 erigon[2228265]: [INFO] [03-12|12:10:40.861] [txpool] stat                            pending=3051 baseFee=1821 queued=2938 alloc=8.9GB sys=16.2GB
Mar 12 12:11:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:11:28.013] P2P                                      app=caplin peers=13
Mar 12 12:12:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:12:28.014] P2P                                      app=caplin peers=11
Mar 12 12:13:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:13:28.014] P2P                                      app=caplin peers=14
Mar 12 12:13:38 anon-r2 erigon[2228265]: [INFO] [03-12|12:13:38.428] [p2p] GoodPeers                          eth68=33 eth67=20 eth66=1
Mar 12 12:13:40 anon-r2 erigon[2228265]: [INFO] [03-12|12:13:40.855] [txpool] stat                            pending=4496 baseFee=1839 queued=3071 alloc=7.0GB sys=16.2GB
Mar 12 12:14:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:14:28.014] P2P                                      app=caplin peers=16
Mar 12 12:15:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:15:28.013] P2P                                      app=caplin peers=19
Mar 12 12:16:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:16:28.013] P2P                                      app=caplin peers=22
Mar 12 12:16:38 anon-r2 erigon[2228265]: [INFO] [03-12|12:16:38.428] [p2p] GoodPeers                          eth68=26 eth67=18 eth66=2
Mar 12 12:16:40 anon-r2 erigon[2228265]: [INFO] [03-12|12:16:40.856] [txpool] stat                            pending=5913 baseFee=1859 queued=3290 alloc=11.3GB sys=16.2GB
Mar 12 12:17:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:17:28.014] P2P                                      app=caplin peers=23
Mar 12 12:18:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:18:28.014] P2P                                      app=caplin peers=25
Mar 12 12:19:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:19:28.013] P2P                                      app=caplin peers=21
Mar 12 12:19:38 anon-r2 erigon[2228265]: [INFO] [03-12|12:19:38.429] [p2p] GoodPeers                          eth68=27 eth67=19 eth66=1
Mar 12 12:19:42 anon-r2 erigon[2228265]: [INFO] [03-12|12:19:42.508] [txpool] stat                            pending=7235 baseFee=1878 queued=3436 alloc=11.6GB sys=16.2GB
Mar 12 12:20:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:20:28.013] P2P                                      app=caplin peers=16
Mar 12 12:21:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:21:28.014] P2P                                      app=caplin peers=31
Mar 12 12:22:16 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:16.749] [3/12 Senders] Started                   from=19418882 to=19418968
Mar 12 12:22:16 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:16.995] [4/12 Execution] Blocks execution        from=19418882 to=19418968
Mar 12 12:22:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:28.013] P2P                                      app=caplin peers=21
Mar 12 12:22:33 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:33.312] [4/12 Execution] Completed on            block=19418968
Mar 12 12:22:33 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:33.313] [5/12 HashState] Promoting plain state   from=19418882 to=19418968
Mar 12 12:22:33 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:33.313] [5/12 HashState] Incremental promotion   from=19418882 to=19418968 codes=true csbucket=AccountChangeSet
Mar 12 12:22:34 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:34.507] [5/12 HashState] Incremental promotion   from=19418882 to=19418968 codes=false csbucket=AccountChangeSet
Mar 12 12:22:37 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:37.855] [5/12 HashState] Incremental promotion   from=19418882 to=19418968 codes=false csbucket=StorageChangeSet
Mar 12 12:22:38 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:38.428] [p2p] GoodPeers                          eth68=26 eth66=1 eth67=21
Mar 12 12:22:40 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:40.855] [txpool] stat                            pending=8395 baseFee=1907 queued=3546 alloc=10.7GB sys=16.2GB
Mar 12 12:22:42 anon-r2 erigon[2228265]: [INFO] [03-12|12:22:42.519] [6/12 IntermediateHashes] Generating intermediate hashes from=19418882 to=19418968
Mar 12 12:23:02 anon-r2 erigon[2228265]: [INFO] [03-12|12:23:02.394] [7/12 CallTraces] Pruned call trace intermediate table from=19328882 to=19328967
Mar 12 12:23:28 anon-r2 erigon[2228265]: [INFO] [03-12|12:23:28.013] P2P                                      app=caplin peers=20
Mar 12 12:23:41 anon-r2 erigon[2228265]: [INFO] [03-12|12:23:41.085] RPC Daemon notified of new headers       from=19418882 to=19418968 hash=0xbfae76baaf1bcc36dc51af6ffac48bb6da32ea8ba86dce8b6de3f36bc148bd40 header sending=115.121µs log sending=262ns
Mar 12 12:23:41 anon-r2 erigon[2228265]: [INFO] [03-12|12:23:41.085] head updated                             hash=0xbfae76baaf1bcc36dc51af6ffac48bb6da32ea8ba86dce8b6de3f36bc148bd40 number=19418968
Mar 12 12:23:41 anon-r2 erigon[2228265]: [INFO] [03-12|12:23:41.396] [snapshots] Prune Blocks                 to=19000000 limit=100
Mar 12 12:23:58 anon-r2 systemd[1]: erigon.service: systemd-oomd killed 68 process(es) in this unit.
Mar 12 12:23:58 anon-r2 systemd[1]: erigon.service: Main process exited, code=killed, status=9/KILL
Mar 12 12:23:58 anon-r2 systemd[1]: erigon.service: Failed with result 'signal'.
Mar 12 12:23:58 anon-r2 systemd[1]: erigon.service: Consumed 9h 9min 5.364s CPU time.
Mar 12 12:24:43 anon-r2 systemd[1]: erigon.service: Scheduled restart job, restart counter is at 1.
Mar 12 12:24:43 anon-r2 systemd[1]: Stopped erigon.service - Erigon Client - Mainnet full.
Mar 12 12:24:43 anon-r2 systemd[1]: erigon.service: Consumed 9h 9min 5.364s CPU time.
Mar 12 12:24:43 anon-r2 systemd[1]: Started erigon.service - Erigon Client - Mainnet full.
Mar 12 12:24:44 anon-r2 erigon[2239284]: [INFO] [03-12|12:24:44.334] logging to file system                   log dir=/storage/ethereum/erigon/data/logs file prefix=erigon log level=info json=false
Mar 12 12:24:44 anon-r2 erigon[2239284]: [INFO] [03-12|12:24:44.335] Build info                               git_branch=HEAD git_tag=v2.58.2 git_commit=125509e483326d77eb810fdc33a64d30ed98e18f