search

ledgerwatch / erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
3.04k stars 1.06k forks source link

Polygon Erigon Syncing From Scratch stalls at block 48077378 across 5 nodes #10886

Open tobidae-cb opened 2 weeks ago

tobidae-cb commented 2 weeks ago

System information

Erigon version: v2.60.1-9471c44e

OS & Version: Linux

Erigon Command (with flags/config):

    EXECUTION_BOOT_NODES="${EXECUTION_BOOT_NODES:-}"
    KEEP_LAST_BLOCKS="${KEEP_LAST_BLOCKS:-2600000}"
    HEIMDALL_URL="http://localhost:1317"

    echo "Starting erigon client for polygon"
    exec /app/erigon \
    --chain "${CHAIN}" \
    --bor.heimdall "${HEIMDALL_URL}" \
    --rpc.txfeecap=0 \
    --http \
    --http.addr 0.0.0.0 \
    --http.vhosts="*" \
    --http.api="engine,eth,web3,net,debug,trace,txpool,erigon,admin,bor" \
    --http.port "${EXECUTION_HTTP_PORT}" \
    --http.corsdomain="*" \
    --http.trace="true" \
    --port "${PEERING_PORT}" \
    --datadir "${DATA_DIR}" \
    --ws \
    --ws.port "${WEBSOCKET_PORT}" \
    --metrics \
    --metrics.addr 0.0.0.0 \
    --private.api.addr 0.0.0.0:9091 \
    --metrics.addr 0.0.0.0 \
    --metrics.port 6062 \
    --pprof \
    --pprof.addr 0.0.0.0 \
    --pprof.port 6063 \
    --bootnodes "${EXECUTION_BOOT_NODES}" \
    --torrent.download.rate=120mb \
    --torrent.staticpeers="${EXECUTION_BOOT_NODES}" \
    --state.cache=16384 \
    --rpc.batch.limit=100000 \
    --torrent.download.slots=6 \
    --torrent.verbosity="4" \
    --batchSize=512M \
    --prune.h.older "${KEEP_LAST_BLOCKS}" \
    --prune.r.older "${KEEP_LAST_BLOCKS}" \
    --prune.t.older "${KEEP_LAST_BLOCKS}" \
    --prune.c.older "${KEEP_LAST_BLOCKS}"

Consensus Layer: Heimdall

Consensus Layer Command (with flags/config):

##### RPC and REST configs #####

# RPC endpoint for ethereum chain
eth_rpc_url = "http://127.0.0.1:9545"

# RPC endpoint for bor chain
bor_rpc_url = "http://127.0.0.1:8545"

# RPC endpoint for tendermint
tendermint_rpc_url = "http://0.0.0.0:26657"

# Heimdall REST server endpoint
heimdall_rest_server = "http://0.0.0.0:1317"

#### Bridge configs ####

# AMQP endpoint
amqp_url = "amqp://guest:guest@127.0.0.1:5672/"

## Poll intervals
checkpoint_poll_interval = "5m0s"
syncer_poll_interval = "1m0s"
noack_poll_interval = "16m50s"
clerk_poll_interval = "10s"
span_poll_interval = "1m0s"

#### gas limits ####
main_chain_gas_limit = "5000000"

#### gas price ####
main_chain_max_gas_price = "400000000000"

##### Timeout Config #####
no_ack_wait_time = "30m0s"

Chain/Network: Mainnet

Expected behaviour

The nodes should be syncing to tip over time.

Actual behaviour

The nodes appear to be stalled at block 48077378 even with 18 peers.

Steps to reproduce the behaviour

Sync mainnet node from scratch with the configuration provided above.

Backtrace

The logs below keep getting printed for the last 4 days now

polygon-0 erigon [INFO] [06-24|15:33:33.224] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=6.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:33:38.293] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:34:00.856] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:34:34.962] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:35:08.855] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:35:24.853] [p2p] GoodPeers                          eth68=12 eth66=1 eth67=5
polygon-0 erigon [INFO] [06-24|15:35:26.340] [mem] memory stats                       Rss=80.3GB Size=0B Pss=80.3GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.1GB PrivateDirty=10.2GB Referenced=80.3GB Anonymous=10.2GB Swap=0B alloc=7.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:35:31.376] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:36:05.488] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:36:33.223] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=5.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:36:39.606] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:37:02.248] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:37:36.395] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:38:10.675] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:38:24.853] [p2p] GoodPeers                          eth66=1 eth68=12 eth67=5
polygon-0 erigon [INFO] [06-24|15:38:26.330] [mem] memory stats                       Rss=80.3GB Size=0B Pss=80.3GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.1GB PrivateDirty=10.2GB Referenced=80.3GB Anonymous=10.2GB Swap=0B alloc=6.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:38:33.634] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:39:07.901] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:39:30.476] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:39:33.223] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=8.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:40:04.450] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.5GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:40:38.507] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:41:01.243] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:41:24.853] [p2p] GoodPeers                          eth68=12 eth67=5 eth66=1
polygon-0 erigon [INFO] [06-24|15:41:26.424] [mem] memory stats                       Rss=80.3GB Size=0B Pss=80.3GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.1GB PrivateDirty=10.2GB Referenced=80.3GB Anonymous=10.2GB Swap=0B alloc=4.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:41:35.402] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:42:09.471] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=5.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:42:32.306] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=6.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:42:33.223] [txpool] stat                            pending=0 baseFee=0 queued=30000 alloc=7.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|15:43:06.382] [5/15 Bodies] Writing block bodies       block_num=56299999 remaining=1889577 alloc=7.9GB sys=9.7GB
AskAlexSharov commented 1 week ago

grep -v Writing block bodies | grep -v p2p | grep -v txpool

tobidae-cb commented 1 week ago

I'm assuming you want to see the non-memory stats info but just in case

+ polygon-0 › erigon
polygon-0 erigon [INFO] [06-23|17:18:00.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=108.6MB wasted/sec=35.1KB remaining=1889576 delivered=2022152 blk/sec=67405 cache=256.0MB alloc=4.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-23|19:13:00.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=41.7MB wasted/sec=35.1KB remaining=1889576 delivered=2047121 blk/sec=68237 cache=256.0MB alloc=5.1GB sys=9.7GB
polygon-0 erigon meta_checktxnid:11415 catch invalid root_page_txnid 4710637 for maindb.mod_txnid 4710634 (workaround for incoherent flaw of unified page/buffer cache)
polygon-0 erigon [INFO] [06-24|00:40:30.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=119.1MB wasted/sec=0B remaining=1889576 delivered=2118420 blk/sec=70614 cache=256.0MB alloc=7.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|06:57:00.150] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=136.5MB wasted/sec=70.1KB remaining=1889576 delivered=2200133 blk/sec=73337 cache=256.0MB alloc=6.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|07:54:30.150] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=20.8MB wasted/sec=0B remaining=1889576 delivered=2212597 blk/sec=73753 cache=256.0MB alloc=8.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|11:46:00.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=83.8MB wasted/sec=0B remaining=1889576 delivered=2262781 blk/sec=75426 cache=256.0MB alloc=7.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|13:16:00.150] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=32.7MB wasted/sec=0B remaining=1889576 delivered=2282379 blk/sec=76079 cache=256.0MB alloc=7.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|16:46:00.153] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=76.2MB wasted/sec=35.1KB remaining=1889576 delivered=2327971 blk/sec=77599 cache=256.0MB alloc=7.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|22:36:30.150] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=127.1MB wasted/sec=105.2KB remaining=1889576 delivered=2403985 blk/sec=80132 cache=256.0MB alloc=7.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-24|23:43:00.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=24.1MB wasted/sec=35.1KB remaining=1889576 delivered=2418417 blk/sec=80613 cache=256.0MB alloc=8.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:20:26.462] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:23:26.404] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:25:30.149] [5/15 Bodies] Downloading block bodies   block_num=56299999 delivery/sec=298.2MB wasted/sec=70.1KB remaining=1889576 delivered=2596890 blk/sec=86563 cache=256.0MB alloc=7.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:26:26.354] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:29:26.353] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:32:26.448] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:35:26.493] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:38:26.352] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:41:26.355] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:44:26.355] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:47:26.366] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:50:26.360] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:53:26.431] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:56:26.422] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|13:59:26.357] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:02:26.356] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.5GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:05:26.348] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:08:26.369] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.7GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:11:26.365] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:14:26.447] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:17:26.437] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:20:26.357] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.7GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:23:26.423] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:26:26.341] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.3GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:29:26.347] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.5GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:32:26.373] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.8GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:35:26.364] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:38:26.368] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:41:26.344] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:44:26.439] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.9GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:47:26.342] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.2GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:50:26.374] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=8.0GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:53:26.355] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:56:26.457] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=5.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|14:59:26.344] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.1GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|15:02:26.437] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=4.6GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|15:05:26.346] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=7.4GB sys=9.7GB
polygon-0 erigon [INFO] [06-25|15:08:26.351] [mem] memory stats                       Rss=80.9GB Size=0B Pss=80.9GB SharedClean=5.0MB SharedDirty=0B PrivateClean=70.8GB PrivateDirty=10.1GB Referenced=80.9GB Anonymous=10.1GB Swap=0B alloc=6.4GB sys=9.7GB
tobidae-cb commented 1 week ago

Here's a different pod

polygon-2 erigon [INFO] [06-25|15:37:10.858] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:12.451] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:12.452] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:13.439] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:15.062] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:15.062] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:16.043] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:17.646] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:17.646] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:18.626] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:20.257] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:20.257] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:21.241] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:22.824] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:22.824] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:23.803] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:25.405] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:25.405] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:26.401] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:28.016] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:28.016] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:28.998] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:30.607] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:30.607] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:31.587] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:33.223] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:33.223] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:34.197] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:35.802] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
polygon-2 erigon [EROR] [06-25|15:37:35.802] Staged Sync                              err="[3/15 BorHeimdall] header not found: 56209000"
polygon-2 erigon [INFO] [06-25|15:37:36.776] [3/15 BorHeimdall] Processing spans...   from=8783 to=9094
polygon-2 erigon [INFO] [06-25|15:37:38.387] [3/15 BorHeimdall] Processing sync events... from=56209000 to=58197424
tobidae-cb commented 1 week ago

Other pods

+ polygon-1 › erigon
polygon-1 erigon meta_checktxnid:11415 catch invalid root_page_txnid 4676552 for maindb.mod_txnid 4676547 (workaround for incoherent flaw of unified page/buffer cache)
polygon-1 erigon [INFO] [06-24|14:14:00.389] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=1.0GB wasted/sec=35.1KB remaining=1889343 delivered=3120005 blk/sec=104000 cache=256.0MB alloc=5.2GB sys=10.3GB
polygon-1 erigon [INFO] [06-24|17:31:30.390] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=107.4MB wasted/sec=35.1KB remaining=1889343 delivered=3174815 blk/sec=105827 cache=256.0MB alloc=8.1GB sys=10.3GB
polygon-1 erigon meta_checktxnid:11415 catch invalid root_page_txnid 4883934 for maindb.mod_txnid 4883928 (workaround for incoherent flaw of unified page/buffer cache)
polygon-1 erigon [INFO] [06-25|07:54:30.389] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=469.0MB wasted/sec=35.1KB remaining=1889343 delivered=3414215 blk/sec=113807 cache=256.0MB alloc=5.1GB sys=10.3GB
+ polygon-3 › erigon
polygon-3 erigon [INFO] [06-24|00:50:41.009] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=835.0MB wasted/sec=18.1MB remaining=1889041 delivered=3674616 blk/sec=122487 cache=256.0MB alloc=8.0GB sys=9.5GB
polygon-3 erigon [INFO] [06-24|05:38:11.010] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=207.3MB wasted/sec=34.2KB remaining=1889041 delivered=3776883 blk/sec=125896 cache=256.0MB alloc=8.1GB sys=9.5GB
polygon-3 erigon [INFO] [06-24|07:08:11.009] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=64.8MB wasted/sec=34.2KB remaining=1889041 delivered=3808827 blk/sec=126960 cache=256.0MB alloc=5.8GB sys=9.5GB
polygon-3 erigon [INFO] [06-24|18:24:11.009] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=486.8MB wasted/sec=68.5KB remaining=1889041 delivered=4048968 blk/sec=134965 cache=256.0MB alloc=4.6GB sys=9.5GB
polygon-3 erigon [INFO] [06-24|20:48:11.010] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=103.7MB wasted/sec=0B remaining=1889041 delivered=4100118 blk/sec=136670 cache=256.0MB alloc=4.6GB sys=9.5GB
polygon-3 erigon [INFO] [06-25|02:28:41.009] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=245.6MB wasted/sec=34.2KB remaining=1889041 delivered=4221294 blk/sec=140709 cache=256.0MB alloc=6.2GB sys=9.5GB
polygon-3 erigon [INFO] [06-25|03:38:11.010] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=50.1MB wasted/sec=34.2KB remaining=1889041 delivered=4245978 blk/sec=141532 cache=256.0MB alloc=6.5GB sys=9.5GB
polygon-3 erigon [INFO] [06-25|07:50:41.010] [5/15 Bodies] Downloading block bodies   block_num=56300016 delivery/sec=181.6MB wasted/sec=0B remaining=1889041 delivered=4335573 blk/sec=144519 cache=256.0MB alloc=7.2GB sys=9.5GB
tobidae-cb commented 1 week ago

@AskAlexSharov thoughts on the potential issue?

AskAlexSharov commented 1 week ago

try integration stage_headers --reset