Open AskAlexSharov opened 2 months ago
@AskAlexSharov the screenshot only shows 4 GB - I imagine it was not taken at the right time?
I added https://github.com/erigontech/erigon/pull/11549 for that.
If you have some time can you restart your Erigon instance with latest code on main
and with export SAVE_HEAP_PROFILE=true
env var please?
I'll run an Amoy instance now too and let it run in the hopes I can reproduce.
Saw it on Amoy snapshotter today. here is what in logs if grep -v
this warning:
[WARN] [08-24|01:07:09.774] Sanitizing invalid bor miner gas price provided=1000000000 updated=25000000000
[WARN] [08-24|01:07:09.774] Sanitizing invalid bor min fee cap provided=1 updated=25000000000.000
[WARN] [08-24|01:09:08.880] [7/9 Execution] Execution failed block=10954598 txNum=36126721 hash=0xbec0aa1033826a45a1e5faba3319445f2ac85849d131f4c0a169c0dfa9baabb6 err="invalid block, txnIdx=1, receiptHash mismatch: 2e8eabdeaf20429d2175676ba2e6400f19a5701a0646c9b62422f4ebff09412b != 51478baf47ac21f86ec338e2e187b1b989787c19e0caea2469b964a54a48fbcf, headerNum=10954598, bec0aa1033826a45a1e5faba3319445f2ac85849d131f4c0a169c0dfa9baabb6"
[EROR] [08-24|01:09:08.908] Staged Sync err="bad block unwinding"
So, the reason is: unwind happened and after unwind - node can't function correctly. So, it bug of unwind.
Recent logs:
[WARN] [08-24|02:27:02.283] [downloader] InsertHeader: Rejected header marked as bad hash=0xbec0aa1033826a45a1e5faba3319445f2ac85849d131f4c0a169c0dfa9baabb6 height=10954598
[WARN] [08-24|02:27:02.428] [downloader] InsertHeader: Rejected header marked as bad hash=0xbec0aa1033826a45a1e5faba3319445f2ac85849d131f4c0a169c0dfa9baabb6 height=10954598
[INFO] [08-24|02:28:52.479] [p2p] GoodPeers eth68=7 eth66=1 eth67=4
[INFO] [08-24|02:32:47.710] [p2p] GoodPeers eth68=3 eth67=4 eth66=1
[INFO] [08-24|02:35:03.373] [p2p] GoodPeers eth68=2 eth67=2 eth66=1
[WARN] [08-24|02:37:14.898] [p2p] No GoodPeers
[INFO] [08-24|02:38:03.573] [2/9 Headers] No block headers to write in this log period number=10954597 blk/second=0.000 alloc=115.1GB sys=123.5GB invalidHeaders=0 rejectedBadHeaders=2546
[INFO] [08-24|02:38:49.178] [mem] memory stats Rss=124.0GB Size=0B Pss=124.0GB SharedClean=2.4MB SharedDirty=0B PrivateClean=24.0KB PrivateDirty=124.0GB Referenced=124.0GB Anonymous=124.0GB Swap=0B alloc=115.1GB sys=123.5GB
[INFO] [08-24|02:39:39.324] Req/resp stats req=5311 reqMin=10954406 reqMax=11094481 skel=25 skelMin=10954596 skelMax=10991652 resp=17034 respMin=10954407 respMax=11094490 dups=2210572 alloc=115.1GB sys=123.5GB
[WARN] [08-24|02:40:16.299] [p2p] No GoodPeers
[WARN] [08-24|02:43:14.321] [p2p] No GoodPeers
@AskAlexSharov are you running the snapshotter with SAVE_HEAP_PROFILE=true
?
Do you see any logs [Experiment] heap profile threshold check
and [Experiment] saving heap profile as near OOM
?
SAVE_HEAP_PROFILE=true
- no. but i can add.
yes please! it will be good to get the heap profile at the right time
alloc=110.0GB
last logs: