search

ethereum / go-ethereum

Go implementation of the Ethereum protocol
https://geth.ethereum.org
GNU Lesser General Public License v3.0
47.3k stars 20.01k forks source link

OOM Kill during state snapshot generation after upgrading to Geth 1.11.5 #27109

Open dasudiy opened 1 year ago

dasudiy commented 1 year ago

I have a node that was previously running in a synchronized state. After updating to Geth 1.11.5, I started experiencing issues with the node. I ran removedb and preserved the ancients before attempting to resynchronize. However, during the state snapshot generation process, my node is repeatedly killed due to Out Of Memory (OOM) errors.

I have tried increasing the swapfile size to 32GB, but Geth still consumes all available memory, leading to OOM kills. I also attempted to set the --cache flag to 1024, but it does not seem to make any noticeable difference.

I'm seeking guidance on how to resolve this issue and prevent OOM kills during state snapshot generation. Thank you.

Apr 18 03:06:38 eth geth[12928]: WARN [04-18|03:06:38.510] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
Apr 18 03:06:43 eth geth[12928]: INFO [04-18|03:06:43.544] Generating state snapshot                root=87ef8e..1c588e in=159e48..5eae12 at=21f32f..dbbacf accounts=17,284,328 slots=108,289,769 storage=8.59GiB dangling=0 elapsed=42m17.367s   eta=95h51m52.445s
Apr 18 03:06:54 eth geth[12928]: INFO [04-18|03:06:54.398] Generating state snapshot                root=87ef8e..1c588e in=159e48..5eae12 at=21fd76..f68ce4 accounts=17,284,328 slots=108,294,891 storage=8.59GiB dangling=0 elapsed=42m28.221s   eta=96h16m28.719s
Apr 18 03:07:03 eth geth[12928]: INFO [04-18|03:07:03.761] Generating state snapshot                root=87ef8e..1c588e in=159e48..5eae12 at=221838..8b1883 accounts=17,284,328 slots=108,308,207 storage=8.59GiB dangling=0 elapsed=42m37.583s   eta=96h37m42.064s
Apr 18 03:07:05 eth geth[12928]: WARN [04-18|03:07:05.485] Ignoring already known beacon payload    number=17,070,786 hash=ca5474..70d8fc age=6m30s
Apr 18 03:07:11 eth geth[12928]: INFO [04-18|03:07:11.950] Generating state snapshot                root=87ef8e..1c588e in=159e48..5eae12 at=221e53..ad533f accounts=17,284,328 slots=108,311,281 storage=8.59GiB dangling=0 elapsed=42m45.773s   eta=96h56m16.002s
Apr 18 03:07:20 eth geth[12928]: INFO [04-18|03:07:20.382] Generating state snapshot                root=87ef8e..1c588e in=159e48..5eae12 at=222813..68dd80 accounts=17,284,328 slots=108,315,948 storage=8.59GiB dangling=0 elapsed=42m54.205s   eta=97h15m22.855s
Apr 18 03:07:22 eth geth[12928]: WARN [04-18|03:07:22.378] Snapshot extension registration failed   peer=d33c3f82 err="peer connected on snap without compatible eth support"
Apr 18 03:07:30 eth geth[12928]: INFO [04-18|03:07:30.323] Generating state snapshot                root=87ef8e..1c588e in=159e48..5eae12 at=222f4b..b72066 accounts=17,284,328 slots=108,319,476 storage=8.59GiB dangling=0 elapsed=43m4.146s    eta=97h37m54.95s
Apr 18 03:07:30 eth geth[12928]: INFO [04-18|03:07:30.977] Chain head was updated                   number=17,070,786 hash=ca5474..70d8fc root=8ff628..02ed48 elapsed=215.484746ms age=6m55s
Apr 18 03:07:31 eth geth[12928]: WARN [04-18|03:07:31.368] Peer delivering stale transactions       peer=ab96ffe34be396ee895da452f3ad64c52e5fd763bd54c83cb0484db8c206b569 rejected=34
Apr 18 03:07:37 eth systemd[1]: geth.service: A process of this unit has been killed by the OOM killer.
Apr 18 03:07:41 eth systemd[1]: geth.service: Main process exited, code=killed, status=9/KILL
Apr 18 03:07:41 eth systemd[1]: geth.service: Failed with result 'oom-kill'.
Apr 18 03:07:41 eth systemd[1]: geth.service: Consumed 1h 13min 50.046s CPU time.
Apr 18 03:07:46 eth systemd[1]: geth.service: Scheduled restart job, restart counter is at 1.
Apr 18 03:07:46 eth systemd[1]: Stopped Geth Execution Client (Mainnet).
Apr 18 03:07:46 eth systemd[1]: geth.service: Consumed 1h 13min 50.046s CPU time.
Apr 18 03:07:46 eth systemd[1]: Started Geth Execution Client (Mainnet).
Apr 18 03:07:46 eth geth[13045]: INFO [04-18|03:07:46.439] Starting Geth on Ethereum mainnet...
Apr 18 03:07:46 eth geth[13045]: INFO [04-18|03:07:46.442] Maximum peer count                       ETH=50 LES=0 total=50
Apr 18 03:07:46 eth geth[13045]: INFO [04-18|03:07:46.444] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Apr 18 03:07:46 eth geth[13045]: INFO [04-18|03:07:46.447] Snap sync requested, enabling --snapshot
Apr 18 03:07:46 eth geth[13045]: INFO [04-18|03:07:46.447] Set global gas cap                       cap=50,000,000
Apr 18 03:07:46 eth geth[13045]: INFO [04-18|03:07:46.448] Allocated trie memory caches             clean=153.00MiB dirty=256.00MiB
Apr 18 03:07:46 eth geth[13045]: INFO [04-18|03:07:46.845] Using leveldb as the backing database
Apr 18 03:07:46 eth geth[13045]: INFO [04-18|03:07:46.845] Allocated cache and file handles         database=/var/lib/geth/geth/chaindata cache=512.00MiB handles=262,144
Apr 18 03:07:47 eth geth[13045]: INFO [04-18|03:07:47.894] Using LevelDB as the backing database
Apr 18 03:07:47 eth geth[13045]: INFO [04-18|03:07:47.950] Opened ancient database                  database=/var/lib/geth/geth/chaindata/ancient/chain readonly=false
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.639] Disk storage enabled for ethash caches   dir=/var/lib/geth/geth/ethash count=3
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.639] Disk storage enabled for ethash DAGs     dir=/home/geth/.ethash        count=2
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.643] Initialising Ethereum protocol           network=1 dbversion=8
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.645]
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646] ---------------------------------------------------------------------------------------------------------------------------------------------------------
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646] Chain ID:  1 (mainnet)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646] Pre-Merge hard forks (block based):
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Homestead:                   #1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - DAO Fork:                    #1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Tangerine Whistle (EIP 150): #2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Spurious Dragon/1 (EIP 155): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Spurious Dragon/2 (EIP 158): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Byzantium:                   #4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Constantinople:              #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Petersburg:                  #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Istanbul:                    #9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Muir Glacier:                #9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Berlin:                      #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - London:                      #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Arrow Glacier:               #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Gray Glacier:                #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646] Merge configured:
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Network known to be merged: true
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Total terminal difficulty:  58750000000000000000000
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646] Post-Merge hard forks (timestamp based):
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]  - Shanghai:                    @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646] ---------------------------------------------------------------------------------------------------------------------------------------------------------
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.646]
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.649] Loaded most recent local block           number=17,070,786 hash=ca5474..70d8fc td=58,750,003,716,598,352,816,469 age=7m13s
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.649] Loaded most recent local finalized block number=17,070,702 hash=dd0b7e..537b26 td=58,750,003,716,598,352,816,469 age=24m13s
Apr 18 03:07:48 eth geth[13045]: INFO [04-18|03:07:48.650] Loaded last fast-sync pivot marker       number=17,069,758
Apr 18 03:07:48 eth geth[13045]: WARN [04-18|03:07:48.651] Head state missing, repairing            number=17,070,786 hash=ca5474..70d8fc snaproot=87ef8e..1c588e
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.088] Loaded most recent local header          number=17,070,786 hash=ca5474..70d8fc td=58,750,003,716,598,352,816,469 age=7m14s
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.088] Loaded most recent local block           number=17,070,606 hash=d6687a..648476 td=58,750,003,716,598,352,816,469 age=43m38s
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.088] Loaded most recent local snap block      number=17,070,786 hash=ca5474..70d8fc td=58,750,003,716,598,352,816,469 age=7m14s
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.089] Loaded most recent local finalized block number=17,070,702 hash=dd0b7e..537b26 td=58,750,003,716,598,352,816,469 age=24m14s
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.089] Loaded last fast-sync pivot marker       number=17,069,758
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.092] Enabling snapshot recovery               chainhead=17,070,606 diskbase=17,070,778
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.094] Loaded snapshot journal                  diskroot=87ef8e..1c588e diffs=unmatched
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.094] Snapshot is not continuous with chain    snaproot=87ef8e..1c588e chainroot=82d8a7..f724dd
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.094] Loaded local transaction journal         transactions=0 dropped=0
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.094] Resuming state snapshot generation       root=87ef8e..1c588e in=159e48..5eae12 at=2173a4..1e1d66 accounts=17,284,328 slots=108,226,266 storage=8.58GiB dangling=0 elapsed="249.036µs"
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.095] Regenerated local transaction journal    transactions=0 accounts=0
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.095] Switch sync mode from snap sync to full sync
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.095] Chain post-merge, sync via beacon client
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.096] Gasprice oracle is ignoring threshold set threshold=2
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Old unclean shutdowns found              count=25
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T07:05:57+0000 age=20h1m52s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T08:55:34+0000 age=18h12m15s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T10:48:20+0000 age=16h19m29s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T12:41:49+0000 age=14h26m
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T14:08:34+0000 age=12h59m15s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T16:19:52+0000 age=10h47m57s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T18:48:31+0000 age=8h19m18s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T21:36:16+0000 age=5h31m33s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-17T23:22:54+0000 age=3h44m55s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-18T01:19:41+0000 age=1h48m8s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Unclean shutdown detected                booted=2023-04-18T03:04:26+0000 age=3m23s
Apr 18 03:07:49 eth geth[13045]: WARN [04-18|03:07:49.096] Engine API enabled                       protocol=eth
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.097] Starting peer-to-peer node               instance=Geth/v1.11.5-stable-a38f4108/linux-amd64/go1.20.2
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.098] Trie missing, state snapshotting paused  root=87ef8e..1c588e in=159e48..5eae12 at=2173a4..1e1d66 accounts=17,284,328 slots=108,226,266 storage=8.58GiB dangling=0 elapsed=3.739ms
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.144] New local node record                    seq=1,662,041,897,926 id=cfcce68fd0196271 ip=127.0.0.1 udp=30303 tcp=30303
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.145] Started P2P networking                   self=enode://688e713339375261742a59c04465379649124b54113acb7b0c21313f670e95fa79ad1bd8018061f943249772d1094e864043169e502a6715011001eec38c9236@127.0.0.1:30303
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.145] IPC endpoint opened                      url=/var/lib/geth/geth.ipc
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.146] Loaded JWT secret file                   path=/var/lib/jwt.hex crc32=0xca4fe9b5
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.146] HTTP server started                      endpoint=127.0.0.1:8545 auth=false prefix= cors= vhosts=localhost
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.146] WebSocket enabled                        url=ws://127.0.0.1:8551
Apr 18 03:07:49 eth geth[13045]: INFO [04-18|03:07:49.146] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=localhost
Apr 18 03:07:55 eth geth[13045]: INFO [04-18|03:07:55.010] New local node record                    seq=1,662,041,897,927 id=cfcce68fd0196271 ip=116.52.68.219 udp=30303 tcp=30303
Apr 18 03:07:55 eth geth[13045]: WARN [04-18|03:07:55.312] Snapshot extension registration failed   peer=409e2bcc err="peer connected on snap without compatible eth support"
Apr 18 03:07:59 eth geth[13045]: INFO [04-18|03:07:59.466] Looking for peers                        peercount=0 tried=14 static=0
Apr 18 03:08:09 eth geth[13045]: INFO [04-18|03:08:09.631] Looking for peers                        peercount=0 tried=24 static=0
Apr 18 03:08:19 eth geth[13045]: INFO [04-18|03:08:19.654] Looking for peers                        peercount=2 tried=30 static=0
Apr 18 03:08:24 eth geth[13045]: WARN [04-18|03:08:24.099] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
Apr 18 03:08:29 eth geth[13045]: INFO [04-18|03:08:29.748] Looking for peers                        peercount=2 tried=32 static=0
Apr 18 03:08:39 eth geth[13045]: INFO [04-18|03:08:39.841] Looking for peers                        peercount=2 tried=34 static=0
Apr 18 03:08:50 eth geth[13045]: INFO [04-18|03:08:50.175] Looking for peers                        peercount=2 tried=35 static=0
Apr 18 03:08:53 eth geth[13045]: WARN [04-18|03:08:53.017] Snapshot extension registration failed   peer=dd74b5b2 err="peer connected on snap without compatible eth support"
Apr 18 03:09:00 eth geth[13045]: INFO [04-18|03:09:00.275] Looking for peers                        peercount=2 tried=35 static=0
Apr 18 03:09:05 eth geth[13045]: WARN [04-18|03:09:05.606] Snapshot extension registration failed   peer=1365675b err="peer connected on snap without compatible eth support"
Apr 18 03:09:10 eth geth[13045]: INFO [04-18|03:09:10.466] Looking for peers                        peercount=3 tried=34 static=0
Apr 18 03:09:20 eth geth[13045]: INFO [04-18|03:09:20.650] Looking for peers                        peercount=3 tried=28 static=0
Apr 18 03:09:30 eth geth[13045]: INFO [04-18|03:09:30.799] Looking for peers                        peercount=3 tried=30 static=0
Apr 18 03:09:40 eth geth[13045]: INFO [04-18|03:09:40.799] Looking for peers                        peercount=4 tried=30 static=0
Apr 18 03:09:50 eth geth[13045]: INFO [04-18|03:09:50.800] Looking for peers                        peercount=2 tried=34 static=0
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.140] State not available, ignoring new payload
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.150] Ignoring payload with missing parent     number=17,070,788 hash=ec68d3..80e67a parent=c7cb3f..993147
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.158] Ignoring payload with missing parent     number=17,070,789 hash=ae80c4..6c4f6b parent=ec68d3..80e67a
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.167] Ignoring payload with missing parent     number=17,070,790 hash=7f9e7f..35139e parent=ae80c4..6c4f6b
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.181] Ignoring payload with missing parent     number=17,070,791 hash=cb19be..62c1e5 parent=7f9e7f..35139e
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.187] Ignoring payload with missing parent     number=17,070,792 hash=e5ba71..20178d parent=cb19be..62c1e5
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.195] Ignoring payload with missing parent     number=17,070,793 hash=a4b964..ed4b24 parent=e5ba71..20178d
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.204] Ignoring payload with missing parent     number=17,070,794 hash=07f696..0d09b5 parent=a4b964..ed4b24
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.210] Ignoring payload with missing parent     number=17,070,795 hash=cd15e8..12273d parent=07f696..0d09b5
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.223] Ignoring payload with missing parent     number=17,070,796 hash=2aa228..9da2bf parent=cd15e8..12273d
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.231] Ignoring payload with missing parent     number=17,070,797 hash=e2634f..0bf8cd parent=2aa228..9da2bf
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.869] Ignoring payload with missing parent     number=17,070,798 hash=35474d..f80496 parent=e2634f..0bf8cd
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.877] Ignoring payload with missing parent     number=17,070,799 hash=1938a2..971f7b parent=35474d..f80496
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.894] Ignoring payload with missing parent     number=17,070,800 hash=0aa049..ecc371 parent=1938a2..971f7b
Apr 18 03:09:54 eth geth[13045]: WARN [04-18|03:09:54.901] Ignoring payload with missing parent     number=17,070,801 hash=989896..35fde6 parent=0aa049..ecc371
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.070] Ignoring payload with missing parent     number=17,070,802 hash=591084..2d3e1c parent=989896..35fde6
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.077] Ignoring payload with missing parent     number=17,070,803 hash=4bf046..104cbd parent=591084..2d3e1c
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.081] Ignoring payload with missing parent     number=17,070,804 hash=8f9ba9..ab46c0 parent=4bf046..104cbd
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.088] Ignoring payload with missing parent     number=17,070,805 hash=0a7b0e..d10945 parent=8f9ba9..ab46c0
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.101] Ignoring payload with missing parent     number=17,070,806 hash=6f23a7..4a3962 parent=0a7b0e..d10945
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.107] Ignoring payload with missing parent     number=17,070,807 hash=c68185..1a33ee parent=6f23a7..4a3962
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.115] Ignoring payload with missing parent     number=17,070,808 hash=5a55df..6e3020 parent=c68185..1a33ee
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.129] Ignoring payload with missing parent     number=17,070,809 hash=705570..6a1db0 parent=5a55df..6e3020
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.135] Ignoring payload with missing parent     number=17,070,810 hash=a0ee93..a7b374 parent=705570..6a1db0
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.142] Ignoring payload with missing parent     number=17,070,811 hash=ed67a0..dfc532 parent=a0ee93..a7b374
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.158] Ignoring payload with missing parent     number=17,070,812 hash=5c6f1c..7fc89b parent=ed67a0..dfc532
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.165] Ignoring payload with missing parent     number=17,070,813 hash=47970a..09622e parent=5c6f1c..7fc89b
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.178] Ignoring payload with missing parent     number=17,070,814 hash=b5f0cf..a83036 parent=47970a..09622e
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.186] Ignoring payload with missing parent     number=17,070,815 hash=461005..6c5b72 parent=b5f0cf..a83036
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.193] Ignoring payload with missing parent     number=17,070,816 hash=b46b6b..ad03a3 parent=461005..6c5b72
Apr 18 03:09:55 eth geth[13045]: INFO [04-18|03:09:55.215] Forkchoice requested sync to new head    number=17,070,816 hash=b46b6b..ad03a3 finalized=unknown
Apr 18 03:09:55 eth geth[13045]: INFO [04-18|03:09:55.619] Syncing beacon headers                   downloaded=29 left=0 eta=0s
Apr 18 03:09:55 eth geth[13045]: INFO [04-18|03:09:55.619] Block synchronisation started
Apr 18 03:09:55 eth geth[13045]: WARN [04-18|03:09:55.622] Retrieved pivot header from local        number=17,070,752 hash=e76167..7d7955 latest=17,070,816 oldest=17,070,787
Apr 18 03:09:55 eth geth[13045]: INFO [04-18|03:09:55.796] Importing sidechain segment              start=17,070,607 end=17,070,787
Apr 18 03:10:00 eth geth[13045]: INFO [04-18|03:10:00.993] Looking for peers                        peercount=3 tried=44 static=0
Apr 18 03:10:01 eth geth[13045]: INFO [04-18|03:10:01.976] Forkchoice requested sync to new head    number=17,070,817 hash=410936..5f3fe5 finalized=unknown
Apr 18 03:10:02 eth geth[13045]: INFO [04-18|03:10:02.847] Forkchoice requested sync to new head    number=17,070,818 hash=45256e..213d5b finalized=unknown
Apr 18 03:10:03 eth geth[13045]: INFO [04-18|03:10:03.748] Forkchoice requested sync to new head    number=17,070,819 hash=3f85b5..3ed23b finalized=unknown
Apr 18 03:10:04 eth geth[13045]: INFO [04-18|03:10:04.154] Forkchoice requested sync to new head    number=17,070,820 hash=b42a89..1952ef finalized=unknown
Apr 18 03:10:04 eth geth[13045]: INFO [04-18|03:10:04.527] Forkchoice requested sync to new head    number=17,070,821 hash=f20094..fe5683 finalized=unknown
Apr 18 03:10:04 eth geth[13045]: INFO [04-18|03:10:04.868] Imported new chain segment               number=17,070,612 hash=3773d8..119dcf blocks=6 txs=845 mgas=77.497 elapsed=9.072s      mgasps=8.542 age=44m41s    dirty=8.29MiB
Apr 18 03:10:04 eth geth[13045]: INFO [04-18|03:10:04.913] Forkchoice requested sync to new head    number=17,070,822 hash=8523b8..1c300a finalized=unknown
Apr 18 03:10:05 eth geth[13045]: INFO [04-18|03:10:05.309] Forkchoice requested sync to new head    number=17,070,823 hash=8dddee..72978e finalized=unknown
Apr 18 03:10:05 eth geth[13045]: INFO [04-18|03:10:05.653] Forkchoice requested sync to new head    number=17,070,824 hash=de7a7f..cc9c06 finalized=unknown
Apr 18 03:10:06 eth geth[13045]: INFO [04-18|03:10:06.014] Forkchoice requested sync to new head    number=17,070,825 hash=80a903..aadd44 finalized=unknown
Apr 18 03:10:07 eth geth[13045]: INFO [04-18|03:10:07.895] Forkchoice requested sync to new head    number=17,070,826 hash=6a48f8..b08db3 finalized=unknown
Apr 18 03:10:08 eth geth[13045]: INFO [04-18|03:10:08.335] Forkchoice requested sync to new head    number=17,070,827 hash=a66060..73fdf9 finalized=unknown
Apr 18 03:10:08 eth geth[13045]: INFO [04-18|03:10:08.625] Forkchoice requested sync to new head    number=17,070,828 hash=f660da..97c053 finalized=unknown
Apr 18 03:10:08 eth geth[13045]: INFO [04-18|03:10:08.879] Forkchoice requested sync to new head    number=17,070,829 hash=fd2fed..0b0c1b finalized=unknown
Apr 18 03:10:11 eth geth[13045]: INFO [04-18|03:10:11.132] Looking for peers                        peercount=4 tried=40 static=0
Apr 18 03:10:13 eth geth[13045]: INFO [04-18|03:10:13.225] Imported new chain segment               number=17,070,621 hash=eb9361..4ee3f4 blocks=9 txs=1305 mgas=135.060 elapsed=8.356s      mgasps=16.162 age=43m2s     dirty=21.29MiB
dasudiy commented 1 year ago

One thing I forgot to mention, I've seen a bunch of "Peer delivering stale transactions" messages in my logs, something I didn't see before. Not sure if it's normal or related to this problem.

Apr 18 03:29:45 eth geth[13045]: INFO [04-18|03:29:45.137] Generating state snapshot                root=d34c15..84c4c8 in=159e48..5eae12 at=7cf278..d714e8 accounts=17,284,328 slots=119,872,199 storage=9.35GiB dangling=0 elapsed=21m56.042s
Apr 18 03:29:51 eth geth[13045]: WARN [04-18|03:29:51.031] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=47
Apr 18 03:29:53 eth geth[13045]: INFO [04-18|03:29:53.427] Generating state snapshot                root=d34c15..84c4c8 in=159e48..5eae12 at=7d3445..a6aba1 accounts=17,284,328 slots=119,904,977 storage=9.36GiB dangling=0 elapsed=22m4.333s
Apr 18 03:29:56 eth geth[13045]: WARN [04-18|03:29:56.037] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=117
Apr 18 03:29:56 eth geth[13045]: WARN [04-18|03:29:56.873] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=72
Apr 18 03:29:59 eth geth[13045]: WARN [04-18|03:29:59.357] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=107
Apr 18 03:30:00 eth geth[13045]: WARN [04-18|03:30:00.444] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=106
Apr 18 03:30:01 eth geth[13045]: INFO [04-18|03:30:01.673] Generating state snapshot                root=d34c15..84c4c8 in=159e48..5eae12 at=7d8ac1..98dcb6 accounts=17,284,328 slots=119,947,987 storage=9.36GiB dangling=0 elapsed=22m12.578s
Apr 18 03:30:02 eth geth[13045]: WARN [04-18|03:30:02.477] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=104
Apr 18 03:30:03 eth geth[13045]: WARN [04-18|03:30:03.405] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=89
Apr 18 03:30:05 eth geth[13045]: WARN [04-18|03:30:05.270] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=111
Apr 18 03:30:06 eth geth[13045]: WARN [04-18|03:30:06.047] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=69
Apr 18 03:30:08 eth geth[13045]: WARN [04-18|03:30:08.108] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=114
Apr 18 03:30:09 eth geth[13045]: WARN [04-18|03:30:09.016] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=84
Apr 18 03:30:09 eth geth[13045]: INFO [04-18|03:30:09.881] Generating state snapshot                root=d34c15..84c4c8 in=159e48..5eae12 at=7dd8bb..4ae9e0 accounts=17,284,328 slots=119,986,906 storage=9.36GiB dangling=0 elapsed=22m20.786s
Apr 18 03:30:15 eth geth[13045]: INFO [04-18|03:30:15.468] Aborting state snapshot generation       root=d34c15..84c4c8 in=159e48..5eae12 at=7dfdd5..6486fc accounts=17,284,328 slots=120,005,346 storage=9.36GiB dangling=0 elapsed=22m26.374s
Apr 18 03:30:15 eth geth[13045]: INFO [04-18|03:30:15.511] Resuming state snapshot generation       root=060676..0c5107 in=159e48..5eae12 at=7dfdd5..6486fc accounts=17,284,328 slots=120,005,346 storage=9.36GiB dangling=0 elapsed=22m26.416s
Apr 18 03:30:18 eth geth[13045]: INFO [04-18|03:30:18.089] Imported new potential chain segment     number=17,070,921 hash=391571..6238a6 blocks=1  txs=115  mgas=12.816  elapsed=6.581s       mgasps=1.947  age=1m55s     dirty=254.73MiB
Apr 18 03:30:20 eth geth[13045]: INFO [04-18|03:30:20.669] Chain head was updated                   number=17,070,921 hash=391571..6238a6 root=11a5d5..67de27 elapsed=41.374779ms  age=1m57s
Apr 18 03:30:23 eth geth[13045]: INFO [04-18|03:30:23.710] Generating state snapshot                root=060676..0c5107 in=159e48..5eae12 at=7e2949..b9817e accounts=17,284,328 slots=120,026,861 storage=9.36GiB dangling=0 elapsed=22m34.615s
Apr 18 03:30:31 eth geth[13045]: INFO [04-18|03:30:31.921] Generating state snapshot                root=060676..0c5107 in=159e48..5eae12 at=7e56bc..e0dff4 accounts=17,284,328 slots=120,049,389 storage=9.37GiB dangling=0 elapsed=22m42.826s
Apr 18 03:30:38 eth geth[13045]: WARN [04-18|03:30:38.509] Peer delivering stale transactions       peer=3361a085d5af828a74982e15181b1ea4ae1888185b94a69b0b6d32bd6097c142 rejected=112
holiman commented 1 year ago

What amount of ram do you have on the machine?

dasudiy commented 1 year ago

What amount of ram do you have on the machine?

12GB

holiman commented 1 year ago

It would be interesting to see what the oom-reaper says geth is using. Usually, in logs that when the OOM-reaper hits, it says what the size was. Could you check if you can see that in the syslog?

fjl commented 1 year ago

Are other processes (beacon node?) running on the same machine?

dasudiy commented 1 year ago

I found these logs, I am also running Prysm on the same machine.

Apr 19 11:29:18 eth kernel: geth invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr 19 11:29:18 eth kernel: CPU: 0 PID: 1182 Comm: geth Not tainted 5.15.0-70-generic #77-Ubuntu
Apr 19 11:29:18 eth kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Apr 19 11:29:18 eth kernel: Call Trace:
Apr 19 11:29:18 eth kernel:  <TASK>
Apr 19 11:29:18 eth kernel:  show_stack+0x52/0x5c
Apr 19 11:29:18 eth kernel:  dump_stack_lvl+0x4a/0x63
Apr 19 11:29:18 eth kernel:  dump_stack+0x10/0x16
Apr 19 11:29:18 eth kernel:  dump_header+0x53/0x228
Apr 19 11:29:18 eth kernel:  oom_kill_process.cold+0xb/0x10
Apr 19 11:29:18 eth kernel:  out_of_memory+0x106/0x2e0
Apr 19 11:29:18 eth kernel:  __alloc_pages_slowpath.constprop.0+0x9b7/0xa80
Apr 19 11:29:18 eth kernel:  __alloc_pages+0x311/0x330
Apr 19 11:29:18 eth kernel:  alloc_pages+0x9e/0x1e0
Apr 19 11:29:18 eth kernel:  __page_cache_alloc+0x7e/0x90
Apr 19 11:29:18 eth kernel:  pagecache_get_page+0x152/0x590
Apr 19 11:29:18 eth kernel:  ? page_cache_ra_unbounded+0x166/0x210
Apr 19 11:29:18 eth kernel:  filemap_fault+0x488/0xab0
Apr 19 11:29:18 eth kernel:  ? filemap_map_pages+0x309/0x400
Apr 19 11:29:18 eth kernel:  __do_fault+0x39/0x120
Apr 19 11:29:18 eth kernel:  do_read_fault+0xeb/0x160
Apr 19 11:29:18 eth kernel:  do_fault+0xa0/0x2e0
Apr 19 11:29:18 eth kernel:  handle_pte_fault+0x1cd/0x240
Apr 19 11:29:18 eth kernel:  __handle_mm_fault+0x405/0x6f0
Apr 19 11:29:18 eth kernel:  handle_mm_fault+0xd8/0x2c0
Apr 19 11:29:18 eth kernel:  do_user_addr_fault+0x1c9/0x670
Apr 19 11:29:18 eth kernel:  exc_page_fault+0x77/0x170
Apr 19 11:29:18 eth kernel:  asm_exc_page_fault+0x27/0x30
Apr 19 11:29:18 eth kernel: RIP: 0033:0x444f20
Apr 19 11:29:18 eth kernel: Code: Unable to access opcode bytes at RIP 0x444ef6.
Apr 19 11:29:18 eth kernel: RSP: 002b:00007f70d6d806b0 EFLAGS: 00010246
Apr 19 11:29:18 eth kernel: RAX: 00007f70d6d806f0 RBX: 00007f70d5507b90 RCX: 00000000000000e9
Apr 19 11:29:18 eth kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000002b77898
Apr 19 11:29:18 eth kernel: RBP: 00007f70d6d80d28 R08: 00000000000000e9 R09: 0000000000000072
Apr 19 11:29:18 eth kernel: R10: 0000000000000000 R11: 0000000000000200 R12: 0000000000000001
Apr 19 11:29:18 eth kernel: R13: 0000000000000040 R14: 000000c0000069c0 R15: 0000000000001000
Apr 19 11:29:18 eth kernel:  </TASK>
Apr 19 11:29:18 eth kernel: Mem-Info:
Apr 19 11:29:18 eth kernel: active_anon:2572618 inactive_anon:372292 isolated_anon:64
Apr 19 11:29:18 eth kernel: Node 0 active_anon:10290472kB inactive_anon:1489168kB active_file:832kB inactive_file:148kB unevictable:27620kB isolated(anon):256kB isolated(file):0kB mapped:8988kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 337920kB writeback_tmp:0kB kernel_stack:3520kB pagetables:88108kB all_unreclaimable? yes
Apr 19 11:29:18 eth kernel: Node 0 DMA free:12352kB min:76kB low:92kB high:108kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15036kB managed:14400kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 19 11:29:18 eth kernel: lowmem_reserve[]: 0 1860 12342 12342 12342
Apr 19 11:29:18 eth kernel: Node 0 DMA32 free:51904kB min:10176kB low:12720kB high:15264kB reserved_highatomic:0KB active_anon:1659684kB inactive_anon:216804kB active_file:0kB inactive_file:104kB unevictable:0kB writepending:0kB present:2078964kB managed:2013184kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 19 11:29:18 eth kernel: lowmem_reserve[]: 0 0 10481 10481 10481
Apr 19 11:29:18 eth kernel: Node 0 Normal free:69388kB min:57328kB low:71660kB high:85992kB reserved_highatomic:12288KB active_anon:8631556kB inactive_anon:1270540kB active_file:720kB inactive_file:0kB unevictable:27620kB writepending:0kB present:11010048kB managed:10741912kB mlocked:27620kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 19 11:29:18 eth kernel: lowmem_reserve[]: 0 0 0 0 0
Apr 19 11:29:18 eth kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 2*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 2*1024kB (U) 3*2048kB (UM) 1*4096kB (M) = 12352kB
Apr 19 11:29:18 eth kernel: Node 0 DMA32: 276*4kB (UME) 416*8kB (UME) 409*16kB (UME) 205*32kB (UME) 57*64kB (UME) 11*128kB (UME) 7*256kB (UE) 2*512kB (UM) 11*1024kB (UM) 6*2048kB (M) 1*4096kB (M) = 53056kB
Apr 19 11:29:18 eth kernel: Node 0 Normal: 7152*4kB (UME) 3831*8kB (UME) 601*16kB (UME) 56*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 70664kB
Apr 19 11:29:18 eth kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 19 11:29:18 eth kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 19 11:29:18 eth kernel: 2995 total pagecache pages
Apr 19 11:29:18 eth kernel: 345 pages in swap cache
Apr 19 11:29:18 eth kernel: Swap cache stats: add 96860343, delete 96865791, find 25757905/49890483
Apr 19 11:29:18 eth kernel: Free swap  = 0kB
Apr 19 11:29:18 eth kernel: Total swap = 16777212kB
Apr 19 11:29:18 eth kernel: 3276012 pages RAM
Apr 19 11:29:18 eth kernel: 0 pages HighMem/MovableOnly
Apr 19 11:29:18 eth kernel: 83638 pages reserved
Apr 19 11:29:18 eth kernel: 0 pages hwpoisoned
Apr 19 11:29:18 eth kernel: Tasks state (memory values in pages):
Apr 19 11:29:18 eth kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 19 11:29:18 eth kernel: [    401]     0   401    56645      499   421888      291          -250 systemd-journal
Apr 19 11:29:18 eth kernel: [    444]     0   444    72328     6775   110592        0         -1000 multipathd
Apr 19 11:29:18 eth kernel: [    453]     0   453     6337      491    73728      378         -1000 systemd-udevd
Apr 19 11:29:18 eth kernel: [    587]   104   587    22339      567    77824      236             0 systemd-timesyn
Apr 19 11:29:18 eth kernel: [    677]   101   677     4030      415    73728      279             0 systemd-network
Apr 19 11:29:18 eth kernel: [    679]   102   679     7261      685    98304     1888             0 systemd-resolve
Apr 19 11:29:18 eth kernel: [    697]   103   697     2148      771    53248      195          -900 dbus-daemon
Apr 19 11:29:18 eth kernel: [    704]     0   704     7530      740   106496     2195             0 networkd-dispat
Apr 19 11:29:18 eth kernel: [    705]  1003   705 64208624   805771 41398272   555664             0 beacon-chain-v4
Apr 19 11:29:18 eth kernel: [    706]  1002   706   508045        0   413696    10906             0 validator-v4.0.
Apr 19 11:29:18 eth kernel: [    710]     0   710     3755      524    69632      241             0 systemd-logind
Apr 19 11:29:18 eth kernel: [    713]     0   713   392491       61   315392     3256          -999 containerd
Apr 19 11:29:18 eth kernel: [    756]     0   756      798      209    45056       33             0 agetty
Apr 19 11:29:18 eth kernel: [    764]     0   764     3856      634    77824      432         -1000 sshd
Apr 19 11:29:18 eth kernel: [    796]     0   796    26801      820   110592     2063             0 unattended-upgr
Apr 19 11:29:18 eth kernel: [    937]     0   937   398750      356   385024     5108          -500 dockerd
Apr 19 11:29:18 eth kernel: [   1181]  1001  1181  6250520  2138001 46501888  3610861             0 geth
Apr 19 11:29:18 eth kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=geth.service,mems_allowed=0,global_oom,task_memcg=/system.slice/geth.service,task=geth,pid=1181,uid=1001
Apr 19 11:29:18 eth kernel: Out of memory: Killed process 1181 (geth) total-vm:25002080kB, anon-rss:8552004kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:45412kB oom_score_adj:0
Apr 19 11:29:21 eth kernel: oom_reaper: reaped process 1181 (geth), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
DavidRomanovizc commented 1 year ago

Hello, @dasudiy ,as I know messages about "Peer delivering stale transactions" in your logs may not be related to OOM errors, these messages are often harmless and can be caused by network congestion or delays in the propagation of transactions between nodes.

dasudiy commented 1 year ago

Hello, @dasudiy ,as I know messages about "Peer delivering stale transactions" in your logs may not be related to OOM errors, these messages are often harmless and can be caused by network congestion or delays in the propagation of transactions between nodes.

thanks for the info, but I don't know how to prevent it. I have googled it, and seems it is caused by my node not fully sync?

fjl commented 1 year ago

The message "Peer delivering stale transactions" indicates peer misbehavior. You can't do anything to stop this, we added the log message just to see how often this particular misbehavior occurs.

catenacap commented 1 year ago

Occurring constantly - i mean continuously with breaks in-between.