Closed vogelito closed 3 months ago
michaelsproul
commented
8 months ago Could be related to this issue, which will be fixed in v4.6.0: https://github.com/sigp/lighthouse/issues/4918
Please try upgrading once the release is available (soon) and let us know if it's resolved
vogelito
commented
8 months ago Will be happy to do so
vogelito
commented
8 months ago I haven't upgraded my node, but it's been 7 days since CPU has been higher than 75.0 on average during a 10m period.
This is CPU over the last 30 days. Each of the spikes that took place before Jan 23 in the chart below had triggered the behavior explained in this ticket. We haven't seen this behavior since Jan 23.
We will update to the latest lighhouse release and comment.
vogelito
commented
8 months ago Spoke to soon. Just had another episode a few hours ago. Upgrading to the latest now.
vogelito
commented
8 months ago lighthouse-beacon[10675]: Jan 31 17:10:49.761 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253
lighthouse-validator[10769]: Jan 31 17:10:53.065 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253The incidents had significantly reduced in frequency, but we still see them in our node. However, today they've come back stronger than never before!
These are the charts from Sunday Feb 11 from 17:00 to 18:00 UTC
Looking at the logs, I see the process is getting killed by systemd.
Feb 11 17:33:17 lighthouse-mainnet current-lighthouse-beacon[21687]: Feb 11 17:33:17.001 INFO Synced slot: 8404064, block: … empty, epoch: 262627, finalized_epoch: 262625, finalized_root: 0x3e85…9ac1, exec_hash:
Feb 11 17:33:26 lighthouse-mainnet current-lighthouse-beacon[21687]: Feb 11 17:33:26.416 INFO New block received root: 0x635ddc157a1e0ec5ab70265eb383c51e64fa8b7789d0896d2a4e8cf8aba950ad, slot: 8404065
Feb 11 17:33:29 lighthouse-mainnet current-lighthouse-beacon[21687]: Feb 11 17:33:29.947 INFO Synced slot: 8404065, block: … empty, epoch: 262627, finalized_epoch: 262625, finalized_root: 0x3e85…9ac1, exec_hash:
Feb 11 17:33:35 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL
Feb 11 17:33:35 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'.
Feb 11 17:33:35 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart.
Feb 11 17:33:35 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 31.
Feb 11 17:33:35 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon.
Feb 11 17:33:35 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon.
Feb 11 17:33:35 lighthouse-mainnet current-lighthouse-beacon[9230]: Feb 11 17:33:35.667 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log"
Feb 11 17:33:35 lighthouse-mainnet current-lighthouse-beacon[9230]: Feb 11 17:33:35.684 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253We're having a second, nastier, incident just now (Feb 13, 17:14 UTC) which is ongoing.
Logs show also a restart:
Feb 13 17:14:05 lighthouse-mainnet current-lighthouse-beacon[2973]: Feb 13 17:14:05.001 INFO Synced slot: 8418368, block: … empty, epoch: 263074, finalized_epoch: 263072, finalized_root: 0xf8b5…d93d, exec_hash:
Feb 13 17:14:13 lighthouse-mainnet current-lighthouse-beacon[2973]: Feb 13 17:14:13.554 INFO New block received root: 0x3e073cc5490e683468f6bf746086c46ac6fb64c90b92943bf44f3a2be0b7bee1, slot: 8418369
Feb 13 17:14:18 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL
Feb 13 17:14:18 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'.
Feb 13 17:14:18 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart.
Feb 13 17:14:18 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 35.
Feb 13 17:14:18 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon.
Feb 13 17:14:18 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon.
Feb 13 17:14:18 lighthouse-mainnet current-lighthouse-beacon[28642]: Feb 13 17:14:18.668 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log"
Feb 13 17:14:18 lighthouse-mainnet current-lighthouse-beacon[28642]: Feb 13 17:14:18.679 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253The server continues to restart:
Feb 13 17:18:44 lighthouse-mainnet current-lighthouse-beacon[28642]: Feb 13 17:18:44.660 WARN Syncing deposit contract block cache est_blocks_remaining: initializing deposits, service: slot_notifier
Feb 13 17:18:53 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL
Feb 13 17:18:53 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'.
Feb 13 17:18:54 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart.
Feb 13 17:18:54 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 36.
Feb 13 17:18:54 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon.
Feb 13 17:18:54 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon.
Feb 13 17:18:54 lighthouse-mainnet current-lighthouse-beacon[29698]: Feb 13 17:18:54.185 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log"
Feb 13 17:18:54 lighthouse-mainnet current-lighthouse-beacon[29698]: Feb 13 17:18:54.203 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253Feb 13 17:23:05 lighthouse-mainnet current-lighthouse-beacon[29698]: Feb 13 17:23:05.001 WARN Syncing deposit contract block cache est_blocks_remaining: initializing deposits, service: slot_notifier
Feb 13 17:23:25 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL
Feb 13 17:23:25 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'.
Feb 13 17:23:25 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart.
Feb 13 17:23:25 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 37.
Feb 13 17:23:25 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon.
Feb 13 17:23:25 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon.
Feb 13 17:23:25 lighthouse-mainnet current-lighthouse-beacon[30700]: Feb 13 17:23:25.435 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log"
Feb 13 17:23:25 lighthouse-mainnet current-lighthouse-beacon[30700]: Feb 13 17:23:25.460 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253Feb 13 17:34:17 lighthouse-mainnet current-lighthouse-beacon[30700]: Feb 13 17:34:17.013 WARN Syncing deposit contract block cache est_blocks_remaining: 760, service: slot_notifier
Feb 13 17:34:26 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL
Feb 13 17:34:26 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'.
Feb 13 17:34:26 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart.
Feb 13 17:34:26 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 38.
Feb 13 17:34:26 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon.
Feb 13 17:34:26 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon.
Feb 13 17:34:26 lighthouse-mainnet current-lighthouse-beacon[745]: Feb 13 17:34:26.932 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log"
Feb 13 17:34:26 lighthouse-mainnet current-lighthouse-beacon[745]: Feb 13 17:34:26.954 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253Feb 13 17:42:41 lighthouse-mainnet current-lighthouse-beacon[745]: Feb 13 17:42:41.014 WARN Syncing deposit contract block cache est_blocks_remaining: 2689, service: slot_notifier
Feb 13 17:43:16 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL
Feb 13 17:43:16 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'.
Feb 13 17:43:16 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart.
Feb 13 17:43:16 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 39.
Feb 13 17:43:16 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon.
Feb 13 17:43:16 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon.
Feb 13 17:43:16 lighthouse-mainnet current-lighthouse-beacon[2575]: Feb 13 17:43:16.424 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log"
Feb 13 17:43:16 lighthouse-mainnet current-lighthouse-beacon[2575]: Feb 13 17:43:16.445 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253
chong-he
commented
8 months ago The incidents had significantly reduced in frequency, but we still see them in our node. However, today they've come back stronger than never before!
These are the charts from Sunday Feb 11 from 17:00 to 18:00 UTC
Looking at the logs, I see the process is getting killed by systemd.
Feb 11 17:33:17 lighthouse-mainnet current-lighthouse-beacon[21687]: Feb 11 17:33:17.001 INFO Synced slot: 8404064, block: … empty, epoch: 262627, finalized_epoch: 262625, finalized_root: 0x3e85…9ac1, exec_hash: Feb 11 17:33:26 lighthouse-mainnet current-lighthouse-beacon[21687]: Feb 11 17:33:26.416 INFO New block received root: 0x635ddc157a1e0ec5ab70265eb383c51e64fa8b7789d0896d2a4e8cf8aba950ad, slot: 8404065 Feb 11 17:33:29 lighthouse-mainnet current-lighthouse-beacon[21687]: Feb 11 17:33:29.947 INFO Synced slot: 8404065, block: … empty, epoch: 262627, finalized_epoch: 262625, finalized_root: 0x3e85…9ac1, exec_hash: Feb 11 17:33:35 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL Feb 11 17:33:35 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'. Feb 11 17:33:35 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart. Feb 11 17:33:35 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 31. Feb 11 17:33:35 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon. Feb 11 17:33:35 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon. Feb 11 17:33:35 lighthouse-mainnet current-lighthouse-beacon[9230]: Feb 11 17:33:35.667 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log" Feb 11 17:33:35 lighthouse-mainnet current-lighthouse-beacon[9230]: Feb 11 17:33:35.684 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253We're having a second, nastier, incident just now (Feb 13, 17:14 UTC) which is ongoing.
Logs show also a restart:
Feb 13 17:14:05 lighthouse-mainnet current-lighthouse-beacon[2973]: Feb 13 17:14:05.001 INFO Synced slot: 8418368, block: … empty, epoch: 263074, finalized_epoch: 263072, finalized_root: 0xf8b5…d93d, exec_hash: Feb 13 17:14:13 lighthouse-mainnet current-lighthouse-beacon[2973]: Feb 13 17:14:13.554 INFO New block received root: 0x3e073cc5490e683468f6bf746086c46ac6fb64c90b92943bf44f3a2be0b7bee1, slot: 8418369 Feb 13 17:14:18 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL Feb 13 17:14:18 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'. Feb 13 17:14:18 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart. Feb 13 17:14:18 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 35. Feb 13 17:14:18 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon. Feb 13 17:14:18 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon. Feb 13 17:14:18 lighthouse-mainnet current-lighthouse-beacon[28642]: Feb 13 17:14:18.668 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log" Feb 13 17:14:18 lighthouse-mainnet current-lighthouse-beacon[28642]: Feb 13 17:14:18.679 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253The server continues to restart:
Feb 13 17:18:44 lighthouse-mainnet current-lighthouse-beacon[28642]: Feb 13 17:18:44.660 WARN Syncing deposit contract block cache est_blocks_remaining: initializing deposits, service: slot_notifier Feb 13 17:18:53 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL Feb 13 17:18:53 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'. Feb 13 17:18:54 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart. Feb 13 17:18:54 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 36. Feb 13 17:18:54 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon. Feb 13 17:18:54 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon. Feb 13 17:18:54 lighthouse-mainnet current-lighthouse-beacon[29698]: Feb 13 17:18:54.185 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log" Feb 13 17:18:54 lighthouse-mainnet current-lighthouse-beacon[29698]: Feb 13 17:18:54.203 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253Feb 13 17:23:05 lighthouse-mainnet current-lighthouse-beacon[29698]: Feb 13 17:23:05.001 WARN Syncing deposit contract block cache est_blocks_remaining: initializing deposits, service: slot_notifier Feb 13 17:23:25 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL Feb 13 17:23:25 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'. Feb 13 17:23:25 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart. Feb 13 17:23:25 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 37. Feb 13 17:23:25 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon. Feb 13 17:23:25 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon. Feb 13 17:23:25 lighthouse-mainnet current-lighthouse-beacon[30700]: Feb 13 17:23:25.435 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log" Feb 13 17:23:25 lighthouse-mainnet current-lighthouse-beacon[30700]: Feb 13 17:23:25.460 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253Feb 13 17:34:17 lighthouse-mainnet current-lighthouse-beacon[30700]: Feb 13 17:34:17.013 WARN Syncing deposit contract block cache est_blocks_remaining: 760, service: slot_notifier Feb 13 17:34:26 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL Feb 13 17:34:26 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'. Feb 13 17:34:26 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart. Feb 13 17:34:26 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 38. Feb 13 17:34:26 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon. Feb 13 17:34:26 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon. Feb 13 17:34:26 lighthouse-mainnet current-lighthouse-beacon[745]: Feb 13 17:34:26.932 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log" Feb 13 17:34:26 lighthouse-mainnet current-lighthouse-beacon[745]: Feb 13 17:34:26.954 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253Feb 13 17:42:41 lighthouse-mainnet current-lighthouse-beacon[745]: Feb 13 17:42:41.014 WARN Syncing deposit contract block cache est_blocks_remaining: 2689, service: slot_notifier Feb 13 17:43:16 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL Feb 13 17:43:16 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'. Feb 13 17:43:16 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart. Feb 13 17:43:16 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 39. Feb 13 17:43:16 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon. Feb 13 17:43:16 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon. Feb 13 17:43:16 lighthouse-mainnet current-lighthouse-beacon[2575]: Feb 13 17:43:16.424 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log" Feb 13 17:43:16 lighthouse-mainnet current-lighthouse-beacon[2575]: Feb 13 17:43:16.445 INFO Lighthouse started version: Lighthouse/v4.6.0-1be5253
It got oom-ed (out of memory error) so it was forced shut down and restarted (set in systemd) The resource usage is quite high too. Do you have lots of validators connected to the beacon node, or doing parallel API queries?
vogelito
commented
8 months ago Do you have lots of validators connected to the beacon node, or doing parallel API queries?
No, there is a single validator connected to the beacon node and nothing else doing API queries to the beacon node.
michaelsproul
commented
8 months ago @vogelito Could you please DM me debug logs and metrics from one of these node on Discord (I am @sproul) or email ($surname@sigmaprime.io). You can find debug logs in your datadir at $datadir/beacon/logs. You can get metrics via curl http://localhost:5054/metrics > metrics.txt. Thanks
vogelito
commented
8 months ago Sent!
vogelito
commented
8 months ago Hi - we’re seeing another big instance of this happening today after quietness for a few days
vogelito
commented
8 months ago It's been pretty bad over the last 48 hours.
Sending logs via discord.
vogelito
commented
7 months ago Started running the new version of lighthouse on Feb 23 20:56:21.569
Feb 23 20:56:21.569 INFO Lighthouse started version: Lighthouse/v5.0.0-b5bae6eJust over 2 hours after upgrading (Feb 23 23:06:07), it got killed:
Feb 23 23:06:05 lighthouse-mainnet current-lighthouse-beacon[805]: Feb 23 23:06:05.002 INFO Synced slot: 8492128, block: 0x346d…c6d8, epoch: 265379, finalized_epoch: 265377, finalized_root: 0xb180…caad, exec_hash: 0x4ec7…6
Feb 23 23:06:07 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Main process exited, code=killed, status=9/KILL
Feb 23 23:06:07 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Failed with result 'signal'.
Feb 23 23:06:07 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Service hold-off time over, scheduling restart.
Feb 23 23:06:07 lighthouse-mainnet systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 1.
Feb 23 23:06:07 lighthouse-mainnet systemd[1]: Stopped Lighthouse Beacon daemon.
Feb 23 23:06:07 lighthouse-mainnet systemd[1]: Started Lighthouse Beacon daemon.
Feb 23 23:06:07 lighthouse-mainnet current-lighthouse-beacon[28353]: Feb 23 23:06:07.677 INFO Logging to file path: "/home/ether/.lighthouse/mainnet/beacon/logs/beacon.log"
Feb 23 23:06:07 lighthouse-mainnet current-lighthouse-beacon[28353]: Feb 23 23:06:07.699 INFO Lighthouse started version: Lighthouse/v5.0.0-b5bae6eSimilar CPU spike to past times:
I increased RAM to 32GB from 16GB to see if that helps...:
vogelito
commented
7 months ago Since increasing the memory of the node to 32GB (on Feb 24 01:31:40.130), we have not seen any beacon node restarts.
michaelsproul
commented
7 months ago Sorry for the slow replies @vogelito. If memory is the main issue then perhaps the improvements we made in https://github.com/sigp/lighthouse/pull/5270 will help (coming in v5.1.0)
vogelito
commented
7 months ago Sounds good! Let’s see how it goes after the new release…
vogelito
commented
7 months ago Some more logs prior to our memory upgrade worth sharing:
$ journalctl -o short-precise -k -b -3 | head -1
-- Logs begin at Wed 2023-12-13 04:27:02 UTC, end at Tue 2024-02-27 17:14:17 UTC. --
$ journalctl -o short-precise -k -b -3 | grep Killed
Dec 14 04:39:05.374989 lighthouse-mainnet kernel: Out of memory: Killed process 962 (current-lightho) total-vm:31573004kB, anon-rss:8349420kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52612kB oom_score_adj:0
Dec 15 03:09:41.520232 lighthouse-mainnet kernel: Out of memory: Killed process 7881 (current-lightho) total-vm:28311960kB, anon-rss:8334552kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47564kB oom_score_adj:0
Dec 15 15:57:51.028155 lighthouse-mainnet kernel: Out of memory: Killed process 6766 (current-lightho) total-vm:26435912kB, anon-rss:8262468kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44748kB oom_score_adj:0
Dec 15 21:36:31.060872 lighthouse-mainnet kernel: Out of memory: Killed process 9498 (current-lightho) total-vm:20153852kB, anon-rss:8277092kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33664kB oom_score_adj:0
Dec 19 07:31:46.322335 lighthouse-mainnet kernel: Out of memory: Killed process 17450 (current-lightho) total-vm:30041128kB, anon-rss:8260064kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:51356kB oom_score_adj:0
Dec 20 00:39:41.340908 lighthouse-mainnet kernel: Out of memory: Killed process 3122 (current-lightho) total-vm:25201120kB, anon-rss:8168340kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43600kB oom_score_adj:0
Dec 20 23:25:29.173859 lighthouse-mainnet kernel: Out of memory: Killed process 29452 (current-lightho) total-vm:29668760kB, anon-rss:8262360kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:49348kB oom_score_adj:0
Dec 20 23:42:26.629822 lighthouse-mainnet kernel: Out of memory: Killed process 31070 (current-lightho) total-vm:21488404kB, anon-rss:8303988kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34136kB oom_score_adj:0
Dec 23 07:31:55.023911 lighthouse-mainnet kernel: Out of memory: Killed process 2401 (current-lightho) total-vm:31416348kB, anon-rss:8059736kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:53212kB oom_score_adj:0
Dec 23 23:51:12.398723 lighthouse-mainnet kernel: Out of memory: Killed process 8667 (current-lightho) total-vm:26071020kB, anon-rss:8111616kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:45052kB oom_score_adj:0
Dec 24 06:34:27.729973 lighthouse-mainnet kernel: Out of memory: Killed process 24621 (current-lightho) total-vm:24044672kB, anon-rss:8052168kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:39392kB oom_score_adj:0
Dec 27 22:27:47.383539 lighthouse-mainnet kernel: Out of memory: Killed process 14365 (current-lightho) total-vm:29696192kB, anon-rss:8054480kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:51336kB oom_score_adj:0
Dec 27 22:35:29.393364 lighthouse-mainnet kernel: Out of memory: Killed process 12159 (current-lightho) total-vm:20550936kB, anon-rss:7937516kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:32576kB oom_score_adj:0
Dec 28 12:26:12.182727 lighthouse-mainnet kernel: Out of memory: Killed process 13902 (current-lightho) total-vm:29064044kB, anon-rss:8068828kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47968kB oom_score_adj:0
Dec 28 16:42:09.283054 lighthouse-mainnet kernel: Out of memory: Killed process 30575 (current-lightho) total-vm:24009104kB, anon-rss:8141112kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41864kB oom_score_adj:0
Dec 29 15:24:38.432971 lighthouse-mainnet kernel: Out of memory: Killed process 20670 (current-lightho) total-vm:26931496kB, anon-rss:8098540kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44604kB oom_score_adj:0
Dec 29 15:39:12.857893 lighthouse-mainnet kernel: Out of memory: Killed process 21699 (current-lightho) total-vm:17494812kB, anon-rss:8083776kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27336kB oom_score_adj:0
Dec 29 19:15:50.241616 lighthouse-mainnet kernel: Out of memory: Killed process 24778 (current-lightho) total-vm:25707544kB, anon-rss:7889248kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43972kB oom_score_adj:0
Dec 30 04:58:21.832916 lighthouse-mainnet kernel: Out of memory: Killed process 6379 (current-lightho) total-vm:26312564kB, anon-rss:7974068kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:45480kB oom_score_adj:0
Dec 30 12:32:45.471101 lighthouse-mainnet kernel: Out of memory: Killed process 2046 (current-lightho) total-vm:21776052kB, anon-rss:7819248kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36872kB oom_score_adj:0
Dec 30 12:51:57.484964 lighthouse-mainnet kernel: Out of memory: Killed process 2510 (current-lightho) total-vm:19869208kB, anon-rss:7908556kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31980kB oom_score_adj:0
Dec 30 15:25:31.820587 lighthouse-mainnet kernel: Out of memory: Killed process 6639 (current-lightho) total-vm:21079724kB, anon-rss:7958536kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35028kB oom_score_adj:0
Jan 01 08:03:51.982826 lighthouse-mainnet kernel: Out of memory: Killed process 7294 (current-lightho) total-vm:28105024kB, anon-rss:7768140kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:46572kB oom_score_adj:0
Jan 02 06:19:52.559764 lighthouse-mainnet kernel: Out of memory: Killed process 12753 (current-lightho) total-vm:26707840kB, anon-rss:7799180kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:45040kB oom_score_adj:0
Jan 02 08:28:02.943288 lighthouse-mainnet kernel: Out of memory: Killed process 7274 (current-lightho) total-vm:18248264kB, anon-rss:7837472kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:29232kB oom_score_adj:0
Jan 02 13:43:09.582812 lighthouse-mainnet kernel: Out of memory: Killed process 2582 (current-lightho) total-vm:22341132kB, anon-rss:7791016kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37436kB oom_score_adj:0
Jan 02 18:05:29.347403 lighthouse-mainnet kernel: Out of memory: Killed process 5230 (current-lightho) total-vm:19855568kB, anon-rss:7907568kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33192kB oom_score_adj:0
Jan 03 04:00:35.761156 lighthouse-mainnet kernel: Out of memory: Killed process 29638 (current-lightho) total-vm:24052084kB, anon-rss:7908904kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40768kB oom_score_adj:0
Jan 03 08:54:59.415792 lighthouse-mainnet kernel: Out of memory: Killed process 27625 (current-lightho) total-vm:20947568kB, anon-rss:7849880kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34824kB oom_score_adj:0
Jan 03 09:07:55.082459 lighthouse-mainnet kernel: Out of memory: Killed process 26058 (current-lightho) total-vm:14558224kB, anon-rss:7864628kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23008kB oom_score_adj:0
Jan 03 09:14:31.094680 lighthouse-mainnet kernel: Out of memory: Killed process 28882 (current-lightho) total-vm:19069696kB, anon-rss:7892756kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:30084kB oom_score_adj:0
Jan 03 14:54:28.058602 lighthouse-mainnet kernel: Out of memory: Killed process 30395 (current-lightho) total-vm:24629136kB, anon-rss:7783592kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41528kB oom_score_adj:0
Jan 04 11:54:14.358449 lighthouse-mainnet kernel: Out of memory: Killed process 5799 (current-lightho) total-vm:28478968kB, anon-rss:7762788kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47812kB oom_score_adj:0
Jan 04 15:44:32.649172 lighthouse-mainnet kernel: Out of memory: Killed process 17679 (current-lightho) total-vm:19177316kB, anon-rss:7933256kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31732kB oom_score_adj:0
Jan 04 21:30:34.038592 lighthouse-mainnet kernel: Out of memory: Killed process 2335 (current-lightho) total-vm:21570220kB, anon-rss:7953924kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37384kB oom_score_adj:0
Jan 04 21:36:47.785311 lighthouse-mainnet kernel: Out of memory: Killed process 11728 (current-lightho) total-vm:15886844kB, anon-rss:7854720kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24940kB oom_score_adj:0
Jan 04 22:22:45.310930 lighthouse-mainnet kernel: Out of memory: Killed process 13104 (current-lightho) total-vm:21301176kB, anon-rss:7926764kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35680kB oom_score_adj:0
Jan 05 01:33:33.920371 lighthouse-mainnet kernel: Out of memory: Killed process 22980 (current-lightho) total-vm:24344312kB, anon-rss:7965900kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40856kB oom_score_adj:0
Jan 05 04:24:31.572123 lighthouse-mainnet kernel: Out of memory: Killed process 31519 (current-lightho) total-vm:20470492kB, anon-rss:7919324kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35456kB oom_score_adj:0
Jan 05 14:53:39.364521 lighthouse-mainnet kernel: Out of memory: Killed process 3313 (current-lightho) total-vm:23739984kB, anon-rss:7969408kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40064kB oom_score_adj:0
Jan 05 21:36:54.493935 lighthouse-mainnet kernel: Out of memory: Killed process 9264 (current-lightho) total-vm:27124364kB, anon-rss:7826864kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47236kB oom_score_adj:0
Jan 06 00:42:06.429796 lighthouse-mainnet kernel: Out of memory: Killed process 31149 (current-lightho) total-vm:23430824kB, anon-rss:7953700kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:39568kB oom_score_adj:0
Jan 06 11:35:08.386709 lighthouse-mainnet kernel: Out of memory: Killed process 5990 (current-lightho) total-vm:24330336kB, anon-rss:7973124kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42072kB oom_score_adj:0
Jan 06 13:04:44.558194 lighthouse-mainnet kernel: Out of memory: Killed process 16458 (current-lightho) total-vm:19864656kB, anon-rss:7840380kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:32588kB oom_score_adj:0
Jan 06 13:11:03.800573 lighthouse-mainnet kernel: Out of memory: Killed process 3358 (current-lightho) total-vm:18518476kB, anon-rss:7916780kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:28640kB oom_score_adj:0
Jan 06 15:44:43.928299 lighthouse-mainnet kernel: Out of memory: Killed process 4805 (current-lightho) total-vm:20810208kB, anon-rss:7862196kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34688kB oom_score_adj:0
Jan 06 18:56:36.111153 lighthouse-mainnet kernel: Out of memory: Killed process 5251 (current-lightho) total-vm:21858356kB, anon-rss:7936456kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:38308kB oom_score_adj:0
Jan 06 21:17:29.648568 lighthouse-mainnet kernel: Out of memory: Killed process 14553 (current-lightho) total-vm:21478220kB, anon-rss:7922228kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:38024kB oom_score_adj:0
Jan 06 23:00:11.853261 lighthouse-mainnet kernel: Out of memory: Killed process 12476 (current-lightho) total-vm:23531308kB, anon-rss:7812116kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40660kB oom_score_adj:0
Jan 07 00:57:03.250833 lighthouse-mainnet kernel: Out of memory: Killed process 2094 (current-lightho) total-vm:21896668kB, anon-rss:7914408kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36884kB oom_score_adj:0
Jan 07 04:12:47.997387 lighthouse-mainnet kernel: Out of memory: Killed process 27245 (current-lightho) total-vm:24889964kB, anon-rss:7906900kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42608kB oom_score_adj:0
Jan 07 04:18:08.371801 lighthouse-mainnet kernel: Out of memory: Killed process 4275 (current-lightho) total-vm:19170088kB, anon-rss:7998840kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31436kB oom_score_adj:0
Jan 07 07:37:00.363853 lighthouse-mainnet kernel: Out of memory: Killed process 5517 (current-lightho) total-vm:20349488kB, anon-rss:7908880kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34828kB oom_score_adj:0
Jan 07 09:18:16.332168 lighthouse-mainnet kernel: Out of memory: Killed process 16779 (current-lightho) total-vm:22712844kB, anon-rss:7955044kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:38144kB oom_score_adj:0
Jan 07 12:19:44.033406 lighthouse-mainnet kernel: Out of memory: Killed process 6054 (current-lightho) total-vm:23145688kB, anon-rss:7844892kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40540kB oom_score_adj:0
Jan 07 13:10:58.024468 lighthouse-mainnet kernel: Out of memory: Killed process 12546 (current-lightho) total-vm:20916036kB, anon-rss:7947052kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35156kB oom_score_adj:0
Jan 08 02:12:54.545541 lighthouse-mainnet kernel: Out of memory: Killed process 23592 (current-lightho) total-vm:23450836kB, anon-rss:7902340kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41124kB oom_score_adj:0
Jan 08 05:23:57.680343 lighthouse-mainnet kernel: Out of memory: Killed process 29265 (current-lightho) total-vm:23332764kB, anon-rss:7834508kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41228kB oom_score_adj:0
Jan 08 18:43:50.517078 lighthouse-mainnet kernel: Out of memory: Killed process 5390 (current-lightho) total-vm:24555540kB, anon-rss:7858000kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41912kB oom_score_adj:0
Jan 09 06:34:28.336240 lighthouse-mainnet kernel: Out of memory: Killed process 15363 (current-lightho) total-vm:26197616kB, anon-rss:7828772kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44868kB oom_score_adj:0
Jan 09 10:54:13.891656 lighthouse-mainnet kernel: Out of memory: Killed process 6309 (current-lightho) total-vm:21929736kB, anon-rss:7846948kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37388kB oom_score_adj:0
Jan 09 10:57:51.265786 lighthouse-mainnet kernel: Out of memory: Killed process 29655 (current-lightho) total-vm:14191844kB, anon-rss:7875276kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:21664kB oom_score_adj:0
Jan 10 22:29:30.702037 lighthouse-mainnet kernel: Out of memory: Killed process 30519 (current-lightho) total-vm:29149748kB, anon-rss:7774348kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:50296kB oom_score_adj:0
Jan 11 03:45:59.942838 lighthouse-mainnet kernel: Out of memory: Killed process 28063 (current-lightho) total-vm:23123168kB, anon-rss:7949012kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37552kB oom_score_adj:0
Jan 11 08:48:27.780359 lighthouse-mainnet kernel: Out of memory: Killed process 29674 (current-lightho) total-vm:25484968kB, anon-rss:7850192kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43108kB oom_score_adj:0
Jan 12 04:32:35.945996 lighthouse-mainnet kernel: Out of memory: Killed process 28635 (current-lightho) total-vm:28153140kB, anon-rss:7833468kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:48908kB oom_score_adj:0
Jan 13 05:17:30.405368 lighthouse-mainnet kernel: Out of memory: Killed process 18180 (current-lightho) total-vm:25788828kB, anon-rss:7625432kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44564kB oom_score_adj:0
Jan 13 05:21:33.736173 lighthouse-mainnet kernel: Out of memory: Killed process 6679 (current-lightho) total-vm:16747376kB, anon-rss:7786468kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25820kB oom_score_adj:0
Jan 13 07:12:29.042158 lighthouse-mainnet kernel: Out of memory: Killed process 7607 (current-lightho) total-vm:16692316kB, anon-rss:7792356kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:28356kB oom_score_adj:0
Jan 14 13:23:56.341548 lighthouse-mainnet kernel: Out of memory: Killed process 31744 (current-lightho) total-vm:25478028kB, anon-rss:7721428kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43648kB oom_score_adj:0
Jan 15 13:36:41.145328 lighthouse-mainnet kernel: Out of memory: Killed process 23845 (current-lightho) total-vm:26612668kB, anon-rss:7810576kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44380kB oom_score_adj:0
Jan 17 03:03:04.908883 lighthouse-mainnet kernel: Out of memory: Killed process 5720 (current-lightho) total-vm:26837712kB, anon-rss:7821788kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47732kB oom_score_adj:0
Jan 17 03:35:16.516670 lighthouse-mainnet kernel: Out of memory: Killed process 24754 (current-lightho) total-vm:22081184kB, anon-rss:7845484kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35988kB oom_score_adj:0
Jan 18 04:19:59.909902 lighthouse-mainnet kernel: Out of memory: Killed process 31571 (current-lightho) total-vm:27495396kB, anon-rss:7727544kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47140kB oom_score_adj:0
Jan 18 12:01:04.498704 lighthouse-mainnet kernel: Out of memory: Killed process 19994 (current-lightho) total-vm:23869344kB, anon-rss:7722680kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41588kB oom_score_adj:0
Jan 20 20:19:40.967670 lighthouse-mainnet kernel: Out of memory: Killed process 20307 (current-lightho) total-vm:27500976kB, anon-rss:7609752kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47248kB oom_score_adj:0
Jan 20 20:26:32.024904 lighthouse-mainnet kernel: Out of memory: Killed process 18453 (current-lightho) total-vm:18459480kB, anon-rss:7733420kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:29236kB oom_score_adj:0
Jan 21 18:24:33.421907 lighthouse-mainnet kernel: Out of memory: Killed process 20057 (current-lightho) total-vm:23303388kB, anon-rss:7696060kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:39644kB oom_score_adj:0
Jan 21 21:24:04.928374 lighthouse-mainnet kernel: Out of memory: Killed process 5832 (current-lightho) total-vm:23399912kB, anon-rss:7714576kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41552kB oom_score_adj:0
Jan 22 12:44:51.580194 lighthouse-mainnet kernel: Out of memory: Killed process 11318 (current-lightho) total-vm:27224828kB, anon-rss:7687956kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:46108kB oom_score_adj:0
Jan 22 18:48:28.898436 lighthouse-mainnet kernel: Out of memory: Killed process 11219 (current-lightho) total-vm:24101796kB, anon-rss:7632308kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40688kB oom_score_adj:0
Jan 22 19:31:38.613777 lighthouse-mainnet kernel: Out of memory: Killed process 22873 (current-lightho) total-vm:21534516kB, anon-rss:7689396kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36736kB oom_score_adj:0
Jan 24 08:10:10.107465 lighthouse-mainnet kernel: Out of memory: Killed process 11097 (current-lightho) total-vm:28246500kB, anon-rss:8337700kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:48708kB oom_score_adj:0
Jan 25 02:11:58.020818 lighthouse-mainnet kernel: Out of memory: Killed process 10260 (current-lightho) total-vm:26112432kB, anon-rss:8351536kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:45136kB oom_score_adj:0
Jan 26 04:58:22.147850 lighthouse-mainnet kernel: Out of memory: Killed process 11277 (current-lightho) total-vm:27708708kB, anon-rss:8217128kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47148kB oom_score_adj:0
Jan 26 11:35:06.439702 lighthouse-mainnet kernel: Out of memory: Killed process 24761 (current-lightho) total-vm:23690920kB, anon-rss:8272292kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41428kB oom_score_adj:0
Jan 26 20:00:52.316294 lighthouse-mainnet kernel: Out of memory: Killed process 11645 (current-lightho) total-vm:28158708kB, anon-rss:8215116kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:48416kB oom_score_adj:0
Jan 27 01:57:56.837604 lighthouse-mainnet kernel: Out of memory: Killed process 20753 (current-lightho) total-vm:28499748kB, anon-rss:8182268kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47564kB oom_score_adj:0
Jan 30 02:43:53.741180 lighthouse-mainnet kernel: Out of memory: Killed process 31002 (current-lightho) total-vm:32346784kB, anon-rss:8096064kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:54084kB oom_score_adj:0
Jan 31 04:58:44.293757 lighthouse-mainnet kernel: Out of memory: Killed process 10101 (current-lightho) total-vm:29959460kB, anon-rss:7967412kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:49360kB oom_score_adj:0
Jan 31 06:02:12.813341 lighthouse-mainnet kernel: Out of memory: Killed process 17578 (current-lightho) total-vm:21328356kB, anon-rss:8005648kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34972kB oom_score_adj:0
Jan 31 08:16:35.810257 lighthouse-mainnet kernel: Out of memory: Killed process 30945 (current-lightho) total-vm:21929128kB, anon-rss:7957384kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35628kB oom_score_adj:0
Jan 31 15:21:02.593999 lighthouse-mainnet kernel: Out of memory: Killed process 26987 (current-lightho) total-vm:26361740kB, anon-rss:7980864kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43788kB oom_score_adj:0
Jan 31 15:25:30.128611 lighthouse-mainnet kernel: Out of memory: Killed process 19112 (current-lightho) total-vm:16735940kB, anon-rss:8013300kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26480kB oom_score_adj:0
Feb 02 08:10:44.253048 lighthouse-mainnet kernel: Out of memory: Killed process 10675 (current-lightho) total-vm:32280016kB, anon-rss:8840360kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:55388kB oom_score_adj:0
Feb 02 18:37:21.706785 lighthouse-mainnet kernel: Out of memory: Killed process 17098 (current-lightho) total-vm:29017792kB, anon-rss:8667292kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:48092kB oom_score_adj:0
Feb 03 15:12:48.307048 lighthouse-mainnet kernel: Out of memory: Killed process 19597 (current-lightho) total-vm:29494928kB, anon-rss:8672084kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:50272kB oom_score_adj:0
Feb 04 03:15:55.952197 lighthouse-mainnet kernel: Out of memory: Killed process 20590 (current-lightho) total-vm:29029884kB, anon-rss:8647332kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47764kB oom_score_adj:0
Feb 04 14:47:10.737046 lighthouse-mainnet kernel: Out of memory: Killed process 10419 (current-lightho) total-vm:30560924kB, anon-rss:8584112kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:50580kB oom_score_adj:0
Feb 04 17:14:13.752217 lighthouse-mainnet kernel: Out of memory: Killed process 26917 (current-lightho) total-vm:25749116kB, anon-rss:8588100kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44092kB oom_score_adj:0
Feb 05 15:57:29.231658 lighthouse-mainnet kernel: Out of memory: Killed process 25391 (current-lightho) total-vm:33086368kB, anon-rss:8539332kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:54356kB oom_score_adj:0
Feb 05 16:42:18.298598 lighthouse-mainnet kernel: Out of memory: Killed process 20723 (current-lightho) total-vm:23823796kB, anon-rss:8519944kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41436kB oom_score_adj:0
Feb 08 00:35:54.228303 lighthouse-mainnet kernel: Out of memory: Killed process 30280 (current-lightho) total-vm:30955488kB, anon-rss:8494692kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52344kB oom_score_adj:0
Feb 08 00:55:19.253069 lighthouse-mainnet kernel: Out of memory: Killed process 23389 (current-lightho) total-vm:23926200kB, anon-rss:8454528kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37508kB oom_score_adj:0
Feb 09 00:48:49.494765 lighthouse-mainnet kernel: Out of memory: Killed process 27459 (current-lightho) total-vm:31994556kB, anon-rss:8521116kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52208kB oom_score_adj:0
Feb 09 00:51:43.995108 lighthouse-mainnet kernel: Out of memory: Killed process 5461 (current-lightho) total-vm:13242560kB, anon-rss:8515848kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:21048kB oom_score_adj:0
Feb 09 00:55:22.239175 lighthouse-mainnet kernel: Out of memory: Killed process 6101 (current-lightho) total-vm:16506376kB, anon-rss:8300156kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25284kB oom_score_adj:0
Feb 09 01:06:06.646860 lighthouse-mainnet kernel: Out of memory: Killed process 6953 (current-lightho) total-vm:15134972kB, anon-rss:8282740kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23404kB oom_score_adj:0
Feb 09 03:40:11.421143 lighthouse-mainnet kernel: Out of memory: Killed process 7715 (current-lightho) total-vm:22935132kB, anon-rss:7914576kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:38580kB oom_score_adj:0
Feb 09 03:44:15.041099 lighthouse-mainnet kernel: Out of memory: Killed process 8350 (current-lightho) total-vm:14083512kB, anon-rss:7983012kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:21484kB oom_score_adj:0
Feb 09 03:48:54.066466 lighthouse-mainnet kernel: Out of memory: Killed process 8998 (current-lightho) total-vm:16751236kB, anon-rss:7937784kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26996kB oom_score_adj:0
Feb 09 03:54:17.515776 lighthouse-mainnet kernel: Out of memory: Killed process 10042 (current-lightho) total-vm:16372344kB, anon-rss:7890740kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25000kB oom_score_adj:0
Feb 09 06:40:40.078556 lighthouse-mainnet kernel: Out of memory: Killed process 11258 (current-lightho) total-vm:25811696kB, anon-rss:7830160kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41244kB oom_score_adj:0
Feb 09 10:56:32.186151 lighthouse-mainnet kernel: Out of memory: Killed process 13924 (current-lightho) total-vm:22461668kB, anon-rss:7813820kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37180kB oom_score_adj:0
Feb 09 17:39:47.980456 lighthouse-mainnet kernel: Out of memory: Killed process 2939 (current-lightho) total-vm:26926152kB, anon-rss:7764100kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44816kB oom_score_adj:0
Feb 09 20:39:13.832390 lighthouse-mainnet kernel: Out of memory: Killed process 23008 (current-lightho) total-vm:23424456kB, anon-rss:7824956kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:38980kB oom_score_adj:0
Feb 09 20:42:18.386825 lighthouse-mainnet kernel: Out of memory: Killed process 28273 (current-lightho) total-vm:13722384kB, anon-rss:7833428kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:20776kB oom_score_adj:0
Feb 10 03:09:27.382634 lighthouse-mainnet kernel: Out of memory: Killed process 28950 (current-lightho) total-vm:21657992kB, anon-rss:7646404kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36680kB oom_score_adj:0
Feb 10 10:26:18.510690 lighthouse-mainnet kernel: Out of memory: Killed process 13516 (current-lightho) total-vm:26227736kB, anon-rss:7653708kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43908kB oom_score_adj:0
Feb 10 10:29:28.007826 lighthouse-mainnet kernel: Out of memory: Killed process 7968 (current-lightho) total-vm:13096364kB, anon-rss:7746328kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:19720kB oom_score_adj:0
Feb 10 10:50:59.851301 lighthouse-mainnet kernel: Out of memory: Killed process 8658 (current-lightho) total-vm:15100864kB, anon-rss:7732644kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22812kB oom_score_adj:0
Feb 10 12:51:42.370932 lighthouse-mainnet kernel: Out of memory: Killed process 10178 (current-lightho) total-vm:20676308kB, anon-rss:7642940kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34676kB oom_score_adj:0
Feb 10 12:54:54.092404 lighthouse-mainnet kernel: Out of memory: Killed process 3320 (current-lightho) total-vm:13158572kB, anon-rss:7655704kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:20676kB oom_score_adj:0
Feb 10 12:58:10.384168 lighthouse-mainnet kernel: Out of memory: Killed process 8867 (geth) total-vm:9924688kB, anon-rss:7619880kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:16052kB oom_score_adj:0
Feb 10 16:53:16.348432 lighthouse-mainnet kernel: Out of memory: Killed process 4006 (current-lightho) total-vm:31119208kB, anon-rss:9693208kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:51444kB oom_score_adj:0
Feb 11 17:33:35.053079 lighthouse-mainnet kernel: Out of memory: Killed process 21687 (current-lightho) total-vm:29761832kB, anon-rss:8918016kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:50332kB oom_score_adj:0
Feb 12 23:44:47.385757 lighthouse-mainnet kernel: Out of memory: Killed process 9230 (current-lightho) total-vm:33797372kB, anon-rss:8642404kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:58380kB oom_score_adj:0
Feb 13 13:24:08.125399 lighthouse-mainnet kernel: Out of memory: Killed process 1529 (current-lightho) total-vm:26827628kB, anon-rss:8486512kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:45400kB oom_score_adj:0
Feb 13 15:12:59.384524 lighthouse-mainnet kernel: Out of memory: Killed process 12504 (current-lightho) total-vm:26210664kB, anon-rss:8532644kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42592kB oom_score_adj:0
Feb 13 17:14:18.039809 lighthouse-mainnet kernel: Out of memory: Killed process 2973 (current-lightho) total-vm:25200536kB, anon-rss:8411604kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42244kB oom_score_adj:0
Feb 13 17:18:53.544211 lighthouse-mainnet kernel: Out of memory: Killed process 28642 (current-lightho) total-vm:20442216kB, anon-rss:8501128kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:30856kB oom_score_adj:0
Feb 13 17:23:24.823285 lighthouse-mainnet kernel: Out of memory: Killed process 29698 (current-lightho) total-vm:19660576kB, anon-rss:8344000kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:29220kB oom_score_adj:0
Feb 13 17:34:26.243643 lighthouse-mainnet kernel: Out of memory: Killed process 30700 (current-lightho) total-vm:25886492kB, anon-rss:8079680kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41712kB oom_score_adj:0
Feb 13 17:43:15.970698 lighthouse-mainnet kernel: Out of memory: Killed process 745 (current-lightho) total-vm:17654680kB, anon-rss:8068092kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:29620kB oom_score_adj:0
Feb 14 02:30:58.101389 lighthouse-mainnet kernel: Out of memory: Killed process 2575 (current-lightho) total-vm:29595344kB, anon-rss:8104084kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:48048kB oom_score_adj:0
Feb 14 02:37:43.284940 lighthouse-mainnet kernel: Out of memory: Killed process 17061 (current-lightho) total-vm:15585148kB, anon-rss:8132840kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22296kB oom_score_adj:0
Feb 14 02:41:07.323661 lighthouse-mainnet kernel: Out of memory: Killed process 18539 (current-lightho) total-vm:13270612kB, anon-rss:8094112kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:20960kB oom_score_adj:0
Feb 14 02:46:45.919990 lighthouse-mainnet kernel: Out of memory: Killed process 19226 (current-lightho) total-vm:17107388kB, anon-rss:7887044kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27844kB oom_score_adj:0
Feb 14 02:49:38.774759 lighthouse-mainnet kernel: Out of memory: Killed process 20453 (current-lightho) total-vm:13726520kB, anon-rss:7794720kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:21352kB oom_score_adj:0
Feb 14 02:54:13.062971 lighthouse-mainnet kernel: Out of memory: Killed process 21085 (current-lightho) total-vm:14820724kB, anon-rss:7786460kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23572kB oom_score_adj:0
Feb 14 06:47:17.489635 lighthouse-mainnet kernel: Out of memory: Killed process 21834 (current-lightho) total-vm:23995600kB, anon-rss:7771956kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:39768kB oom_score_adj:0
Feb 14 06:50:25.762694 lighthouse-mainnet kernel: Out of memory: Killed process 6264 (current-lightho) total-vm:15160344kB, anon-rss:7771400kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22652kB oom_score_adj:0
Feb 14 06:58:15.314460 lighthouse-mainnet kernel: Out of memory: Killed process 6975 (current-lightho) total-vm:20487580kB, anon-rss:7641816kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31620kB oom_score_adj:0
Feb 14 07:05:49.234057 lighthouse-mainnet kernel: Out of memory: Killed process 8623 (current-lightho) total-vm:20855812kB, anon-rss:7788900kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31964kB oom_score_adj:0
Feb 14 07:09:47.937425 lighthouse-mainnet kernel: Out of memory: Killed process 10102 (current-lightho) total-vm:18092636kB, anon-rss:7786284kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27188kB oom_score_adj:0
Feb 14 07:18:32.392538 lighthouse-mainnet kernel: Out of memory: Killed process 4690 (geth) total-vm:10190392kB, anon-rss:7662400kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:16084kB oom_score_adj:0
Feb 14 13:44:55.038149 lighthouse-mainnet kernel: Out of memory: Killed process 11016 (current-lightho) total-vm:31168524kB, anon-rss:9496632kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:53444kB oom_score_adj:0
Feb 16 09:52:52.025826 lighthouse-mainnet kernel: Out of memory: Killed process 28549 (current-lightho) total-vm:35021604kB, anon-rss:8780084kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:56436kB oom_score_adj:0
Feb 16 14:08:34.012660 lighthouse-mainnet kernel: Out of memory: Killed process 3431 (current-lightho) total-vm:24676800kB, anon-rss:8810196kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42032kB oom_score_adj:0
Feb 16 17:39:44.951038 lighthouse-mainnet kernel: Out of memory: Killed process 24831 (current-lightho) total-vm:24336612kB, anon-rss:8722828kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:39916kB oom_score_adj:0
Feb 17 04:11:25.962138 lighthouse-mainnet kernel: Out of memory: Killed process 4494 (current-lightho) total-vm:29225048kB, anon-rss:8646708kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:49752kB oom_score_adj:0
Feb 17 19:03:25.197879 lighthouse-mainnet kernel: Out of memory: Killed process 7885 (current-lightho) total-vm:30197404kB, anon-rss:8617648kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:49272kB oom_score_adj:0
Feb 17 19:10:04.125632 lighthouse-mainnet kernel: Out of memory: Killed process 1787 (current-lightho) total-vm:18962228kB, anon-rss:8603272kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31692kB oom_score_adj:0
Feb 17 23:40:25.747432 lighthouse-mainnet kernel: Out of memory: Killed process 3002 (current-lightho) total-vm:25430064kB, anon-rss:8271820kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42148kB oom_score_adj:0
Feb 17 23:46:43.652241 lighthouse-mainnet kernel: Out of memory: Killed process 26898 (current-lightho) total-vm:19602116kB, anon-rss:8359416kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31260kB oom_score_adj:0
Feb 18 02:56:34.878090 lighthouse-mainnet kernel: Out of memory: Killed process 28266 (current-lightho) total-vm:26317292kB, anon-rss:8272428kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44212kB oom_score_adj:0
Feb 18 02:59:14.439495 lighthouse-mainnet kernel: Out of memory: Killed process 3852 (current-lightho) total-vm:13258688kB, anon-rss:8246928kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:19344kB oom_score_adj:0
Feb 18 03:44:38.178427 lighthouse-mainnet kernel: Out of memory: Killed process 4489 (current-lightho) total-vm:19427168kB, anon-rss:8285596kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:30960kB oom_score_adj:0
Feb 18 11:34:56.737688 lighthouse-mainnet kernel: Out of memory: Killed process 14098 (current-lightho) total-vm:29324772kB, anon-rss:8260724kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:48280kB oom_score_adj:0
Feb 18 11:41:04.161660 lighthouse-mainnet kernel: Out of memory: Killed process 16064 (current-lightho) total-vm:20582044kB, anon-rss:8280788kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31980kB oom_score_adj:0
Feb 18 11:47:10.253071 lighthouse-mainnet kernel: Out of memory: Killed process 17390 (current-lightho) total-vm:19893220kB, anon-rss:8252200kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33576kB oom_score_adj:0
Feb 18 12:50:18.343675 lighthouse-mainnet kernel: Out of memory: Killed process 18655 (current-lightho) total-vm:22067820kB, anon-rss:8057836kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36636kB oom_score_adj:0
Feb 18 12:53:40.126809 lighthouse-mainnet kernel: Out of memory: Killed process 32095 (current-lightho) total-vm:16136852kB, anon-rss:8015308kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24180kB oom_score_adj:0
Feb 18 12:57:38.004617 lighthouse-mainnet kernel: Out of memory: Killed process 320 (current-lightho) total-vm:15584792kB, anon-rss:8061228kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24924kB oom_score_adj:0
Feb 18 13:00:32.675244 lighthouse-mainnet kernel: Out of memory: Killed process 1228 (current-lightho) total-vm:15782992kB, anon-rss:8016060kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23936kB oom_score_adj:0
Feb 18 13:11:18.986342 lighthouse-mainnet kernel: Out of memory: Killed process 1863 (current-lightho) total-vm:18616836kB, anon-rss:7989176kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:30832kB oom_score_adj:0
Feb 18 14:02:11.728314 lighthouse-mainnet kernel: Out of memory: Killed process 4173 (current-lightho) total-vm:28473100kB, anon-rss:8101540kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:46052kB oom_score_adj:0
Feb 18 14:12:51.307578 lighthouse-mainnet kernel: Out of memory: Killed process 14991 (current-lightho) total-vm:19129900kB, anon-rss:8044112kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31632kB oom_score_adj:0
Feb 18 15:39:06.242563 lighthouse-mainnet kernel: Out of memory: Killed process 17259 (current-lightho) total-vm:18821308kB, anon-rss:8072876kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:32120kB oom_score_adj:0
Feb 18 15:42:08.005553 lighthouse-mainnet kernel: Out of memory: Killed process 2941 (current-lightho) total-vm:12825112kB, anon-rss:8076728kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:19548kB oom_score_adj:0
Feb 18 15:47:30.114750 lighthouse-mainnet kernel: Out of memory: Killed process 3623 (current-lightho) total-vm:18467140kB, anon-rss:8101808kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:30420kB oom_score_adj:0
Feb 18 15:53:54.991224 lighthouse-mainnet kernel: Out of memory: Killed process 4832 (current-lightho) total-vm:16604440kB, anon-rss:8123244kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27448kB oom_score_adj:0
Feb 18 15:59:20.834368 lighthouse-mainnet kernel: Out of memory: Killed process 6196 (current-lightho) total-vm:19444704kB, anon-rss:8096756kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:30332kB oom_score_adj:0
Feb 18 16:02:57.369147 lighthouse-mainnet kernel: Out of memory: Killed process 7366 (current-lightho) total-vm:16944844kB, anon-rss:8058044kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27188kB oom_score_adj:0
Feb 18 16:06:49.211359 lighthouse-mainnet kernel: Out of memory: Killed process 8228 (current-lightho) total-vm:13665476kB, anon-rss:8052184kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22176kB oom_score_adj:0
Feb 18 16:16:56.260654 lighthouse-mainnet kernel: Out of memory: Killed process 8965 (current-lightho) total-vm:21931704kB, anon-rss:8099156kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36136kB oom_score_adj:0
Feb 18 16:19:38.303496 lighthouse-mainnet kernel: Out of memory: Killed process 11206 (current-lightho) total-vm:14181224kB, anon-rss:8099724kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22600kB oom_score_adj:0
Feb 18 16:23:52.510397 lighthouse-mainnet kernel: Out of memory: Killed process 11843 (current-lightho) total-vm:17876904kB, anon-rss:8077616kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:28964kB oom_score_adj:0
Feb 18 16:27:15.067364 lighthouse-mainnet kernel: Out of memory: Killed process 12822 (current-lightho) total-vm:15692664kB, anon-rss:8080148kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25508kB oom_score_adj:0
Feb 18 16:32:08.672834 lighthouse-mainnet kernel: Out of memory: Killed process 13585 (current-lightho) total-vm:17234680kB, anon-rss:8053360kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26316kB oom_score_adj:0
Feb 18 16:39:21.792681 lighthouse-mainnet kernel: Out of memory: Killed process 14619 (current-lightho) total-vm:19102012kB, anon-rss:7898292kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:29504kB oom_score_adj:0
Feb 18 16:47:49.002461 lighthouse-mainnet kernel: Out of memory: Killed process 15852 (current-lightho) total-vm:17621328kB, anon-rss:8125588kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27120kB oom_score_adj:0
Feb 18 16:51:12.096064 lighthouse-mainnet kernel: Out of memory: Killed process 17673 (current-lightho) total-vm:15147304kB, anon-rss:8139440kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22280kB oom_score_adj:0
Feb 18 16:57:34.771927 lighthouse-mainnet kernel: Out of memory: Killed process 18406 (current-lightho) total-vm:22244172kB, anon-rss:8091960kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35408kB oom_score_adj:0
Feb 18 17:10:47.128175 lighthouse-mainnet kernel: Out of memory: Killed process 19735 (current-lightho) total-vm:21221388kB, anon-rss:8044724kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35012kB oom_score_adj:0
Feb 18 17:18:07.157514 lighthouse-mainnet kernel: Out of memory: Killed process 22583 (current-lightho) total-vm:20671584kB, anon-rss:8006156kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34776kB oom_score_adj:0
Feb 18 20:00:35.921588 lighthouse-mainnet kernel: Out of memory: Killed process 24155 (current-lightho) total-vm:25488608kB, anon-rss:8174692kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41848kB oom_score_adj:0
Feb 18 20:05:39.291893 lighthouse-mainnet kernel: Out of memory: Killed process 26027 (current-lightho) total-vm:22152816kB, anon-rss:8205068kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34808kB oom_score_adj:0
Feb 18 20:20:25.730383 lighthouse-mainnet kernel: Out of memory: Killed process 27141 (current-lightho) total-vm:22893816kB, anon-rss:8015868kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36232kB oom_score_adj:0
Feb 18 20:29:03.757517 lighthouse-mainnet kernel: Out of memory: Killed process 30170 (current-lightho) total-vm:19596100kB, anon-rss:8029828kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:32488kB oom_score_adj:0
Feb 18 21:10:40.321746 lighthouse-mainnet kernel: Out of memory: Killed process 31958 (current-lightho) total-vm:19685104kB, anon-rss:8063224kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33276kB oom_score_adj:0
Feb 18 21:43:00.580860 lighthouse-mainnet kernel: Out of memory: Killed process 8288 (current-lightho) total-vm:23399948kB, anon-rss:8009500kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:38596kB oom_score_adj:0
Feb 18 21:52:55.808520 lighthouse-mainnet kernel: Out of memory: Killed process 15201 (current-lightho) total-vm:21777068kB, anon-rss:7987056kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36700kB oom_score_adj:0
Feb 18 22:02:26.837884 lighthouse-mainnet kernel: Out of memory: Killed process 17380 (current-lightho) total-vm:15467240kB, anon-rss:8009064kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25060kB oom_score_adj:0
Feb 19 00:23:00.367947 lighthouse-mainnet kernel: Out of memory: Killed process 19465 (current-lightho) total-vm:26159124kB, anon-rss:8012256kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43400kB oom_score_adj:0
Feb 19 00:26:20.304958 lighthouse-mainnet kernel: Out of memory: Killed process 16718 (current-lightho) total-vm:13827388kB, anon-rss:8098840kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:20916kB oom_score_adj:0
Feb 19 00:29:43.189647 lighthouse-mainnet kernel: Out of memory: Killed process 17366 (current-lightho) total-vm:13242748kB, anon-rss:8172504kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:21756kB oom_score_adj:0
Feb 19 03:47:54.803324 lighthouse-mainnet kernel: Out of memory: Killed process 18152 (current-lightho) total-vm:21088668kB, anon-rss:7996524kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34940kB oom_score_adj:0
Feb 19 04:13:25.095041 lighthouse-mainnet kernel: Out of memory: Killed process 27327 (current-lightho) total-vm:21107764kB, anon-rss:8041068kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33856kB oom_score_adj:0
Feb 19 04:16:21.624863 lighthouse-mainnet kernel: Out of memory: Killed process 32739 (current-lightho) total-vm:14030824kB, anon-rss:8027652kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:20960kB oom_score_adj:0
Feb 19 04:21:08.321792 lighthouse-mainnet kernel: Out of memory: Killed process 976 (current-lightho) total-vm:17503460kB, anon-rss:8060420kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27880kB oom_score_adj:0
Feb 19 05:55:52.024605 lighthouse-mainnet kernel: Out of memory: Killed process 1975 (current-lightho) total-vm:24768016kB, anon-rss:8028840kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41836kB oom_score_adj:0
Feb 19 06:00:33.937991 lighthouse-mainnet kernel: Out of memory: Killed process 22469 (current-lightho) total-vm:18698828kB, anon-rss:8051484kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:30836kB oom_score_adj:0
Feb 19 06:04:36.908755 lighthouse-mainnet kernel: Out of memory: Killed process 23529 (current-lightho) total-vm:16257996kB, anon-rss:7981088kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25092kB oom_score_adj:0
Feb 19 06:09:01.983427 lighthouse-mainnet kernel: Out of memory: Killed process 24431 (current-lightho) total-vm:16587328kB, anon-rss:7890268kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26324kB oom_score_adj:0
Feb 19 06:17:34.560029 lighthouse-mainnet kernel: Out of memory: Killed process 25394 (current-lightho) total-vm:19516604kB, anon-rss:7946732kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:30936kB oom_score_adj:0
Feb 19 07:06:10.031083 lighthouse-mainnet kernel: Out of memory: Killed process 27239 (current-lightho) total-vm:24800812kB, anon-rss:7911232kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40540kB oom_score_adj:0
Feb 19 07:11:21.849100 lighthouse-mainnet kernel: Out of memory: Killed process 5407 (current-lightho) total-vm:15433056kB, anon-rss:7842232kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24360kB oom_score_adj:0
Feb 19 07:19:07.874928 lighthouse-mainnet kernel: Out of memory: Killed process 6512 (current-lightho) total-vm:21653732kB, anon-rss:7984256kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34064kB oom_score_adj:0
Feb 19 09:46:21.757874 lighthouse-mainnet kernel: Out of memory: Killed process 8213 (current-lightho) total-vm:22166296kB, anon-rss:8054560kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35552kB oom_score_adj:0
Feb 19 10:24:37.776294 lighthouse-mainnet kernel: Out of memory: Killed process 6912 (current-lightho) total-vm:25527596kB, anon-rss:8012880kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42252kB oom_score_adj:0
Feb 19 11:47:31.831965 lighthouse-mainnet kernel: Out of memory: Killed process 15073 (current-lightho) total-vm:22476412kB, anon-rss:7980824kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36504kB oom_score_adj:0
Feb 19 13:03:46.087963 lighthouse-mainnet kernel: Out of memory: Killed process 12567 (geth) total-vm:9652832kB, anon-rss:7377296kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:15532kB oom_score_adj:0
Feb 20 09:20:44.501902 lighthouse-mainnet kernel: Out of memory: Killed process 32637 (current-lightho) total-vm:30430108kB, anon-rss:9067768kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:50884kB oom_score_adj:0
Feb 20 10:18:17.089065 lighthouse-mainnet kernel: Out of memory: Killed process 13651 (current-lightho) total-vm:24810780kB, anon-rss:9094088kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42916kB oom_score_adj:0
Feb 20 13:27:02.933878 lighthouse-mainnet kernel: Out of memory: Killed process 25829 (current-lightho) total-vm:31211512kB, anon-rss:8940432kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:53020kB oom_score_adj:0
Feb 21 01:59:26.856154 lighthouse-mainnet kernel: Out of memory: Killed process 830 (current-lightho) total-vm:29681780kB, anon-rss:8733340kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:49396kB oom_score_adj:0
Feb 21 02:07:23.435208 lighthouse-mainnet kernel: Out of memory: Killed process 29376 (current-lightho) total-vm:21156528kB, anon-rss:8765152kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33668kB oom_score_adj:0
Feb 21 04:20:02.698654 lighthouse-mainnet kernel: Out of memory: Killed process 31073 (current-lightho) total-vm:26287044kB, anon-rss:8633696kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42032kB oom_score_adj:0
Feb 21 04:24:36.532449 lighthouse-mainnet kernel: Out of memory: Killed process 26536 (current-lightho) total-vm:16732792kB, anon-rss:8603208kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26112kB oom_score_adj:0
Feb 21 09:07:47.113629 lighthouse-mainnet kernel: Out of memory: Killed process 27536 (current-lightho) total-vm:27495452kB, anon-rss:8549816kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:47288kB oom_score_adj:0
Feb 21 09:11:33.182717 lighthouse-mainnet kernel: Out of memory: Killed process 22431 (current-lightho) total-vm:15121316kB, anon-rss:8455516kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23068kB oom_score_adj:0
Feb 21 09:18:53.877959 lighthouse-mainnet kernel: Out of memory: Killed process 23275 (current-lightho) total-vm:19350648kB, anon-rss:8504508kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31316kB oom_score_adj:0
Feb 21 15:06:09.803462 lighthouse-mainnet kernel: Out of memory: Killed process 24755 (current-lightho) total-vm:27754968kB, anon-rss:8439964kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:46912kB oom_score_adj:0
Feb 22 02:50:10.623605 lighthouse-mainnet kernel: Out of memory: Killed process 991 (current-lightho) total-vm:28966396kB, anon-rss:8445940kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:49120kB oom_score_adj:0
Feb 22 03:29:59.746980 lighthouse-mainnet kernel: Out of memory: Killed process 19370 (current-lightho) total-vm:21869212kB, anon-rss:8538576kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34704kB oom_score_adj:0
Feb 22 10:31:00.539622 lighthouse-mainnet kernel: Out of memory: Killed process 27836 (current-lightho) total-vm:31435908kB, anon-rss:8503864kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:51996kB oom_score_adj:0
Feb 22 10:37:06.731705 lighthouse-mainnet kernel: Out of memory: Killed process 19957 (current-lightho) total-vm:20175396kB, anon-rss:8553632kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33088kB oom_score_adj:0
Feb 22 10:44:18.803799 lighthouse-mainnet kernel: Out of memory: Killed process 21290 (current-lightho) total-vm:19592076kB, anon-rss:8577740kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31532kB oom_score_adj:0
Feb 22 10:49:02.498521 lighthouse-mainnet kernel: Out of memory: Killed process 22871 (current-lightho) total-vm:16363600kB, anon-rss:8479044kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26740kB oom_score_adj:0
Feb 22 12:07:05.663206 lighthouse-mainnet kernel: Out of memory: Killed process 23915 (current-lightho) total-vm:22145720kB, anon-rss:8519652kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36060kB oom_score_adj:0
Feb 22 12:41:54.072584 lighthouse-mainnet kernel: Out of memory: Killed process 7852 (current-lightho) total-vm:21257864kB, anon-rss:8465072kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33476kB oom_score_adj:0
Feb 22 12:54:29.852180 lighthouse-mainnet kernel: Out of memory: Killed process 10633 (current-lightho) total-vm:24021844kB, anon-rss:8606760kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:38956kB oom_score_adj:0
Feb 22 13:00:08.110041 lighthouse-mainnet kernel: Out of memory: Killed process 13518 (current-lightho) total-vm:20712352kB, anon-rss:8626168kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33008kB oom_score_adj:0
Feb 23 09:42:41.854665 lighthouse-mainnet kernel: Out of memory: Killed process 14725 (current-lightho) total-vm:27611156kB, anon-rss:8744484kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:45472kB oom_score_adj:0
$ journalctl -o short-precise -k -b -2 | head -1
-- Logs begin at Wed 2023-12-13 04:27:02 UTC, end at Tue 2024-02-27 17:15:13 UTC. --
$ journalctl -o short-precise -k -b -2 | grep Killed
$ journalctl -o short-precise -k -b -1 | head -1
-- Logs begin at Wed 2023-12-13 04:27:02 UTC, end at Tue 2024-02-27 17:15:41 UTC. --
$ journalctl -o short-precise -k -b -1 | grep Killed
Feb 23 23:06:06.748577 lighthouse-mainnet kernel: Out of memory: Killed process 805 (current-lightho) total-vm:22455880kB, anon-rss:8280768kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35272kB oom_score_adj:0
$ journalctl -o short-precise -k -b -0 | head -1
-- Logs begin at Wed 2023-12-13 04:27:02 UTC, end at Tue 2024-02-27 17:16:17 UTC. --
$ journalctl -o short-precise -k -b -0 | grep Killed
$ This is what the last event looks like:
Feb 23 23:06:06.669413 lighthouse-mainnet kernel: tokio-runtime-w invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Feb 23 23:06:06.713910 lighthouse-mainnet kernel: CPU: 1 PID: 1323 Comm: tokio-runtime-w Not tainted 5.4.0-1103-aws #111~18.04.1-Ubuntu
Feb 23 23:06:06.714946 lighthouse-mainnet kernel: Hardware name: Amazon EC2 m5.xlarge/, BIOS 1.0 10/16/2017
Feb 23 23:06:06.714993 lighthouse-mainnet kernel: Call Trace:
Feb 23 23:06:06.715024 lighthouse-mainnet kernel: dump_stack+0x6d/0x8b
Feb 23 23:06:06.715919 lighthouse-mainnet kernel: dump_header+0x4f/0x200
Feb 23 23:06:06.717302 lighthouse-mainnet kernel: oom_kill_process+0xec/0x140
Feb 23 23:06:06.718712 lighthouse-mainnet kernel: out_of_memory+0x117/0x570
Feb 23 23:06:06.718748 lighthouse-mainnet kernel: __alloc_pages_slowpath+0xada/0xec0
Feb 23 23:06:06.719909 lighthouse-mainnet kernel: __alloc_pages_nodemask+0x2cd/0x320
Feb 23 23:06:06.720685 lighthouse-mainnet kernel: alloc_pages_vma+0x88/0x230
Feb 23 23:06:06.721687 lighthouse-mainnet kernel: __handle_mm_fault+0x87e/0x1240
Feb 23 23:06:06.723651 lighthouse-mainnet kernel: handle_mm_fault+0xcb/0x210
Feb 23 23:06:06.725606 lighthouse-mainnet kernel: __do_page_fault+0x2a1/0x4d0
Feb 23 23:06:06.727345 lighthouse-mainnet kernel: do_page_fault+0x2c/0xe0
Feb 23 23:06:06.728337 lighthouse-mainnet kernel: do_async_page_fault+0x54/0x70
Feb 23 23:06:06.729596 lighthouse-mainnet kernel: async_page_fault+0x34/0x40
Feb 23 23:06:06.729630 lighthouse-mainnet kernel: RIP: 0033:0x7f8f2fd64ca0
Feb 23 23:06:06.729651 lighthouse-mainnet kernel: Code: fe 6f 4e 20 c5 fe 6f 56 40 c5 fe 6f 5e 60 48 81 c6 80 00 00 00 48 81 ea 80 00 00 00 c5 fd e7 07 c5 fd e7 4f 20 c5 fd e7 57 40 <c5> fd e7 5f 60 48 81 c7 80 00 00 00 48 81 fa 80 00 00 00 77 a0 0f
Feb 23 23:06:06.729674 lighthouse-mainnet kernel: RSP: 002b:00007f8f07ffd6d8 EFLAGS: 00010202
Feb 23 23:06:06.730963 lighthouse-mainnet kernel: RAX: 00007f89cc847010 RBX: 00007f8ef80116c0 RCX: 00007f89db7c8e57
Feb 23 23:06:06.730995 lighthouse-mainnet kernel: RDX: 0000000004b27e57 RSI: 00007f89f4ba5020 RDI: 00007f89d6ca0fa0
Feb 23 23:06:06.731019 lighthouse-mainnet kernel: RBP: 000000000ef81e67 R08: fffffffffffffff0 R09: 0000000000000000
Feb 23 23:06:06.731040 lighthouse-mainnet kernel: R10: 00007f89db7c8e77 R11: 00007f89cc847010 R12: 00007f8ef80116d0
Feb 23 23:06:06.732457 lighthouse-mainnet kernel: R13: 0000000000000000 R14: 0000000000000004 R15: 00007f89cc847010
Feb 23 23:06:06.732495 lighthouse-mainnet kernel: Mem-Info:
Feb 23 23:06:06.732515 lighthouse-mainnet kernel: active_anon:3851622 inactive_anon:41 isolated_anon:0
active_file:330 inactive_file:282 isolated_file:0
unevictable:0 dirty:5 writeback:0 unstable:0
slab_reclaimable:51405 slab_unreclaimable:16486
mapped:124 shmem:192 pagetables:14787 bounce:0
free:33160 free_pcp:422 free_cma:0
Feb 23 23:06:06.732534 lighthouse-mainnet kernel: Node 0 active_anon:15406488kB inactive_anon:164kB active_file:1320kB inactive_file:1128kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:496kB dirty:20kB writeback:0kB shmem:768kB shmem_thp:
Feb 23 23:06:06.732618 lighthouse-mainnet kernel: Node 0 DMA free:15908kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_s
Feb 23 23:06:06.732639 lighthouse-mainnet kernel: lowmem_reserve[]: 0 2971 15501 15501 15501
Feb 23 23:06:06.733491 lighthouse-mainnet kernel: Node 0 DMA32 free:62648kB min:12940kB low:16172kB high:19404kB active_anon:2968068kB inactive_anon:0kB active_file:200kB inactive_file:520kB unevictable:0kB writepending:0kB present:3129256kB managed:30637
Feb 23 23:06:06.735013 lighthouse-mainnet kernel: lowmem_reserve[]: 0 0 12530 12530 12530
Feb 23 23:06:06.735056 lighthouse-mainnet kernel: Node 0 Normal free:54084kB min:54572kB low:68212kB high:81852kB active_anon:12437496kB inactive_anon:164kB active_file:1136kB inactive_file:972kB unevictable:0kB writepending:0kB present:13150208kB managed
Feb 23 23:06:06.735081 lighthouse-mainnet kernel: lowmem_reserve[]: 0 0 0 0 0
Feb 23 23:06:06.735841 lighthouse-mainnet kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Feb 23 23:06:06.735874 lighthouse-mainnet kernel: Node 0 DMA32: 646*4kB (UME) 536*8kB (UE) 1037*16kB (UME) 489*32kB (UME) 201*64kB (UME) 49*128kB (UE) 10*256kB (UE) 4*512kB (ME) 0*1024kB 0*2048kB 0*4096kB = 62856kB
Feb 23 23:06:06.735904 lighthouse-mainnet kernel: Node 0 Normal: 50*4kB (UH) 2081*8kB (UEH) 1334*16kB (UEH) 477*32kB (UEH) 1*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 53520kB
Feb 23 23:06:06.736675 lighthouse-mainnet kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 23 23:06:06.736721 lighthouse-mainnet kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 23 23:06:06.737672 lighthouse-mainnet kernel: 809 total pagecache pages
Feb 23 23:06:06.737700 lighthouse-mainnet kernel: 0 pages in swap cache
Feb 23 23:06:06.737726 lighthouse-mainnet kernel: Swap cache stats: add 0, delete 0, find 0/0
Feb 23 23:06:06.738529 lighthouse-mainnet kernel: Free swap = 0kB
Feb 23 23:06:06.738559 lighthouse-mainnet kernel: Total swap = 0kB
Feb 23 23:06:06.739155 lighthouse-mainnet kernel: 4073864 pages RAM
Feb 23 23:06:06.739186 lighthouse-mainnet kernel: 0 pages HighMem/MovableOnly
Feb 23 23:06:06.740047 lighthouse-mainnet kernel: 94145 pages reserved
Feb 23 23:06:06.740084 lighthouse-mainnet kernel: 0 pages cma reserved
Feb 23 23:06:06.740111 lighthouse-mainnet kernel: 0 pages hwpoisoned
Feb 23 23:06:06.741187 lighthouse-mainnet kernel: Tasks state (memory values in pages):
Feb 23 23:06:06.741216 lighthouse-mainnet kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Feb 23 23:06:06.741232 lighthouse-mainnet kernel: [ 431] 0 431 35412 166 270336 0 0 systemd-journal
Feb 23 23:06:06.741803 lighthouse-mainnet kernel: [ 442] 0 442 24428 45 98304 0 0 lvmetad
Feb 23 23:06:06.742636 lighthouse-mainnet kernel: [ 453] 0 453 10795 240 114688 0 -1000 systemd-udevd
Feb 23 23:06:06.742675 lighthouse-mainnet kernel: [ 501] 62583 501 35447 128 180224 0 0 systemd-timesyn
Feb 23 23:06:06.743402 lighthouse-mainnet kernel: [ 660] 100 660 20013 164 176128 0 0 systemd-network
Feb 23 23:06:06.743439 lighthouse-mainnet kernel: [ 683] 101 683 17656 151 184320 0 0 systemd-resolve
Feb 23 23:06:06.743463 lighthouse-mainnet kernel: [ 798] 0 798 1140 16 57344 0 0 acpid
Feb 23 23:06:06.743493 lighthouse-mainnet kernel: [ 805] 1001 805 5613970 2070192 36118528 0 0 current-lightho
Feb 23 23:06:06.743543 lighthouse-mainnet kernel: [ 811] 0 811 27622 79 114688 0 0 irqbalance
Feb 23 23:06:06.744895 lighthouse-mainnet kernel: [ 814] 0 814 42814 2038 225280 0 0 networkd-dispat
Feb 23 23:06:06.745421 lighthouse-mainnet kernel: [ 815] 0 815 40271 33 90112 0 0 lxcfs
Feb 23 23:06:06.746191 lighthouse-mainnet kernel: [ 819] 0 819 7938 74 106496 0 0 cron
Feb 23 23:06:06.746225 lighthouse-mainnet kernel: [ 822] 103 822 12527 170 139264 0 -900 dbus-daemon
Feb 23 23:06:06.747078 lighthouse-mainnet kernel: [ 832] 0 832 328821 1967 192512 0 0 amazon-ssm-agen
Feb 23 23:06:06.747754 lighthouse-mainnet kernel: [ 840] 0 840 7084 52 102400 0 0 atd
Feb 23 23:06:06.747805 lighthouse-mainnet kernel: [ 844] 0 844 17611 154 172032 0 0 systemd-logind
Feb 23 23:06:06.747827 lighthouse-mainnet kernel: [ 848] 102 848 65762 258 167936 0 0 rsyslogd
Feb 23 23:06:06.747871 lighthouse-mainnet kernel: [ 854] 1001 854 3251274 1728238 19312640 0 0 geth
Feb 23 23:06:06.747897 lighthouse-mainnet kernel: [ 855] 1001 855 359922 4117 1003520 0 0 current-lightho
Feb 23 23:06:06.747918 lighthouse-mainnet kernel: [ 857] 111 857 631145 21246 876544 0 0 agent
Feb 23 23:06:06.747942 lighthouse-mainnet kernel: [ 861] 111 861 492766 6771 479232 0 0 process-agent
Feb 23 23:06:06.747968 lighthouse-mainnet kernel: [ 862] 111 862 498409 5442 409600 0 0 trace-agent
Feb 23 23:06:06.747990 lighthouse-mainnet kernel: [ 863] 0 863 369757 2257 270336 0 -900 snapd
Feb 23 23:06:06.748012 lighthouse-mainnet kernel: [ 865] 0 865 71999 222 196608 0 0 accounts-daemon
Feb 23 23:06:06.748034 lighthouse-mainnet kernel: [ 873] 0 873 18076 182 180224 0 -1000 sshd
Feb 23 23:06:06.748056 lighthouse-mainnet kernel: [ 883] 0 883 46923 1975 258048 0 0 unattended-upgr
Feb 23 23:06:06.748471 lighthouse-mainnet kernel: [ 903] 0 903 4105 37 73728 0 0 agetty
Feb 23 23:06:06.748500 lighthouse-mainnet kernel: [ 910] 0 910 3724 32 69632 0 0 agetty
Feb 23 23:06:06.748528 lighthouse-mainnet kernel: [ 925] 0 925 72864 266 208896 0 0 polkitd
Feb 23 23:06:06.748549 lighthouse-mainnet kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/lighthouse-beacon.service,task=current-lightho,pid=805,uid=1001
Feb 23 23:06:06.748577 lighthouse-mainnet kernel: Out of memory: Killed process 805 (current-lightho) total-vm:22455880kB, anon-rss:8280768kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35272kB oom_score_adj:0@vogelito I think https://github.com/sigp/lighthouse/pull/5270 should help quite a bit. Lighthouse is not using particularly excessive amounts of memory (which would be indicative of a bug). All your OOM logs show it gettting killed at around 8GB:
Feb 23 09:42:41.854665 lighthouse-mainnet kernel: Out of memory: Killed process 14725 (current-lightho) total-vm:27611156kB, anon-rss:8744484kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:45472kB oom_score_adj:0
(anon-rss is the relevant value on each line)
On our infrastructure Lighthouse uses 4-8GB regularly, so I expect the spikes to 8GB were too much for your node with the 16GB hard limit. Geth is using the rest of the memory by the looks of things, its RSS shown in the kernel backtrace is 1728238 pages * 4096 bytes/page = 7.07 GB. Maybe once v5.1.0 is out you can experiment with going back to 16GB.
vogelito
commented
7 months ago Sounds like a plan to me....
vogelito
commented
7 months ago Upgraded to Lighthouse/v5.1.0-10a38a8 and have gone back to 16GB.
Mar 11 15:45:15.687 INFO Lighthouse started version: Lighthouse/v5.1.0-10a38a8
Will report back!
vogelito
commented
7 months ago It's already been killed:
Mar 12 22:21:21.515047 lighthouse-mainnet kernel: Out of memory: Killed process 876 (current-lightho) total-vm:35104264kB, anon-rss:9031688kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:57820kB oom_score_adj:0Seems like Geth was using ~6.38GB:
Mar 12 22:21:21.116911 lighthouse-mainnet kernel: tokio-runtime-w invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Mar 12 22:21:21.432808 lighthouse-mainnet kernel: CPU: 3 PID: 956 Comm: tokio-runtime-w Not tainted 5.4.0-1103-aws #111~18.04.1-Ubuntu
Mar 12 22:21:21.440793 lighthouse-mainnet kernel: Hardware name: Amazon EC2 m5.xlarge/, BIOS 1.0 10/16/2017
Mar 12 22:21:21.440837 lighthouse-mainnet kernel: Call Trace:
Mar 12 22:21:21.442894 lighthouse-mainnet kernel: dump_stack+0x6d/0x8b
Mar 12 22:21:21.447463 lighthouse-mainnet kernel: dump_header+0x4f/0x200
Mar 12 22:21:21.447491 lighthouse-mainnet kernel: oom_kill_process+0xec/0x140
Mar 12 22:21:21.447514 lighthouse-mainnet kernel: out_of_memory+0x117/0x570
Mar 12 22:21:21.447530 lighthouse-mainnet kernel: __alloc_pages_slowpath+0xada/0xec0
Mar 12 22:21:21.450248 lighthouse-mainnet kernel: __alloc_pages_nodemask+0x2cd/0x320
Mar 12 22:21:21.454392 lighthouse-mainnet kernel: alloc_pages_current+0x6a/0xe0
Mar 12 22:21:21.454428 lighthouse-mainnet kernel: __page_cache_alloc+0x6a/0xa0
Mar 12 22:21:21.454445 lighthouse-mainnet kernel: pagecache_get_page+0xab/0x2c0
Mar 12 22:21:21.454465 lighthouse-mainnet kernel: filemap_fault+0x685/0xb80
Mar 12 22:21:21.456044 lighthouse-mainnet kernel: ? unlock_page_memcg+0x12/0x20
Mar 12 22:21:21.460499 lighthouse-mainnet kernel: ? page_add_file_rmap+0x13a/0x180
Mar 12 22:21:21.462288 lighthouse-mainnet kernel: ? ttwu_do_wakeup+0x1e/0x150
Mar 12 22:21:21.464224 lighthouse-mainnet kernel: ? xas_load+0xc/0x80
Mar 12 22:21:21.464256 lighthouse-mainnet kernel: ? xas_find+0x16f/0x1b0
Mar 12 22:21:21.464274 lighthouse-mainnet kernel: ? filemap_map_pages+0x181/0x3b0
Mar 12 22:21:21.464303 lighthouse-mainnet kernel: ext4_filemap_fault+0x31/0x50
Mar 12 22:21:21.464323 lighthouse-mainnet kernel: __do_fault+0x57/0x158
Mar 12 22:21:21.464342 lighthouse-mainnet kernel: __handle_mm_fault+0xdae/0x1240
Mar 12 22:21:21.467120 lighthouse-mainnet kernel: handle_mm_fault+0xcb/0x210
Mar 12 22:21:21.467159 lighthouse-mainnet kernel: __do_page_fault+0x2a1/0x4d0
Mar 12 22:21:21.467176 lighthouse-mainnet kernel: do_page_fault+0x2c/0xe0
Mar 12 22:21:21.470459 lighthouse-mainnet kernel: do_async_page_fault+0x54/0x70
Mar 12 22:21:21.470487 lighthouse-mainnet kernel: async_page_fault+0x34/0x40
Mar 12 22:21:21.472508 lighthouse-mainnet kernel: RIP: 0033:0x555e4925e160
Mar 12 22:21:21.472537 lighthouse-mainnet kernel: Code: Bad RIP value.
Mar 12 22:21:21.472552 lighthouse-mainnet kernel: RSP: 002b:00007f0b435f0e48 EFLAGS: 00010206
Mar 12 22:21:21.475406 lighthouse-mainnet kernel: RAX: 0000000000000940 RBX: 00007f0b435f0f70 RCX: 0000000000000000
Mar 12 22:21:21.475430 lighthouse-mainnet kernel: RDX: 00007f0b4071c080 RSI: 00007f0b4071c1a0 RDI: 00007f0b435f0e60
Mar 12 22:21:21.475447 lighthouse-mainnet kernel: RBP: 0000000000000003 R08: 00000000031a881a R09: 0000000000000000
Mar 12 22:21:21.475466 lighthouse-mainnet kernel: R10: 000000000001ae61 R11: 0000000000000202 R12: 00007f0b4071c1a0
Mar 12 22:21:21.477859 lighthouse-mainnet kernel: R13: 00007f0b4071c080 R14: 00007f0b4071c000 R15: 00007f0b435f4da0
Mar 12 22:21:21.480482 lighthouse-mainnet kernel: Mem-Info:
Mar 12 22:21:21.480523 lighthouse-mainnet kernel: active_anon:3869200 inactive_anon:41 isolated_anon:0
active_file:233 inactive_file:212 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:71313 slab_unreclaimable:17262
mapped:0 shmem:192 pagetables:19304 bounce:0
free:33300 free_pcp:666 free_cma:0
Mar 12 22:21:21.480558 lighthouse-mainnet kernel: Node 0 active_anon:15476800kB inactive_anon:164kB active_file:932kB inactive_file:848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:768kB shmem_thp: 0kB s
Mar 12 22:21:21.480585 lighthouse-mainnet kernel: Node 0 DMA free:15908kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_s
Mar 12 22:21:21.482561 lighthouse-mainnet kernel: lowmem_reserve[]: 0 2971 15669 15669 15669
Mar 12 22:21:21.482593 lighthouse-mainnet kernel: Node 0 DMA32 free:63112kB min:12800kB low:16000kB high:19200kB active_anon:2943248kB inactive_anon:0kB active_file:0kB inactive_file:268kB unevictable:0kB writepending:0kB present:3129256kB managed:3063720
Mar 12 22:21:21.482615 lighthouse-mainnet kernel: lowmem_reserve[]: 0 0 12698 12698 12698
Mar 12 22:21:21.485742 lighthouse-mainnet kernel: Node 0 Normal free:54180kB min:54712kB low:68388kB high:82064kB active_anon:12533552kB inactive_anon:164kB active_file:640kB inactive_file:772kB unevictable:0kB writepending:0kB present:13324288kB managed:
Mar 12 22:21:21.485780 lighthouse-mainnet kernel: lowmem_reserve[]: 0 0 0 0 0
Mar 12 22:21:21.485805 lighthouse-mainnet kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Mar 12 22:21:21.489267 lighthouse-mainnet kernel: Node 0 DMA32: 1263*4kB (UME) 756*8kB (UME) 976*16kB (UME) 553*32kB (UME) 208*64kB (UME) 44*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 63356kB
Mar 12 22:21:21.489302 lighthouse-mainnet kernel: Node 0 Normal: 164*4kB (UEH) 1103*8kB (UMH) 1352*16kB (UEH) 713*32kB (UMEH) 7*64kB (H) 1*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54504kB
Mar 12 22:21:21.489323 lighthouse-mainnet kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 12 22:21:21.492523 lighthouse-mainnet kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 12 22:21:21.495569 lighthouse-mainnet kernel: 632 total pagecache pages
Mar 12 22:21:21.495598 lighthouse-mainnet kernel: 0 pages in swap cache
Mar 12 22:21:21.500788 lighthouse-mainnet kernel: Swap cache stats: add 0, delete 0, find 0/0
Mar 12 22:21:21.500822 lighthouse-mainnet kernel: Free swap = 0kB
Mar 12 22:21:21.504089 lighthouse-mainnet kernel: Total swap = 0kB
Mar 12 22:21:21.505795 lighthouse-mainnet kernel: 4117384 pages RAM
Mar 12 22:21:21.505824 lighthouse-mainnet kernel: 0 pages HighMem/MovableOnly
Mar 12 22:21:21.505846 lighthouse-mainnet kernel: 94657 pages reserved
Mar 12 22:21:21.505861 lighthouse-mainnet kernel: 0 pages cma reserved
Mar 12 22:21:21.507479 lighthouse-mainnet kernel: 0 pages hwpoisoned
Mar 12 22:21:21.507500 lighthouse-mainnet kernel: Tasks state (memory values in pages):
Mar 12 22:21:21.507519 lighthouse-mainnet kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mar 12 22:21:21.507534 lighthouse-mainnet kernel: [ 435] 0 435 81478 181 638976 0 0 systemd-journal
Mar 12 22:21:21.507554 lighthouse-mainnet kernel: [ 443] 0 443 24428 44 98304 0 0 lvmetad
Mar 12 22:21:21.507647 lighthouse-mainnet kernel: [ 450] 0 450 10795 266 122880 0 -1000 systemd-udevd
Mar 12 22:21:21.507664 lighthouse-mainnet kernel: [ 504] 62583 504 35447 127 184320 0 0 systemd-timesyn
Mar 12 22:21:21.507681 lighthouse-mainnet kernel: [ 681] 100 681 20013 164 180224 0 0 systemd-network
Mar 12 22:21:21.507701 lighthouse-mainnet kernel: [ 700] 101 700 17655 149 176128 0 0 systemd-resolve
Mar 12 22:21:21.507718 lighthouse-mainnet kernel: [ 819] 102 819 65762 324 155648 0 0 rsyslogd
Mar 12 22:21:21.507733 lighthouse-mainnet kernel: [ 823] 0 823 7084 51 94208 0 0 atd
Mar 12 22:21:21.507749 lighthouse-mainnet kernel: [ 829] 0 829 476450 2066 217088 0 0 amazon-ssm-agen
Mar 12 22:21:21.507765 lighthouse-mainnet kernel: [ 832] 0 832 1140 15 57344 0 0 acpid
Mar 12 22:21:21.507786 lighthouse-mainnet kernel: [ 834] 103 834 12515 174 143360 0 -900 dbus-daemon
Mar 12 22:21:21.507808 lighthouse-mainnet kernel: [ 860] 0 860 71999 215 196608 0 0 accounts-daemon
Mar 12 22:21:21.507825 lighthouse-mainnet kernel: [ 862] 0 862 370878 2700 299008 0 -900 snapd
Mar 12 22:21:21.507842 lighthouse-mainnet kernel: [ 869] 1001 869 360373 4377 1011712 0 0 current-lightho
Mar 12 22:21:21.507861 lighthouse-mainnet kernel: [ 875] 0 875 58738 72 90112 0 0 lxcfs
Mar 12 22:21:21.507878 lighthouse-mainnet kernel: [ 876] 1001 876 8776066 2257922 59207680 0 0 current-lightho
Mar 12 22:21:21.507892 lighthouse-mainnet kernel: [ 881] 0 881 42814 2037 237568 0 0 networkd-dispat
Mar 12 22:21:21.507905 lighthouse-mainnet kernel: [ 883] 1001 883 2259399 1558386 13746176 0 0 geth
Mar 12 22:21:21.507921 lighthouse-mainnet kernel: [ 884] 0 884 17611 155 172032 0 0 systemd-logind
Mar 12 22:21:21.507940 lighthouse-mainnet kernel: [ 885] 0 885 7938 76 102400 0 0 cron
Mar 12 22:21:21.507959 lighthouse-mainnet kernel: [ 886] 111 886 658078 20371 884736 0 0 agent
Mar 12 22:21:21.507973 lighthouse-mainnet kernel: [ 888] 111 888 498345 5598 405504 0 0 trace-agent
Mar 12 22:21:21.507989 lighthouse-mainnet kernel: [ 889] 111 889 511071 7133 483328 0 0 process-agent
Mar 12 22:21:21.511197 lighthouse-mainnet kernel: [ 890] 0 890 27622 79 114688 0 0 irqbalance
Mar 12 22:21:21.511229 lighthouse-mainnet kernel: [ 904] 0 904 4105 37 73728 0 0 agetty
Mar 12 22:21:21.511250 lighthouse-mainnet kernel: [ 907] 0 907 46923 1976 266240 0 0 unattended-upgr
Mar 12 22:21:21.511268 lighthouse-mainnet kernel: [ 909] 0 909 3724 32 73728 0 0 agetty
Mar 12 22:21:21.511288 lighthouse-mainnet kernel: [ 911] 0 911 72866 266 200704 0 0 polkitd
Mar 12 22:21:21.514995 lighthouse-mainnet kernel: [ 920] 0 920 18076 183 184320 0 -1000 sshd
Mar 12 22:21:21.515027 lighthouse-mainnet kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/lighthouse-beacon.service,task=current-lightho,pid=876,uid=1001
Mar 12 22:21:21.515047 lighthouse-mainnet kernel: Out of memory: Killed process 876 (current-lightho) total-vm:35104264kB, anon-rss:9031688kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:57820kB oom_score_adj:0Let me know if you need anything else on my end to be helpful in your debugging.
vogelito
commented
7 months ago Another crash:
Mar 14 14:46:57.148134 lighthouse-mainnet kernel: Out of memory: Killed process 301 (current-lightho) total-vm:35219888kB, anon-rss:8467008kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:59344kB oom_score_adj:0@vogelito Please go back to 32GB or try --state-cache-size 2 until we have a chance to ship tree-states
michaelsproul
commented
7 months ago If you can DM me debug logs again too, we can take a look. Lighthouse shouldn't really be hitting 9GB.
vogelito
commented
7 months ago Looking progressively worse....
Mar 12 22:21:21.515047 lighthouse-mainnet kernel: Out of memory: Killed process 876 (current-lightho) total-vm:35104264kB, anon-rss:9031688kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:57820kB oom_score_adj:0
Mar 14 14:46:57.148134 lighthouse-mainnet kernel: Out of memory: Killed process 301 (current-lightho) total-vm:35219888kB, anon-rss:8467008kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:59344kB oom_score_adj:0
Mar 14 19:41:15.171903 lighthouse-mainnet kernel: Out of memory: Killed process 2832 (current-lightho) total-vm:30916028kB, anon-rss:8464560kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52508kB oom_score_adj:0
Mar 14 20:26:07.847889 lighthouse-mainnet kernel: Out of memory: Killed process 30751 (current-lightho) total-vm:22252708kB, anon-rss:8478196kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:35352kB oom_score_adj:0
Mar 14 23:06:11.396173 lighthouse-mainnet kernel: Out of memory: Killed process 7544 (current-lightho) total-vm:31396520kB, anon-rss:8458808kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:49940kB oom_score_adj:0
Mar 15 07:25:19.078513 lighthouse-mainnet kernel: Out of memory: Killed process 7675 (current-lightho) total-vm:27629916kB, anon-rss:8504864kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:44760kB oom_score_adj:0
Mar 15 07:28:23.647117 lighthouse-mainnet kernel: Out of memory: Killed process 11684 (current-lightho) total-vm:18758968kB, anon-rss:8511072kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:28824kB oom_score_adj:0
Mar 16 05:30:16.986444 lighthouse-mainnet kernel: Out of memory: Killed process 12359 (current-lightho) total-vm:33380104kB, anon-rss:8204916kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:57464kB oom_score_adj:0
Mar 16 05:33:36.360417 lighthouse-mainnet kernel: Out of memory: Killed process 21460 (current-lightho) total-vm:16910420kB, anon-rss:8244316kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26340kB oom_score_adj:0
Mar 16 05:37:06.611366 lighthouse-mainnet kernel: Out of memory: Killed process 22179 (current-lightho) total-vm:17776308kB, anon-rss:8051056kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26868kB oom_score_adj:0
Mar 16 05:39:42.313179 lighthouse-mainnet kernel: Out of memory: Killed process 22961 (current-lightho) total-vm:15998480kB, anon-rss:8076232kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23888kB oom_score_adj:0
Mar 16 05:43:11.010371 lighthouse-mainnet kernel: Out of memory: Killed process 23518 (current-lightho) total-vm:18287132kB, anon-rss:8072088kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:28052kB oom_score_adj:0
Mar 16 05:46:54.689850 lighthouse-mainnet kernel: Out of memory: Killed process 24302 (current-lightho) total-vm:17821696kB, anon-rss:7972692kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:28172kB oom_score_adj:0
Mar 16 05:50:57.681507 lighthouse-mainnet kernel: Out of memory: Killed process 25023 (current-lightho) total-vm:21318680kB, anon-rss:8133908kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:32208kB oom_score_adj:0
Mar 16 07:31:45.762864 lighthouse-mainnet kernel: Out of memory: Killed process 25892 (current-lightho) total-vm:24242492kB, anon-rss:8042644kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:39620kB oom_score_adj:0
Mar 16 07:35:12.328095 lighthouse-mainnet kernel: Out of memory: Killed process 14209 (current-lightho) total-vm:17607628kB, anon-rss:8008580kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24112kB oom_score_adj:0
Mar 16 09:01:27.654294 lighthouse-mainnet kernel: Out of memory: Killed process 15007 (current-lightho) total-vm:25444068kB, anon-rss:8025444kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40804kB oom_score_adj:0
Mar 16 09:20:41.302240 lighthouse-mainnet kernel: Out of memory: Killed process 32499 (current-lightho) total-vm:23868480kB, anon-rss:7977484kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36900kB oom_score_adj:0
Mar 16 11:47:51.887782 lighthouse-mainnet kernel: Out of memory: Killed process 4114 (current-lightho) total-vm:23958932kB, anon-rss:7956472kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:38140kB oom_score_adj:0
Mar 16 11:56:53.523057 lighthouse-mainnet kernel: Out of memory: Killed process 1587 (current-lightho) total-vm:16979648kB, anon-rss:7898700kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25964kB oom_score_adj:0
Mar 16 11:59:31.338756 lighthouse-mainnet kernel: Out of memory: Killed process 3468 (current-lightho) total-vm:15750220kB, anon-rss:7975808kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24248kB oom_score_adj:0
Mar 16 12:03:44.444103 lighthouse-mainnet kernel: Out of memory: Killed process 4032 (current-lightho) total-vm:21042276kB, anon-rss:7872768kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31368kB oom_score_adj:0
Mar 16 15:44:33.154132 lighthouse-mainnet kernel: Out of memory: Killed process 4951 (current-lightho) total-vm:25663796kB, anon-rss:7884196kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42736kB oom_score_adj:0
Mar 16 15:47:30.945806 lighthouse-mainnet kernel: Out of memory: Killed process 17282 (current-lightho) total-vm:15657760kB, anon-rss:7861508kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23096kB oom_score_adj:0
Mar 16 15:50:29.246304 lighthouse-mainnet kernel: Out of memory: Killed process 17969 (current-lightho) total-vm:17295396kB, anon-rss:7823948kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25520kB oom_score_adj:0
Mar 16 15:53:09.328822 lighthouse-mainnet kernel: Out of memory: Killed process 18641 (current-lightho) total-vm:16614920kB, anon-rss:7886880kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24380kB oom_score_adj:0
Mar 16 15:55:44.456669 lighthouse-mainnet kernel: Out of memory: Killed process 19210 (current-lightho) total-vm:16176884kB, anon-rss:7840144kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24080kB oom_score_adj:0
Mar 16 15:59:42.115135 lighthouse-mainnet kernel: Out of memory: Killed process 19824 (current-lightho) total-vm:19793940kB, anon-rss:7845452kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:29648kB oom_score_adj:0I've restarted with --state-cache-size 2
DM'ed you logs on Discord
vogelito
commented
7 months ago The issue continues after the restart, though at a lesser frequency
Mar 17 00:22:55.515286 lighthouse-mainnet kernel: Out of memory: Killed process 6281 (current-lightho) total-vm:25785068kB, anon-rss:7768916kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42996kB oom_score_adj:0
Mar 17 02:11:42.141526 lighthouse-mainnet kernel: Out of memory: Killed process 25879 (current-lightho) total-vm:26377572kB, anon-rss:7728276kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42932kB oom_score_adj:0
Mar 17 02:35:09.640541 lighthouse-mainnet kernel: Out of memory: Killed process 883 (geth) total-vm:10206832kB, anon-rss:7675288kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:15812kB oom_score_adj:0
Mar 19 06:02:06.920665 lighthouse-mainnet kernel: Out of memory: Killed process 15613 (current-lightho) total-vm:38453400kB, anon-rss:8520308kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:65052kB oom_score_adj:0
Mar 19 07:52:36.720328 lighthouse-mainnet kernel: Out of memory: Killed process 31380 (current-lightho) total-vm:31337044kB, anon-rss:8420012kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52560kB oom_score_adj:0
Mar 19 07:55:19.908627 lighthouse-mainnet kernel: Out of memory: Killed process 22029 (current-lightho) total-vm:18002800kB, anon-rss:8590904kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27280kB oom_score_adj:0
Mar 19 11:09:19.149939 lighthouse-mainnet kernel: Out of memory: Killed process 22652 (current-lightho) total-vm:30756432kB, anon-rss:8373716kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:51304kB oom_score_adj:0
Mar 19 11:34:58.588615 lighthouse-mainnet kernel: Out of memory: Killed process 29517 (current-lightho) total-vm:24076624kB, anon-rss:8287376kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37604kB oom_score_adj:0
Mar 20 06:40:41.834013 lighthouse-mainnet kernel: Out of memory: Killed process 2383 (current-lightho) total-vm:31202568kB, anon-rss:8431996kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52672kB oom_score_adj:0
Mar 20 09:27:11.229139 lighthouse-mainnet kernel: Out of memory: Killed process 8475 (current-lightho) total-vm:28790268kB, anon-rss:8382596kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:46396kB oom_score_adj:0@vogelito Can you please DM me a dump of curl http://localhost:5054/metrics? :pray:
We don't have any imminent memory improvements coming, so Lighthouse's memory usage is expected to remain spikey until we merge tree-states (a month or two). I'd recommend going back to 32GB in the meantime so you can avoid this issue
michaelsproul
commented
7 months ago Here's stable vs tree-states:
vogelito
commented
6 months ago tree-states looks promising :)
I've sent you the output of curl http://localhost:5054/metrics
log of restarts, still running 16GB, will restart to 32GB now until tree-states is out:
Mar 17 00:22:55.515286 lighthouse-mainnet kernel: Out of memory: Killed process 6281 (current-lightho) total-vm:25785068kB, anon-rss:7768916kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42996kB oom_score_adj:0
Mar 17 02:11:42.141526 lighthouse-mainnet kernel: Out of memory: Killed process 25879 (current-lightho) total-vm:26377572kB, anon-rss:7728276kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42932kB oom_score_adj:0
Mar 17 02:35:09.640541 lighthouse-mainnet kernel: Out of memory: Killed process 883 (geth) total-vm:10206832kB, anon-rss:7675288kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:15812kB oom_score_adj:0
Mar 19 06:02:06.920665 lighthouse-mainnet kernel: Out of memory: Killed process 15613 (current-lightho) total-vm:38453400kB, anon-rss:8520308kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:65052kB oom_score_adj:0
Mar 19 07:52:36.720328 lighthouse-mainnet kernel: Out of memory: Killed process 31380 (current-lightho) total-vm:31337044kB, anon-rss:8420012kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52560kB oom_score_adj:0
Mar 19 07:55:19.908627 lighthouse-mainnet kernel: Out of memory: Killed process 22029 (current-lightho) total-vm:18002800kB, anon-rss:8590904kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27280kB oom_score_adj:0
Mar 19 11:09:19.149939 lighthouse-mainnet kernel: Out of memory: Killed process 22652 (current-lightho) total-vm:30756432kB, anon-rss:8373716kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:51304kB oom_score_adj:0
Mar 19 11:34:58.588615 lighthouse-mainnet kernel: Out of memory: Killed process 29517 (current-lightho) total-vm:24076624kB, anon-rss:8287376kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37604kB oom_score_adj:0
Mar 20 06:40:41.834013 lighthouse-mainnet kernel: Out of memory: Killed process 2383 (current-lightho) total-vm:31202568kB, anon-rss:8431996kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52672kB oom_score_adj:0
Mar 20 09:27:11.229139 lighthouse-mainnet kernel: Out of memory: Killed process 8475 (current-lightho) total-vm:28790268kB, anon-rss:8382596kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:46396kB oom_score_adj:0
Mar 21 07:50:54.630157 lighthouse-mainnet kernel: Out of memory: Killed process 9957 (current-lightho) total-vm:30062344kB, anon-rss:8367032kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:51508kB oom_score_adj:0
Mar 21 07:54:06.640733 lighthouse-mainnet kernel: Out of memory: Killed process 24467 (current-lightho) total-vm:17029156kB, anon-rss:8374632kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22628kB oom_score_adj:0
Mar 21 07:58:20.626826 lighthouse-mainnet kernel: Out of memory: Killed process 25208 (current-lightho) total-vm:16239840kB, anon-rss:8410564kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23812kB oom_score_adj:0
Mar 21 09:33:23.339752 lighthouse-mainnet kernel: Out of memory: Killed process 25939 (current-lightho) total-vm:23098864kB, anon-rss:8437492kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36180kB oom_score_adj:0
Mar 22 02:11:48.504258 lighthouse-mainnet kernel: Out of memory: Killed process 12915 (current-lightho) total-vm:29413420kB, anon-rss:8354972kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:49144kB oom_score_adj:0
Mar 22 10:18:05.928344 lighthouse-mainnet kernel: Out of memory: Killed process 20676 (current-lightho) total-vm:25884368kB, anon-rss:8270768kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42488kB oom_score_adj:0
Mar 22 18:11:51.244319 lighthouse-mainnet kernel: Out of memory: Killed process 22591 (current-lightho) total-vm:27889468kB, anon-rss:8340576kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:46768kB oom_score_adj:0
Mar 22 18:14:28.609367 lighthouse-mainnet kernel: Out of memory: Killed process 21337 (current-lightho) total-vm:16601488kB, anon-rss:8377068kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24472kB oom_score_adj:0
Mar 22 18:17:00.403712 lighthouse-mainnet kernel: Out of memory: Killed process 21906 (current-lightho) total-vm:15204164kB, anon-rss:8254120kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22960kB oom_score_adj:0
Mar 22 18:20:03.889184 lighthouse-mainnet kernel: Out of memory: Killed process 22520 (current-lightho) total-vm:18439936kB, anon-rss:7857744kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27916kB oom_score_adj:0
Mar 22 20:32:30.754269 lighthouse-mainnet kernel: Out of memory: Killed process 23222 (current-lightho) total-vm:25873412kB, anon-rss:7750792kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41584kB oom_score_adj:0
Mar 22 20:35:08.528444 lighthouse-mainnet kernel: Out of memory: Killed process 17628 (current-lightho) total-vm:16482720kB, anon-rss:7817124kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23844kB oom_score_adj:0
Mar 23 04:13:20.762438 lighthouse-mainnet kernel: Out of memory: Killed process 20503 (geth) total-vm:10450532kB, anon-rss:7648720kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:16112kB oom_score_adj:0
Mar 24 08:30:14.202207 lighthouse-mainnet kernel: Out of memory: Killed process 18194 (current-lightho) total-vm:36080440kB, anon-rss:8670632kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:60488kB oom_score_adj:0
Mar 24 10:00:30.598687 lighthouse-mainnet kernel: Out of memory: Killed process 2099 (current-lightho) total-vm:23547924kB, anon-rss:8671872kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37996kB oom_score_adj:0
Mar 24 10:02:43.658333 lighthouse-mainnet kernel: Out of memory: Killed process 20375 (current-lightho) total-vm:15924252kB, anon-rss:8705604kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22384kB oom_score_adj:0
Mar 24 10:06:05.486224 lighthouse-mainnet kernel: Out of memory: Killed process 20889 (current-lightho) total-vm:16831952kB, anon-rss:7973964kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24272kB oom_score_adj:0
Mar 24 10:10:39.085558 lighthouse-mainnet kernel: Out of memory: Killed process 21634 (current-lightho) total-vm:18529856kB, anon-rss:8163200kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27920kB oom_score_adj:0
Mar 24 10:14:23.276147 lighthouse-mainnet kernel: Out of memory: Killed process 22365 (current-lightho) total-vm:16653472kB, anon-rss:8272312kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25268kB oom_score_adj:0
Mar 24 10:19:40.147616 lighthouse-mainnet kernel: Out of memory: Killed process 22986 (current-lightho) total-vm:22927264kB, anon-rss:8200972kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:32724kB oom_score_adj:0
Mar 24 10:22:57.029910 lighthouse-mainnet kernel: Out of memory: Killed process 24115 (current-lightho) total-vm:15159336kB, anon-rss:8172540kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:21368kB oom_score_adj:0
Mar 24 10:25:33.464185 lighthouse-mainnet kernel: Out of memory: Killed process 24783 (current-lightho) total-vm:15541484kB, anon-rss:8285380kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22348kB oom_score_adj:0
Mar 24 10:28:24.159259 lighthouse-mainnet kernel: Out of memory: Killed process 25394 (current-lightho) total-vm:17009280kB, anon-rss:8237272kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:24792kB oom_score_adj:0
Mar 24 14:08:33.531334 lighthouse-mainnet kernel: Out of memory: Killed process 26015 (current-lightho) total-vm:26979640kB, anon-rss:8069824kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42912kB oom_score_adj:0
Mar 24 14:21:22.117092 lighthouse-mainnet kernel: Out of memory: Killed process 5860 (current-lightho) total-vm:22014840kB, anon-rss:8142632kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:33352kB oom_score_adj:0
Mar 24 16:35:44.427766 lighthouse-mainnet kernel: Out of memory: Killed process 8523 (current-lightho) total-vm:22998360kB, anon-rss:8086492kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36620kB oom_score_adj:0
Mar 24 16:38:17.580694 lighthouse-mainnet kernel: Out of memory: Killed process 3349 (current-lightho) total-vm:15156408kB, anon-rss:8105440kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:21224kB oom_score_adj:0
Mar 24 16:41:13.895464 lighthouse-mainnet kernel: Out of memory: Killed process 3916 (current-lightho) total-vm:15759748kB, anon-rss:7934672kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:22832kB oom_score_adj:0
Mar 24 18:11:44.878939 lighthouse-mainnet kernel: Out of memory: Killed process 4577 (current-lightho) total-vm:25848872kB, anon-rss:7783336kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41428kB oom_score_adj:0
Mar 24 18:30:54.994419 lighthouse-mainnet kernel: Out of memory: Killed process 22923 (current-lightho) total-vm:21008932kB, anon-rss:7775636kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:34172kB oom_score_adj:0
Mar 24 22:34:21.572601 lighthouse-mainnet kernel: Out of memory: Killed process 26896 (current-lightho) total-vm:26444808kB, anon-rss:7719180kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43428kB oom_score_adj:0
Mar 25 02:31:22.150839 lighthouse-mainnet kernel: Out of memory: Killed process 11570 (current-lightho) total-vm:25405424kB, anon-rss:7767380kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41956kB oom_score_adj:0
Mar 25 04:06:52.459149 lighthouse-mainnet kernel: Out of memory: Killed process 13767 (geth) total-vm:10613732kB, anon-rss:7656164kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:15844kB oom_score_adj:0
Mar 26 06:34:08.645146 lighthouse-mainnet kernel: Out of memory: Killed process 27180 (current-lightho) total-vm:33813168kB, anon-rss:8825552kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:56836kB oom_score_adj:0
Mar 26 11:02:56.278656 lighthouse-mainnet kernel: Out of memory: Killed process 12371 (current-lightho) total-vm:25519852kB, anon-rss:8639128kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42176kB oom_score_adj:0
Mar 26 12:38:57.526988 lighthouse-mainnet kernel: Out of memory: Killed process 2076 (current-lightho) total-vm:26520504kB, anon-rss:8548796kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:43308kB oom_score_adj:0
Mar 26 15:27:03.266016 lighthouse-mainnet kernel: Out of memory: Killed process 21776 (current-lightho) total-vm:25789040kB, anon-rss:8467728kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:42244kB oom_score_adj:0
Mar 26 22:27:46.948274 lighthouse-mainnet kernel: Out of memory: Killed process 23704 (current-lightho) total-vm:31071748kB, anon-rss:8429644kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:52548kB oom_score_adj:0
Mar 26 22:30:49.803845 lighthouse-mainnet kernel: Out of memory: Killed process 11620 (current-lightho) total-vm:17278600kB, anon-rss:8464288kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:25556kB oom_score_adj:0
Mar 26 23:18:53.974323 lighthouse-mainnet kernel: Out of memory: Killed process 12294 (current-lightho) total-vm:23197516kB, anon-rss:8469392kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:36988kB oom_score_adj:0
Mar 26 23:22:09.110826 lighthouse-mainnet kernel: Out of memory: Killed process 22093 (current-lightho) total-vm:15693224kB, anon-rss:8428944kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:23712kB oom_score_adj:0
Mar 26 23:27:09.229725 lighthouse-mainnet kernel: Out of memory: Killed process 22825 (current-lightho) total-vm:18007132kB, anon-rss:8453636kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:27208kB oom_score_adj:0
Mar 27 02:05:20.204051 lighthouse-mainnet kernel: Out of memory: Killed process 23653 (current-lightho) total-vm:27635108kB, anon-rss:8403908kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:46428kB oom_score_adj:0
Mar 27 02:38:25.939542 lighthouse-mainnet kernel: Out of memory: Killed process 23296 (current-lightho) total-vm:19037096kB, anon-rss:8403524kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:31380kB oom_score_adj:0
Mar 27 05:42:57.123799 lighthouse-mainnet kernel: Out of memory: Killed process 29871 (current-lightho) total-vm:29282496kB, anon-rss:8453920kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:50112kB oom_score_adj:0
Mar 27 05:49:37.416610 lighthouse-mainnet kernel: Out of memory: Killed process 2955 (current-lightho) total-vm:21152248kB, anon-rss:8412908kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:32868kB oom_score_adj:0
Mar 27 07:31:54.075969 lighthouse-mainnet kernel: Out of memory: Killed process 4413 (current-lightho) total-vm:23331844kB, anon-rss:8311144kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:37200kB oom_score_adj:0
Mar 27 08:22:54.029771 lighthouse-mainnet kernel: Out of memory: Killed process 25352 (current-lightho) total-vm:23772748kB, anon-rss:8324296kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:39908kB oom_score_adj:0
Mar 27 09:52:32.520902 lighthouse-mainnet kernel: Out of memory: Killed process 3437 (current-lightho) total-vm:25679068kB, anon-rss:8296668kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:41520kB oom_score_adj:0
Mar 27 11:41:20.650367 lighthouse-mainnet kernel: Out of memory: Killed process 21620 (current-lightho) total-vm:24423416kB, anon-rss:8257648kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:40956kB oom_score_adj:0
Mar 27 15:12:56.458906 lighthouse-mainnet kernel: Out of memory: Killed process 21433 (current-lightho) total-vm:18413324kB, anon-rss:8263312kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:26880kB oom_score_adj:0I updated to the Tree People release last Thursday (June 13). I see a smaller and less choppy memory footprint but with a clear upwards trend. See chart below:
Do we want to give 16GB RAM a try?
michaelsproul
commented
4 months ago @vogelito It should plateau. It won't grow indefinitely
michaelsproul
commented
4 months ago Try the 16GB RAM, I think it should be OK now
vogelito
commented
4 months ago Alright, back to 16GB RAM. Will report any findings....
vogelito
commented
4 months ago No restarts in the last 28 hours running 5.2.0 on 16GB RAM.
Will report again in a week.
Jun 17 12:52:08 lighthouse-mainnet current-lighthouse-beacon[931]: Jun 17 12:52:08.501 INFO Lighthouse started version: Lighthouse/v5.2.0-f1d88baI can confirm that everything look very nice and stable on a 16GB machine for the first time in 7 months :)
Thanks for the hard work!!
Let me know if you need anything from me or if I can be helpful in any way!
michaelsproul
commented
3 months ago Great to hear @vogelito! Do you mind if we close this issue?
vogelito
commented
3 months ago Of course! Well earned close!!
chong-he
commented
3 months ago Closing the issue as discussed above
vogelito
commented
3 months ago Although definitely no longer an issue, wanted to give you visibility over a couple of OOM kills that have taken place since I downgraded to a 16GB machine
journalctl -o short-precise -k | grep Killed
Jun 30 17:33:22.158209 lighthouse-mainnet kernel: Out of memory: Killed process 19687 (current-lightho) total-vm:36258504kB, anon-rss:7817800kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:61060kB oom_score_adj:0
Jun 30 17:37:00.223682 lighthouse-mainnet kernel: Out of memory: Killed process 7968 (current-lightho) total-vm:19702936kB, anon-rss:7889132kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:28376kB oom_score_adj:0
Jul 02 20:00:40.593911 lighthouse-mainnet kernel: Out of memory: Killed process 8796 (current-lightho) total-vm:38959468kB, anon-rss:7724908kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:63932kB oom_score_adj:0June 30th 17:24:00 until 18:24:00
July 02 19:30:00 unti 20:30:00
vogelito
commented
3 months ago As always, happy to help debug.
Description
5 weeks ago I started noticing my validator node missing attestations. I then realized these missed attestations matched moments where CPU usage in my node was high. I then added a probe to see what process in my node was spiking the CPU usage and realized it was mostly due to higher CPU usage from my beacon node. I also realized these periods matched periods of higher than usual disk reading and writing.
This is new behavior that started roughly 5 weeks ago.
Version
I'm downloading the release binaries from Github and currently running:
Present Behaviour
I have missed attestations on each of these periods of high CPU usage:
Expected Behaviour
I shouldn't be missing attestations.
Steps to resolve
I'm unaware of what steps I can take to resolve this, but happy to work with the devs on resolving them :)