Potential virtual memory leak

solajim commented 19 hours ago

Summary

Overview

Ombi seems to be running into an issue where it tries to consume more virtual memory than is physically possible (263G). As a result, it keeps getting flagged by the system resource manager as needing to die, and it gets killed. I've implemented a hotfix via cron, but it's messy and doesn't solve the base issue.

The odd part is that the system itself isn't burdened - memory and CPU are all nominal. For whatever reason, linux thinks that Ombi is chewing up 263G of memory.

Hotfix

/etc/cron.d\# touch oom_disable /ect/cron.d\# echo '*/1 * * * * root pgrep -f "Ombi\ --storage\ [long storage path]" | while read PID; do echo -17 > /proc/$PID/oom_adj; done'

This just persistently keeps the process from being killed. Not the best solution.

Environment setup

Linux version - Ubuntu 20.04.6 LTS (I know I know, it's time to upgrade)
Daemon - launched at boot by root via script su daemon-ombi -s /bin/bash -c 'cd /home/daemon-ombi/bin && /home/daemon-ombi/bin/Ombi --storage "/media/[long storage path]" --host "http://0.0.0.0:40011" --baseurl "/ombi"' &
Relevant Binaries
- MySQL Ver 8.0.40-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))
- Ombi - 4.44.1 (automatically downloaded and installed via bash script from git stable into /home/daemon-ombi/bin)
- Emby - Version [4.8.10.0] Emby Server is up to date
Platform - See log output below
Relevant Error logs

2024-11-22 14:49:55.180 -05:00 [INF] Default Quartz.NET properties loaded from embedded resource file 2024-11-22 14:49:55.255 -05:00 [INF] Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl 2024-11-22 14:49:55.255 -05:00 [INF] Quartz Scheduler created 2024-11-22 14:49:55.255 -05:00 [INF] RAMJobStore initialized. 2024-11-22 14:49:55.256 -05:00 [INF] Quartz Scheduler 3.6.2.0 - 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED' initialized 2024-11-22 14:49:55.256 -05:00 [INF] Using thread pool 'Quartz.Simpl.DefaultThreadPool', size: 10 2024-11-22 14:49:55.256 -05:00 [INF] Using job store 'Quartz.Simpl.RAMJobStore', supports persistence: False, clustered: False 2024-11-22 14:49:55.257 -05:00 [INF] JobFactory set to: Ombi.Schedule.IoCJobFactory 2024-11-22 14:49:55.304 -05:00 [INF] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.

Relevant Picture

Ombi Version

4.44.1

What platform(s) does this occur on?

Linux

What database are you using?

MySQL

Relevant log output

00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:5910] (rev 05)
        Subsystem: ASUSTeK Computer Inc. Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [1043:1bb0]
        Kernel driver in use: skl_uncore
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
        Kernel driver in use: pcieport
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 630 [8086:591b] (rev 04)
        DeviceName:  Onboard IGD
        Subsystem: ASUSTeK Computer Inc. HD Graphics 630 [1043:1bb0]
        Kernel driver in use: i915
        Kernel modules: i915
00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
        Subsystem: ASUSTeK Computer Inc. Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [1043:1bb0]
00:14.0 USB controller [0c03]: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller [8086:a12f] (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller [1043:201f]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
00:14.2 Signal processing controller [1180]: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem [8086:a131] (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family Thermal Subsystem [1043:1bb0]
        Kernel driver in use: intel_pch_thermal
        Kernel modules: intel_pch_thermal
00:15.0 Signal processing controller [1180]: Intel Corporation 100 Series/C230 Series Chipset Family Serial IO I2C Controller #0 [8086:a160] (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family Serial IO I2C Controller [1043:1bb0]
        Kernel driver in use: intel-lpss
        Kernel modules: intel_lpss_pci
00:16.0 Communication controller [0780]: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 [8086:a13a] (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family MEI Controller [1043:1bb0]
        Kernel driver in use: mei_me
        Kernel modules: mei_me
00:17.0 RAID bus controller [0104]: Intel Corporation 82801 Mobile SATA Controller [RAID mode] [8086:282a] (rev 31)
        Subsystem: ASUSTeK Computer Inc. 82801 Mobile SATA Controller [RAID mode] [1043:1bb0]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 [8086:a114] (rev f1)
        Kernel driver in use: pcieport
00:1c.5 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #6 [8086:a115] (rev f1)
        Kernel driver in use: pcieport
00:1c.6 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #7 [8086:a116] (rev f1)
        Kernel driver in use: pcieport
00:1f.0 ISA bridge [0601]: Intel Corporation HM175 Chipset LPC/eSPI Controller [8086:a152] (rev 31)
        Subsystem: ASUSTeK Computer Inc. HM175 Chipset LPC/eSPI Controller [1043:1bb0]
00:1f.2 Memory controller [0580]: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller [8086:a121] (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family Power Management Controller [1043:1bb0]
00:1f.3 Audio device [0403]: Intel Corporation CM238 HD Audio Controller [8086:a171] (rev 31)
        Subsystem: ASUSTeK Computer Inc. CM238 HD Audio Controller [1043:1bb0]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation 100 Series/C230 Series Chipset Family SMBus [8086:a123] (rev 31)
        Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family SMBus [1043:1bb0]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
01:00.0 3D controller [0302]: NVIDIA Corporation GP107M [GeForce GTX 1050 Mobile] [10de:1c8d] (rev a1)
        Subsystem: ASUSTeK Computer Inc. GP107M [GeForce GTX 1050 Mobile] [1043:1bb0]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
        Subsystem: ASUSTeK Computer Inc. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1043:200f]
        Kernel driver in use: r8169
        Kernel modules: r8169
03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5229 PCI Express Card Reader [10ec:5229] (rev 01)
        Subsystem: ASUSTeK Computer Inc. RTS5229 PCI Express Card Reader [1043:202f]
        Kernel driver in use: rtsx_pci
        Kernel modules: rtsx_pci
04:00.0 Network controller [0280]: Intel Corporation Wireless 8265 / 8275 [8086:24fd] (rev 78)
        Subsystem: Intel Corporation Dual Band Wireless-AC 8265 [8086:0010]
        Kernel driver in use: iwlwifi
        Kernel modules: iwlwifi

solajim commented 18 hours ago

Something I forgot to mention - it lists as trying to consume 263G as soon as the process starts -- there's no waiting time.

tidusjar commented 17 hours ago

So this is VIRT memory usage. This is something that the underlying technology (.net) does. Here’s an explanation https://github.com/dotnet/runtime/issues/9199

but this shouldn’t really be an issue or causing any issues

Ombi-app / Ombi