Launching Steam immediately caused `fossilize_replay` to consume all available memory

[rhoot]

Not sure whether to treat this as a Steam or fossilize issue, but trying here first:

When I launched Steam today, my system got very laggy and unresponsive. Eventually it froze completely (video, audio, everything) for a few seconds until oom-killer kicked in. So I closed Steam and opened it again while keeping an eye on RAM.

I have 32 GiB of RAM plus 8 GiB of swap. Before opening Steam 29 GiB of RAM was available. Within 5 seconds fossilize_replay had consumed all of it, as well as the swap. oom-killer then kicked in and killed some processes, a few seconds later it was at 100% usage again, repeat for a couple of minutes.

System information

Computer Information: Manufacturer: EVGA Corp. Model: X570 FTW WIFI Form Factor: Desktop No Touch Input Detected Processor Information: CPU Vendor: AuthenticAMD CPU Brand: AMD Ryzen 9 5950X 16-Core Processor CPU Family: 0x19 CPU Model: 0x21 CPU Stepping: 0x0 CPU Type: 0x0 Speed: 5083 MHz 32 logical processors 16 physical processors Hyper-threading: Supported FCMOV: Supported SSE2: Supported SSE3: Supported SSSE3: Supported SSE4a: Supported SSE41: Supported SSE42: Supported AES: Supported AVX: Supported AVX2: Supported AVX512F: Unsupported AVX512PF: Unsupported AVX512ER: Unsupported AVX512CD: Unsupported AVX512VNNI: Unsupported SHA: Supported CMPXCHG16B: Supported LAHF/SAHF: Supported PrefetchW: Unsupported Operating System Version: "EndeavourOS Linux" (64 bit) Kernel Name: Linux Kernel Version: 6.4.12-zen1-1-zen X Server Vendor: The X.Org Foundation X Server Release: 12302000 X Window Manager: KWin Steam Runtime Version: steam-runtime_0.20230606.51628 Video Card: Driver: AMD AMD Radeon RX 6900 XT (navi21, LLVM 15.0.7, DRM 3.52, 6.4.12-zen1-1-zen) Driver Version: 4.6 (Compatibility Profile) Mesa 23.1.6 OpenGL Version: 4.6 Desktop Color Depth: 24 bits per pixel Monitor Refresh Rate: 174 Hz VendorID: 0x10de DeviceID: 0x1e07 Revision Not Detected Number of Monitors: 2 Number of Logical Video Cards: 2 Primary Display Resolution: 3440 x 1440 Desktop Resolution: 3440 x 2520 Primary Display Size: 31.89" x 13.78" (34.72" diag), 81.0cm x 35.0cm (88.2cm diag) Primary VRAM: 16384 MB Sound card: Audio device: USB Mixer Memory: RAM: 32018 Mb VR Hardware: VR Headset: None detected Miscellaneous: UI Language: English LANG: en_US.UTF-8 Total Hard Disk Space Available: 255884 MB Largest Free Hard Disk Block: 82750 MB Storage: Number of SSDs: 2 SSD sizes: 2000G,1000G Number of HDDs: 0 Number of removable drives: 0

[kisak-valve]

Hello @rhoot, can you check if rebuilding mesa with https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24949 helps (or test mesa 23.1.5)?

[rhoot]

It's hard to say whether building with that patch applied helped or not. The issue worked itself out after a couple of minutes yesterday, so presumably the only things to compile today were the files for whatever games got updated.

So immediately after launching Steam today I didn't have this issue. But then some game updates got installed and memory use started to rise again. For the most part fossilize seems to sit around a much more reasonable 1 GiB of use, but there were a few spikes that brought it up right to the limit and then drop again almost immediately. I think the highest I saw was around 30 GiB used out of 31.3 (on my system, not just fossilize). Similar situation to yesterday: Only about 3 GiB was used before launching Steam.

Next time it happens I'll try to see if i can figure out which game may have triggered those spikes from the fossilize command line. I didn't realize it contained the appid in the path of an argument passed into it until after it had stopped spiking.

Edit: Actually I'll just try wiping some shader caches later.

[rhoot]

Okay, so I disabled the shader cache and re-enabled it again (in Steam settings). That caused it to start compiling some shaders. It did rise back up to using basically all my memory (on mesa 23.1.6, with the patch from that MR applied). The game it was compiling shaders for was Deep Rock Galactic (appid 548430).

Eventually it dropped back down a bit, but then just... kind of stalled out:

You can see the remnants of the memory usage in that screenshot too. How the swap is completely full, and 100% of the free memory is now used for cache. As best I can tell, Steam was using the CPU to download cached shaders, so that CPU usage is likely expected.

But the fossilize processes just stayed sleeping like that for several minutes, until I eventually closed Steam. Once I loaded it up again I managed to snap this before my system completely froze for a few seconds again until oom-killer kicked in and killed some processes:

Edit: For reference/comparison, this is after closing Steam:

[rhoot]

I just tried building/installing mesa 23.1.5. I have had shaders compiling for over an hour now, and no fossilize_replay process has ever gone above 544M in the resident set. So it definitely seems like a regression in 23.1.6.

[kakra]

You can see the remnants of the memory usage in that screenshot too. How the swap is completely full, and 100% of the free memory is now used for cache. As best I can tell, Steam was using the CPU to download cached shaders, so that CPU usage is likely expected.

This looks a lot like the memory behavior I'm seeing from 6.x kernels not only with fossilize but all sorts of processes. Especially here, it looks like fossilize does not use shared memory at all, and memory usage is too high by a factor of 10.

See if your kernel has /sys/kernel/mm/transparent_hugepage and if yes, run as root after a fresh reboot and before starting Steam:

echo within_size >/sys/kernel/mm/transparent_hugepage/shmem_enabled
echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag
echo 64 >/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
echo 8 >/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap
echo 32 >/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared

This should reduce memory pressure but if you're still seeing high swap pressure, try booting with kernel cmdline cgroup_disable=memory and then try this test again. How to adjust the kernel cmdline or set the above settings permanently, is specific to your distribution.

So it definitely seems like a regression in 23.1.6.

This is very possible, too.

[rhoot]

This should reduce memory pressure but if you're still seeing high swap pressure, [...]

To be clear, swap didn't start filling up until my physical RAM had been fully consumed.

Downgrading to mesa 23.1.5 without changing anything else about the system also caused shared memory usage to go up. This is what it looks like after the downgrade:

[NextGenRyo]

I am suffering from this exact same problem after a system update on manjaro. In that update was mesa 23.1.6. Any updates on this?

[kisak-valve]

This is a mesa/RADV regression limited to mesa 23.1.6.

Caused by https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24579, and should be fixed by https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24949 and https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24896.

The practical fix is to update to mesa 23.1.7 or newer, which includes these.

There's nothing more to be done on Fossilize's side.

[rhoot]

The practical fix is to update to mesa 23.1.7 or newer, which includes these.

There's nothing more to be done on Fossilize's side.

Yep, 23.1.7 seems to fix it. Thanks!