kakra
commented
1 year ago Show cat /proc/buddyinfo before and after the problem occurs. I'm seeing similar behavior since some of the latest updates for kernel 6.1. While I don't get the exact same behavior, I'm seeing the kernel generating memory pressure, flushing cache and sending memory to swap despite multiple gigabytes of memory announced as free, resulting in IO thrashing.
buddyinfo shows how many pages of which size are free (column 1 = order 0 = 4k 2^0, column 2 = 4k 2^1 = 8k, then 16k, 32k...), and if the kernel cannot find a free block that can hold the allocation, it will swap data out or fail the allocation. This is because the kernel itself allocates physical pages not virtual memory (thus, it cannot split big blocks of allocations and merge them virtually into one continuous range of memory: if something doesn't fit, it doesn't fit, this is called memory fragmentation).
Disabling the memory cgroup controller (and using generational LRU instead) somewhat mitigated that for me, setting transparent huge pages to madvise also helped, but it is still not fully fixed for me.
What you want to see in buddyinfo is high numbers in the high order columns. If it peaks in the low order columns, the kernel should (and will) try to compact movable pages into larger blocks, thus defragmenting free space. This can take a few seconds, so you may want to take multiple snapshots of buddyinfo. If pages do not migrate to higher order free space, you should check if your system has a lot of non-movable pages or huge pages.
User-space pages are movable (because they are addressed indirectly through page table lookups). Buffers for hardware are usually not movable. Page cache is, I think, neither movable but reclaimable.
Your error indicates it's trying to get an order 9 allocation (4k * 2^9 = 2M) so buddyinfo probably doesn't have anything beyond the 9th column.
Something bad is going on with memory allocations in the kernel since 6.1.
tytgatlieven
Zygo
Dear I upgraded my system from linux 6.2.13 to 6.3.1. This resulted in the error messages as below in my logs. There are no crashes.
This has also been reported here
In the reply to above it is mentioned that a out-of-memory condition could trigger the issue. In my setup I have about 30GB of free RAM memory, so this shouldn't be the case.
Feel free to contact me for more info/tests.