OOM killer triggers very early/aggressively on armv7 6.1.x

gavinbeatty commented 1 year ago

Describe the bug

On armv7 builds (but not aarch64 ones), OOM killer triggers and kills numerous processes when memory usage goes above a certain threshold, but that threshold is far below the volatile memory available (the threshold is not consistent, I have experienced it anywhere from 10-50% of volatile memory). This seems to coincide with my distribution changing to a 6.1.x kernel in the last 2 months, and definitely did not happen for any 5.15.x kernel. (And to repeat, this does not occur for 6.1.x aarch64)

The threshold is really very low: just running a zsh shell with zimfw might trigger it. Just running a low usage gitea service occasionally triggers it. It is impossible to run "heavier" software such as media servers as they always trigger a cascade of OOM kills. While creating this issue just now, I experienced it when running inxi -F, and was locked out of the system (it's common for both systemd and ssh to be killed when it triggers), and had to hard reset power.

Steps to reproduce the behaviour

I am using archlinuxarm, installed according to the armv7 instructions https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-4)
Install a 6.1.x build of this kernel (pacman -S linux-rpi linux-rpi-headers)
Install some software than can use a decent amount of memory (e.g., pacman -S ffmpeg)
Start process(es) that use a few hundred MBs of memory. I made a loop to spawn 30 concurrent processes re-encoding video, with ffmpeg -i ./example.mkv ./example1.mp4 &, and a sleep 1 in between
Watch memory usage gradually increase as additional ffmpeg processes start, using htop or btm or free in a loop
If logged in via ssh/mosh, prepare for disconnect or timeout as a number of important processes are OOM killed, including sshd/mosh-server
Note that memory usage never approached full when OOM kills were triggered

Device (s)

Raspberry Pi 4 Mod. B

System

Archlinuxarm armv7h

uname -a
Linux dill 6.1.20-1-rpi-ARCH #1 SMP Tue Mar 21 12:49:06 MDT 2023 armv7l GNU/Linux

grep Model /proc/cpuinfo
Model           : Raspberry Pi 4 Model B Rev 1.1

vcgencmd version
Mar 17 2023 10:50:39
Copyright (c) 2012 Broadcom
version 82f3750a65fadae9a38077e3c2e217ad158c8d54 (clean) (release) (start)

rpi-eeprom-update
BOOTLOADER: up to date
   CURRENT: Wed Jan 11 05:40:52 PM UTC 2023 (1673458852)
    LATEST: Wed Jan 11 05:40:52 PM UTC 2023 (1673458852)
   RELEASE: critical (/lib/firmware/raspberrypi/bootloader/critical)

  VL805_FW: Dedicated VL805 EEPROM
     VL805: up to date
   CURRENT: 000138c0
    LATEST: 000138c0

Logs

From journalctl (wrapped)

Mar 22 09:34:30 dill kernel: Out of memory: Killed process 2169 (gitea) total-vm:843364kB, anon-rss:80128kB, file-rss:65252kB, shmem-rss:0kB, UID:972 pgtables:412kB oom_score_adj:0
Mar 22 09:34:30 dill kernel: bash invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
Mar 22 09:34:30 dill kernel: CPU: 2 PID: 2241 Comm: bash Tainted: P         C O       6.1.20-1-rpi-ARCH #1
Mar 22 09:34:30 dill kernel: Hardware name: BCM2711
Mar 22 09:34:30 dill kernel:  unwind_backtrace from show_stack+0x18/0x1c
Mar 22 09:34:30 dill kernel:  show_stack from dump_stack_lvl+0x90/0xac
Mar 22 09:34:30 dill kernel:  dump_stack_lvl from dump_header+0x54/0x1fc
Mar 22 09:34:30 dill kernel:  dump_header from oom_kill_process+0x23c/0x248
Mar 22 09:34:30 dill kernel:  oom_kill_process from out_of_memory+0x218/0x34c
Mar 22 09:34:30 dill kernel:  out_of_memory from __alloc_pages+0xa98/0x1044
Mar 22 09:34:30 dill kernel:  __alloc_pages from __pmd_alloc+0x3c/0x1d8
Mar 22 09:34:30 dill kernel:  __pmd_alloc from pgd_alloc+0x2e0/0x32c
Mar 22 09:34:30 dill kernel:  pgd_alloc from mm_init+0x100/0x1cc
Mar 22 09:34:30 dill kernel:  mm_init from alloc_bprm+0x84/0x28c
Mar 22 09:34:30 dill kernel:  alloc_bprm from do_execveat_common+0x74/0x1e0
Mar 22 09:34:30 dill kernel:  do_execveat_common from sys_execve+0x38/0x40
Mar 22 09:34:30 dill kernel:  sys_execve from ret_fast_syscall+0x0/0x1c
Mar 22 09:34:30 dill kernel: Exception stack(0xf4499fa8 to 0xf4499ff0)
Mar 22 09:34:30 dill kernel: 9fa0:                   008fa170 004e44e0 008fa170 008fa930 008fbd28 00000000
Mar 22 09:34:30 dill kernel: 9fc0: 008fa170 004e44e0 008fa170 0000000b 008fa930 008fbd28 00000020 004ee330
Mar 22 09:34:30 dill kernel: 9fe0: 004e480c bedc7794 00432874 b6d5dc4c
Mar 22 09:34:30 dill kernel: Mem-Info:
Mar 22 09:34:30 dill kernel: active_anon:321709 inactive_anon:2285 isolated_anon:0
                              active_file:41305 inactive_file:17260 isolated_file:0
                              unevictable:395 dirty:609 writeback:0
                              slab_reclaimable:3453 slab_unreclaimable:14828
                              mapped:15676 shmem:2254 pagetables:2420
                              sec_pagetables:0 bounce:0
                              kernel_misc_reclaimable:0
                              free:554755 free_pcp:170 free_cma:126429
Mar 22 09:34:30 dill kernel: Node 0 active_anon:1286836kB inactive_anon:9140kB active_file:165220kB inactive_file:69040kB unevictable:1580kB      isolated(anon):0kB isolated(file):0kB mapped:62704kB dirty:2436kB writeback:0kB shmem:9016kB writeback_tmp:0kB kernel_stack:4712kB pagetables:    9680kB sec_pagetables:0kB all_unreclaimable? yes
Mar 22 09:34:30 dill kernel: DMA free:508904kB boost:0kB min:3232kB low:4040kB high:4848kB reserved_highatomic:0KB active_anon:0kB inactive_anon: 0kB active_file:504kB inactive_file:55364kB unevictable:0kB writepending:176kB present:786432kB managed:663844kB mlocked:0kB bounce:0kB free_pcp: 680kB local_pcp:252kB free_cma:505716kB
Mar 22 09:34:30 dill kernel: lowmem_reserve[]: 0 0 3188 3188
Mar 22 09:34:30 dill kernel: DMA: 22*4kB (UMEC) 8*8kB (UME) 25*16kB (UMEC) 20*32kB (UMEC) 17*64kB (UMEC) 4*128kB (UE) 3*256kB (UC) 1*512kB (C) 1* 1024kB (C) 0*2048kB 123*4096kB (C) = 508904kB
Mar 22 09:34:30 dill kernel: 61185 total pagecache pages
Mar 22 09:34:30 dill kernel: 0 pages in swap cache
Mar 22 09:34:30 dill kernel: Free swap  = 16777212kB
Mar 22 09:34:30 dill kernel: Total swap = 16777212kB
Mar 22 09:34:30 dill kernel: 1012736 pages RAM
Mar 22 09:34:30 dill kernel: 816128 pages HighMem/MovableOnly
Mar 22 09:34:30 dill kernel: 30647 pages reserved
Mar 22 09:34:30 dill kernel: 131072 pages cma reserved
Mar 22 09:34:30 dill kernel: Tasks state (memory values in pages):
Mar 22 09:34:30 dill kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Mar 22 09:34:30 dill kernel: [    223]     0   223     5634     1931    65536        0          -250 systemd-journal
Mar 22 09:34:30 dill kernel: [    225]     0   225     7163     1940    61440        0         -1000 systemd-udevd
Mar 22 09:34:30 dill kernel: [    360]   980   360     4508     2815    61440        0             0 systemd-resolve
Mar 22 09:34:30 dill kernel: [    361]   979   361     6053     1656    65536        0             0 systemd-timesyn
Mar 22 09:34:30 dill kernel: [    366]    81   366     2173      886    45056        0          -900 dbus-broker-lau
Mar 22 09:34:30 dill kernel: [    367]    81   367      860      521    28672        0          -900 dbus-broker
Mar 22 09:34:30 dill kernel: [    368] 61876   368      577      193    32768        0             0 earlyoom
Mar 22 09:34:30 dill kernel: [    369]     0   369    14863     7462   131072        0             0 firewalld
Mar 22 09:34:30 dill kernel: [    370]     0   370     3715     1603    53248        0             0 systemd-logind
Mar 22 09:34:30 dill kernel: [    372]   981   372     3906     1848    65536        0             0 systemd-network
Mar 22 09:34:30 dill kernel: [    386]   974   386    33335     5007   196608        0             0 minidlnad
Mar 22 09:34:30 dill kernel: [    387]     0   387     2423     1598    40960        0         -1000 sshd
Mar 22 09:34:30 dill kernel: [    389]     0   389     1097      199    36864        0             0 agetty
Mar 22 09:34:30 dill kernel: [    857]  1000   857     3480     1137    45056        0             0 tmux: server
Mar 22 09:34:30 dill kernel: [    860]  1000   860     2443     1287    36864        0             0 bash
Mar 22 09:34:30 dill kernel: [   1079]  1000  1079     5516     2448    65536        0             0 mosh-server
Mar 22 09:34:30 dill kernel: [   1080]  1000  1080     3187      965    53248        0             0 tmux: client
Mar 22 09:34:30 dill kernel: [   1120]  1000  1120     2309     1061    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1125]  1000  1125     3369     1287    53248        0             0 htop
Mar 22 09:34:30 dill kernel: [   1126]  1000  1126     2309     1055    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1132]  1000  1132     9096     1640    94208        0             0 btm
Mar 22 09:34:30 dill kernel: [   1155]  1000  1155     2309     1152    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1299]  1000  1299     2309     1061    40960        0             0 bash
Mar 22 09:34:30 dill kernel: [   1360]  1000  1360   106887    32086   557056        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   1390]  1000  1390     2309     1147    40960        0             0 bash
Mar 22 09:34:30 dill kernel: [   1447]  1000  1447     2309     1109    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1486]  1000  1486     2309     1061    40960        0             0 bash
Mar 22 09:34:30 dill kernel: [   1499]  1000  1499   106887    32056   557056        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   1526]  1000  1526     2309     1109    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1568]  1000  1568     2309     1109    40960        0             0 bash
Mar 22 09:34:30 dill kernel: [   1624]  1000  1624     2309     1055    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1631]  1000  1631     2309     1061    36864        0             0 bash
Mar 22 09:34:30 dill kernel: [   1645]  1000  1645   106880    32031   552960        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   1677]  1000  1677   106881    31954   561152        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   1719]  1000  1719     2309     1061    40960        0             0 bash
Mar 22 09:34:30 dill kernel: [   1732]  1000  1732   106784    31848   548864        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   1736]  1000  1736     2309     1061    36864        0             0 bash
Mar 22 09:34:30 dill kernel: [   1742]  1000  1742     2309     1054    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1748]  1000  1748     2309     1061    49152        0             0 bash
Mar 22 09:34:30 dill kernel: [   1780]  1000  1780     2309     1061    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1786]  1000  1786     2309     1061    36864        0             0 bash
Mar 22 09:34:30 dill kernel: [   1794]  1000  1794     2309     1061    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   1805]  1000  1805   106637    31616   561152        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   1822]  1000  1822   106603    31717   557056        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   1868]  1000  1868   106564    31560   557056        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   1883]  1000  1883   106530    31550   552960        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   2000]  1000  2000   106530    31352   565248        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   2018]  1000  2018   106456    31021   561152        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   2033]  1000  2033     2309     1061    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   2098]  1000  2098   106456    30924   561152        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   2102]  1000  2102     2309     1061    49152        0             0 bash
Mar 22 09:34:30 dill kernel: [   2129]  1000  2129     2309     1055    40960        0             0 bash
Mar 22 09:34:30 dill kernel: [   2136]  1000  2136     2309     1061    40960        0             0 bash
Mar 22 09:34:30 dill kernel: [   2145]  1000  2145     2309     1061    36864        0             0 bash
Mar 22 09:34:30 dill kernel: [   2150]  1000  2150     2309     1061    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   2159]  1000  2159     2309     1055    45056        0             0 bash
Mar 22 09:34:30 dill kernel: [   2178]  1000  2178     2309     1061    40960        0             0 bash
Mar 22 09:34:30 dill kernel: [   2208]  1000  2208    90682    12887   430080        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   2214]  1000  2214    33643     3968   200704        0             0 ffmpeg
Mar 22 09:34:30 dill kernel: [   2241]  1000  2241     2309      247    28672        0             0 bash
Mar 22 09:34:30 dill kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=ffmpeg,     pid=1360,uid=1000

Additional context

There are several other reports of this issue in this archlinuxarm forum post, along with lots more logs and /boot/*.txt, etc., and some info from one of the maintainers, who cannot find a distro specific cause: https://archlinuxarm.org/forum/viewtopic.php?f=23&t=16377

gavinbeatty commented 1 year ago

For some reason, a very reliable way I've been able to trigger it is to reinstall mkinitcpio (pacman -S mkinitcpio)

popcornmix commented 1 year ago

RPiOS is our primary focus here. If you can reproduce your issue using a clean RPiOS image, then it will be a lot easier for us to reproduce.

My initial thought is MGLRU is one of the big new features of 6.1 kernel. Firstly is it enabled on archlinux? On RPiOS it is:

$ cat /sys/kernel/mm/lru_gen/enabled
0x0001

if it is you could try disabling it and see if behaviour changes.

$ echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled
0

gavinbeatty commented 1 year ago

I'll try disabling that now, just going to post the following info that I wrote before I saw your post:

Looking in more detail at each stacktrace, including lots of new ones I've been able to generate thanks to being able to reliably cause it with pacman reinstalls, it seems to be a kernel issue where the following happens:

userspace makes a filesystem syscall like statx, openat, faccessat, etc.
after context switch to kernel, it ends up calling __alloc_pages which spuriously fails and calls either out_of_memory or warn_alloc
the userspace process is killed

So systemd is not involved at all, and it can be triggered with non-destructive and non-memory intensive programs. Non-mutating filesystem use is a reliable source of the issue.

I can now reliably reproduce it with a simple find command that issues stat-type syscalls, like find / -amin -1.

pelwell commented 1 year ago

If you are using one of the vc4-*-v3d overlays, does appending ,cma-320 make any difference?

gavinbeatty commented 1 year ago

@popcornmix Thanks, it was enabled, and after disabling, the OOMs are far less frequent, but not entirely gone. I re-enabled the media server processes, etc., and was testing it out. For a while, everything worked fine, but I did eventually see the same "syscall leads to __alloc_pages, out_of_memory" chain, this time with sendto (so not the VFS layer). Previously some OOM would occur within 1 second of running the media server.

Mar 22 13:49:47 dill kernel: Hardware name: BCM2711
Mar 22 13:49:47 dill kernel:  unwind_backtrace from show_stack+0x18/0x1c
Mar 22 13:49:47 dill kernel:  show_stack from dump_stack_lvl+0x90/0xac
Mar 22 13:49:47 dill kernel:  dump_stack_lvl from dump_header+0x54/0x1fc
Mar 22 13:49:47 dill kernel:  dump_header from oom_kill_process+0x23c/0x248
Mar 22 13:49:47 dill kernel:  oom_kill_process from out_of_memory+0x218/0x34c
Mar 22 13:49:47 dill kernel:  out_of_memory from __alloc_pages+0xa98/0x1044
Mar 22 13:49:47 dill kernel:  __alloc_pages from skb_page_frag_refill+0xd4/0xf8
Mar 22 13:49:47 dill kernel:  skb_page_frag_refill from sk_page_frag_refill+0x1c/0x94
Mar 22 13:49:47 dill kernel:  sk_page_frag_refill from tcp_sendmsg_locked+0x1fc/0xa9c
Mar 22 13:49:47 dill kernel:  tcp_sendmsg_locked from tcp_sendmsg+0x30/0x44
Mar 22 13:49:47 dill kernel:  tcp_sendmsg from sock_sendmsg+0x3c/0x4c
Mar 22 13:49:47 dill kernel:  sock_sendmsg from __sys_sendto+0x110/0x154
Mar 22 13:49:47 dill kernel:  __sys_sendto from __sys_trace_return+0x0/0x10
Mar 22 13:49:47 dill kernel: Exception stack(0xf4759fa8 to 0xf4759ff0)
Mar 22 13:49:47 dill kernel: 9fa0:                   00000000 00000000 00000028 b26e08e0 00010000 00004000
Mar 22 13:49:47 dill kernel: 9fc0: 00000000 00000000 00000000 00000122 b24c5bcc b26e08e0 00000000 b26e08e0
Mar 22 13:49:47 dill kernel: 9fe0: b24c5b38 b24c5b28 b6d17f37 b6d176f8
Mar 22 13:49:47 dill kernel: Mem-Info:
Mar 22 13:49:47 dill kernel: active_anon:621 inactive_anon:84478 isolated_anon:0
                              active_file:36887 inactive_file:53736 isolated_file:27
                              unevictable:395 dirty:7615 writeback:69
                              slab_reclaimable:4231 slab_unreclaimable:23640
                              mapped:35696 shmem:2253 pagetables:834
                              sec_pagetables:0 bounce:0
                              kernel_misc_reclaimable:0
                              free:755526 free_pcp:70 free_cma:129347
Mar 22 13:49:47 dill kernel: Node 0 active_anon:2484kB inactive_anon:337912kB active_file:147548kB inactive_file:214944kB unevictable:1580kB isola
ted(anon):0kB isolated(file):108kB mapped:142784kB dirty:30460kB writeback:276kB shmem:9012kB writeback_tmp:0kB kernel_stack:3344kB pagetables:333
6kB sec_pagetables:0kB all_unreclaimable? no
Mar 22 13:49:47 dill kernel: DMA free:531136kB boost:0kB min:3232kB low:4040kB high:4848kB reserved_highatomic:4096KB active_anon:0kB inactive_ano
n:0kB active_file:528kB inactive_file:0kB unevictable:0kB writepending:44kB present:786432kB managed:663844kB mlocked:0kB bounce:0kB free_pcp:280k
B local_pcp:0kB free_cma:517388kB
Mar 22 13:49:47 dill kernel: lowmem_reserve[]: 0 0 3188 3188
Mar 22 13:49:47 dill kernel: DMA: 583*4kB (UEHC) 856*8kB (UEHC) 195*16kB (UEHC) 65*32kB (UEHC) 30*64kB (UHC) 10*128kB (HC) 4*256kB (C) 7*512kB (C)
 3*1024kB (C) 3*2048kB (C) 122*4096kB (C) = 531116kB
Mar 22 13:49:47 dill kernel: 92358 total pagecache pages
Mar 22 13:49:47 dill kernel: 0 pages in swap cache
Mar 22 13:49:47 dill kernel: Free swap  = 16777212kB
Mar 22 13:49:47 dill kernel: Total swap = 16777212kB
Mar 22 13:49:47 dill kernel: 1012736 pages RAM
Mar 22 13:49:47 dill kernel: 816128 pages HighMem/MovableOnly
Mar 22 13:49:47 dill kernel: 30647 pages reserved
Mar 22 13:49:47 dill kernel: 131072 pages cma reserved

For now, I've made that config permanent by running systemd-tmpfiles --create sysfs-memory-fix.conf with the following config:

# /etc/tmpfiles.d/sysfs-memory-fix.conf
w- /sys/kernel/mm/lru_gen/enabled - - - - 0

@pelwell Thanks for the suggestion, I will try that shortly.

gavinbeatty commented 1 year ago

@pelwell Hard to say conclusively, but I think that is a further improvement. With the CMA change, I have experienced one crash, originating in sendto again -- but just one. I can't say for certain it had an effect given I've only let it run for an hour each with "mm/lru_gen disabled" and "mm/lru_gen disabled + cma-320" and might have just been unlucky with the former.

And just to be clear, I originally had dtoverlay=vc4-kms-v3d; now dtoverlay=vc4-kms-v3d,cma-320.

I'll ask the archlinuxarm maintainers if all this information points to any distro-specific issues. I'm not sure I'll be able to try a fresh repro in RPiOS soon, but I'm happy to try more suggestions with my existing install.

gavinbeatty commented 1 year ago

By the way, what is the default linux kernel version for RPiOS right now? Is it < 6.1.x?

graysky2 commented 1 year ago

@popcornmix - I maintain the linux-rpi kernel for Arch ARM. The config is RPiOS's bcm2711_defconfig target, see here. I then append some options to that before running make oldconfig to harmonize/setup the distro config. This was a suggestion by @pelwell, see here.

Those are the only changes vs. RPiOS's bcm2711_defconfig with the exception of enabling CONFIG_BCM2835_THERMAL=y which cannot be set by the method. It is set after by call to scripts/config as you see here. In any case, I do not believe anything in archarm.diffconfig would be driving OOM triggers.

MGLRU is enabled on Arch ARM/armv7h and aarch64:

% cat /sys/kernel/mm/lru_gen/enabled 
0x0001

% zgrep -i lru /proc/config.gz 
CONFIG_LRU_GEN=y
CONFIG_LRU_GEN_ENABLED=y
# CONFIG_LRU_GEN_STATS is not set
CONFIG_LRU_CACHE=m

gavinbeatty commented 1 year ago

In the new "mm/lru_gen disabled + cma-320" configuration I have a novel __alloc_pages failure:

Mar 22 21:10:35 dill earlyoom[361]: mem avail:  3056 of  3836 MiB (79.66%), swap free: 16373 of 16383 MiB (99.94%)
Mar 22 21:11:57 dill kernel: ksoftirqd/0: page allocation failure: order:0, mode:0xa20(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
Mar 22 21:11:57 dill kernel: CPU: 0 PID: 13 Comm: ksoftirqd/0 Tainted: P         C O       6.1.20-1-rpi-ARCH #1
Mar 22 21:11:57 dill kernel: Hardware name: BCM2711
Mar 22 21:11:57 dill kernel:  unwind_backtrace from show_stack+0x18/0x1c
Mar 22 21:11:57 dill kernel:  show_stack from dump_stack_lvl+0x90/0xac
Mar 22 21:11:57 dill kernel:  dump_stack_lvl from warn_alloc+0x110/0x1b0
Mar 22 21:11:57 dill kernel:  warn_alloc from __alloc_pages+0xef8/0x1044
Mar 22 21:11:57 dill kernel:  __alloc_pages from new_slab+0x384/0x43c
Mar 22 21:11:57 dill kernel:  new_slab from ___slab_alloc+0x3e8/0xa0c
Mar 22 21:11:57 dill kernel:  ___slab_alloc from kmem_cache_alloc+0x470/0x4d4
Mar 22 21:11:57 dill kernel:  kmem_cache_alloc from __netdev_alloc_skb+0xe4/0x198
Mar 22 21:11:57 dill kernel:  __netdev_alloc_skb from bcmgenet_rx_refill+0x38/0x1ec
Mar 22 21:11:57 dill kernel:  bcmgenet_rx_refill from bcmgenet_rx_poll+0x1d4/0x468
Mar 22 21:11:57 dill kernel:  bcmgenet_rx_poll from __napi_poll.constprop.0+0x34/0x1d0
Mar 22 21:11:57 dill kernel:  __napi_poll.constprop.0 from net_rx_action+0x344/0x3b0
Mar 22 21:11:57 dill kernel:  net_rx_action from __do_softirq+0x15c/0x3dc
Mar 22 21:11:57 dill kernel:  __do_softirq from run_ksoftirqd+0x44/0x5c
Mar 22 21:11:57 dill kernel:  run_ksoftirqd from smpboot_thread_fn+0xc0/0x15c
Mar 22 21:11:57 dill kernel:  smpboot_thread_fn from kthread+0xd8/0xf4
Mar 22 21:11:57 dill kernel:  kthread from ret_from_fork+0x14/0x34
Mar 22 21:11:57 dill kernel: Exception stack(0xf085dfb0 to 0xf085dff8)
Mar 22 21:11:57 dill kernel: dfa0:                                     00000000 00000000 00000000 00000000
Mar 22 21:11:57 dill kernel: dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mar 22 21:11:57 dill kernel: dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
Mar 22 21:11:57 dill kernel: Mem-Info:
Mar 22 21:11:57 dill kernel: active_anon:1427 inactive_anon:93802 isolated_anon:0
                              active_file:84493 inactive_file:606758 isolated_file:1
                              unevictable:395 dirty:0 writeback:10
                              slab_reclaimable:14479 slab_unreclaimable:39039
                              mapped:34487 shmem:467 pagetables:937
                              sec_pagetables:0 bounce:0
                              kernel_misc_reclaimable:0
                              free:85635 free_pcp:209 free_cma:80196
Mar 22 21:11:57 dill kernel: Node 0 active_anon:5708kB inactive_anon:375208kB active_file:337972kB inactive_file:2427032kB unevictable:1580kB isolated(anon):0kB isolated(file):4kB mapped:137948kB dirty:0kB writeback:40kB shmem:1868kB writeback_tmp:0kB kernel_stack:2992kB pagetables:3748kB sec_pagetables:0kB all_unreclaimable? no
Mar 22 21:11:57 dill kernel: DMA free:321404kB boost:7272kB min:10504kB low:11312kB high:12120kB reserved_highatomic:0KB active_anon:0kB inactive_anon:40kB active_file:216kB inactive_file:0kB unevictable:0kB writepending:28kB present:786432kB managed:664228kB mlocked:0kB bounce:0kB free_pcp:836kB local_pcp:68kB free_cma:320784kB
Mar 22 21:11:57 dill kernel: lowmem_reserve[]: 0 0 3188 3188
Mar 22 21:11:57 dill kernel: DMA: 117*4kB (UEC) 21*8kB (EC) 26*16kB (UEC) 21*32kB (UEC) 13*64kB (C) 15*128kB (C) 12*256kB (C) 5*512kB (C) 4*1024kB (C) 4*2048kB (C) 73*4096kB (C) = 321404kB
Mar 22 21:11:57 dill kernel: 691807 total pagecache pages
Mar 22 21:11:57 dill kernel: 148 pages in swap cache
Mar 22 21:11:57 dill kernel: Free swap  = 16766460kB
Mar 22 21:11:57 dill kernel: Total swap = 16777212kB
Mar 22 21:11:57 dill kernel: 1012736 pages RAM
Mar 22 21:11:57 dill kernel: 816128 pages HighMem/MovableOnly
Mar 22 21:11:57 dill kernel: 30551 pages reserved
Mar 22 21:11:57 dill kernel: 81920 pages cma reserved
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0
Mar 22 21:11:57 dill kernel: SLUB: Unable to allocate memory on node -1, gfp=0xa20(GFP_ATOMIC)
Mar 22 21:11:57 dill kernel:   cache: skbuff_head_cache, object size: 184, buffer size: 192, default order: 0, min order: 0
Mar 22 21:11:57 dill kernel:   node 0: slabs: 48, objs: 1008, free: 0

I didn't notice any system degradation, and [ksoftirqd/0] is still present in ps aux, but seems related.

popcornmix commented 1 year ago

By the way, what is the default linux kernel version for RPiOS right now? Is it < 6.1.x?

An updated RPiOS bullseye is using the 6.1 kernel. And we are not seeing large numbers of OOM reports on RPiOS.

flipreverse commented 1 year ago

@popcornmix I'm running Raspbian 11 on armv7l, and seeing the same issue: The oom killer runs way to early. I can hardly clone the linux kernel soruces. I'm switching back to aarch64, and try that one.

terminet85 commented 1 year ago

Same story here. Cannot do btrbk backup anymore. OOM Killer comes up when btrfs send-receive is working.

popcornmix commented 1 year ago

does running echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled before doing the failing operation help?

terminet85 commented 1 year ago

does running echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled before doing the failing operation help?

Yeah that solved my OOM killer with btrbk.

Edit: To double-check I've reenabled lru_gen echoing 1, and re-run btrbk and worked too... So I cannot replicate OOM Killer with btrbk ever.

graysky2 commented 1 year ago

does running echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled

@popcornmix Does this translate into # CONFIG_LRU_GEN_ENABLED is not set in the config? Is that a good way to fix this bug at least on armv7h?

@terminet85 - are you comfortable building the kernel from the PKGBUILD? If so, would you try this and report back:

# assuming you have cloned https://github.com/archlinuxarm/PKGBUILDs
cd core/inux-rpi
sed -i 's/CONFIG_LRU_GEN_ENABLED=y/# CONFIG_LRU_GEN_ENABLED is not set/' config
updpkgsums
makepkg -srci

terminet85 commented 1 year ago

@graysky2 I don't building kernel I'm using raspbian 11

popcornmix commented 1 year ago

@popcornmix Does this translate into # CONFIG_LRU_GEN_ENABLED is not set in the config?

Yes

Is that a good way to fix this bug at least on armv7h?

I don't know if it's clear it's a bug. MGLRU changes the behaviour of a system in low memory conditions, and there are likely to be use cases that have preferable behaviour with this enabled or disabled.

There are definitely performance improvements available when this is enabled.

But, yes, changing that config setting should make the kernel go back to the previous behaviour (no MGLRU) and is something worth considering.

graysky2 commented 1 year ago

I don't building kernel I'm using raspbian 11

For those using Arch ARM, please update and report back. linux-rpi-6.1.21-2 for armv7h has MGLRU compiled in but disabled by default.

terminet85 commented 1 year ago

I don't building kernel I'm using raspbian 11

For those using Arch ARM, please update and report back. linux-rpi-6.1.21-2 for armv7h has MGLRU compiled in but disabled by default.

That is the same keeping this:

# cat  /sys/kernel/mm/lru_gen/enabled
0x0000

right?

toofishes commented 1 year ago

I'm seeing this as well. Is everyone that is affected by this on a 4 GB or greater RPi 4? I'm wondering if this is only triggered on higher memory setups. I saw this myself on a 4 GB Pi:

Mar 14 22:45:02 cork munin-node[17468]: 2023/03/14-22:45:02 CONNECT TCP Peer: "[::ffff:10.9.0.1]:54226" Local: "[::ffff:10.9.0.2]:4949"
Mar 14 22:45:04 cork kernel: find invoked oom-killer: gfp_mask=0xcd0(GFP_KERNEL|__GFP_RECLAIMABLE), order=0, oom_score_adj=0
Mar 14 22:45:04 cork kernel: CPU: 0 PID: 17513 Comm: find Tainted: G         C         6.1.16-3-rpi-ARCH #1
Mar 14 22:45:04 cork kernel: Hardware name: BCM2711
Mar 14 22:45:04 cork kernel:  unwind_backtrace from show_stack+0x18/0x1c
Mar 14 22:45:04 cork kernel:  show_stack from dump_stack_lvl+0x90/0xac
Mar 14 22:45:04 cork kernel:  dump_stack_lvl from dump_header+0x54/0x1fc
Mar 14 22:45:04 cork kernel:  dump_header from oom_kill_process+0x23c/0x248
Mar 14 22:45:04 cork kernel:  oom_kill_process from out_of_memory+0x218/0x34c
Mar 14 22:45:04 cork kernel:  out_of_memory from __alloc_pages+0xa98/0x1044
Mar 14 22:45:04 cork kernel:  __alloc_pages from new_slab+0x384/0x43c
...
Mar 14 22:45:04 cork kernel: Mem-Info:
Mar 14 22:45:04 cork kernel: active_anon:99244 inactive_anon:243 isolated_anon:0
                              active_file:39172 inactive_file:68153 isolated_file:0
                              unevictable:0 dirty:4 writeback:0
                              slab_reclaimable:5594 slab_unreclaimable:7341
                              mapped:24772 shmem:219 pagetables:1861
                              sec_pagetables:0 bounce:0
                              kernel_misc_reclaimable:0
                              free:741220 free_pcp:144 free_cma:129007
Mar 14 22:45:04 cork kernel: Node 0 active_anon:396976kB inactive_anon:972kB active_file:156688kB inactive_file:272612kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:99088kB dirty:16kB writeback:0kB shmem:876kB writeback_tmp:0kB kernel_stack:2104kB pagetables:7444kB sec_pagetables:0kB all_unreclaimable? yes
Mar 14 22:45:04 cork kernel: DMA free:520420kB boost:4096kB min:7328kB low:8136kB high:8944kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:612kB inactive_file:76868kB unevictable:0kB writepending:16kB present:786432kB managed:664308kB mlocked:0kB bounce:0kB free_pcp:576kB local_pcp:380kB free_cma:516028kB
Mar 14 22:45:04 cork kernel: lowmem_reserve[]: 0 0 3136 3136
Mar 14 22:45:04 cork kernel: DMA: 423*4kB (UEC) 153*8kB (UEC) 74*16kB (UEC) 21*32kB (UEC) 5*64kB (C) 2*128kB (C) 2*256kB (C) 1*512kB (C) 0*1024kB 1*2048kB (C) 125*4096kB (C) = 520420kB
Mar 14 22:45:04 cork kernel: 107545 total pagecache pages
Mar 14 22:45:04 cork kernel: 0 pages in swap cache
Mar 14 22:45:04 cork kernel: Free swap  = 262140kB
Mar 14 22:45:04 cork kernel: Total swap = 262140kB
Mar 14 22:45:04 cork kernel: 999424 pages RAM
Mar 14 22:45:04 cork kernel: 802816 pages HighMem/MovableOnly
Mar 14 22:45:04 cork kernel: 30531 pages reserved
Mar 14 22:45:04 cork kernel: 131072 pages cma reserved
...
Mar 14 22:45:04 cork kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=homebridge,pid=389,uid=1001
Mar 14 22:45:04 cork kernel: Out of memory: Killed process 389 (homebridge) total-vm:249024kB, anon-rss:77820kB, file-rss:43336kB, shmem-rss:0kB, UID:1001 pgtables:1080kB oom_score_adj:200
Mar 14 22:45:04 cork kernel: systemd-journal invoked oom-killer: gfp_mask=0xcd0(GFP_KERNEL|__GFP_RECLAIMABLE), order=0, oom_score_adj=-250
Mar 14 22:45:04 cork kernel: CPU: 3 PID: 200 Comm: systemd-journal Tainted: G         C         6.1.16-3-rpi-ARCH #1
Mar 14 22:45:04 cork kernel: Hardware name: BCM2711
Mar 14 22:45:04 cork kernel:  unwind_backtrace from show_stack+0x18/0x1c
Mar 14 22:45:04 cork kernel:  show_stack from dump_stack_lvl+0x90/0xac
Mar 14 22:45:04 cork kernel:  dump_stack_lvl from dump_header+0x54/0x1fc
Mar 14 22:45:04 cork kernel:  dump_header from oom_kill_process+0x23c/0x248
Mar 14 22:45:04 cork kernel:  oom_kill_process from out_of_memory+0x218/0x34c
Mar 14 22:45:04 cork kernel:  out_of_memory from __alloc_pages+0xa98/0x1044
Mar 14 22:45:04 cork kernel:  __alloc_pages from new_slab+0x384/0x43c
...

rvk01 commented 1 year ago

Is everyone that is affected by this on a 4 GB or greater RPi 4?

Yes, for me on a 8 GB RPI 4. I went back to the 64 bit kernel where this is not a problem. But the 64 bit kernel in combination with 32 bit userland gives problems with PHP/PDO_Firebird for which a fix is pending. But now I also have problems with docker (memory crashes during startup of a blanc nginx container). I want to go back to the 32 bit kernel but the OOM-killer was so aggressive that of my 8GB it killed ALL processes even upto where there was only 100MB used. It just kept on killing everything.

Maybe I should just bite the bullet and go full 64 bit.

gavinbeatty commented 1 year ago

@graysky2 I deleted my armv7h instance's tmpfiles.d config, and upgraded my aarch64 and armv7h instances:

armv7h: cat /sys/kernel/mm/lru_gen/enabled shows 0x0000 as desired/expected, no OOM issues so far ✔️
aarch64: cat /sys/kernel/mm/lru_gen/enabled shows 0x0001 as desired/expected, never had OOM issues there, so that seems best ✔️

Thanks!

@toofishes 4GB model, yes. It would be interesting if anyone with a <4GB model could run sudo sh -c "while sleep 1 ; do find / -amin -1 ; done" for a while to see if it's possible to trigger it there. But I would guess the <4GB models are far less popular, so absence of reports might not mean much.

@popcornmix There are at least two reports of OOM issues on RPiOS/Raspbian here now. I think it's likely that there is an unsigned overflow issue in the MGLRU.

terminet85 commented 1 year ago

that is affected by this on a 4 GB or greater RPi 4

Mine is 8Gb

Keeping /sys/kernel/mm/lru_gen/enabled to 0 seems like OOM Killer doesn't come up. I'll go ahead and report.

rajo commented 1 year ago

RPi4 user with 4GB here, running bullseye. After the latest update, 6.1.19-v7l+ was installed and I am victim of the oom-killer as well. As I use the RPi as a NAS and smart home central and many other things, oom basically behaves like a shotgun and randomly kills services and renders it pretty unusable. Switching to aarch64 isn't an option either, as - for whatever reason - the kernel-headers aren't available for it, so I cannot compile neccessary addtional modules for my smart home stuff.

For now, I'm trying with disabling the lru__gen stuff, fingers crossed that this will help. (But I would appreciate a more proper fix here)

terminet85 commented 1 year ago

For now, I'm trying with disabling the lru__gen stuff, fingers crossed that this will help. (But I would appreciate a more proper fix here)

Rock solid from almost few days with lru_gen disabled. MGLRU is pretty involved...

flipreverse commented 1 year ago

RPi4 user with 4GB here, running bullseye. After the latest update, 6.1.19-v7l+ was installed and I am victim of the oom-killer as well. As I use the RPi as a NAS and smart home central and many other things, oom basically behaves like a shotgun and randomly kills services and renders it pretty unusable. Switching to aarch64 isn't an option either, as - for whatever reason - the kernel-headers aren't available for it, so I cannot compile neccessary addtional modules for my smart home stuff.

For now, I'm trying with disabling the lru__gen stuff, fingers crossed that this will help. (But I would appreciate a more proper fix here)

Providing those headers isn't a hard task. But annoying for sure. I can assist you if you want.

@popcornmix I have a 4GB Pi4 as well. I'll give the workaround a try, and report back.

flipreverse commented 1 year ago

@popcornmix I do not have any long-term data yet. However, with this workaround (disabled lru_gen) my system is stable for at least 15 minutes. Without disabling lru_gen, the OOM killer kicks in right after startup when my Home Assistant instance is started.

terminet85 commented 1 year ago

Switching to aarch64 isn't an option either, as - for whatever reason - the kernel-headers aren't available for it, so I cannot compile neccessary addtional modules for my smart home stuff.

raspberrypi-kernel-headers matches the userland arch (dpkg --print-architecture) You need raspberrypi-kernel-headers_1.20230317-1_arm64.deb

popcornmix commented 1 year ago

I note that some users have reported running 6.1.19-v7l+ (i.e. 32-bit kernel) and gettin OOM.

Can users with this issue confirm if you are running: 32-bit distro + 32-bit kernel 32-bit distro + 64-bit kernel 64-bit distro + 64-bit kernel

If the OOM issue is occurring on 32-bit kernel, does it still occur with 64-bit kernel?

flipreverse commented 1 year ago

I note that some users have reported running 6.1.19-v7l+ (i.e. 32-bit kernel) and gettin OOM.

Can users with this issue confirm if you are running: 32-bit distro + 32-bit kernel 32-bit distro + 64-bit kernel 64-bit distro + 64-bit kernel

If the OOM issue is occurring on 32-bit kernel, does it still occur with 64-bit kernel?

I'm running a 32-bit distro. Using a 32-bit kernel, the OOM issue occurs. The issue does not occur when running the 64-bit kernel.

gavinbeatty commented 1 year ago

My report was 32-bit distro/userland + 32-bit kernel with frequent cascading OOMs when MGLRU is enabled.

file /usr/bin/uname
/usr/bin/uname: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=948aec51a4b77a3a8bd22538d111c11a659d822c, for GNU/Linux 3.2.0, stripped
uname -a
Linux dill 6.1.21-2-rpi-ARCH #1 SMP Sat Mar 25 01:25:04 MDT 2023 armv7l GNU/Linux

Another instance of mine is running 64-bit distro/userland + 64-bit kernel without frequent cascading OOMs even when MGLRU is enabled.

 file /usr/bin/uname
/usr/bin/uname: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=b81d64f754a56431f3c1dca1e9e1ee7fdc0005d6, for GNU/Linux 3.7.0, stripped
uname -a
Linux stone 6.1.21-2-rpi-ARCH #1 SMP PREEMPT Sat Mar 25 01:11:01 MDT 2023 aarch64 GNU/Linux

rvk01 commented 1 year ago

On raspberry pi 4 8GB. 32-bit distro + 32-bit kernel = OOM-killer kills everything to 100MB 32-bit distro + 64-bit kernel = no issues 64-bit distro + 64-bit kernel = don't know

I did have an issue earlier with "32-bit distro + 64-bit kernel" but that's resolved (PDO Firebird driver issues). The Docker issues came from earlier attempts to disable the OOM-killer but it's working fine now.

popcornmix commented 1 year ago

@graysky2 does arch provide a way of installing the 64-bit kernel on 32-bit image? On RPiOS we provide both kernels and arm_bit64=0/arm_bit64=0 can specify which to use. It would be useful to confirm if OOM is only an issue for 32-bit kernel use (regardless of whether image is 32 or 64-bit) when we are considering what the best defaults are on each version of kernel.

graysky2 commented 1 year ago

@popcornmix - I haven't tried that before. @kmihelich - is this possible?

kmihelich commented 1 year ago

@popcornmix - I haven't tried that before. @kmihelich - is this possible?

You can try with: pacman -U --arch aarch64 linux-rpi-6.1.21-2-aarch64.pkg.tar.xz

josepmaria79 commented 1 year ago

I upgraded to latest linux-rpi-6.1.21-2-armv7h.pkg.tar.xz and oom kills triggered aggressively the same way as previous 6.1 kernels. Instead, I tried with linux-rpi-5.15.84-1-armv7h.pkg.tar.xz from 2022 and oom kills triggered anyway, but not so often. Finally, I tried with linux-rpi-5.10.83-1-armv7h.pkg.tar.xz from 2021 and oom kills disappeared completely!

terminet85 commented 1 year ago

If the OOM issue is occurring on 32-bit kernel, does it still occur with 64-bit kernel?

Switched to 64bit kernel with a 32bit userland and OOM Killer issue doesn't come up. Instead on kernel/userland 32 bit, without lru_gen occurs.

I switched to 64bit kernel 'cause would be the default setup to DietPI distro.

josepmaria79 commented 1 year ago

@popcornmix - I haven't tried that before. @kmihelich - is this possible?

You can try with: pacman -U --arch aarch64 linux-rpi-6.1.21-2-aarch64.pkg.tar.xz

In that case, after that, should I change something in config.txt and pacman.conf That is: is it needed to specify 64bit at boot? And: will future upgrades keep the aarch64 for this package?

josepmaria79 commented 1 year ago

I upgraded to latest linux-rpi-6.1.21-2-armv7h.pkg.tar.xz and oom kills triggered aggressively the same way as previous 6.1 kernels. Instead, I tried with linux-rpi-5.15.84-1-armv7h.pkg.tar.xz from 2022 and oom kills triggered anyway, but not so often. Finally, I tried with linux-rpi-5.10.83-1-armv7h.pkg.tar.xz from 2021 and oom kills disappeared completely!

Interesting that issue happens with old 5.15 but not with 5.10 Why?

rajo commented 1 year ago

32bit/32bit here - after 24 hours I can confirm that disabling lru is helping as OOM didn't occur any more.

popcornmix commented 1 year ago

Based on feedback here, we've disabled LRU_GEN_ENABLED for 32-bit kernel, but left it enabled for 64-bit, as it seemed the reports of OOM were predominantly using 32-bit kernels.

We'll continue to keep an eye for OOM reports, and if there are any changes that sound like MGLRU on 32-bit behaviour may be improved and can be reconsidered.

To clarify, once the source changes reach built kernels (rpi-update build is in progress): On 32-bit kernels MGLRU is disabled by default and can be enabled manually with

echo 1 | sudo tee /sys/kernel/mm/lru_gen/enabled

On 64-bit kernels MGLRU is enabled by default and can be disabled manually with

echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled

lategoodbye commented 1 year ago

Anyone able to reproduce the OOM issue on a Raspberry Pi 3 (Plus) with LRU_GEN_ENABLED?

Since this issue seems specific for Raspberry Pi 4 32 bit, maybe the issue is related to CONFIG_ARM_LPAE.

graysky2 commented 1 year ago

Thanks for the support, all. For the record, Arch ARM has it compiled in but not-enabled by default for armv7h. For aarch64, it is both compiled in and enabled.

pelwell commented 1 year ago

Any more datapoints would be useful. Please state model, memory, 32- or 64-bit, OOM yes or no.

graysky2 commented 1 year ago

RPi4B/8G, 64-bit, no OOM (ArchARM) RPi4B/4G, 64-bit, no OOM (ArchARM) RPi4B/2G, 64-bit, no OOM (ArchARM and OpenWRT)

popcornmix commented 1 year ago

apt kernel now has MGLRU disabled by default on 32-bit. Once the dust settles we might try enabling it again on non-pi4 32-bit (as the suggestion that MGLRU+LPAE may be the bad combination seems plausible).

yuzhaogoogle commented 1 year ago

Hi all,

Based on the logs in this thread, e.g.,

Mar 22 09:34:30 dill kernel: 1012736 pages RAM
Mar 22 09:34:30 dill kernel: 816128 pages HighMem/MovableOnly

It seems the 32-bit kernel was built with the default CONFIG_ VMSPLIT_3G. Long story short: the OOM kills were caused by the depletion of the cold memory in the DMA zone because its size was too small. On the other hand, the highmem zone was too large.

This can cause unnecessary OOM kills for both MGLRU and non-MGLRU configurations on 32-bit platforms. But MGLRU is more prone to it (OpenWrt reported this too).

Is it possible for Pi to switch to CONFIG_ VMSPLIT_2G? @popcornmix @pelwell

And please feel free to tag me. Thanks.

pelwell commented 1 year ago

@yuzhaogoogle As you may know, we use CONFIG_ VMSPLIT_2G on Pi 2 and Pi 3, where limiting the application address space to 2GB (as it does) has less effect because they can only address 1GB of RAM in total. Pi 4s exist with 4 and 8GB of RAM, so a 2GB address space limit will prevent some large applications from running that would otherwise have worked. I welcome @popcornmix's input here, but the preferred route forward has to be moving to a 64-bit kernel with none of these addressing limitation (but a few compatibility issues for some ioctl interfaces when running with a 32-bit userspace, as we're discovering).

popcornmix commented 1 year ago

Yes, 64-bit for Pi4 is the preferred option

If someone manually switches to 32-bit kernel on Pi4 (for compatibility with code or drivers that handle mixed userland/kernel badly) then it seems there are two options (that probably avoid OOM): CONFIG_ VMSPLIT3G with no MGLRU CONFIG VMSPLIT_2G with MGLRU

As this setup is likely to affect a very small percentage of users it may not be too critical, and the former option has the advantage of being well tested.

This sounds like further evidence that the OOM issue here doesn't affect Pi0-3, so I'm still keen to enable MGLRU again on those platforms (in rpi-update and we'll keep an eye out for issues reported).

yuzhaogoogle commented 1 year ago

Pi 4s exist with 4 and 8GB of RAM, so a 2GB address space limit will prevent some large applications from running that would otherwise have worked.

Yes, this makes senses.

I welcome @popcornmix's input here, but the preferred route forward has to be moving to a 64-bit kernel with none of these addressing limitation (but a few compatibility issues for some ioctl interfaces when running with a 32-bit userspace, as we're discovering).

Agreed.

raspberrypi / linux