search

jiangcuo / Proxmox-Arm64

Proxmox VE & PBS unofficial arm64 version
GNU Affero General Public License v3.0
387 stars 41 forks source link

pveproxy oom on ps225-8G #31

Closed jiangcuo closed 1 year ago

jiangcuo commented 1 year ago 
[   64.647345] SLUB: Unable to allocate memory on node -1, gfp=0x15004c0(GFP_KERNEL_ACCOUNT|__GFP_RETRY_MAYFAIL)
[   64.647347]   cache: kmalloc-256(351:pveproxy.service), object size: 256, buffer size: 256, default order: 1, min order: 0
[   64.647348]   node 0: slabs: 3, objs: 96, free: 0
[   65.342797] pveproxy worker invoked oom-killer: gfp_mask=0x14080c0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),  order=0, oom_score_adj=0
[   65.342801] pveproxy worker cpuset=/ mems_allowed=0
[   65.342806] CPU: 7 PID: 7318 Comm: pveproxy worker Tainted: G           O    4.14.79 #1
[   65.342807] Hardware name: BRCM BRCM-SR/BRCM-SR, BIOS 0.1 Feb 22 2019
[   65.342809] Call trace:
[   65.342819] [<ffff000008089a98>] dump_backtrace+0x0/0x398
[   65.342821] [<ffff000008089e44>] show_stack+0x14/0x20
[   65.342825] [<ffff00000883c0fc>] dump_stack+0x90/0xb4
[   65.342828] [<ffff000008171abc>] dump_header+0x8c/0x1d0
[   65.342830] [<ffff000008170f1c>] oom_kill_process+0x274/0x520
[   65.342831] [<ffff0000081717ec>] out_of_memory+0xfc/0x2d0
[   65.342833] [<ffff000008176988>] __alloc_pages_nodemask+0xb10/0xba0
[   65.342835] [<ffff00000819ee14>] __handle_mm_fault+0xe04/0x10f8
[   65.342836] [<ffff00000819f1d0>] handle_mm_fault+0xc8/0x170
[   65.342839] [<ffff000008855938>] do_page_fault+0x1f0/0x418
[   65.342840] [<ffff000008855b9c>] do_translation_fault+0x3c/0x44
[   65.342841] [<ffff000008081434>] do_mem_abort+0x3c/0xa0
[   65.342842] [<ffff000008081520>] do_el0_ia_bp_hardening+0x38/0x98
[   65.342843] Exception stack(0xffff00000af83ec0 to 0xffff00000af84000)
[   65.342845] 3ec0: 0000ffffb8e86c80 0000000000000000 0000ffffb8c64240 51d87ebacc46acae
[   65.342846] 3ee0: 0000aaab05b78380 0000000000000002 0000000000000007 0000000000000000
[   65.342848] 3f00: 0000ffffba7d6640 00000000ffffffff 0000000000000000 0000000000000038
[   65.342849] 3f20: 0000ffffb8e88c80 0000000000000000 0000ffffd353ec11 0000000000000000
[   65.342850] 3f40: 0000000000000000 0000000000000000 0000000000000011 0000aaab05ba0b80
[   65.342851] 3f60: 0000ffffba6b1000 00000000000001aa 0000ffffba6af000 0000ffffba6b2068
[   65.342852] 3f80: 0000000000000000 0000ffffba6af000 0000ffffba7d6638 0000ffffba6b06b8
[   65.342853] 3fa0: 000000000000000c 0000ffffd353f000 0000ffffba57678c 0000ffffd353f000
[   65.342854] 3fc0: 0000ffffb8c64240 0000000060000000 000000000000000f 00000000ffffffff
[   65.342855] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   65.342857] [<ffff000008083688>] el0_ia+0x18/0x1c
[   65.342858] Mem-Info:
[   65.342862] active_anon:261137 inactive_anon:7078 isolated_anon:0
                active_file:356 inactive_file:364 isolated_file:0
                unevictable:0 dirty:0 writeback:0 unstable:0
                slab_reclaimable:4533 slab_unreclaimable:13344
                mapped:7659 shmem:7172 pagetables:1634 bounce:0
                free:21266 free_pcp:564 free_cma:12805
[   65.342866] Node 0 active_anon:1044548kB inactive_anon:28312kB active_file:1424kB inactive_file:1456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:30636kB dirty:0kB writeback:0kB shmem:28688kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 4096kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[   65.342870] DMA free:63616kB min:8616kB low:10768kB high:12920kB active_anon:1044548kB inactive_anon:28312kB active_file:1512kB inactive_file:1504kB unevictable:0kB writepending:0kB present:2063232kB managed:1452560kB mlocked:0kB kernel_stack:2160kB pagetables:6500kB bounce:0kB free_pcp:2256kB local_pcp:200kB free_cma:51220kB
[   65.342870] lowmem_reserve[]: 0 5995 5995
[   65.342874] Normal free:21448kB min:36436kB low:45544kB high:54652kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:6287360kB managed:6139560kB mlocked:0kB kernel_stack:1168kB pagetables:36kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[   65.342874] lowmem_reserve[]: 0 0 0
[   65.342876] DMA: 729*4kB (UMEC) 252*8kB (UMEC) 86*16kB (UMEC) 45*32kB (UEC) 27*64kB (UEC) 9*128kB (UEC) 1*256kB (E) 1*512kB (U) 1*1024kB (C) 1*2048kB (E) 12*4096kB (C) = 63620kB
[   65.342887] Normal: 116*4kB (UMH) 135*8kB (UMEH) 78*16kB (UME) 55*32kB (UMEH) 28*64kB (UMEH) 8*128kB (UME) 3*256kB (MEH) 4*512kB (MEH) 1*1024kB (E) 1*2048kB (M) 2*4096kB (M) = 21448kB
[   65.342898] Node 0 hugepages_total=3072 hugepages_free=3072 hugepages_surp=0 hugepages_size=2048kB
[   65.342899] 7930 total pagecache pages
[   65.342899] 2087648 pages RAM
[   65.342900] 0 pages HighMem/MovableOnly
[   65.342900] 189618 pages reserved
[   65.342901] 16384 pages cma reserved
[   65.342901] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[   65.342904] [ 1213]     0  1213     8175      229      16       4        0          -250 systemd-journal
[   65.342906] [ 1804]     0  1804     5615     1161      10       3        0         -1000 systemd-udevd
[   65.342907] [ 6352]   105  6352     2020      110       8       3        0             0 rpcbind
[   65.342909] [ 6354]   103  6354     1953      133       7       4        0          -900 dbus-daemon
[   65.342911] [ 6358]     0  6358    37721       42       9       4        0         -1000 lxcfs
[   65.342912] [ 6360]     0  6360    69177      565      15       5        0             0 pve-lxc-syscall
[   65.342913] [ 6366]     0  6366    55219      272      12       4        0             0 rsyslogd
[   65.342915] [ 6367]     0  6367     2557      253       9       3        0             0 smartd
[   65.342916] [ 6368]     0  6368     3590      206      11       4        0             0 systemd-logind
[   65.342917] [ 6372]     0  6372     1007       34       6       4        0             0 qmeventd
[   65.342918] [ 6373]     0  6373      451       15       5       3        0         -1000 watchdog-mux
[   65.342920] [ 6962]     0  6962     2996      160      10       4        0             0 lxc-monitord
[   65.342921] [ 6969]     0  6969      624       26       5       3        0             0 agetty
[   65.342922] [ 6972]     0  6972     2742      166       9       4        0             0 login
[   65.342923] [ 7233]     0  7233    43790      271      23       4        0             0 rrdcached
[   65.342924] [ 7248]   108  7248     4702      240      13       3        0             0 exim4
[   65.342926] [ 7251]     0  7251   158458     8075      67       5        0             0 pmxcfs
[   65.342927] [ 7257]     0  7257     1395       52       7       3        0             0 cron
[   65.342928] [ 7269]     0  7269     4034      336      13       3        0             0 systemd
[   65.342929] [ 7270]     0  7270    42170      758      18       5        0             0 (sd-pam)
[   65.342931] [ 7285]     0  7285     1461      140       7       3        0             0 bash
[   65.342932] [ 7289]     0  7289     3407      241      11       4        0         -1000 sshd
[   65.342933] [ 7290]     0  7290    36525    21872      67       4        0             0 pve-firewall
[   65.342934] [ 7291]     0  7291    35166    20539      65       4        0             0 pvestatd
[   65.342936] [ 7305]     0  7305    54273    31163      90       4        0             0 pvedaemon
[   65.342937] [ 7306]     0  7306    54400    31230      91       4        0             0 pvedaemon worke
[   65.342938] [ 7307]     0  7307    54396    31230      91       4        0             0 pvedaemon worke
[   65.342939] [ 7308]     0  7308    54396    31230      91       4        0             0 pvedaemon worke
[   65.342940] [ 7313]     0  7313    51240    25233      82       3        0             0 pve-ha-crm
[   65.342942] [ 7314]     0  7314     3993      309      11       4        0             0 sshd
[   65.342943] [ 7316]    33  7316    54621    31528      91       4        0             0 pveproxy
[   65.342944] [ 7317]    33  7317    54833    31653      92       4        0             0 pveproxy worker
[   65.342945] [ 7318]    33  7318    54834    31856      92       4        0             0 pveproxy worker
[   65.342947] [ 7319]    33  7319    54833    31927      92       4        0             0 pveproxy worker
[   65.342948] [ 7325]    33  7325    19039    12350      39       4        0             0 spiceproxy
[   65.342949] [ 7326]    33  7326    19187    12416      39       4        0             0 spiceproxy work
[   65.342950] [ 7327]     0  7327    51083    25088      80       3        0             0 pve-ha-lrm
[   65.342952] [ 7333]     0  7333    49886    25689      82       4        0             0 pvescheduler
[   65.342953] [ 7356]     0  7356      988      132       6       4        0             0 bash
[   65.342954] [ 7374]     0  7374     1680       58       8       3        0             0 systemctl
[   65.342955] [ 7375]     0  7375     3468      148      10       4        0             0 systemd-tty-ask
[   65.342956] [ 7376]     0  7376    54564    31470     100       4        0             0 pveproxy
[   65.342958] Out of memory: Kill process 7319 (pveproxy worker) score 16 or sacrifice child
[   65.342964] Killed process 7319 (pveproxy worker) total-vm:219332kB, anon-rss:126612kB, file-rss:1096kB, shmem-rss:0kB
[   65.357145] oom_reaper: reaped process 7319 (pveproxy worker), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[   66.174295] pveproxy worker cpuset=/ mems_allowed=0
[   66.179335] CPU: 6 PID: 7317 Comm: pveproxy worker Tainted: G           O    4.14.79 #1
[   66.187592] Hardware name: BRCM BRCM-SR/BRCM-SR, BIOS 0.1 Feb 22 2019
[   66.194235] Call trace:
[   66.196765] [<ffff000008089a98>] dump_backtrace+0x0/0x398
[   66.202334] [<ffff000008089e44>] show_stack+0x14/0x20
[   66.207546] [<ffff00000883c0fc>] dump_stack+0x90/0xb4
[   66.212757] [<ffff000008175d78>] warn_alloc+0xe8/0x180
[   66.218056] [<ffff00000817696c>] __alloc_pages_nodemask+0xaf4/0xba0
[   66.224522] [<ffff0000081bf060>] new_slab+0x360/0x428
[   66.229733] [<ffff0000081c0ecc>] ___slab_alloc.constprop.33+0x254/0x338
[   66.236556] [<ffff0000081c147c>] kmem_cache_alloc+0x124/0x160
[   66.242485] [<ffff000008720f6c>] __alloc_skb+0x3c/0x168
[   66.247874] [<ffff00000872281c>] alloc_skb_with_frags+0x54/0x218
[   66.254070] [<ffff000008719f08>] sock_alloc_send_pskb+0x1e8/0x200
[   66.260356] [<ffff0000087fb7d0>] unix_dgram_sendmsg+0xe8/0x5e0
[   66.266372] [<ffff000008717260>] sock_sendmsg+0x18/0x30
[   66.271760] [<ffff0000087172e4>] sock_write_iter+0x6c/0xb0
[   66.277418] [<ffff0000081db3d4>] __vfs_write+0xac/0x118
[   66.282807] [<ffff0000081db634>] vfs_write+0xa4/0x1b0
[   66.288016] [<ffff0000081db8f8>] SyS_write+0x48/0xb0
[   66.293136] Exception stack(0xffff00000ae03ec0 to 0xffff00000ae04000)
[   66.299779] 3ec0: 0000000000000007 0000aaab0a9d8640 0000000000000031 0000ffffba4f5800
[   66.307856] 3ee0: 0000aaaad9fda1d4 0000000010004403 0000000000000068 0000aaab035ef398
[   66.315933] 3f00: 0000000000000040 0000aaab0a9e8600 0000000000000000 0000000000000000
[   66.324010] 3f20: 0000aaab035ef398 0000aaab035ef3b0 0000000000000000 0000000000000038
[   66.332087] 3f40: 0000aaaada1c0408 0000ffffba6c7430 0000000000000000 0000000000000007
[   66.340164] 3f60: 0000aaab0a9e8600 0000000000000000 0000ffffba4f5f00 0000aaab0a9e8618
[   66.348241] 3f80: 0000aaab036086b0 0000000000000031 0000aaab0a9e8618 0000000000000007
[   66.356318] 3fa0: 0000000000000000 0000ffffd353ee10 0000aaaad9fa2ac8 0000ffffd353ee10
[   66.364394] 3fc0: 0000ffffba6c7460 0000000060000000 0000000000000007 0000000000000040
[   66.372471] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   66.380549] [<ffff000008083ac0>] el0_svc_naked+0x34/0x38
[   66.386035] warn_alloc_show_mem: 1 callbacks suppressed
[   66.386036] Mem-Info:
[   66.393775] active_anon:210887 inactive_anon:7078 isolated_anon:0
                active_file:202 inactive_file:2662 isolated_file:0
                unevictable:0 dirty:21 writeback:0 unstable:0
                slab_reclaimable:4557 slab_unreclaimable:13370
                mapped:8758 shmem:7172 pagetables:1350 bounce:0
                free:69205 free_pcp:604 free_cma:13948
[   66.428385] Node 0 active_anon:843548kB inactive_anon:28312kB active_file:808kB inactive_file:10648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:35032kB dirty:84kB writeback:0kB shmem:28688kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 4096kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[   66.457078] DMA free:255496kB min:8616kB low:10768kB high:12920kB active_anon:843548kB inactive_anon:28312kB active_file:808kB inactive_file:10748kB unevictable:0kB writepending:92kB present:2063232kB managed:1452560kB mlocked:0kB kernel_stack:2112kB pagetables:5364kB bounce:0kB free_pcp:2284kB local_pcp:552kB free_cma:55792kB
[   66.486932] lowmem_reserve[]: 0 5995 5995
[   66.491074] Normal free:21324kB min:36436kB low:45544kB high:54652kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:6287360kB managed:6139560kB mlocked:0kB kernel_stack:1168kB pagetables:36kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB
[   66.518957] lowmem_reserve[]: 0 0 0
[   66.522559] DMA: 4216*4kB (UMEC) 2983*8kB (UMEC) 2048*16kB (UMEC) 1353*32kB (UMEC) 775*64kB (UMEC) 162*128kB (UMEC) 37*256kB (MEC) 9*512kB (UM) 1*1024kB (C) 2*2048kB (EC) 12*4096kB (C) = 255480kB
[   66.540512] Normal: 85*4kB (UMH) 135*8kB (UMEH) 78*16kB (UME) 55*32kB (UMEH) 28*64kB (UMEH) 8*128kB (UME) 3*256kB (MEH) 4*512kB (MEH) 1*1024kB (E) 1*2048kB (M) 2*4096kB (M) = 21324kB
[   66.557299] Node 0 hugepages_total=3072 hugepages_free=3072 hugepages_surp=0 hugepages_size=2048kB
[   66.566543] 10119 total pagecache pages
[   66.570501] 2087648 pages RAM
[   66.573565] 0 pages HighMem/MovableOnly
[   66.577523] 189618 pages reserved
[   66.580943] 16384 pages cma reserved
[   66.584634] SLUB: Unable to allocate memory on node -1, gfp=0x15004c0(GFP_KERNEL_ACCOUNT|__GFP_RETRY_MAYFAIL)
[   66.594864]   cache: kmalloc-256(351:pveproxy.service), object size: 256, buffer size: 256, default order: 1, min order: 0
[   66.606256]   node 0: slabs: 3, objs: 96, free: 0
jiangcuo commented 1 year ago

ps225 default boot with hugepages.