search

rumca-js / Django-link-archive

Self-hostable link database
https://rumca-js.github.io
GNU General Public License v3.0
72 stars 4 forks source link

Celery "out of memory" bug #395

Closed rumca-js closed 1 month ago

rumca-js commented 1 month ago

Observed via "dmesg"

[89075.584285] Out of memory: Killed process 116013 (celery) total-vm:1452108kB, anon-rss:1217700kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:2584kB oom_score_adj:0 [89221.136738] sd 1:0:0:0: [sdc] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s [89221.136758] sd 1:0:0:0: [sdc] tag#0 Sense Key : 0x5 [current] [89221.136766] sd 1:0:0:0: [sdc] tag#0 ASC=0x20 ASCQ=0x0 [89221.136775] sd 1:0:0:0: [sdc] tag#0 CDB: opcode=0x93 93 08 00 00 00 00 00 01 37 c8 00 00 50 38 00 00 [89221.136782] critical target error, dev sdc, sector 79816 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 2 [92575.926614] EXT4-fs (sdc1): error count since last fsck: 1 [92575.926667] EXT4-fs (sdc1): initial error at time 1561008307: __ext4_get_inode_loc:4617: inode 166199867: block 664797251 [92575.926741] EXT4-fs (sdc1): last error at time 1561008307: ext4_get_inode_loc:4617: inode 166199867: block 664797251 [123565.452985] gvfs-afc-volume invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|GFP_COMP), order=0, oom_score_adj=0 [123565.453015] CPU: 0 PID: 1871 Comm: gvfs-afc-volume Tainted: G C 6.1.21-v8+ #1642 [123565.453022] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)

[123565.453162] Mem-Info: [123565.453168] active_anon:639797 inactive_anon:273738 isolated_anon:0 active_file:10 inactive_file:175 isolated_file:0 unevictable:4182 dirty:5 writeback:0 slab_reclaimable:19094 slab_unreclaimable:9578 mapped:37272 shmem:42894 pagetables:5644 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:7002 free_pcp:1493 free_cma:153 [123565.453181] Node 0 active_anon:2559188kB inactive_anon:1094952kB active_file:40kB inactive_file:552kB unevictable:16728kB isolated(anon):0kB isolated(file):0kB mapped:149088kB dirty:20kB writeback:0kB shmem:171576kB writeback_tmp:0kB kernel_stack:7776kB pagetables:22576kB sec_pagetables:0kB all_unreclaimable? no [123565.453193] DMA free:15388kB boost:0kB min:3732kB low:4664kB high:5596kB reserved_highatomic:0KB active_anon:448792kB inactive_anon:395736kB active_file:0kB inactive_file:136kB unevictable:584kB writepending:16kB present:969728kB managed:885392kB mlocked:0kB bounce:0kB free_pcp:2844kB local_pcp:600kB free_cma:612kB [123565.453206] lowmem_reserve[]: 0 2928 2928 2928 [123565.453229] DMA32 free:12620kB boost:0kB min:12648kB low:15808kB high:18968kB reserved_highatomic:0KB active_anon:824540kB inactive_anon:1985072kB active_file:1220kB inactive_file:0kB unevictable:16144kB writepending:4kB present:3080192kB managed:2998976kB mlocked:16kB bounce:0kB free_pcp:3128kB local_pcp:736kB free_cma:0kB [123565.453240] lowmem_reserve[]: 0 0 0 0 [123565.453262] DMA: 1144kB (UEC) 458kB (UMEC) 13016kB (UMEC) 9832kB (UME) 4764kB (UME) 17128kB (UME) 10256kB (UME) 3512kB (UM) 01024kB 02048kB 04096kB = 15312kB [123565.453345] DMA32: 14kB (M) 2178kB (U) 34916kB (UE) 11432kB (UE) 2764kB (UE) 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 12700kB [123565.453415] 53608 total pagecache pages [123565.453420] 10074 pages in swap cache [123565.453424] Free swap = 228kB [123565.453429] Total swap = 102396kB [123565.453433] 1012480 pages RAM [123565.453437] 0 pages HighMem/MovableOnly [123565.453441] 41388 pages reserved [123565.453445] 131072 pages cma reserved [123565.453449] Tasks state (memory values in pages):

[123565.454238] [ 162320] 1000 162320 42951 34296 372736 0 0 celery [123565.454245] [ 162329] 1000 162329 356717 298176 2596864 0 0 celery [123565.454252] [ 162330] 1000 162330 301399 242841 2203648 0 0 celery [123565.454259] [ 162331] 1000 162331 54664 44678 471040 0 0 celery [123565.454266] [ 162333] 1000 162333 152651 142900 1372160 0 0 celery [123565.454273] [ 162334] 1000 162334 43335 34249 372736 0 0 celery

[123565.453026] Call trace: [123565.453029] dump_backtrace+0x120/0x130 [123565.453039] show_stack+0x20/0x30 [123565.453044] dump_stack_lvl+0x8c/0xb8 [123565.453052] dump_stack+0x18/0x34 [123565.453058] dump_header+0x4c/0x21c [123565.453064] oom_kill_process+0x2a8/0x2b0 [123565.453073] out_of_memory+0xf0/0x350 [123565.453079] alloc_pages_slowpath.constprop.158+0x7d4/0xbc0 [123565.453086] alloc_pages+0x2a8/0x318 [123565.453090] folio_alloc+0x1c/0x28 [123565.453095] __filemap_get_folio+0x214/0x5f0 [123565.453100] filemap_fault+0x644/0xc40 [123565.453105] do_fault+0x44/0x170 [123565.453112] __handle_mm_fault+0x508/0x9c0 [123565.453117] handle_mm_fault+0x1c4/0x2e0 [123565.453123] do_page_fault+0x1f4/0x480 [123565.453130] do_translation_fault+0x54/0x70 [123565.453136] do_mem_abort+0x48/0x98 [123565.453143] el0_ia+0x70/0x100 [123565.453149] el0t_64_sync_handler+0x78/0xc0 [123565.453155] el0t_64_sync+0x18c/0x190 [123565.453162] Mem-Info: [123565.453168] active_anon:639797 inactive_anon:273738 isolated_anon:0 active_file:10 inactive_file:175 isolated_file:0 unevictable:4182 dirty:5 writeback:0 slab_reclaimable:19094 slab_unreclaimable:9578 mapped:37272 shmem:42894 pagetables:5644 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:7002 free_pcp:1493 free_cma:153 [123565.453181] Node 0 active_anon:2559188kB inactive_anon:

[123565.454484] Out of memory: Killed process 162329 (celery) total-vm:1426868kB, anon-rss:1192704kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:2536kB oom_score_adj:0

Notable sources:

Resolution:

rumca-js commented 1 month ago

I think it works now, with max memory celery limit