Closed evilbulgarian closed 1 year ago
If you're referring to the log that you put it's not an OOM it's a concurrent access error.
Not sure why preset-rtsp-restream would cause that for you but we've not had this reported by anyone else. In any case, go2rtc should be restarted by frigate after this happens.
I have the exact same issue:
Apr 23 03:48:16 UNRAIDSRV kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=f600278ac505d12709e4d3e3e2e69e92153a96e34ec320ddbb20e21e38b7a842,mems_allowed=0,oom_memcg=/docker/f600278ac505d12709e4d3e3e2e69e92153a96e34ec320ddbb20e21e38b7a842,task_memcg=/docker/f600278ac505d12709e4d3e3e2e69e92153a96e34ec320ddbb20e21e38b7a842,task=ffmpeg,pid=20642,uid=0 Apr 23 03:48:16 UNRAIDSRV kernel: Memory cgroup out of memory: Killed process 20642 (ffmpeg) total-vm:1227048kB, anon-rss:311204kB, file-rss:74416kB, shmem-rss:0kB, UID:0 pgtables:1104kB oom_score_adj:0
I have the same config and different cameras and also 4 Wize
@johnvprcam24573455623 please make your own issue, like I pointed to above: the issue OP is pointing to does not have anything to do with OOM. So you don't have the same issue
also as an extra note, wyze cams are known to cause problems, so that's probably why
Hi, Nick,
the error is a oom-kill:
kernel: oom-kill
I do have oom issue, I just pasted the relevant frigate logs during the oom kill. Here is the oom as well:
[98361.467531] database invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[98361.467538] CPU: 3 PID: 19213 Comm: database Not tainted 6.3.0 #90
[98361.467541] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5603 07/28/2020
[98361.467543] Call Trace:
[98361.467545] <TASK>
[98361.467546] dump_header+0x51/0x270
[98361.467550] oom_kill_process+0x175/0x410
[98361.467553] out_of_memory+0x39d/0x580
[98361.467555] __alloc_pages_slowpath+0x935/0xef0
[98361.467558] __alloc_pages+0x212/0x2b0
[98361.467561] __filemap_get_folio+0x2d8/0x490
[98361.467563] filemap_fault+0x23e/0x510
[98361.467566] __do_fault+0x99/0x170
[98361.467568] do_fault+0x173/0x7e0
[98361.467570] handle_mm_fault+0xa87/0x1270
[98361.467572] exc_page_fault+0x2ab/0x450
[98361.467576] asm_exc_page_fault+0x26/0x30
[98361.467579] RIP: 0033:0x55a5e23f6f70
[98361.467584] Code: Unable to access opcode bytes at 0x55a5e23f6f46.
[98361.467585] RSP: 002b:00007fa769a01ea8 EFLAGS: 00010206
[98361.467588] RAX: 0000000000000000 RBX: 000055a5e264a7b8 RCX: 000055a5e264a788
[98361.467589] RDX: 000055a5e264a7b8 RSI: 000000000000005c RDI: 0000000000000006
[98361.467591] RBP: 00007fa769a01ef0 R08: 00007fa769a01e60 R09: 0000000000000000
[98361.467592] R10: 0000000000000000 R11: 0000000000000000 R12: 000055a5e264a788
[98361.467593] R13: 0000000064485af8 R14: 00007fa769a01eb0 R15: 0000000064485b03
[98361.467595] </TASK>
[98361.467596] Mem-Info:
[98361.467597] active_anon:1774692 inactive_anon:2032936 isolated_anon:0
active_file:3738 inactive_file:2371 isolated_file:0
unevictable:1987 dirty:120 writeback:0
slab_reclaimable:23669 slab_unreclaimable:32707
mapped:1759 shmem:819 pagetables:20303
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:33229 free_pcp:396 free_cma:0
[98361.467602] Node 0 active_anon:7098768kB inactive_anon:8131744kB active_file:15472kB inactive_file:8964kB unevictable:7948kB isolated(anon):0kB isolated(file):0kB mapped:7036kB dirty:480kB writeback:0kB shmem:3276kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:17344kB pagetables:81212kB sec_pagetables:0kB all_unreclaimable? no
[98361.467606] DMA free:15360kB boost:0kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[98361.467611] lowmem_reserve[]: 0 3443 15661 15661
[98361.467614] DMA32 free:63328kB boost:0kB min:14844kB low:18552kB high:22260kB reserved_highatomic:0KB active_anon:3302004kB inactive_anon:125424kB active_file:0kB inactive_file:1916kB unevictable:0kB writepending:16kB present:3596572kB managed:3531028kB mlocked:0kB bounce:0kB free_pcp:512kB local_pcp:0kB free_cma:0kB
[98361.467618] lowmem_reserve[]: 0 0 12217 12217
[98361.467621] Normal free:54228kB boost:0kB min:52672kB low:65840kB high:79008kB reserved_highatomic:2048KB active_anon:3795288kB inactive_anon:8007796kB active_file:348kB inactive_file:23008kB unevictable:7948kB writepending:464kB present:12828672kB managed:12513636kB mlocked:7948kB bounce:0kB free_pcp:1072kB local_pcp:0kB free_cma:0kB
[98361.467625] lowmem_reserve[]: 0 0 0 0
[98361.467627] DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB
[98361.467636] DMA32: 1035*4kB (UME) 752*8kB (UME) 458*16kB (UME) 291*32kB (UE) 94*64kB (UE) 37*128kB (UME) 13*256kB (UE) 28*512kB (UME) 6*1024kB (UM) 1*2048kB (M) 0*4096kB = 63404kB
[98361.467647] Normal: 1780*4kB (UE) 1218*8kB (UME) 658*16kB (UME) 421*32kB (UME) 110*64kB (UME) 50*128kB (UE) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54560kB
[98361.467657] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[98361.467659] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[98361.467660] 14039 total pagecache pages
[98361.467661] 5379 pages in swap cache
[98361.467662] Free swap = 12kB
[98361.467662] Total swap = 16777208kB
[98361.467663] 4110310 pages RAM
[98361.467664] 0 pages HighMem/MovableOnly
[98361.467664] 95304 pages reserved
[98361.467665] 0 pages hwpoisoned
[98361.467665] Tasks state (memory values in pages):
[98361.467666] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[98361.467671] [ 1660] 0 1660 1726 496 57344 64 -1000 sshd
[98361.467674] [ 1721] 0 1721 4821 480 65536 384 0 systemd-udevd
[98361.467678] [ 3451] 0 3451 816 550 45056 96 0 dhcpcd
[98361.467681] [ 3926] 998 3926 1228 416 49152 64 0 dbus-daemon
[98361.467683] [ 4035] 61 4035 1224 480 45056 224 0 avahi-daemon
[98361.467685] [ 4036] 61 4036 1075 352 45056 32 0 avahi-daemon
[98361.467687] [ 4069] 0 4069 54566 439 73728 416 0 rsyslogd
[98361.467690] [ 4103] 0 4103 1690 387 49152 32 0 crond
[98361.467692] [ 4135] 997 4135 2758 1139 61440 256 0 stubby
[98361.467694] [ 4167] 101 4167 1520 452 53248 0 0 dnsmasq
[98361.467696] [ 4199] 102 4199 5034 256 65536 2432 0 ddclient - slee
[98361.467698] [ 4230] 0 4230 468424 1011 372736 4503 0 containerd
[98361.467700] [ 4304] 482 4304 1847 1152 53248 128 0 mosquitto
[98361.467702] [ 4397] 60 4397 525592 1640 602112 37152 0 mysqld
[98361.467704] [ 6020] 0 6020 4648 266 57344 224 0 nginx
[98361.467706] [ 6021] 103 6021 4919 1066 61440 352 0 nginx
[98361.467708] [ 6182] 123 6182 18347 1933 57344 0 0 ntpd
[98361.467710] [ 6284] 0 6284 5083 384 86016 4384 0 pgld
[98361.467712] [ 6328] 215 6328 627 448 40960 0 0 radvd
[98361.467714] [ 6329] 0 6329 627 288 40960 0 0 radvd
[98361.467716] [ 6359] 0 6359 2499 416 61440 96 0 rsync
[98361.467718] [ 6392] 0 6392 18763 522 135168 544 0 smbd
[98361.467720] [ 6396] 0 6396 18341 433 131072 576 0 smbd-notifyd
[98361.467722] [ 6397] 0 6397 18339 401 114688 576 0 cleanupd
[98361.467724] [ 6400] 0 6400 16335 548 102400 416 0 nmbd
[98361.467726] [ 6443] 104 6443 515443 2142 311296 1477 0 zabbix_agent2
[98361.467728] [ 6484] 0 6484 1447 480 45056 0 0 agetty
[98361.467730] [ 6485] 0 6485 1447 512 49152 0 0 agetty
[98361.467732] [ 6486] 0 6486 1447 448 49152 0 0 agetty
[98361.467734] [ 6487] 0 6487 1447 480 45056 32 0 agetty
[98361.467736] [ 6488] 0 6488 1447 448 45056 32 0 agetty
[98361.467738] [ 6489] 0 6489 1447 512 45056 0 0 agetty
[98361.467739] [ 6767] 0 6767 9264 448 73728 96 0 elogind-daemon
[98361.467742] [ 7431] 1000 7431 3836 416 69632 1504 0 tmux: server
[98361.467744] [ 7432] 1000 7432 2746 664 57344 672 0 zsh
[98361.467746] [ 7492] 1000 7492 2789 562 61440 736 0 zsh
[98361.467748] [ 7546] 1000 7546 2710 604 61440 640 0 zsh
[98361.467750] [ 7604] 1000 7604 79772 3489 393216 36192 0 rtorrent main
[98361.467752] [ 21487] 65534 21487 2798 621 69632 161 0 rsync
[98361.467754] [ 10922] 1000 10922 1447 352 49152 0 0 tail
[98361.467757] [ 17450] 0 17450 623039 1031 512000 7103 0 dockerd
[98361.467759] [ 17679] 0 17679 288815 352 126976 160 0 docker-proxy
[98361.467761] [ 17692] 0 17692 288815 352 139264 160 0 docker-proxy
[98361.467763] [ 17705] 0 17705 270446 320 126976 160 0 docker-proxy
[98361.467765] [ 17723] 0 17723 307376 320 147456 192 0 docker-proxy
[98361.467767] [ 17751] 0 17751 180179 520 118784 576 1 containerd-shim
[98361.467768] [ 17775] 0 17775 180243 538 122880 576 1 containerd-shim
[98361.467770] [ 17804] 0 17804 180179 552 110592 512 1 containerd-shim
[98361.467772] [ 17831] 0 17831 53 0 28672 0 0 s6-svscan
[98361.467774] [ 17832] 0 17832 975 64 49152 64 0 entrypoint-coll
[98361.467777] [ 17833] 0 17833 255 32 28672 0 0 docker-init
[98361.467779] [ 17917] 0 17917 270446 352 126976 160 0 docker-proxy
[98361.467781] [ 17929] 0 17929 344242 320 163840 192 0 docker-proxy
[98361.467783] [ 17943] 0 17943 270446 320 131072 160 0 docker-proxy
[98361.467785] [ 17953] 0 17953 288879 320 135168 160 0 docker-proxy
[98361.467786] [ 17968] 0 17968 270382 320 126976 160 0 docker-proxy
[98361.467788] [ 17985] 0 17985 180243 602 114688 480 1 containerd-shim
[98361.467790] [ 18030] 0 18030 180179 521 118784 509 1 containerd-shim
[98361.467792] [ 18079] 0 18079 307376 352 151552 192 0 docker-proxy
[98361.467794] [ 18102] 0 18102 307376 352 143360 416 0 docker-proxy
[98361.467796] [ 18127] 0 18127 1551 32 53248 96 0 su
[98361.467798] [ 18131] 0 18131 937 96 40960 32 0 cron
[98361.467800] [ 18137] 1000 18137 45069 315 434176 38464 0 flexget
[98361.467802] [ 18139] 0 18139 179213 126 122880 1440 0 watchtower
[98361.467804] [ 18155] 0 18155 288879 320 135168 192 0 docker-proxy
[98361.467806] [ 18195] 0 18195 204 0 45056 0 0 tini
[98361.467808] [ 18196] 0 18196 87313 716 3305472 14509 0 node
[98361.467810] [ 18258] 0 18258 270446 352 131072 192 0 docker-proxy
[98361.467812] [ 18265] 0 18265 53 0 28672 0 0 s6-supervise
[98361.467814] [ 18266] 0 18266 50 0 24576 0 0 s6-linux-init-s
[98361.467816] [ 18274] 0 18274 53 0 28672 0 0 s6-supervise
[98361.467818] [ 18275] 0 18275 53 0 28672 0 0 s6-supervise
[98361.467819] [ 18276] 0 18276 53 0 28672 0 0 s6-supervise
[98361.467821] [ 18277] 0 18277 53 0 28672 0 0 s6-supervise
[98361.467823] [ 18278] 0 18278 53 0 28672 0 0 s6-supervise
[98361.467825] [ 18279] 0 18279 53 0 28672 0 0 s6-supervise
[98361.467827] [ 18280] 0 18280 53 0 28672 0 0 s6-supervise
[98361.467829] [ 18281] 0 18281 53 0 28672 0 0 s6-supervise
[98361.467830] [ 18282] 0 18282 53 0 24576 0 0 s6-supervise
[98361.467832] [ 18291] 0 18291 47 0 24576 0 0 s6-ipcserverd
[98361.467834] [ 18292] 0 18292 131 0 28672 0 0 s6-fdholderd
[98361.467836] [ 18336] 65534 18336 69 0 36864 0 0 s6-log
[98361.467838] [ 18337] 65534 18337 69 0 40960 0 0 s6-log
[98361.467840] [ 18338] 65534 18338 69 32 40960 0 0 s6-log
[98361.467842] [ 18345] 0 18345 181793 1993 139264 3263 0 go2rtc
[98361.467844] [ 18361] 0 18361 973 32 45056 32 0 bash
[98361.467846] [ 18366] 0 18366 773263 3344 933888 31744 0 python3
[98361.467849] [ 18378] 0 18378 145420 192 110592 448 0 nginx
[98361.467851] [ 18394] 0 18394 211097 140 221184 608 0 nginx
[98361.467852] [ 18395] 0 18395 211097 108 221184 608 0 nginx
[98361.467854] [ 18396] 0 18396 211097 108 221184 608 0 nginx
[98361.467856] [ 18401] 0 18401 211097 172 221184 576 0 nginx
[98361.467858] [ 18413] 0 18413 211097 172 221184 608 0 nginx
[98361.467860] [ 18442] 0 18442 211097 108 221184 608 0 nginx
[98361.467862] [ 18464] 0 18464 211097 108 221184 640 0 nginx
[98361.467864] [ 18465] 0 18465 211097 76 221184 640 0 nginx
[98361.467866] [ 18652] 0 18652 180179 644 118784 576 1 containerd-shim
[98361.467868] [ 18718] 0 18718 52 0 32768 0 0 s6-svscan
[98361.467870] [ 18867] 0 18867 53 0 32768 0 0 s6-supervise
[98361.467871] [ 18868] 0 18868 50 0 28672 0 0 s6-linux-init-s
[98361.467873] [ 18890] 0 18890 53 0 32768 0 0 s6-supervise
[98361.467875] [ 18891] 0 18891 53 0 32768 0 0 s6-supervise
[98361.467877] [ 18892] 0 18892 53 0 32768 0 0 s6-supervise
[98361.467879] [ 18893] 0 18893 53 0 32768 0 0 s6-supervise
[98361.467881] [ 18894] 0 18894 53 0 32768 0 0 s6-supervise
[98361.467882] [ 18895] 0 18895 53 0 32768 0 0 s6-supervise
[98361.467884] [ 18896] 0 18896 53 0 32768 0 0 s6-supervise
[98361.467886] [ 18903] 0 18903 47 0 28672 0 0 s6-ipcserverd
[98361.467888] [ 18921] 0 18921 975 96 45056 64 0 bash
[98361.467890] [ 18927] 0 18927 937 64 45056 32 0 cron
[98361.467892] [ 19084] 0 19084 975 96 45056 64 0 bash
[98361.467894] [ 19090] 0 19090 975 64 36864 64 0 bash
[98361.467896] [ 19115] 33 19115 1112 0 40960 160 0 lighttpd
[98361.467898] [ 19125] 999 19125 975 96 53248 64 0 bash
[98361.467900] [ 19126] 999 19126 79635 271 159744 864 0 pihole-FTL
[98361.467902] [ 19148] 33 19148 49474 96 139264 1056 0 php-cgi
[98361.467904] [ 19150] 33 19150 49474 36 106496 1056 0 php-cgi
[98361.467906] [ 19152] 33 19152 49474 36 106496 1056 0 php-cgi
[98361.467908] [ 19153] 33 19153 49474 36 106496 1056 0 php-cgi
[98361.467910] [ 19154] 33 19154 49474 32 106496 1056 0 php-cgi
[98361.467911] [ 19170] 0 19170 213767 495 368640 18016 0 frigate.logger
[98361.467913] [ 19176] 0 19176 3564 160 61440 1216 0 python3
[98361.467915] [ 19179] 0 19179 300073 731 487424 19168 0 frigate.output
[98361.467917] [ 19185] 0 19185 31705 96 176128 704 0 ffmpeg
[98361.467919] [ 19186] 0 19186 375385 1036 528384 20608 0 frigate.process
[98361.467921] [ 19187] 0 19187 374785 739 524288 20416 0 frigate.process
[98361.467923] [ 19188] 0 19188 31705 96 167936 736 0 ffmpeg
[98361.467925] [ 19189] 0 19189 321482 1502 434176 18496 0 frigate.capture
[98361.467927] [ 19193] 0 19193 318106 556 450560 19808 0 frigate.capture
[98361.467929] [ 19197] 0 19197 237312 2768 622592 7072 0 ffmpeg
[98361.467931] [ 16209] 0 16209 7934561 3765380 61620224 3849504 0 ffmpeg
[98361.467933] [ 17368] 1000 17368 2787 589 61440 672 0 zsh
[98361.467936] [ 22347] 0 22347 362803 384 176128 256 0 docker-proxy
[98361.467938] [ 22361] 0 22361 180179 618 118784 512 1 containerd-shim
[98361.467939] [ 22381] 0 22381 53 0 28672 0 0 s6-svscan
[98361.467941] [ 22468] 0 22468 53 0 28672 0 0 s6-supervise
[98361.467943] [ 22470] 0 22470 50 0 24576 0 0 s6-linux-init-s
[98361.467945] [ 22477] 0 22477 53 0 28672 0 0 s6-supervise
[98361.467947] [ 22478] 0 22478 53 0 28672 0 0 s6-supervise
[98361.467949] [ 22486] 0 22486 47 0 24576 0 0 s6-ipcserverd
[98361.467951] [ 22517] 0 22517 53 0 24576 0 0 s6-supervise
[98361.467953] [ 22519] 0 22519 368073 10635 1032192 49888 0 python3
[98361.467955] [ 23807] 1000 23807 271172 256 131072 384 0 idalv-rest
[98361.467957] [ 23859] 1000 23859 1881 416 49152 64 0 kodi-standalone
[98361.467959] [ 23861] 1000 23861 1914 416 57344 96 0 kodi
[98361.467961] [ 23866] 1000 23866 628248 4604 1282048 49376 0 kodi-gbm
[98361.467963] [ 25950] 0 25950 731283 1129 692224 36457 0 frigate.detecto
[98361.467965] [ 28490] 0 28490 622 32 40960 0 0 sleep
[98361.467967] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=0020a45490e1cb39c279c9a9ead8996ebac7867643743bfaa4d486d78b373d78,mems_allowed=0,global_oom,task_memcg=/docker/ea829defadeb33e4dd8a3134f460e276a863a86bd23b5b277e20afff4319ca28,task=ffmpeg,pid=16209,uid=0
[98361.468001] Out of memory: Killed process 16209 (ffmpeg) total-vm:31738244kB, anon-rss:15061264kB, file-rss:256kB, shmem-rss:0kB, UID:0 pgtables:60176kB oom_score_adj:0
@johnvprcam24573455623 please make your own issue, like I pointed to above: the issue OP is pointing to does not have anything to do with OOM. So you don't have the same issue
also as an extra note, wyze cams are known to cause problems, so that's probably why
Those logs were from go2rtc and not frigate, and they did not indicate an oom so there was nothing pointing to that.
We'd need to show an htop output or something similar showing which process in frigate is using the memory
Its the ffmpeg process which is in my output above: [98361.467666] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [98361.467931] [ 16209] 0 16209 7934561 3765380 61620224 3849504 0 ffmpeg
That doesn't give useful info though, we need to know which ffmpeg command is it because there's many different types of ffmpeg processes throughout frigate
Same here and it also crashed my BOX:
Memory cgroup out of memory: Killed process 6126 (ffmpeg) total-vm:3433428kB, anon-rss:2454116kB, file-rss:64204kB, shmem-rss:0kB, UID:0 pgtables:5308kB oom_score_adj:0
Can we activate any additional logs that can help throubleshoot?
This is the complete logs of the crash
Apr 28 09:17:19 UNRAIDSRV kernel: CPU: 7 PID: 30278 Comm: frigate.process Tainted: G O 5.19.17-Unraid #2
Apr 28 09:17:19 UNRAIDSRV kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4, BIOS P4.20 06/18/2020
Apr 28 09:17:19 UNRAIDSRV kernel: Call Trace:
Apr 28 09:17:19 UNRAIDSRV kernel: <TASK>
Apr 28 09:17:19 UNRAIDSRV kernel: dump_stack_lvl+0x44/0x5c
Apr 28 09:17:19 UNRAIDSRV kernel: dump_header+0x4a/0x1ff
Apr 28 09:17:19 UNRAIDSRV kernel: oom_kill_process+0x80/0x111
Apr 28 09:17:19 UNRAIDSRV kernel: out_of_memory+0x3e8/0x41a
Apr 28 09:17:19 UNRAIDSRV kernel: mem_cgroup_out_of_memory+0x7c/0xb2
Apr 28 09:17:19 UNRAIDSRV kernel: try_charge_memcg+0x44a/0x55e
Apr 28 09:17:19 UNRAIDSRV kernel: ? select_task_rq_fair+0xb38/0xb4e
Apr 28 09:17:19 UNRAIDSRV kernel: obj_cgroup_charge_pages+0x28/0x4d
Apr 28 09:17:19 UNRAIDSRV kernel: obj_cgroup_charge+0x70/0xd7
Apr 28 09:17:19 UNRAIDSRV kernel: slab_pre_alloc_hook.constprop.0+0x8e/0x14b
Apr 28 09:17:19 UNRAIDSRV kernel: ? __alloc_file+0x26/0x93
Apr 28 09:17:19 UNRAIDSRV kernel: kmem_cache_alloc+0x48/0x143
Apr 28 09:17:19 UNRAIDSRV kernel: ? rcuwait_wake_up+0x2d/0x38
Apr 28 09:17:19 UNRAIDSRV kernel: __alloc_file+0x26/0x93
Apr 28 09:17:19 UNRAIDSRV kernel: alloc_empty_file+0x7f/0xd5
Apr 28 09:17:19 UNRAIDSRV kernel: path_openat+0x4b/0xaa9
Apr 28 09:17:19 UNRAIDSRV kernel: ? current_time+0x5a/0x7a
Apr 28 09:17:19 UNRAIDSRV kernel: ? atime_needs_update+0x9c/0xf7
Apr 28 09:17:19 UNRAIDSRV kernel: do_filp_open+0x55/0xb8
Apr 28 09:17:19 UNRAIDSRV kernel: ? pipe_read+0x30a/0x33e
Apr 28 09:17:19 UNRAIDSRV kernel: ? __handle_mm_fault+0xa6d/0xc7d
Apr 28 09:17:19 UNRAIDSRV kernel: ? slab_post_alloc_hook+0x4d/0x15e
Apr 28 09:17:19 UNRAIDSRV kernel: ? getname_flags+0x29/0x152
Apr 28 09:17:19 UNRAIDSRV kernel: ? kmem_cache_alloc+0x11a/0x143
Apr 28 09:17:19 UNRAIDSRV kernel: do_sys_openat2+0x6c/0xd9
Apr 28 09:17:19 UNRAIDSRV kernel: do_sys_open+0x3a/0x5a
Apr 28 09:17:19 UNRAIDSRV kernel: do_syscall_64+0x6b/0x81
Apr 28 09:17:19 UNRAIDSRV kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
Apr 28 09:17:19 UNRAIDSRV kernel: RIP: 0033:0x154cb2b97e41
Apr 28 09:17:19 UNRAIDSRV kernel: Code: 44 24 18 31 c0 41 83 e2 40 75 3e 89 f0 25 00 00 41 00 3d 00 00 41 00 74 30 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 3f 48 8b 54 24 18 64 48 2b 14 25 28 00 00 00
Apr 28 09:17:19 UNRAIDSRV kernel: RSP: 002b:00007ffc8ac70110 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
Apr 28 09:17:19 UNRAIDSRV kernel: RAX: ffffffffffffffda RBX: 00007ffc8ac70170 RCX: 0000154cb2b97e41
Apr 28 09:17:19 UNRAIDSRV kernel: RDX: 00000000000a0002 RSI: 00007ffc8ac70170 RDI: 00000000ffffff9c
Apr 28 09:17:19 UNRAIDSRV kernel: RBP: 0000000000000180 R08: 0000000000000000 R09: 0000000000000000
Apr 28 09:17:19 UNRAIDSRV kernel: R10: 0000000000000000 R11: 0000000000000287 R12: 0000000000000002
Apr 28 09:17:19 UNRAIDSRV kernel: R13: 0000154c800916f0 R14: 0000154c80091720 R15: 00000000009636f0
Apr 28 09:17:19 UNRAIDSRV kernel: </TASK>
Apr 28 09:17:19 UNRAIDSRV kernel: memory: usage 3145728kB, limit 3145728kB, failcnt 81395
Apr 28 09:17:19 UNRAIDSRV kernel: memory+swap: usage 3145728kB, limit 6291456kB, failcnt 0
Apr 28 09:17:19 UNRAIDSRV kernel: kmem: usage 36076kB, limit 9007199254740988kB, failcnt 0
Apr 28 09:17:19 UNRAIDSRV kernel: Memory cgroup stats for /docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c:
Apr 28 09:17:19 UNRAIDSRV kernel: anon 3157348352
Apr 28 09:17:19 UNRAIDSRV kernel: file 26935296
Apr 28 09:17:19 UNRAIDSRV kernel: kernel 36941824
Apr 28 09:17:19 UNRAIDSRV kernel: kernel_stack 7766016
Apr 28 09:17:19 UNRAIDSRV kernel: pagetables 18161664
Apr 28 09:17:19 UNRAIDSRV kernel: percpu 10064
Apr 28 09:17:19 UNRAIDSRV kernel: sock 0
Apr 28 09:17:19 UNRAIDSRV kernel: vmalloc 0
Apr 28 09:17:19 UNRAIDSRV kernel: shmem 26791936
Apr 28 09:17:19 UNRAIDSRV kernel: file_mapped 5079040
Apr 28 09:17:19 UNRAIDSRV kernel: file_dirty 0
Apr 28 09:17:19 UNRAIDSRV kernel: file_writeback 0
Apr 28 09:17:19 UNRAIDSRV kernel: swapcached 0
Apr 28 09:17:19 UNRAIDSRV kernel: anon_thp 312475648
Apr 28 09:17:19 UNRAIDSRV kernel: file_thp 0
Apr 28 09:17:19 UNRAIDSRV kernel: shmem_thp 0
Apr 28 09:17:19 UNRAIDSRV kernel: inactive_anon 3174850560
Apr 28 09:17:19 UNRAIDSRV kernel: active_anon 9289728
Apr 28 09:17:19 UNRAIDSRV kernel: inactive_file 143360
Apr 28 09:17:19 UNRAIDSRV kernel: active_file 0
Apr 28 09:17:19 UNRAIDSRV kernel: unevictable 0
Apr 28 09:17:19 UNRAIDSRV kernel: slab_reclaimable 956032
Apr 28 09:17:19 UNRAIDSRV kernel: slab_unreclaimable 8774296
Apr 28 09:17:19 UNRAIDSRV kernel: slab 9730328
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_refault_anon 0
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_refault_file 225956
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_activate_anon 0
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_activate_file 19929
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_restore_anon 0
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_restore_file 3925
Apr 28 09:17:19 UNRAIDSRV kernel: workingset_nodereclaim 0
Apr 28 09:17:19 UNRAIDSRV kernel: pgfault 369855614
Apr 28 09:17:19 UNRAIDSRV kernel: pgmajfault 1336
Apr 28 09:17:19 UNRAIDSRV kernel: pgrefill 64608
Apr 28 09:17:19 UNRAIDSRV kernel: pgscan 554608
Apr 28 09:17:19 UNRAIDSRV kernel: pgsteal 261286
Apr 28 09:17:19 UNRAIDSRV kernel: pgactivate 106265120
Apr 28 09:17:19 UNRAIDSRV kernel: pgdeactivate 64212
Apr 28 09:17:19 UNRAIDSRV kernel: pglazyfree 0
Apr 28 09:17:19 UNRAIDSRV kernel: pglazyfreed 0
Apr 28 09:17:19 UNRAIDSRV kernel: thp_fault_alloc 17074
Apr 28 09:17:19 UNRAIDSRV kernel: thp_collapse_alloc 143
Apr 28 09:17:19 UNRAIDSRV kernel: Tasks state (memory values in pages):
Apr 28 09:17:19 UNRAIDSRV kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29675] 0 29675 53 16 24576 0 0 s6-svscan
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29747] 0 29747 53 14 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29748] 0 29748 50 1 24576 0 0 s6-linux-init-s
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29756] 0 29756 53 16 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29757] 0 29757 53 15 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29758] 0 29758 53 16 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29759] 0 29759 53 15 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29760] 0 29760 53 15 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29761] 0 29761 53 15 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29762] 0 29762 53 16 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29763] 0 29763 53 16 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29764] 0 29764 53 16 24576 0 0 s6-supervise
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29773] 0 29773 47 1 24576 0 0 s6-ipcserverd
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29774] 0 29774 131 41 24576 0 0 s6-fdholderd
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29812] 65534 29812 69 27 36864 0 0 s6-log
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29813] 65534 29813 69 28 36864 0 0 s6-log
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29814] 65534 29814 70 27 36864 0 0 s6-log
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29821] 0 29821 181857 6962 143360 0 0 go2rtc
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29830] 0 29830 973 730 40960 0 0 bash
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29833] 0 29833 618176 44055 905216 0 0 python3
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29849] 0 29849 145420 3479 102400 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29866] 0 29866 161947 1584 212992 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29867] 0 29867 161947 1584 212992 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29868] 0 29868 161947 1584 212992 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29875] 0 29875 161947 1584 212992 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29897] 0 29897 161947 1584 212992 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29932] 0 29932 161947 1584 212992 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29962] 0 29962 161947 1584 212992 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 29970] 0 29970 161947 1584 212992 0 0 nginx
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30266] 0 30266 212817 19824 376832 0 0 frigate.logger
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30272] 0 30272 3564 2647 65536 0 0 python3
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30273] 0 30273 301653 26449 479232 0 0 frigate.detecto
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30275] 0 30275 349154 24376 524288 0 0 frigate.output
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30278] 0 30278 366610 25271 536576 0 0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30279] 0 30279 31705 5386 172032 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30280] 0 30280 374564 33602 602112 0 0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30284] 0 30284 371525 30250 573440 0 0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30285] 0 30285 31705 4897 163840 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30286] 0 30286 369682 28568 565248 0 0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30287] 0 30287 31705 4486 163840 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30288] 0 30288 316513 23315 495616 0 0 frigate.process
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30289] 0 30289 316552 22757 475136 0 0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30292] 0 30292 324793 23723 466944 0 0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30293] 0 30293 31705 5347 163840 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30297] 0 30297 317199 25091 487424 0 0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30298] 0 30298 257592 29999 622592 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30303] 0 30303 31705 5371 163840 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30304] 0 30304 320153 23173 462848 0 0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30310] 0 30310 317199 22945 483328 0 0 frigate.capture
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30312] 0 30312 308966 31568 647168 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 30315] 0 30315 31705 5881 167936 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 27058] 0 27058 872202 627124 5423104 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 31302] 0 31302 306464 26832 626688 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 3524] 0 3524 32871 6892 196608 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 4124] 0 4124 622 213 40960 0 0 sleep
Apr 28 09:17:19 UNRAIDSRV kernel: [ 4506] 0 4506 31668 3866 163840 0 0 ffmpeg
Apr 28 09:17:19 UNRAIDSRV kernel: [ 4514] 0 4514 31645 2414 147456 0 0 ffprobe
Apr 28 09:17:19 UNRAIDSRV kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,mems_allowed=0,oom_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task=ffmpeg,pid=27058,uid=0
Apr 28 09:17:19 UNRAIDSRV kernel: Memory cgroup out of memory: Killed process 27058 (ffmpeg) total-vm:3488808kB, anon-rss:2443960kB, file-rss:64536kB, shmem-rss:0kB, UID:0 pgtables:5296kB oom_score_adj:0
Apr 28 09:18:46 UNRAIDSRV kernel: ffmpeg invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Apr 28 09:18:46 UNRAIDSRV kernel: CPU: 0 PID: 4705 Comm: ffmpeg Tainted: G O 5.19.17-Unraid #2
Apr 28 09:18:46 UNRAIDSRV kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4, BIOS P4.20 06/18/2020
Apr 28 09:18:46 UNRAIDSRV kernel: Call Trace:
Apr 28 09:18:46 UNRAIDSRV kernel: <TASK>
Apr 28 09:18:46 UNRAIDSRV kernel: dump_stack_lvl+0x44/0x5c
Apr 28 09:18:46 UNRAIDSRV kernel: dump_header+0x4a/0x1ff
Apr 28 09:18:46 UNRAIDSRV kernel: oom_kill_process+0x80/0x111
Apr 28 09:18:46 UNRAIDSRV kernel: out_of_memory+0x3e8/0x41a
Apr 28 09:18:46 UNRAIDSRV kernel: mem_cgroup_out_of_memory+0x7c/0xb2
Apr 28 09:18:46 UNRAIDSRV kernel: try_charge_memcg+0x44a/0x55e
Apr 28 09:18:46 UNRAIDSRV kernel: charge_memcg+0x29/0x71
Apr 28 09:18:46 UNRAIDSRV kernel: __mem_cgroup_charge+0x29/0x41
Apr 28 09:18:46 UNRAIDSRV kernel: __handle_mm_fault+0x813/0xc7d
Apr 28 09:18:46 UNRAIDSRV kernel: ? __default_send_IPI_dest_field+0x32/0x50
Apr 28 09:18:46 UNRAIDSRV kernel: handle_mm_fault+0x113/0x1d7
Apr 28 09:18:46 UNRAIDSRV kernel: do_user_addr_fault+0x36a/0x514
Apr 28 09:18:46 UNRAIDSRV kernel: exc_page_fault+0xfc/0x11e
Apr 28 09:18:46 UNRAIDSRV kernel: asm_exc_page_fault+0x22/0x30
Apr 28 09:18:46 UNRAIDSRV kernel: RIP: 0033:0x15042db317f7
Apr 28 09:18:46 UNRAIDSRV kernel: Code: 48 ff c7 48 01 fe 48 8d 54 11 80 0f 1f 84 00 00 00 00 00 c5 fe 6f 0e c5 fe 6f 56 20 c5 fe 6f 5e 40 c5 fe 6f 66 60 48 83 ee 80 <c5> fd 7f 0f c5 fd 7f 57 20 c5 fd 7f 5f 40 c5 fd 7f 67 60 48 83 ef
Apr 28 09:18:46 UNRAIDSRV kernel: RSP: 002b:00007ffdda03ea68 EFLAGS: 00010207
Apr 28 09:18:46 UNRAIDSRV kernel: RAX: 000055772af8f730 RBX: 000015042bfe7800 RCX: 000055772af8f730
Apr 28 09:18:46 UNRAIDSRV kernel: RDX: 000055772af8fe30 RSI: 000015042bfe7090 RDI: 000055772af8f740
Apr 28 09:18:46 UNRAIDSRV kernel: RBP: 0000000000000780 R08: 0000000000000000 R09: 0000000000000000
Apr 28 09:18:46 UNRAIDSRV kernel: R10: 0000000000000000 R11: 000000000000021b R12: 0000000000000800
Apr 28 09:18:46 UNRAIDSRV kernel: R13: 000000000000021c R14: 0000000000000f00 R15: 00000000000001eb
Apr 28 09:18:46 UNRAIDSRV kernel: </TASK>
Apr 28 09:18:46 UNRAIDSRV kernel: memory: usage 3145728kB, limit 3145728kB, failcnt 92888
Apr 28 09:18:46 UNRAIDSRV kernel: memory+swap: usage 3145728kB, limit 6291456kB, failcnt 0
Apr 28 09:18:46 UNRAIDSRV kernel: kmem: usage 35904kB, limit 9007199254740988kB, failcnt 0
Apr 28 09:18:46 UNRAIDSRV kernel: Memory cgroup stats for /docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c:
Apr 28 09:18:46 UNRAIDSRV kernel: anon 3159777280
Apr 28 09:18:46 UNRAIDSRV kernel: file 24682496
Apr 28 09:18:46 UNRAIDSRV kernel: kernel 36765696
Apr 28 09:18:46 UNRAIDSRV kernel: kernel_stack 7667712
Apr 28 09:18:46 UNRAIDSRV kernel: pagetables 17879040
Apr 28 09:18:46 UNRAIDSRV kernel: percpu 10064
Apr 28 09:18:46 UNRAIDSRV kernel: sock 0
Apr 28 09:18:46 UNRAIDSRV kernel: vmalloc 0
Apr 28 09:18:46 UNRAIDSRV kernel: shmem 24584192
Apr 28 09:18:46 UNRAIDSRV kernel: file_mapped 5079040
Apr 28 09:18:46 UNRAIDSRV kernel: file_dirty 0
Apr 28 09:18:46 UNRAIDSRV kernel: file_writeback 0
Apr 28 09:18:46 UNRAIDSRV kernel: swapcached 0
Apr 28 09:18:46 UNRAIDSRV kernel: anon_thp 784334848
Apr 28 09:18:46 UNRAIDSRV kernel: file_thp 0
Apr 28 09:18:46 UNRAIDSRV kernel: shmem_thp 0
Apr 28 09:18:46 UNRAIDSRV kernel: inactive_anon 3175759872
Apr 28 09:18:46 UNRAIDSRV kernel: active_anon 8585216
Apr 28 09:18:46 UNRAIDSRV kernel: inactive_file 45056
Apr 28 09:18:46 UNRAIDSRV kernel: active_file 0
Apr 28 09:18:46 UNRAIDSRV kernel: unevictable 0
Apr 28 09:18:46 UNRAIDSRV kernel: slab_reclaimable 1285840
Apr 28 09:18:46 UNRAIDSRV kernel: slab_unreclaimable 8689776
Apr 28 09:18:46 UNRAIDSRV kernel: slab 9975616
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_refault_anon 0
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_refault_file 254111
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_activate_anon 0
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_activate_file 22716
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_restore_anon 0
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_restore_file 3925
Apr 28 09:18:46 UNRAIDSRV kernel: workingset_nodereclaim 0
Apr 28 09:18:46 UNRAIDSRV kernel: pgfault 372540282
Apr 28 09:18:46 UNRAIDSRV kernel: pgmajfault 1453
Apr 28 09:18:46 UNRAIDSRV kernel: pgrefill 71669
Apr 28 09:18:46 UNRAIDSRV kernel: pgscan 588191
Apr 28 09:18:46 UNRAIDSRV kernel: pgsteal 290931
Apr 28 09:18:46 UNRAIDSRV kernel: pgactivate 106916000
Apr 28 09:18:46 UNRAIDSRV kernel: pgdeactivate 71272
Apr 28 09:18:46 UNRAIDSRV kernel: pglazyfree 0
Apr 28 09:18:46 UNRAIDSRV kernel: pglazyfreed 0
Apr 28 09:18:46 UNRAIDSRV kernel: thp_fault_alloc 17469
Apr 28 09:18:46 UNRAIDSRV kernel: thp_collapse_alloc 143
Apr 28 09:18:46 UNRAIDSRV kernel: Tasks state (memory values in pages):
Apr 28 09:18:46 UNRAIDSRV kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29675] 0 29675 53 16 24576 0 0 s6-svscan
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29747] 0 29747 53 14 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29748] 0 29748 50 1 24576 0 0 s6-linux-init-s
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29756] 0 29756 53 16 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29757] 0 29757 53 15 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29758] 0 29758 53 16 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29759] 0 29759 53 15 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29760] 0 29760 53 15 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29761] 0 29761 53 15 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29762] 0 29762 53 16 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29763] 0 29763 53 16 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29764] 0 29764 53 16 24576 0 0 s6-supervise
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29773] 0 29773 47 1 24576 0 0 s6-ipcserverd
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29774] 0 29774 131 41 24576 0 0 s6-fdholderd
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29812] 65534 29812 69 27 36864 0 0 s6-log
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29813] 65534 29813 69 28 36864 0 0 s6-log
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29814] 65534 29814 70 27 36864 0 0 s6-log
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29821] 0 29821 181857 7116 143360 0 0 go2rtc
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29830] 0 29830 973 729 40960 0 0 bash
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29833] 0 29833 618176 44055 905216 0 0 python3
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29849] 0 29849 145420 3477 102400 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29866] 0 29866 161947 1582 212992 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29867] 0 29867 161947 1582 212992 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29868] 0 29868 161947 1582 212992 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29875] 0 29875 161947 1582 212992 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29897] 0 29897 161947 1582 212992 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29932] 0 29932 161947 1582 212992 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29962] 0 29962 161947 1582 212992 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 29970] 0 29970 161947 1582 212992 0 0 nginx
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30266] 0 30266 212817 19820 376832 0 0 frigate.logger
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30272] 0 30272 3564 2644 65536 0 0 python3
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30273] 0 30273 301653 26410 479232 0 0 frigate.detecto
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30275] 0 30275 349154 24375 524288 0 0 frigate.output
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30278] 0 30278 366610 25268 536576 0 0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30279] 0 30279 31705 5380 172032 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30280] 0 30280 374564 33600 602112 0 0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30284] 0 30284 371525 30248 573440 0 0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30285] 0 30285 31705 4891 163840 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30286] 0 30286 369682 28567 565248 0 0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30287] 0 30287 31705 4480 163840 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30288] 0 30288 316513 23312 495616 0 0 frigate.process
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30289] 0 30289 316552 22754 475136 0 0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30292] 0 30292 324793 23720 466944 0 0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30293] 0 30293 31705 5341 163840 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30297] 0 30297 317199 25089 487424 0 0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30298] 0 30298 257592 29958 622592 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30303] 0 30303 31705 5365 163840 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30304] 0 30304 320153 23170 462848 0 0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30310] 0 30310 316439 22944 483328 0 0 frigate.capture
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30312] 0 30312 307946 31652 647168 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 30315] 0 30315 31705 5875 167936 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 31302] 0 31302 306464 26822 626688 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 3524] 0 3524 32871 6940 196608 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 4705] 0 4705 824549 629241 5427200 0 0 ffmpeg
Apr 28 09:18:46 UNRAIDSRV kernel: [ 5670] 0 5670 622 216 40960 0 0 sleep
Apr 28 09:18:46 UNRAIDSRV kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,mems_allowed=0,oom_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task=ffmpeg,pid=4705,uid=0
Apr 28 09:18:46 UNRAIDSRV kernel: Memory cgroup out of memory: Killed process 4705 (ffmpeg) total-vm:3298196kB, anon-rss:2452320kB, file-rss:64644kB, shmem-rss:0kB, UID:0 pgtables:5300kB oom_score_adj:0
Apr 28 09:20:24 UNRAIDSRV kernel: ffmpeg invoked oom-killer: gfp_mask=0x8c40(GFP_NOFS|__GFP_NOFAIL), order=0, oom_score_adj=0
Apr 28 09:20:24 UNRAIDSRV kernel: CPU: 6 PID: 6126 Comm: ffmpeg Tainted: G O 5.19.17-Unraid #2
Apr 28 09:20:24 UNRAIDSRV kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4, BIOS P4.20 06/18/2020
Apr 28 09:20:24 UNRAIDSRV kernel: Call Trace:
Apr 28 09:20:24 UNRAIDSRV kernel: <TASK>
Apr 28 09:20:24 UNRAIDSRV kernel: dump_stack_lvl+0x44/0x5c
Apr 28 09:20:24 UNRAIDSRV kernel: dump_header+0x4a/0x1ff
Apr 28 09:20:24 UNRAIDSRV kernel: oom_kill_process+0x80/0x111
Apr 28 09:20:24 UNRAIDSRV kernel: out_of_memory+0x3e8/0x41a
Apr 28 09:20:24 UNRAIDSRV kernel: mem_cgroup_out_of_memory+0x7c/0xb2
Apr 28 09:20:24 UNRAIDSRV kernel: try_charge_memcg+0x44a/0x55e
Apr 28 09:20:24 UNRAIDSRV kernel: ? get_page_from_freelist+0x6ff/0x82d
Apr 28 09:20:24 UNRAIDSRV kernel: charge_memcg+0x29/0x71
Apr 28 09:20:24 UNRAIDSRV kernel: __mem_cgroup_charge+0x29/0x41
Apr 28 09:20:24 UNRAIDSRV kernel: __filemap_add_folio+0xb9/0x34b
Apr 28 09:20:24 UNRAIDSRV kernel: ? lruvec_page_state+0x43/0x43
Apr 28 09:20:24 UNRAIDSRV kernel: filemap_add_folio+0x37/0x91
Apr 28 09:20:24 UNRAIDSRV kernel: __filemap_get_folio+0x1a4/0x1ff
Apr 28 09:20:24 UNRAIDSRV kernel: pagecache_get_page+0x13/0x8c
Apr 28 09:20:24 UNRAIDSRV kernel: alloc_extent_buffer+0x12d/0x38b
Apr 28 09:20:24 UNRAIDSRV kernel: ? bio_add_page+0x5a/0x80
Apr 28 09:20:24 UNRAIDSRV kernel: read_tree_block+0x21/0x7f
Apr 28 09:20:24 UNRAIDSRV kernel: read_block_for_search+0x200/0x27d
Apr 28 09:20:24 UNRAIDSRV kernel: btrfs_search_slot+0x6f7/0x7c5
Apr 28 09:20:24 UNRAIDSRV kernel: btrfs_lookup_csum+0x5b/0xfd
Apr 28 09:20:24 UNRAIDSRV kernel: btrfs_lookup_bio_sums+0x1f4/0x4a2
Apr 28 09:20:24 UNRAIDSRV kernel: btrfs_submit_data_bio+0x102/0x18b
Apr 28 09:20:24 UNRAIDSRV kernel: read_pages+0x4a/0xe9
Apr 28 09:20:24 UNRAIDSRV kernel: page_cache_ra_unbounded+0x10c/0x13f
Apr 28 09:20:24 UNRAIDSRV kernel: filemap_fault+0x2e7/0x524
Apr 28 09:20:24 UNRAIDSRV kernel: __do_fault+0x30/0x6e
Apr 28 09:20:24 UNRAIDSRV kernel: __handle_mm_fault+0x9a5/0xc7d
Apr 28 09:20:24 UNRAIDSRV kernel: ? _raw_spin_unlock_irq+0x1a/0x2f
Apr 28 09:20:24 UNRAIDSRV kernel: handle_mm_fault+0x113/0x1d7
Apr 28 09:20:24 UNRAIDSRV kernel: do_user_addr_fault+0x36a/0x514
Apr 28 09:20:24 UNRAIDSRV kernel: exc_page_fault+0xfc/0x11e
Apr 28 09:20:24 UNRAIDSRV kernel: asm_exc_page_fault+0x22/0x30
Apr 28 09:20:24 UNRAIDSRV kernel: RIP: 0033:0x1505b00e22dc
Apr 28 09:20:24 UNRAIDSRV kernel: Code: 8a e3 dc ff 31 d2 83 78 44 03 0f 84 ee 04 00 00 41 0f b6 87 08 05 00 00 c1 e2 05 b9 72 00 00 00 48 89 ee 4d 8d 8f e8 00 00 00 <f3> 0f 10 05 60 29 4a 00 83 e0 df 4c 89 cf 09 d0 49 8d 97 ec 04 00
Apr 28 09:20:24 UNRAIDSRV kernel: RSP: 002b:00007fff99f13b00 EFLAGS: 00010256
Apr 28 09:20:24 UNRAIDSRV kernel: RAX: 0000000000000000 RBX: 000055bc1b1e1050 RCX: 0000000000000072
Apr 28 09:20:24 UNRAIDSRV kernel: RDX: 0000000000000000 RSI: 00007fff99f13de0 RDI: 0000000000000031
Apr 28 09:20:24 UNRAIDSRV kernel: RBP: 00007fff99f13de0 R08: 0000000000000007 R09: 000055bc8e791228
Apr 28 09:20:24 UNRAIDSRV kernel: R10: 70ace78c2583f88f R11: 0000000000000000 R12: 0000000000000000
Apr 28 09:20:24 UNRAIDSRV kernel: R13: 0000000000000000 R14: 00007fff99f13c90 R15: 000055bc8e791140
Apr 28 09:20:24 UNRAIDSRV kernel: </TASK>
Apr 28 09:20:24 UNRAIDSRV kernel: memory: usage 3145728kB, limit 3145728kB, failcnt 104946
Apr 28 09:20:24 UNRAIDSRV kernel: memory+swap: usage 3145728kB, limit 6291456kB, failcnt 0
Apr 28 09:20:24 UNRAIDSRV kernel: kmem: usage 35964kB, limit 9007199254740988kB, failcnt 0
Apr 28 09:20:24 UNRAIDSRV kernel: Memory cgroup stats for /docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c:
Apr 28 09:20:24 UNRAIDSRV kernel: anon 3162849280
Apr 28 09:20:24 UNRAIDSRV kernel: file 21549056
Apr 28 09:20:24 UNRAIDSRV kernel: kernel 36827136
Apr 28 09:20:24 UNRAIDSRV kernel: kernel_stack 7700480
Apr 28 09:20:24 UNRAIDSRV kernel: pagetables 17899520
Apr 28 09:20:24 UNRAIDSRV kernel: percpu 10064
Apr 28 09:20:24 UNRAIDSRV kernel: sock 0
Apr 28 09:20:24 UNRAIDSRV kernel: vmalloc 0
Apr 28 09:20:24 UNRAIDSRV kernel: shmem 21434368
Apr 28 09:20:24 UNRAIDSRV kernel: file_mapped 5079040
Apr 28 09:20:24 UNRAIDSRV kernel: file_dirty 0
Apr 28 09:20:24 UNRAIDSRV kernel: file_writeback 0
Apr 28 09:20:24 UNRAIDSRV kernel: swapcached 0
Apr 28 09:20:24 UNRAIDSRV kernel: anon_thp 809500672
Apr 28 09:20:24 UNRAIDSRV kernel: file_thp 0
Apr 28 09:20:24 UNRAIDSRV kernel: shmem_thp 0
Apr 28 09:20:24 UNRAIDSRV kernel: inactive_anon 3175710720
Apr 28 09:20:24 UNRAIDSRV kernel: active_anon 8548352
Apr 28 09:20:24 UNRAIDSRV kernel: inactive_file 114688
Apr 28 09:20:24 UNRAIDSRV kernel: active_file 0
Apr 28 09:20:24 UNRAIDSRV kernel: unevictable 0
Apr 28 09:20:24 UNRAIDSRV kernel: slab_reclaimable 1274104
Apr 28 09:20:24 UNRAIDSRV kernel: slab_unreclaimable 8701640
Apr 28 09:20:24 UNRAIDSRV kernel: slab 9975744
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_refault_anon 0
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_refault_file 284955
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_activate_anon 0
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_activate_file 23324
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_restore_anon 0
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_restore_file 3963
Apr 28 09:20:24 UNRAIDSRV kernel: workingset_nodereclaim 0
Apr 28 09:20:24 UNRAIDSRV kernel: pgfault 375927252
Apr 28 09:20:24 UNRAIDSRV kernel: pgmajfault 1573
Apr 28 09:20:24 UNRAIDSRV kernel: pgrefill 76636
Apr 28 09:20:24 UNRAIDSRV kernel: pgscan 621258
Apr 28 09:20:24 UNRAIDSRV kernel: pgsteal 321885
Apr 28 09:20:24 UNRAIDSRV kernel: pgactivate 107798065
Apr 28 09:20:24 UNRAIDSRV kernel: pgdeactivate 76233
Apr 28 09:20:24 UNRAIDSRV kernel: pglazyfree 0
Apr 28 09:20:24 UNRAIDSRV kernel: pglazyfreed 0
Apr 28 09:20:24 UNRAIDSRV kernel: thp_fault_alloc 17912
Apr 28 09:20:24 UNRAIDSRV kernel: thp_collapse_alloc 143
Apr 28 09:20:24 UNRAIDSRV kernel: Tasks state (memory values in pages):
Apr 28 09:20:24 UNRAIDSRV kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29675] 0 29675 53 16 24576 0 0 s6-svscan
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29747] 0 29747 53 14 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29748] 0 29748 50 1 24576 0 0 s6-linux-init-s
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29756] 0 29756 53 16 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29757] 0 29757 53 15 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29758] 0 29758 53 16 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29759] 0 29759 53 15 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29760] 0 29760 53 15 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29761] 0 29761 53 15 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29762] 0 29762 53 16 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29763] 0 29763 53 16 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29764] 0 29764 53 16 24576 0 0 s6-supervise
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29773] 0 29773 47 1 24576 0 0 s6-ipcserverd
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29774] 0 29774 131 41 24576 0 0 s6-fdholderd
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29812] 65534 29812 69 27 36864 0 0 s6-log
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29813] 65534 29813 69 28 36864 0 0 s6-log
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29814] 65534 29814 70 27 36864 0 0 s6-log
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29821] 0 29821 181857 7306 143360 0 0 go2rtc
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29830] 0 29830 973 729 40960 0 0 bash
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29833] 0 29833 618176 44057 905216 0 0 python3
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29849] 0 29849 145420 3477 102400 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29866] 0 29866 161947 1582 212992 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29867] 0 29867 161947 1582 212992 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29868] 0 29868 161947 1582 212992 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29875] 0 29875 161947 1582 212992 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29897] 0 29897 161947 1582 212992 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29932] 0 29932 161947 1582 212992 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29962] 0 29962 161947 1582 212992 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 29970] 0 29970 161947 1582 212992 0 0 nginx
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30266] 0 30266 212817 19818 376832 0 0 frigate.logger
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30272] 0 30272 3564 2643 65536 0 0 python3
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30273] 0 30273 301653 26373 479232 0 0 frigate.detecto
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30275] 0 30275 349154 24375 524288 0 0 frigate.output
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30278] 0 30278 366610 25268 536576 0 0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30279] 0 30279 31705 5339 172032 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30280] 0 30280 374628 33601 606208 0 0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30284] 0 30284 371525 30248 573440 0 0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30285] 0 30285 31705 4858 163840 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30286] 0 30286 369682 28567 565248 0 0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30287] 0 30287 31705 4444 163840 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30288] 0 30288 316513 23311 495616 0 0 frigate.process
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30289] 0 30289 316552 22753 475136 0 0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30292] 0 30292 324793 23719 466944 0 0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30293] 0 30293 31705 5305 163840 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30297] 0 30297 317199 25089 487424 0 0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30298] 0 30298 257592 29886 622592 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30303] 0 30303 31705 5332 163840 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30304] 0 30304 320153 23169 462848 0 0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30310] 0 30310 316439 22944 483328 0 0 frigate.capture
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30312] 0 30312 307946 31582 647168 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 30315] 0 30315 31705 5843 167936 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 31302] 0 31302 306976 26775 634880 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 3524] 0 3524 32871 6863 196608 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 6126] 0 6126 858357 629580 5435392 0 0 ffmpeg
Apr 28 09:20:24 UNRAIDSRV kernel: [ 7102] 0 7102 622 213 40960 0 0 sleep
Apr 28 09:20:24 UNRAIDSRV kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,mems_allowed=0,oom_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task_memcg=/docker/e67310a09985ceac4e8d2d0de8bb11b31ac2dd1c2f832ea1edee01d942cf6c0c,task=ffmpeg,pid=6126,uid=0
Apr 28 09:20:24 UNRAIDSRV kernel: Memory cgroup out of memory: Killed process 6126 (ffmpeg) total-vm:3433428kB, anon-rss:2454116kB, file-rss:64204kB, shmem-rss:0kB, UID:0 pgtables:5308kB oom_score_adj:0
We need to try reducing the configs to the bare minimum and see where this happens when adding things back.
Just use one camera. Remove the hwaccel args. Remove the go2rtc config.
If this works without any oom issues, then add things back one at a time in this order:
Let's see where the oom errors come up.
@johnvprcam24573455623 unless your config is identical to the one on this issue, I have no idea if these steps are relevant for you.
I am also having this issue. It started right after enabling the hwaccel and seems to happen randomly. Let me know if there is anything I can do to help debug.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
So some more observations:
yes, this is a known issue with the currently included go2rtc 1.2.0. You can update the internal go2rtc to a newer version which many users have said fixes the issue https://docs.frigate.video/configuration/advanced#custom-go2rtc-version
Thank you both. I can confirm 2 of my cameras are pieces of crap (ezviz) on WiFi and have constant ffmpeg errors with and without go2rtc. For now I have just set a memory limit for my pod to prevent it from effecting my whole system. I will probably hold out until go2rtc is updated officially or in an RC.
I am wondering if we have an ETA for the latest go2rtc update in Frigate?
Thanks!
It has already been updated in the dev branch, will be part of 0.13
Here are some more logs after upgrade to latest dev container image: dev-7e5abe7
2023-06-13 14:20:32.141309371 [2023-06-13 14:20:32] watchdog.front_door ERROR : Ffmpeg process crashed unexpectedly for front_door. 2023-06-13 14:20:32.141354140 [2023-06-13 14:20:32] watchdog.front_door ERROR : The following ffmpeg logs include the last 100 lines prior to exit. 2023-06-13 14:20:32.141423074 [2023-06-13 14:20:32] ffmpeg.front_door.detect ERROR : [rtsp @ 0x55e1df52e100] RTP: PT=60: bad cseq 9055 expected=9e97 2023-06-13 14:20:32.141788835 [2023-06-13 14:20:32] ffmpeg.front_door.detect ERROR : [segment @ 0x55e1df541640] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly 2023-06-13 14:20:32.141816981 [2023-06-13 14:20:32] ffmpeg.front_door.detect ERROR : [out_1_0 @ 0x55e1df850480] 100 buffers queued in out_1_0, something may be wrong. 2023-06-13 14:20:32.141862378 [2023-06-13 14:20:32] ffmpeg.front_door.detect ERROR : [out_1_0 @ 0x55e1df850480] 1000 buffers queued in out_1_0, something may be wrong.
to be clear I was saying the OOM is fixed, the actual ffmpeg issues will come down to your camera
to be clear I was saying the OOM is fixed, the actual ffmpeg issues will come down to your camera
roger thank you!
What is weird is that using mpv the camera playback is perfectly smooth and fine..
❯ mpv -v rtsp://admin:xxx@spycam1.denof.sin/live
[cplayer] Command line options: '-v' 'rtsp://admin:xxx@spycam1.denof.sin/live'
[cplayer] mpv 0.35.1 Copyright © 2000-2023 mpv/MPlayer/mplayer2 projects
[cplayer] built on UNKNOWN
[cplayer] FFmpeg library versions:
[cplayer] libavutil 56.70.100
[cplayer] libavcodec 58.134.100
[cplayer] libavformat 58.76.100
[cplayer] libswscale 5.9.100
[cplayer] libavfilter 7.110.100
[cplayer] libswresample 3.9.100
[cplayer] FFmpeg version: 4.4.4
[cplayer]
[cplayer] Configuration: meson configure build -Dprefix=/usr -Dbuildtype=plain -Doptimization=plain
[cplayer] List of enabled features: alsa cdda cplayer cplugins dmabuf-interop-gl dmabuf-interop-pl dmabuf-wayland drm drm-is-kms dvdnav egl egl-drm egl-helpers egl-x11 ffmpeg gbm gl gl-wayland gl-x11 glibc-thread-name glob glob-posix gpl html-build iconv jpeg lcms2 libarchive libass libavdevice libdl libm libmpv-shared libplacebo libplacebo-next librt linux-fstatfs luajit manpage-build memfd_create noexecstack pipewire plain-gl posix pulse shaderc stdatomic threads uchardet vaapi vaapi-drm vaapi-egl vaapi-libplacebo vaapi-wayland vaapi-x-egl vaapi-x11 vector vt.h vulkan wayland wayland_protocols_1_24 x11 xv zlib
[cplayer] Built with NDEBUG.
[cplayer] Reading config file /etc/mpv/encoding-profiles.conf
[cplayer] Applying profile 'default'...
[cplayer] Reading config file /home/vladi/.config/mpv/config
[cplayer] Applying profile 'default'...
[cplayer] Setting option 'hwdec' = 'vaapi' (flags = 4)
[cplayer] Setting option 'vo' = 'gpu' (flags = 4)
[cplayer] Setting option 'gpu-context' = 'wayland' (flags = 4)
[cplayer] Setting option 'ao' = 'pulse' (flags = 4)
[cplayer] Setting option 'v' = '' (flags = 8)
[cplayer] Waiting for scripts...
[osd/libass] libass API version: 0x1701000
[osd/libass] libass source: tarball: 0.17.1
[osd/libass] Shaper: FriBidi 1.0.12 (SIMPLE) HarfBuzz-ng 7.3.0 (COMPLEX)
[osd/libass] Setting up fonts...
[osd/libass] Using font provider fontconfig
[osd/libass] Done.
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Done loading scripts.
[cplayer] Running hook: ytdl_hook/on_load
[ytdl_hook] ytdl:// hook
[ytdl_hook] not a ytdl:// url
[ffmpeg] Opening rtsp://admin:xxx@spycam1.denof.sin/live
[cplayer] Set property: shared-script-properties -> 1
[demux] Trying demuxers for level=request.
[cplayer] Set property: shared-script-properties -> 1
[osd/libass] libass API version: 0x1701000
[osd/libass] libass source: tarball: 0.17.1
[osd/libass] Shaper: FriBidi 1.0.12 (SIMPLE) HarfBuzz-ng 7.3.0 (COMPLEX)
[osd/libass] Setting up fonts...
[lavf] Found 'rtsp' at score=100 size=0 (forced).
[lavf] Broken FFmpeg RTSP API => not setting timeout.
[osd/libass] Using font provider fontconfig
[osd/libass] Done.
[lavf] Could not set AVOption tls_verify='0'
[lavf] Could not set AVOption icy='1'
[lavf] avformat_find_stream_info() finished after 0 bytes.
[demux] Detected file format: rtsp (libavformat)
[demux] Stream is not seekable.
[cplayer] Opening done: rtsp://admin:xxxx@spycam1.denof.sin/live
[cplayer] Running hook: ytdl_hook/on_preloaded
[lavf] select track 0
[lavf] select track 1
[cplayer] (+) Video --vid=1 (h264 1920x1080 20.000fps)
[cplayer] (+) Audio --aid=1 (pcm_alaw 1ch 16000Hz)
[display-tags] File tags:
[display-tags] Comment: live
[display-tags] Title: Session streamed by the WYZE Media Server
[vo/gpu/opengl] Initializing GPU context 'wayland'
[vo/gpu/wayland] Registered for protocol wl_shm
[vo/gpu/wayland] Registered for protocol zwp_linux_dmabuf_v1
[vo/gpu/wayland] Registered for protocol wl_compositor
[vo/gpu/wayland] Registered for protocol wl_subcompositor
[vo/gpu/wayland] Registered for protocol wl_data_device_manager
[vo/gpu/wayland] Registered for protocol wp_viewporter
[vo/gpu/wayland] Registered for protocol xdg_wm_base
[vo/gpu/wayland] Registered for protocol wl_seat
[vo/gpu/wayland] Registered for protocol wp_presentation
[vo/gpu/wayland] Registered for protocol zxdg_decoration_manager_v1
[vo/gpu/wayland] Registered for protocol zwp_idle_inhibit_manager_v1
[vo/gpu/wayland] Registered for protocol wl_output
[vo/gpu/wayland] Registered for protocol wl_output
[vo/gpu/wayland] Registered output Acer Technologies VG271U (0x31):
[vo/gpu/wayland] x: 0px, y: 0px
[vo/gpu/wayland] w: 2560px (600mm), h: 1440px (340mm)
[vo/gpu/wayland] scale: 1
[vo/gpu/wayland] Hz: 119.998000
[vo/gpu/wayland] Registered output Ancor Communications Inc ASUS MG279 (0x32):
[vo/gpu/wayland] x: 0px, y: 0px
[vo/gpu/wayland] w: 2560px (600mm), h: 1440px (340mm)
[vo/gpu/wayland] scale: 1
[vo/gpu/wayland] Hz: 119.878000
[vo/gpu/wayland] Enabling server decorations
[vo/gpu/opengl] EGL_VERSION=1.5
[vo/gpu/opengl] EGL_VENDOR=Mesa Project
[vo/gpu/opengl] EGL_CLIENT_APIS=OpenGL OpenGL_ES
[vo/gpu/opengl] Trying to create Desktop OpenGL context.
[vo/gpu/wayland] GL_VERSION='4.6 (Core Profile) Mesa 23.0.3'
[vo/gpu/wayland] Detected desktop OpenGL 4.6.
[vo/gpu/wayland] GL_VENDOR='AMD'
[vo/gpu/wayland] GL_RENDERER='AMD Radeon RX 5700 XT (navi10, LLVM 15.0.7, DRM 3.52, 6.3.7)'
[vo/gpu/wayland] GL_SHADING_LANGUAGE_VERSION='4.60'
[vo/gpu/wayland] Loaded extension GL_KHR_debug.
[vo/gpu] Testing FBO format rgba16f
[vo/gpu] Using FBO format rgba16f.
[vo/gpu] No advanced processing required. Enabling dumb mode.
[vd] Container reported FPS: 20.000000
[vd] Codec list:
[vd] h264 - H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
[vd] h264_v4l2m2m (h264) - V4L2 mem2mem H.264 decoder wrapper
[vd] Opening decoder h264
[vd] Looking at hwdec h264-vaapi...
[vo/gpu] Loading hwdec drivers for format: 'vaapi'
[vo/gpu] Loading hwdec driver 'vaapi'
[vo/gpu/vaapi] using EGL dmabuf interop
[vo/gpu/vaapi] Trying to open a x11 VA display...
[vo/gpu/vaapi] Trying to open a wayland VA display...
[vo/gpu/vaapi/vaapi] Initialized VAAPI: version 1.18
[vo/gpu/vaapi] Going to probe surface formats (may log bogus errors)...
[vo/gpu/vaapi] Done probing surface formats.
[vd] Trying hardware decoding via h264-vaapi.
[vd] Selected codec: h264 (H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10)
[vf] User filter list:
[vf] (empty)
[ad] Codec list:
[ad] pcm_alaw - PCM A-law / G.711 A-law
[ad] Opening decoder pcm_alaw
[ad] Requesting 1 threads for decoding.
[ad] Selected codec: pcm_alaw (PCM A-law / G.711 A-law)
[af] User filter list:
[af] (empty)
[cplayer] Starting playback...
[vd] Pixel formats supported by decoder: vaapi_vld yuv420p
[vd] Codec profile: Main (0x4d)
[vd] Requesting pixfmt 'vaapi_vld' from decoder.
[vd] Using hardware decoding (vaapi).
[vd] Decoder format: 1920x1080 [0:1] vaapi[nv12] bt.709/bt.709/bt.1886/limited/auto CL=mpeg2/4/h264
[vf] [in] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [userdeint] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [userdeint] (disabled)
[vf] [autorotate] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [autorotate] (disabled)
[vf] [convert] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vf] [convert] (disabled)
[vf] [out] 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[af] [in] 16000Hz mono 1ch s16
[af] [userspeed] 16000Hz mono 1ch s16
[af] [userspeed] (disabled)
[af] [convert] 16000Hz mono 1ch s16
[ao] Trying audio driver 'pulse'
[ao/pulse] requested format: 16000 Hz, mono channels, s16
[ao/pulse] Library version: 16.1.0
[ao/pulse] Proto: 35
[ao/pulse] Server proto: 4294967295
[ao/pulse] Channel layouts:
[ao/pulse] - #fl
[ao/pulse] - #fr
[ao/pulse] - #fc
[ao/pulse] - #lfe
[ao/pulse] - #bl
[ao/pulse] - #br
[ao/pulse] - #flc
[ao/pulse] - #frc
[ao/pulse] - #bc
[ao/pulse] - #sl
[ao/pulse] - #sr
[ao/pulse] - #tc
[ao/pulse] - #tfl
[ao/pulse] - #tfc
[ao/pulse] - #tfr
[ao/pulse] - #tbl
[ao/pulse] - #tbc
[ao/pulse] - #tbr
[ao/pulse] result: mono
[ao/pulse] device buffer: 1600 samples.
[ao/pulse] using soft-buffer of 3200 samples.
[cplayer] AO: [pulse] 16000Hz mono 1ch s16
[cplayer] AO: Description: PulseAudio audio output
[af] [convert] (disabled)
[af] [out] 16000Hz mono 1ch s16
[cplayer] VO: [gpu] 1920x1080 vaapi[nv12]
[cplayer] VO: Description: Shader-based GPU Renderer
[vo/gpu] reconfig to 1920x1080 vaapi[nv12] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[vo/gpu/wayland] Reconfiguring!
[vo/gpu] Resize: 1920x1080
[vo/gpu] Window size: 1920x1080 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 1920x1080
[vo/gpu] Video scale: 1.000000/1.000000
[vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[vo/gpu] Video borders: l=0 t=0 r=0 b=0
[vo/gpu] Testing FBO format rgba16f
[vo/gpu] Using FBO format rgba16f.
[vo/gpu] No advanced processing required. Enabling dumb mode.
[vo/gpu/wayland] Handling resize on the egl side
[vo/gpu] Resize: 1920x1080
[vo/gpu] Window size: 1920x1080 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 1920x1080
[vo/gpu] Video scale: 1.000000/1.000000
[vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[vo/gpu] Video borders: l=0 t=0 r=0 b=0
[vo/gpu] Reported display depth: 10
[vo/gpu/wayland] Given DND offer with mime type text/plain;charset=utf-8
[vo/gpu/wayland] Received a new DND offer. Releasing the previous offer.
[vo/gpu/wayland] Surface entered output Acer Technologies VG271U (0x31), scale = 1, refresh rate = 119.998000 Hz
[vo/gpu/wayland] Resizing due to xdg from 1920x1080 to 826x678
[cplayer] first video frame after restart shown
[vo/gpu] Resize: 1920x1080
[vo/gpu] Window size: 1920x1080 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 1920x1080
[vo/gpu] Video scale: 1.000000/1.000000
[vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[vo/gpu] Video borders: l=0 t=0 r=0 b=0
[vo/gpu] Reported display depth: 10
[vo/gpu/wayland] Handling resize on the egl side
[vo/gpu] Resize: 826x678
[vo/gpu] Window size: 826x678 (Borders: l=0 t=0 r=0 b=0)
[vo/gpu] Video source: 1920x1080 (1:1)
[vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 107) 826x464
[vo/gpu] Video scale: 0.430208/0.429630
[vo/gpu] OSD borders: l=0 t=107 r=0 b=107
[vo/gpu] Video borders: l=0 t=107 r=0 b=107
[vo/gpu] Reported display depth: 10
[vo/gpu] Assuming 119.998000 FPS for display sync.
[cplayer] audio ready
[cplayer] starting audio playback
[cplayer] playback restart complete @ 1.851000, audio=playing, video=playing
[vo/gpu/wayland] Enabling idle inhibitor
[ao/pulse] starting AO
[statusline] AV: 00:00:01 / 00:00:03 (58%) A-V: 0.003
[cplayer] Set property: shared-script-properties -> 1
[cplayer] Set property: shared-script-properties -> 1
[statusline] AV: 00:00:07 / 00:00:08 (84%) A-V: -0.005
[vo/gpu/wayland] Received a new DND offer. Releasing the previous offer.
[statusline] AV: 00:00:40 / 00:00:41 (97%) A-V: 0.000
[vo/gpu/wayland] Received a new DND offer. Releasing the previous offer.
[statusline] AV: 00:00:40 / 00:00:41 (97%) A-V: 0.000
[cplayer] Set property: shared-script-properties -> 1
[statusline] AV: 00:00:40 / 00:00:41 (97%) A-V: 0.000
[osd/libass] fontselect: (sans-serif, 400, 0) -> /usr/share/fonts/noto/NotoSans-Regular.ttf, 0, NotoSans-Regular
[osd/libass] fontselect: (mpv-osd-symbols, 400, 0) -> mpv-osd-symbols-Regular, 0, mpv-osd-symbols-Regular
[vo/gpu] Reallocating OSD texture to 1024x256.
[statusline] AV: 00:00:40 / 00:00:42 (97%) A-V: -0.023
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Describe the problem you are having
I am having an oom kill of ffmpeg process when I have preset-rtsp-restrem in the input_args, removing it ffmpeg: inputs:
Version
0.12.0-rc2
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
No response
Operating system
Other Linux
Install method
Docker CLI
Coral version
USB
Network connection
Wired
Camera make and model
wyze
Any other information that may be helpful
No response