search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.73k stars 1.7k forks source link

Memory leak in 0.13.2-6476f8a #10650

Closed andrewhack closed 6 months ago

andrewhack commented 6 months ago

Discussed in https://github.com/blakeblackshear/frigate/discussions/10649

Originally posted by **andrewhack** March 24, 2024 After upgrading to 0.13.2-6476f8a I have frequent errors in all my frigate servers. Frigate in Proxmox containers (LXC) with 2Gb RAM each. Increasing the RAM doesn't help. Call Trace: [46862.703189] [46862.703194] dump_stack_lvl+0x4a/0x63 [46862.703203] dump_stack+0x10/0x16 [46862.703207] dump_header+0x53/0x225 [46862.703214] oom_kill_process.cold+0xb/0x10 [46862.703219] out_of_memory+0x1dc/0x530 [46862.703232] mem_cgroup_out_of_memory+0x145/0x160 [46862.703238] try_charge_memcg+0x69d/0x760 [46862.703245] charge_memcg+0x45/0xb0 [46862.703249] __mem_cgroup_charge+0x2d/0x90 [46862.703253] __add_to_page_cache_locked+0x2e1/0x360 [46862.703258] ? scan_shadow_nodes+0x40/0x40 [46862.703265] add_to_page_cache_lru+0x4d/0xd0 [46862.703269] pagecache_get_page+0x1c2/0x5c0 [46862.703274] filemap_fault+0x488/0xb10 [46862.703277] ? filemap_map_pages+0x139/0x660 [46862.703282] __do_fault+0x3c/0x120 [46862.703287] __handle_mm_fault+0xde8/0x1610 [46862.703293] handle_mm_fault+0xd8/0x2c0 [46862.703297] do_user_addr_fault+0x1c2/0x660 [46862.703304] ? exit_to_user_mode_prepare+0x90/0x1b0 [46862.703311] exc_page_fault+0x77/0x170 [46862.703318] asm_exc_page_fault+0x27/0x30 [46862.703325] RIP: 0033:0x7fd9d3a1f9f0 [46862.703334] Code: Unable to access opcode bytes at RIP 0x7fd9d3a1f9c6. [46862.703336] RSP: 002b:00007ffec21c80e8 EFLAGS: 00010246 [46862.703340] RAX: 0000000000000000 RBX: 0000560af8de2f48 RCX: 0000000000000000 [46862.703343] RDX: ffffffffffffff80 RSI: 000000000000000b RDI: 0000560af8de2f10 [46862.703346] RBP: 00007fd9d3bae240 R08: 0000000000000000 R09: 0000000000000000 [46862.703348] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [46862.703350] R13: 0000560af8de3430 R14: 0000560af8de2f10 R15: 0000560af8de3a40 [46862.703356] [46862.703371] memory: usage 2097252kB, limit 2097152kB, failcnt 30836 [46862.703375] swap: usage 0kB, limit 0kB, failcnt 0 [46862.703377] Memory cgroup stats for /lxc/2203: [46862.703399] anon 2009395200 [46862.703399] file 51322880 [46862.703399] kernel_stack 14172160 [46862.703399] pagetables 22544384 [46862.703399] percpu 2007424 [46862.703399] sock 23662592 [46862.703399] shmem 51138560 [46862.703399] file_mapped 14696448 [46862.703399] file_dirty 0 [46862.703399] file_writeback 45056 [46862.703399] swapcached 0 [46862.703399] anon_thp 127926272 [46862.703399] file_thp 0 [46862.703399] shmem_thp 0 [46862.703399] inactive_anon 2043564032 [46862.703399] active_anon 16969728 [46862.703399] inactive_file 159744 [46862.703399] active_file 24576 [46862.703399] unevictable 0 [46862.703399] slab_reclaimable 1844824 [46862.703399] slab_unreclaimable 19508376 [46862.703399] slab 21353200 [46862.703399] workingset_refault_anon 0 [46862.703399] workingset_refault_file 1548838 [46862.703399] workingset_activate_anon 0 [46862.703399] workingset_activate_file 106220 [46862.703399] workingset_restore_anon 0 [46862.703406] Tasks state (memory values in pages): [46862.703407] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [46862.703411] [ 824978] 0 824978 41397 1631 94208 0 0 systemd [46862.703425] [ 825778] 0 825778 1842 282 53248 0 0 agetty [46862.703430] [ 839725] 0 839725 3006 578 65536 0 0 login [46862.703435] [3918613] 0 3918613 2511 678 53248 0 0 bash [46862.703457] [3918651] 0 3918651 3012 585 57344 0 0 top [46862.703463] [ 825604] 0 825604 27477 2292 241664 0 -250 systemd-journal [46862.703469] [ 825626] 104 825626 4012 841 77824 0 0 systemd-network [46862.703474] [ 825650] 0 825650 24970 902 69632 0 0 dhclient [46862.703479] [ 825744] 107 825744 1975 517 57344 0 0 rpcbind [46862.703484] [ 825745] 105 825745 6025 1869 90112 0 0 systemd-resolve [46862.703516] [ 825753] 0 825753 2152 408 53248 0 0 cron [46862.703522] [ 825754] 106 825754 2062 605 57344 0 -900 dbus-daemon [46862.703527] [ 825756] 0 825756 55199 495 77824 0 0 rsyslogd [46862.703532] [ 825757] 0 825757 3350 1012 61440 0 0 systemd-logind [46862.703537] [ 825771] 0 825771 339365 4650 282624 0 -999 containerd [46862.703542] [ 827736] 0 827736 180082 1315 110592 0 -998 containerd-shim [46862.703547] [3918798] 0 3918798 290946 1463 167936 0 -998 runc [46862.703552] [ 826044] 110 826044 4651 526 81920 0 0 exim4 [46862.703557] [ 825768] 0 825768 90050 5532 237568 0 0 ruby [46862.703562] [ 825795] 112 825795 5335 473 69632 0 0 zabbix_agentd [46862.703566] [ 825805] 112 825805 5335 509 69632 0 0 zabbix_agentd [46862.703571] [ 825806] 112 825806 5390 1036 81920 0 0 zabbix_agentd [46862.703575] [ 825807] 112 825807 5390 1055 81920 0 0 zabbix_agentd [46862.703579] [ 825808] 112 825808 5390 1052 81920 0 0 zabbix_agentd [46862.703584] [ 825809] 112 825809 5392 1026 81920 0 0 zabbix_agentd [46862.703598] [ 825775] 0 825775 1842 280 53248 0 0 agetty [46862.703603] [ 826119] 0 826119 628963 12518 585728 0 -500 dockerd [46862.703609] [ 827648] 0 827648 417930 482 147456 0 -500 docker-proxy [46862.703614] [ 827655] 0 827655 399497 514 139264 0 -500 docker-proxy [46862.703618] [ 827670] 0 827670 417930 504 143360 0 -500 docker-proxy [46862.703623] [ 827676] 0 827676 399497 492 135168 0 -500 docker-proxy [46862.703628] [ 827691] 0 827691 417994 493 143360 0 -500 docker-proxy [46862.703633] [ 827697] 0 827697 417930 498 139264 0 -500 docker-proxy [46862.703638] [ 827713] 0 827713 399561 579 143360 0 -500 docker-proxy [46862.703643] [ 827718] 0 827718 399497 512 135168 0 -500 docker-proxy [46862.703648] [ 827756] 0 827756 52 5 28672 0 0 s6-svscan [46862.703652] [ 827796] 0 827796 53 5 24576 0 0 s6-supervise [46862.703657] [ 827798] 0 827798 50 1 24576 0 0 s6-linux-init-s [46862.703661] [ 827811] 0 827811 53 4 28672 0 0 s6-supervise [46862.703665] [ 827812] 0 827812 53 5 28672 0 0 s6-supervise [46862.703669] [ 827813] 0 827813 53 5 28672 0 0 s6-supervise [46862.703673] [ 827814] 0 827814 53 5 28672 0 0 s6-supervise [46862.703678] [ 827815] 0 827815 53 6 28672 0 0 s6-supervise [46862.703682] [ 827816] 0 827816 53 5 28672 0 0 s6-supervise [46862.703686] [ 827817] 0 827817 53 5 28672 0 0 s6-supervise [46862.703690] [ 827818] 0 827818 53 5 28672 0 0 s6-supervise [46862.703694] [ 827819] 0 827819 53 5 28672 0 0 s6-supervise [46862.703699] [ 827831] 0 827831 131 16 28672 0 0 s6-fdholderd [46862.703713] [ 827833] 0 827833 47 1 24576 0 0 s6-ipcserverd [46862.703718] [ 828029] 65534 828029 69 26 36864 0 0 s6-log [46862.703722] [ 828032] 65534 828032 69 1 40960 0 0 s6-log [46862.703727] [ 828036] 65534 828036 77 32 40960 0 0 s6-log [46862.703731] [ 828040] 0 828040 310467 11535 233472 0 0 go2rtc [46862.703736] [ 828058] 0 828058 973 436 53248 0 0 bash [46862.703740] [ 828061] 0 828061 433874 105637 1572864 0 0 python3 [46862.703744] [ 828073] 0 828073 148012 2102 110592 0 0 nginx [46862.703749] [ 828116] 0 828116 213774 1162 229376 0 0 nginx [46862.703753] [ 828117] 0 828117 213774 1164 229376 0 0 nginx [46862.703757] [ 828118] 0 828118 213774 1096 229376 0 0 nginx [46862.703762] [ 828119] 0 828119 213774 1109 229376 0 0 nginx [46862.703766] [ 828121] 0 828121 213774 1182 229376 0 0 nginx [46862.703770] [ 828122] 0 828122 213774 1094 229376 0 0 nginx [46862.703774] [ 828123] 0 828123 213774 1201 229376 0 0 nginx [46862.703779] [ 828124] 0 828124 213774 1169 229376 0 0 nginx [46862.703783] [ 828125] 0 828125 213774 1083 229376 0 0 nginx [46862.703787] [ 828126] 0 828126 213774 1209 229376 0 0 nginx [46862.703792] [ 828127] 0 828127 213774 1221 229376 0 0 nginx [46862.703796] [ 828128] 0 828128 213774 1196 229376 0 0 nginx [46862.703800] [ 828129] 0 828129 213774 1214 229376 0 0 nginx [46862.703805] [ 828130] 0 828130 213774 1196 229376 0 0 nginx [46862.703809] [ 828131] 0 828131 213774 1186 229376 0 0 nginx [46862.703822] [ 828132] 0 828132 213774 1209 229376 0 0 nginx [46862.703849] [ 828133] 0 828133 148062 876 94208 0 0 nginx [46862.703853] [ 829778] 0 829778 36119 2883 184320 0 0 ffmpeg [46862.703858] [ 830210] 0 830210 36119 2902 192512 0 0 ffmpeg [46862.703862] [ 830436] 0 830436 124174 24589 458752 0 0 frigate.logger [46862.703867] [ 830838] 0 830838 244103 48820 843776 0 0 frigate.recordi [46862.703871] [ 830847] 0 830847 3781 1849 69632 0 0 python3 [46862.703875] [ 830848] 0 830848 253906 33639 626688 0 0 frigate.detecto [46862.703880] [ 830851] 0 830851 351842 32173 684032 0 0 frigate.output [46862.703885] [ 830859] 0 830859 31705 2322 163840 0 0 ffmpeg [46862.703889] [ 830871] 0 830871 31705 2291 167936 0 0 ffmpeg [46862.703893] [ 830872] 0 830872 31705 2288 167936 0 0 ffmpeg [46862.703898] [ 830873] 0 830873 31705 2297 167936 0 0 ffmpeg [46862.703903] [ 830874] 0 830874 31705 2273 167936 0 0 ffmpeg [46862.703907] [ 830917] 0 830917 287727 43797 745472 0 0 frigate.process [46862.703912] [ 830921] 0 830921 287809 43700 749568 0 0 frigate.process [46862.703916] [ 830922] 0 830922 287873 43846 749568 0 0 frigate.process [46862.703920] [ 830924] 0 830924 280641 38990 765952 0 0 frigate.capture [46862.703935] [ 830925] 0 830925 294844 41118 679936 0 0 frigate.capture [46862.703940] [ 830928] 0 830928 284017 36768 634880 0 0 frigate.capture [46862.703945] [ 830933] 0 830933 280641 37883 659456 0 0 frigate.capture [46862.703950] [ 830936] 0 830936 281815 37687 688128 0 0 frigate.audio_m [46862.703956] [ 830943] 0 830943 122583 14126 319488 0 0 ffmpeg [46862.703961] [1514787] 0 1514787 36119 2904 196608 0 0 ffmpeg [46862.703966] [3053514] 0 3053514 121186 13462 299008 0 0 ffmpeg [46862.703970] [3542320] 0 3542320 36349 3144 192512 0 0 ffmpeg [46862.703975] [3651746] 0 3651746 245059 147719 1392640 0 0 ffmpeg [46862.703980] [3831390] 0 3831390 123183 37964 491520 0 0 ffmpeg [46862.703985] [3831408] 0 3831408 37121 3822 204800 0 0 ffmpeg [46862.703989] [3914504] 0 3914504 622 115 40960 0 0 sleep [46862.703994] [3919309] 0 3919309 272513 1118 151552 0 0 runc:[2:INIT] [46862.703999] [ 833560] 0 833560 3338 1178 65536 0 -1000 sshd [46862.704004] [ 841965] 0 841965 43593 11895 315392 0 0 puppet [46862.704010] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=mcollective.service,mems_allowed=0-1,oom_memcg=/lxc/2203,task_memcg=/lxc/2203/ns/system.slice/docker-8c770fc842fc6939e0d2afd9ed8baaac97a81b38bebaa1856a5da4ae383e7316.scope,task=ffmpeg,pid=3651746,uid=0 [46862.704080] Memory cgroup out of memory: Killed process 3651746 (ffmpeg) total-vm:980236kB, anon-rss:583108kB, file-rss:7768kB, shmem-rss:0kB, UID:0 pgtables:1360kB oom_score_adj:0
NickM-27 commented 6 months ago

this is not a feature request