search

SvenVD / rpisurv

Raspberry Pi surveillance
https://community.rpisurv.net
GNU General Public License v2.0
621 stars 101 forks source link

vlc memory leak #170

Closed si458 closed 11 months ago

si458 commented 2 years ago

Hi All, im not sure what to do about this but im having an issue where after its running for a few hours, its starts memory leaking so much so its crashing my pi hard withint 24hours its a Raspberry Pi 2 B Model, i installed the raspbian buster full os version, setup version 3 and it runs fine for a few hours then suddenly crashes, i then activated snmp to watch the pi remotely, and i can see the memory usage just starts to climb after awhile and then every 5mins coming from the vlc processes? the streams are from dahua DVR, sub stream (352x288), the display is a DVI monitor (used hdmi to dvi cable) resolution 1024x768 only 4 streams any suggestions?

si458 commented 1 year ago

few extra logs from debug, might be helpful not sure

Nov  2 10:43:08 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:08 - l_default - DEBUG - free malloc gpu mem value is 9M
Nov  2 10:43:08 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:08 - l_default - DEBUG - total available malloc gpu mem value is 12M
Nov  2 10:43:08 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:08 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:9M outputmem_bytes:9437184.0
Nov  2 10:43:08 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:08 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:12M outputmem_bytes:12582912.0
Nov  2 10:43:09 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:09 - l_default - DEBUG - free reloc gpu mem value is 466M
Nov  2 10:43:09 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:09 - l_default - DEBUG - total available reloc gpu mem value is 492M
Nov  2 10:43:09 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:09 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:466M outputmem_bytes:488636416.0
Nov  2 10:43:09 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:09 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:492M outputmem_bytes:515899392.0
Nov  2 10:43:09 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:09 - l_default - DEBUG - Screen: screen_manager_0_screen1 active_run_time: 18 / 30
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - free malloc gpu mem value is 9M
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - total available malloc gpu mem value is 12M
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:9M outputmem_bytes:9437184.0
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:12M outputmem_bytes:12582912.0
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - free reloc gpu mem value is 466M
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - total available reloc gpu mem value is 492M
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:466M outputmem_bytes:488636416.0
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:492M outputmem_bytes:515899392.0
Nov  2 10:43:10 fthcctvpi rpisurv[7007]: 2022/11/02 10:43:10 - l_default - DEBUG - Screen: screen_manager_0_screen1 active_run_time: 19 / 30
SvenVD commented 1 year ago

Describe "so much so its crashing my pi hard withint 24hours". What does happen? Kernel panic?

si458 commented 1 year ago

Describe "so much so its crashing my pi hard withint 24hours". What does happen? Kernel panic?

It's causing the pi to lock up, and the video streams to freeze

It doesn't seem to kernel panic?

Just the memory usage climbs then the swap starts to climb too til they are both at 100% then the pi stops responding,

But i have noticed if I leave it alone for say 5 hours it does eventually kill the vlc process itself and sorta return to normal

Also the ssh is still accessible, but take about 10 mins to login remotely and then another 10 mins to type the command systemctl restart rpisurv and 10 more mins to run the command

The only fix I have found at the moment is to run a cron every 3 hours and restart rpisurv to avoid the memory leak

SvenVD commented 1 year ago

normally oomkiller should kick in do you see something in the syslogs? Other then that what does top say before it freezes? Does this also occur with one stream?

si458 commented 1 year ago

@SvenVD im checking now for you, i will edit this post with logs as i find stuff but i dont see anything weird and the oomkiller only seems to kick in after about 5 hours when the menmorys at 100% and swaps at 100% too

EDIT: i have spotted these from todays logs but not sure if its revelant or not?

Nov  3 01:33:23 fthcctvpi rpisurv[21485]: [73800668] main input error: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 500 ms)
Nov  3 01:33:23 fthcctvpi rpisurv[21485]: [7350e998] main decoder error: Timestamp conversion failed for 1667439201485001: no reference clock
Nov  3 01:33:23 fthcctvpi rpisurv[21485]: [7350e998] main decoder error: Could not convert timestamp 0 for MMAL decoder
Nov  3 01:33:24 fthcctvpi rpisurv[21485]: [739006b0] main input error: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 500 ms)
Nov  3 01:33:24 fthcctvpi rpisurv[21485]: [7350e998] main decoder error: buffer deadlock prevented
Nov  3 01:33:24 fthcctvpi rpisurv[21485]: [7350ea88] main decoder error: buffer deadlock prevented
Nov  3 01:34:23 fthcctvpi rpisurv[21485]: [739006b0] main input error: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 513 ms)
Nov  3 01:34:23 fthcctvpi rpisurv[21485]: [7350ea88] main decoder error: Timestamp conversion failed for 1667439261854001: no reference clock
Nov  3 01:34:23 fthcctvpi rpisurv[21485]: [7350ea88] main decoder error: Could not convert timestamp 0 for MMAL decoder
Nov  3 01:34:23 fthcctvpi rpisurv[21485]: [73800668] main input error: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 507 ms)
Nov  3 01:34:24 fthcctvpi rpisurv[21485]: [7350ea88] main decoder error: buffer deadlock prevented
Nov  3 01:34:24 fthcctvpi rpisurv[21485]: [7350e998] main decoder error: buffer deadlock prevented

EDIT:

logs from yesterday when the pi crashed due to no memory left (i had debug enabled to help with checking)

Nov  2 05:47:18 fthcctvpi rpisurv[396]: 2022/11/02 05:47:18 - l_default - DEBUG - free reloc gpu mem value is 466M
Nov  2 05:47:18 fthcctvpi rpisurv[396]: 2022/11/02 05:47:18 - l_default - DEBUG - total available reloc gpu mem value is 492M
Nov  2 05:47:18 fthcctvpi rpisurv[396]: 2022/11/02 05:47:18 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:466M outputmem_bytes:488636416.0
Nov  2 05:47:18 fthcctvpi rpisurv[396]: 2022/11/02 05:47:18 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:492M outputmem_bytes:515899392.0
Nov  2 05:47:18 fthcctvpi rpisurv[396]: 2022/11/02 05:47:18 - l_default - DEBUG - Screen: screen_manager_0_screen1 active_run_time: 29 / 30
Nov  2 05:47:22 fthcctvpi rpisurv[396]: 2022/11/02 05:47:21 - l_default - DEBUG - free malloc gpu mem value is 9M
Nov  2 05:47:22 fthcctvpi rpisurv[396]: 2022/11/02 05:47:22 - l_default - DEBUG - total available malloc gpu mem value is 12M
Nov  2 05:47:22 fthcctvpi rpisurv[396]: 2022/11/02 05:47:22 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:9M outputmem_bytes:9437184.0
Nov  2 05:47:22 fthcctvpi rpisurv[396]: 2022/11/02 05:47:22 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:12M outputmem_bytes:12582912.0
Nov  2 05:47:26 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - free reloc gpu mem value is 466M
Nov  2 05:47:26 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - total available reloc gpu mem value is 492M
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:466M outputmem_bytes:488636416.0
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:492M outputmem_bytes:515899392.0
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - Screen: screen_manager_0_screen1 active_run_time: 36 / 30
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - screen_manager_0: only one screen configured, do not rotate
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - Screen: reset_active_timer screen_manager_0_screen1
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - Screen: screen_manager_0_screen1 start_of_active_time: 1667368045.4960794
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - MAIN screen_manager_0: after rotate_next start update_active_screen
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - screen_manager_0: update_active_screen screen_manager_0_screen1
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - Screen: Start polling connectivity for the camera_streams part of screen: screen_manager_0_screen1
Nov  2 05:47:27 fthcctvpi rpisurv[396]: 2022/11/02 05:47:25 - l_default - DEBUG - CameraStream: screen_manager_0_screen1_cam_stream1 rtsp://<hidden_username>:<hidden_password>@192.168.52.123:554/cam/realmonitor?channel=3&subtype=1 Connectable
Nov  2 05:47:28 fthcctvpi rpisurv[396]: 2022/11/02 05:47:27 - l_default - DEBUG - CameraStream: screen_manager_0_screen1_cam_stream2 rtsp://<hidden_username>:<hidden_password>@192.168.52.123:554/cam/realmonitor?channel=8&subtype=1 Connectable
Nov  2 05:47:28 fthcctvpi rpisurv[396]: 2022/11/02 05:47:28 - l_default - DEBUG - CameraStream: screen_manager_0_screen1_cam_stream3 rtsp://<hidden_username>:<hidden_password>@192.168.52.123:554/cam/realmonitor?channel=13&subtype=1 Connectable
Nov  2 05:47:30 fthcctvpi rpisurv[396]: 2022/11/02 05:47:29 - l_default - DEBUG - CameraStream: screen_manager_0_screen1_cam_stream4 rtsp://<hidden_username>:<hidden_password>@192.168.52.123:554/cam/realmonitor?channel=14&subtype=1 Connectable
Nov  2 05:47:30 fthcctvpi rpisurv[396]: 2022/11/02 05:47:29 - l_default - DEBUG - screen_manager_0: update_connectable_camera_streams, disable_probing_for_all_streams is off for this screen, so using probes screen_manager_0_screen1
Nov  2 05:47:30 fthcctvpi rpisurv[396]: 2022/11/02 05:47:30 - l_default - DEBUG - Screen: Connectable camera streams stayed the same, from 4 to 4, screen: screen_manager_0_screen1 does not need full redraw
Nov  2 05:47:30 fthcctvpi rpisurv[396]: 2022/11/02 05:47:30 - l_default - DEBUG - CameraStream: This stream screen_manager_0_screen1_cam_stream1 is not an imageurl, skip refreshing imageurl
Nov  2 05:47:30 fthcctvpi rpisurv[396]: 2022/11/02 05:47:30 - l_default - DEBUG - CameraStream: This stream screen_manager_0_screen1_cam_stream2 is not an imageurl, skip refreshing imageurl
Nov  2 05:47:30 fthcctvpi rpisurv[396]: 2022/11/02 05:47:30 - l_default - DEBUG - CameraStream: This stream screen_manager_0_screen1_cam_stream3 is not an imageurl, skip refreshing imageurl
Nov  2 05:47:30 fthcctvpi rpisurv[396]: 2022/11/02 05:47:30 - l_default - DEBUG - CameraStream: This stream screen_manager_0_screen1_cam_stream4 is not an imageurl, skip refreshing imageurl
Nov  2 05:47:34 fthcctvpi rpisurv[396]: 2022/11/02 05:47:33 - l_default - DEBUG - free malloc gpu mem value is 9M
Nov  2 05:47:34 fthcctvpi rpisurv[396]: 2022/11/02 05:47:34 - l_default - DEBUG - total available malloc gpu mem value is 12M
Nov  2 05:47:35 fthcctvpi rpisurv[396]: 2022/11/02 05:47:34 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:9M outputmem_bytes:9437184.0
Nov  2 05:47:37 fthcctvpi rpisurv[396]: 2022/11/02 05:47:36 - l_default - DEBUG - convert_gpumem_string_to_bytes: inputmem:12M outputmem_bytes:12582912.0
Nov  2 05:48:01 fthcctvpi rpisurv[396]: [73800668] [73800668] [73a006b0] [73700668] main input error: main input error: main input error: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 936 ms)main input error:
Nov  2 05:48:04 fthcctvpi rpisurv[396]: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 943 ms)ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 938 ms)
Nov  2 05:48:07 fthcctvpi rpisurv[396]: ES_OUT_SET_(GROUP_)PCR  is called too late (pts_delay increased to 959 ms)
Nov  2 05:48:11 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 05:48:15 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 05:48:23 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 05:48:36 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 05:49:18 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 05:49:46 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 05:50:27 fthcctvpi kernel: [57014.470638] INFO: task kworker/1:0:14169 blocked for more than 122 seconds.
Nov  2 05:50:28 fthcctvpi kernel: [57014.470667]       Tainted: G         C        5.10.103-v7+ #1529
Nov  2 05:50:28 fthcctvpi kernel: [57014.470677] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov  2 05:50:28 fthcctvpi kernel: [57014.470689] task:kworker/1:0     state:D stack:    0 pid:14169 ppid:     2 flags:0x00000000
Nov  2 05:50:28 fthcctvpi kernel: [57014.470735] Workqueue: events_freezable mmc_rescan
Nov  2 05:50:28 fthcctvpi kernel: [57014.470755] Backtrace: 
Nov  2 05:50:28 fthcctvpi kernel: [57014.470787] [<809fd310>] (__schedule) from [<809fdce8>] (schedule+0x68/0xe4)
Nov  2 05:50:28 fthcctvpi kernel: [57014.470830]  r10:816da000 r9:ffffe000 r8:00000000 r7:00000000 r6:60000013 r5:8de92e80
Nov  2 05:50:28 fthcctvpi kernel: [57014.470842]  r4:ffffe000
Nov  2 05:50:28 fthcctvpi kernel: [57014.470861] [<809fdc80>] (schedule) from [<80808350>] (__mmc_claim_host+0xe0/0x238)
Nov  2 05:50:28 fthcctvpi kernel: [57014.470874]  r5:816da218 r4:00000002
Nov  2 05:50:28 fthcctvpi kernel: [57014.470894] [<80808270>] (__mmc_claim_host) from [<808084e0>] (mmc_get_card+0x38/0x3c)
Nov  2 05:50:28 fthcctvpi kernel: [57014.470911]  r10:00000000 r9:00000000 r8:00000040 r7:9ff91d00 r6:816da218 r5:00000000
Nov  2 05:50:28 fthcctvpi kernel: [57014.470923]  r4:816da800
Nov  2 05:50:28 fthcctvpi kernel: [57014.470942] [<808084a8>] (mmc_get_card) from [<808121b0>] (mmc_sd_detect+0x24/0x7c)
Nov  2 05:50:28 fthcctvpi kernel: [57014.470956]  r5:816da000 r4:816da000
Nov  2 05:50:28 fthcctvpi kernel: [57014.470976] [<8081218c>] (mmc_sd_detect) from [<8080ad44>] (mmc_rescan+0xdc/0x3b0)
Nov  2 05:50:28 fthcctvpi kernel: [57014.470989]  r5:816da000 r4:816da27c
Nov  2 05:50:28 fthcctvpi kernel: [57014.471012] [<8080ac68>] (mmc_rescan) from [<8013bb34>] (process_one_work+0x250/0x5a0)
Nov  2 05:50:28 fthcctvpi kernel: [57014.471029]  r9:00000000 r8:00000040 r7:9ff91d00 r6:9ff8e6c0 r5:8df34180 r4:816da27c
Nov  2 05:50:28 fthcctvpi kernel: [57014.471048] [<8013b8e4>] (process_one_work) from [<8013bee4>] (worker_thread+0x60/0x5c4)
Nov  2 05:50:28 fthcctvpi kernel: [57014.471067]  r10:9ff8e6c0 r9:80f03d00 r8:9ff8e6d8 r7:00000008 r6:9ff8e6c0 r5:8df34194
Nov  2 05:50:28 fthcctvpi kernel: [57014.471078]  r4:8df34180
Nov  2 05:50:28 fthcctvpi kernel: [57014.471097] [<8013be84>] (worker_thread) from [<801439c4>] (kthread+0x170/0x174)
Nov  2 05:50:28 fthcctvpi kernel: [57014.471115]  r10:83b81e74 r9:8df34180 r8:8013be84 r7:81ff8000 r6:00000000 r5:8a32de00
Nov  2 05:50:28 fthcctvpi kernel: [57014.471126]  r4:9097ec00
Nov  2 05:50:28 fthcctvpi kernel: [57014.471143] [<80143854>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Nov  2 05:50:28 fthcctvpi kernel: [57014.471156] Exception stack(0x81ff9fb0 to 0x81ff9ff8)
Nov  2 05:50:28 fthcctvpi kernel: [57014.471171] 9fa0:                                     00000000 00000000 00000000 00000000
Nov  2 05:50:28 fthcctvpi kernel: [57014.471187] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov  2 05:50:28 fthcctvpi kernel: [57014.471202] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Nov  2 05:50:28 fthcctvpi kernel: [57014.471219]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143854
Nov  2 05:50:28 fthcctvpi kernel: [57014.471230]  r4:8a32de00
Nov  2 05:50:31 fthcctvpi kernel: [57018.856190] kworker/2:1H: page allocation failure: order:0, mode:0x40800(GFP_NOWAIT|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0
Nov  2 05:50:31 fthcctvpi kernel: [57018.856250] CPU: 2 PID: 16957 Comm: kworker/2:1H Tainted: G         C        5.10.103-v7+ #1529
Nov  2 05:50:31 fthcctvpi kernel: [57018.856257] Hardware name: BCM2835
Nov  2 05:50:31 fthcctvpi kernel: [57018.856280] Workqueue: kblockd blk_mq_run_work_fn
Nov  2 05:50:31 fthcctvpi kernel: [57018.856292] Backtrace: 
Nov  2 05:50:31 fthcctvpi kernel: [57018.856316] [<809f41d4>] (dump_backtrace) from [<809f4564>] (show_stack+0x20/0x24)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856330]  r7:ffffffff r6:00000000 r5:60000093 r4:80fe5ef0
Nov  2 05:50:31 fthcctvpi kernel: [57018.856345] [<809f4544>] (show_stack) from [<809f8770>] (dump_stack+0xcc/0xf8)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856361] [<809f86a4>] (dump_stack) from [<80306450>] (warn_alloc+0xec/0x17c)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856374]  r10:00040800 r9:80f05008 r8:ffffe000 r7:80d1950c r6:00000000 r5:00000000
Nov  2 05:50:31 fthcctvpi kernel: [57018.856383]  r4:80f05008 r3:a27fa1b6
Nov  2 05:50:31 fthcctvpi kernel: [57018.856397] [<80306364>] (warn_alloc) from [<803075a0>] (__alloc_pages_nodemask+0x10c0/0x1184)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856405]  r3:00000000 r2:80d1950c
Nov  2 05:50:31 fthcctvpi kernel: [57018.856417]  r8:00000000 r7:00000000 r6:00000008 r5:00000001 r4:00000800
Nov  2 05:50:31 fthcctvpi kernel: [57018.856431] [<803064e0>] (__alloc_pages_nodemask) from [<80317668>] (allocate_slab+0x2c4/0x360)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856444]  r10:ffffffff r9:00000001 r8:00000000 r7:00000000 r6:00000008 r5:81401b40
Nov  2 05:50:31 fthcctvpi kernel: [57018.856451]  r4:00000000
Nov  2 05:50:31 fthcctvpi kernel: [57018.856465] [<803173a4>] (allocate_slab) from [<8031ab64>] (___slab_alloc.constprop.27+0x32c/0x56c)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856479]  r10:9ffa56f0 r9:00000154 r8:81401b40 r7:806bf284 r6:00000900 r5:81400ec0
Nov  2 05:50:31 fthcctvpi kernel: [57018.856487]  r4:00000000 r3:003beee0
Nov  2 05:50:31 fthcctvpi kernel: [57018.856500] [<8031a838>] (___slab_alloc.constprop.27) from [<8031ae14>] (__slab_alloc.constprop.26+0x70/0x80)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856514]  r10:000a4220 r9:00000154 r8:9ffa56f0 r7:806bf284 r6:00000900 r5:81401b40
Nov  2 05:50:31 fthcctvpi kernel: [57018.856521]  r4:60000013
Nov  2 05:50:31 fthcctvpi kernel: [57018.856534] [<8031ada4>] (__slab_alloc.constprop.26) from [<8031b960>] (__kmalloc+0x578/0x5c4)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856546]  r8:00000900 r7:80f0581c r6:00000000 r5:00000000 r4:81401b40
Nov  2 05:50:31 fthcctvpi kernel: [57018.856566] [<8031b3e8>] (__kmalloc) from [<806bf284>] (bcm2835_dma_create_cb_chain+0x64/0x308)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856580]  r10:000d0418 r9:00000000 r8:7e202040 r7:00000002 r6:00000020 r5:81735440
Nov  2 05:50:31 fthcctvpi kernel: [57018.856587]  r4:00000000
Nov  2 05:50:31 fthcctvpi kernel: [57018.856603] [<806bf220>] (bcm2835_dma_create_cb_chain) from [<806bfa80>] (bcm2835_dma_prep_slave_sg+0x140/0x318)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856616]  r10:40000000 r9:00000000 r8:81fcb800 r7:00000002 r6:00000020 r5:00000000
Nov  2 05:50:31 fthcctvpi kernel: [57018.856623]  r4:81735440
Nov  2 05:50:31 fthcctvpi kernel: [57018.856642] [<806bf940>] (bcm2835_dma_prep_slave_sg) from [<8082a2a0>] (bcm2835_sdhost_request+0x4a4/0x70c)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856656]  r10:806bf940 r9:81f9e37c r8:816da340 r7:810f2dcc r6:80f05008 r5:81f9e2a8
Nov  2 05:50:31 fthcctvpi kernel: [57018.856663]  r4:816da000
Nov  2 05:50:31 fthcctvpi kernel: [57018.856681] [<80829dfc>] (bcm2835_sdhost_request) from [<80809298>] (__mmc_start_request+0x88/0x198)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856694]  r10:81f9e200 r9:816da000 r8:00000000 r7:816da000 r6:81f9e2a8 r5:81f9e2a8
Nov  2 05:50:31 fthcctvpi kernel: [57018.856701]  r4:816da000
Nov  2 05:50:31 fthcctvpi kernel: [57018.856716] [<80809210>] (__mmc_start_request) from [<8080943c>] (mmc_start_request+0x94/0xbc)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856728]  r7:816da000 r6:00000000 r5:81f9e2a8 r4:816da000
Nov  2 05:50:31 fthcctvpi kernel: [57018.856743] [<808093a8>] (mmc_start_request) from [<8081d61c>] (mmc_blk_mq_issue_rq+0x388/0x9e8)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856754]  r7:816da000 r6:81ee4e08 r5:81f9e2a8 r4:80f05008
Nov  2 05:50:31 fthcctvpi kernel: [57018.856768] [<8081d294>] (mmc_blk_mq_issue_rq) from [<8081e090>] (mmc_mq_queue_rq+0x158/0x2a0)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856782]  r10:00000002 r9:81ee4ee0 r8:816da000 r7:816da800 r6:81ee4e10 r5:81f9e200
Nov  2 05:50:31 fthcctvpi kernel: [57018.856789]  r4:81ee4e08
Nov  2 05:50:31 fthcctvpi kernel: [57018.856804] [<8081df38>] (mmc_mq_queue_rq) from [<805fccb4>] (blk_mq_dispatch_rq_list+0x12c/0x7f4)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856818]  r10:81f9e200 r9:81eaa800 r8:81f80000 r7:00000001 r6:81eaa800 r5:8ddede3c
Nov  2 05:50:31 fthcctvpi kernel: [57018.856825]  r4:81f9e230
Nov  2 05:50:31 fthcctvpi kernel: [57018.856842] [<805fcb88>] (blk_mq_dispatch_rq_list) from [<80602778>] (__blk_mq_do_dispatch_sched+0x174/0x2b4)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856855]  r10:81eaa804 r9:81ee5e00 r8:00000000 r7:80f05008 r6:81eaa800 r5:8ddede3c
Nov  2 05:50:31 fthcctvpi kernel: [57018.856862]  r4:0000000a
Nov  2 05:50:31 fthcctvpi kernel: [57018.856878] [<80602604>] (__blk_mq_do_dispatch_sched) from [<80602b90>] (__blk_mq_sched_dispatch_requests+0x17c/0x1b0)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856891]  r10:00000000 r9:00000000 r8:81eaa804 r7:80f05008 r6:8ddede84 r5:00000001
Nov  2 05:50:31 fthcctvpi kernel: [57018.856899]  r4:81eaa800
Nov  2 05:50:31 fthcctvpi kernel: [57018.856914] [<80602a14>] (__blk_mq_sched_dispatch_requests) from [<80602e50>] (blk_mq_sched_dispatch_requests+0x44/0x6c)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856927]  r8:000000a0 r7:9ffa7800 r6:9ffa28c0 r5:00000001 r4:81eaa800
Nov  2 05:50:31 fthcctvpi kernel: [57018.856942] [<80602e0c>] (blk_mq_sched_dispatch_requests) from [<805fa114>] (__blk_mq_run_hw_queue+0xb0/0x14c)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856951]  r5:00000001 r4:81eaa800
Nov  2 05:50:31 fthcctvpi kernel: [57018.856963] [<805fa064>] (__blk_mq_run_hw_queue) from [<805fa1d8>] (blk_mq_run_work_fn+0x28/0x2c)
Nov  2 05:50:31 fthcctvpi kernel: [57018.856973]  r5:8df3ed00 r4:81eaa840
Nov  2 05:50:31 fthcctvpi kernel: [57018.856989] [<805fa1b0>] (blk_mq_run_work_fn) from [<8013bb34>] (process_one_work+0x250/0x5a0)
Nov  2 05:50:31 fthcctvpi kernel: [57018.857003] [<8013b8e4>] (process_one_work) from [<8013bee4>] (worker_thread+0x60/0x5c4)
Nov  2 05:50:31 fthcctvpi kernel: [57018.857017]  r10:9ffa28c0 r9:80f03d00 r8:9ffa28d8 r7:00000008 r6:9ffa28c0 r5:8df3ed14
Nov  2 05:50:31 fthcctvpi kernel: [57018.857024]  r4:8df3ed00
Nov  2 05:50:31 fthcctvpi kernel: [57018.857040] [<8013be84>] (worker_thread) from [<801439c4>] (kthread+0x170/0x174)
Nov  2 05:50:31 fthcctvpi kernel: [57018.857053]  r10:867b3e74 r9:8df3ed00 r8:8013be84 r7:8ddec000 r6:00000000 r5:853d9780
Nov  2 05:50:31 fthcctvpi kernel: [57018.857060]  r4:8a32dec0
Nov  2 05:50:31 fthcctvpi kernel: [57018.857074] [<80143854>] (kthread) from [<801000ec>] (ret_from_fork+0x14/0x28)
Nov  2 05:50:31 fthcctvpi kernel: [57018.857083] Exception stack(0x8ddedfb0 to 0x8ddedff8)
Nov  2 05:50:31 fthcctvpi kernel: [57018.857094] dfa0:                                     00000000 00000000 00000000 00000000
Nov  2 05:50:31 fthcctvpi kernel: [57018.857106] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov  2 05:50:31 fthcctvpi kernel: [57018.857116] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
Nov  2 05:50:31 fthcctvpi kernel: [57018.857130]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80143854
Nov  2 05:50:31 fthcctvpi kernel: [57018.857138]  r4:853d9780
Nov  2 05:50:31 fthcctvpi kernel: [57018.857144] Mem-Info:
Nov  2 05:50:31 fthcctvpi kernel: [57018.857164] active_anon:53784 inactive_anon:53811 isolated_anon:0
Nov  2 05:50:31 fthcctvpi kernel: [57018.857164]  active_file:695 inactive_file:561 isolated_file:57
Nov  2 05:50:31 fthcctvpi kernel: [57018.857164]  unevictable:4 dirty:0 writeback:0
Nov  2 05:50:31 fthcctvpi kernel: [57018.857164]  slab_reclaimable:2736 slab_unreclaimable:3772
Nov  2 05:50:31 fthcctvpi kernel: [57018.857164]  mapped:964 shmem:6297 pagetables:945 bounce:0
Nov  2 05:50:31 fthcctvpi kernel: [57018.857164]  free:4375 free_pcp:102 free_cma:595
Nov  2 05:50:31 fthcctvpi kernel: [57018.857181] Node 0 active_anon:215136kB inactive_anon:215244kB active_file:2780kB inactive_file:2244kB unevictable:16kB isolated(anon):0kB isolated(file):228kB mapped:3856kB dirty:0kB writeback:0kB shmem:25188kB writeback_tmp:0kB kernel_stack:1592kB all_unreclaimable? no
Nov  2 05:50:31 fthcctvpi kernel: [57018.857203] DMA free:17500kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:215136kB inactive_anon:215244kB active_file:1996kB inactive_file:2728kB unevictable:16kB writepending:0kB present:524288kB managed:503216kB mlocked:16kB pagetables:3780kB bounce:0kB free_pcp:408kB local_pcp:8kB free_cma:2380kB
Nov  2 05:50:31 fthcctvpi kernel: [57018.857209] lowmem_reserve[]: 0 0 0
Nov  2 05:50:31 fthcctvpi kernel: [57018.857230] DMA: 1119*4kB (UMEC) 558*8kB (UMEC) 321*16kB (UMEC) 32*32kB (UME) 24*64kB (UME) 5*128kB (UME) 2*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17788kB
Nov  2 05:50:31 fthcctvpi kernel: [57018.857321] 7713 total pagecache pages
Nov  2 05:50:31 fthcctvpi kernel: [57018.857330] 67 pages in swap cache
Nov  2 05:50:31 fthcctvpi kernel: [57018.857338] Swap cache stats: add 33583, delete 33515, find 91915/94129
Nov  2 05:50:31 fthcctvpi kernel: [57018.857345] Free swap  = 0kB
Nov  2 05:50:31 fthcctvpi kernel: [57018.857350] Total swap = 102396kB
Nov  2 05:50:31 fthcctvpi kernel: [57018.857356] 131072 pages RAM
Nov  2 05:50:31 fthcctvpi kernel: [57018.857362] 0 pages HighMem/MovableOnly
Nov  2 05:50:31 fthcctvpi kernel: [57018.857367] 5268 pages reserved
Nov  2 05:50:31 fthcctvpi kernel: [57018.857373] 16384 pages cma reserved
Nov  2 05:50:31 fthcctvpi kernel: [57018.857386] SLUB: Unable to allocate memory on node -1, gfp=0x900(GFP_NOWAIT|__GFP_ZERO)
Nov  2 05:50:31 fthcctvpi kernel: [57018.857395]   cache: kmalloc-512, object size: 512, buffer size: 512, default order: 1, min order: 0
Nov  2 05:50:31 fthcctvpi kernel: [57018.857405]   node 0: slabs: 88, objs: 1304, free: 0
Nov  2 05:50:31 fthcctvpi kernel: [57018.871813] SLUB: Unable to allocate memory on node -1, gfp=0x900(GFP_NOWAIT|__GFP_ZERO)
Nov  2 05:50:31 fthcctvpi kernel: [57018.871836]   cache: kmalloc-512, object size: 512, buffer size: 512, default order: 1, min order: 0
Nov  2 05:50:31 fthcctvpi kernel: [57018.871847]   node 0: slabs: 88, objs: 1304, free: 0

EDIT: you can see the oomkiller didnt kick in for 2 hours

Nov  2 07:16:18 fthcctvpi kernel: [62163.944082] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=vlc,pid=1094,uid=0
Nov  2 07:16:18 fthcctvpi kernel: [62163.944264] Out of memory: Killed process 1094 (vlc) total-vm:586568kB, anon-rss:378384kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:498kB oom_score_adj:0
Nov  2 07:16:18 fthcctvpi kernel: [62164.214550] oom_reaper: reaped process 1094 (vlc), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: bits received from HRNG source: 540064
Nov  2 07:16:22 fthcctvpi systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)!
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: bits sent to kernel pool: 483584
Nov  2 07:16:22 fthcctvpi systemd[1]: systemd-journald.service: Killing process 113 (systemd-journal) with signal SIGABRT.
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: entropy added to kernel pool: 483584
Nov  2 07:16:22 fthcctvpi systemd-timesyncd[247]: Timed out waiting for reply from 63.250.53.99:123 (3.debian.pool.ntp.org).
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: FIPS 140-2 successes: 27
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: FIPS 140-2 failures: 0
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: FIPS 140-2(2001-10-10) Poker: 0
Nov  2 07:16:22 fthcctvpi rpisurv[396]: [7350eb78] [7360e998] main decoder error: main decoder error: buffer deadlock preventedbuffer deadlock prevented
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: FIPS 140-2(2001-10-10) Runs: 0
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: FIPS 140-2(2001-10-10) Long run: 0
Nov  2 07:16:22 fthcctvpi rpisurv[396]: [7350eb78] main decoder error: buffer deadlock prevented
Nov  2 07:16:22 fthcctvpi rngd[293]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Nov  2 07:16:22 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 07:16:22 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 07:16:22 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 07:16:22 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 07:16:22 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 07:16:22 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
Nov  2 07:16:22 fthcctvpi rpisurv[396]: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred
si458 commented 1 year ago

@SvenVD ok added logs for you, hope they help! the only thing i can think of is after checking the logs is the AUDIO, i did yesterday morning ater i restarted the pi about 10am(ish) change the audio setting from HDMI to 3.5 because the HDMI has an adapter to DVI so the is no audio (i have a cron every 3 hours restart rpisurv but i have just disabled this now to see if the memory leaks again) but i can see from the logs again, after the oomkiller happened, rpisurv restarted vlc with the 'no-audio' option?

Nov  2 07:16:24 fthcctvpi rpisurv[396]: 2022/11/02 07:16:23 - core.worker - DEBUG - Starting stream screen_manager_0_screen1_cam_stream1 with commandline ['/usr/bin/cvlc', '--aspect-ratio=512:384', '--vout', 'mmal_vout', '--network-caching', '500', '--no-video-title-show', '--mmal-display=hdmi-1', '--input-timeshift-granularity=0', '--repeat', '--mmal-vout-transparent', '--mmal-vout-window', '512x384+0+0', '--mmal-layer', '2000000000', '--no-audio', 'rtsp://USER:PASSWORD!@192.168.52.123:554/cam/realmonitor?channel=3&subtype=1']
Nov  2 07:16:24 fthcctvpi rpisurv[396]: 2022/11/02 07:16:24 - l_default - DEBUG - screen_manager_0: update_active_screen screen_manager_0_screen1
Nov  2 07:16:24 fthcctvpi rpisurv[396]: 2022/11/02 07:16:24 - l_default - DEBUG - Screen: Start polling connectivity for the camera_streams part of screen: screen_manager_0_screen1
si458 commented 1 year ago

@SvenVD right so it still ran outta memory again even with audio switched to the 3.5m jack not hdmi, i also tried uninstalling vlc 3.0.17 and installing an old version 3.0.12 from a repo i found but it was missing the mmla patch, now i cant get vlc to install again with mmla patch from the raspbian repo, so i need to format the card again when im next on site, groan...

SvenVD commented 1 year ago

It seems to be a problem with you sdcard https://github.com/raspberrypi/linux/issues/2810

Memory is complex, Linux always tries to use all available memory for caching so it will fill over time. You should measure immediate freeable memory which is your true available memory.

si458 commented 1 year ago

It seems to be a problem with you sdcard https://github.com/raspberrypi/linux/issues/2810

Memory is complex, Linux always tries to use all available memory for caching so it will fill over time. You should measure immediate freeable memory which is your true available memory.

Hmmm OK I do think the micro sd card is about 4 years old so maybe a new one I can try?

SvenVD commented 1 year ago

I would try a another sdcard to see if that solves the issue

si458 commented 1 year ago

I would try a another sdcard to see if that solves the issue

ive bought a class 10 sd card brand new (all i could get my hand on this minute) so driving to the site today to swap it out, which OS should i use? buster or bullseye? i did have bullseye working no problem after changing it to the fake graphics but i no your support buster only?

si458 commented 1 year ago

ok fresh install on a brand new class 10 micro sd card and im seeing this error? Nov 6 12:33:01 fthcctvpi rpisurv[371]: 2022/11/06 12:33:01 - l_default - ERROR - Skipping calculating free malloc memory because of TimeoutExpired(['/usr/bin/vcgencmd', 'get_mem', 'malloc'], 1)

si458 commented 1 year ago

@SvenVD right ok so after 12 hours, the memory started climbing again! but weirdly enough its only a single VLC instance that has this memory leak? (im running 4 cameras) so ive changed the rtsp stream thats seems to be causing this memory leak to a different camera to see if it happens again? (the camera that has this issue does indeed have a feed issue to the DVR unit itself, it flickers on and off and wavey lines, i think the BNC cable is broke, but the RTRSP stream is to the DVR itself and not the camera as the camera is BNC based)

SvenVD commented 1 year ago

I only tested things on buster and buster is still supported by raspberry pi

SvenVD commented 1 year ago

ok fresh install on a brand new class 10 micro sd card and im seeing this error? Nov 6 12:33:01 fthcctvpi rpisurv[371]: 2022/11/06 12:33:01 - l_default - ERROR - Skipping calculating free malloc memory because of TimeoutExpired(['/usr/bin/vcgencmd', 'get_mem', 'malloc'], 1)

This seems to mean your gpu is overloaded for some reason

SvenVD commented 1 year ago

@SvenVD right ok so after 12 hours, the memory started climbing again! but weirdly enough its only a single VLC instance that has this memory leak? (im running 4 cameras) so ive changed the rtsp stream thats seems to be causing this memory leak to a different camera to see if it happens again? (the camera that has this issue does indeed have a feed issue to the DVR unit itself, it flickers on and off and wavey lines, i think the BNC cable is broke, but the RTRSP stream is to the DVR itself and not the camera as the camera is BNC based)

Does it still crash with the new sdcard?

si458 commented 1 year ago

Does it still crash with the new sdcard

yes it still crashes with the new sdcard and i have the GPU set to 512mb, and i installed buster, i have also disabled the GPU memory checks for the moment

SvenVD commented 1 year ago

Do you still get the events_freezable mmc_rescan and mmc errors? Other than that

the streams are from dahua DVR, sub stream (352x288), the display is a DVI monitor (used hdmi to dvi cable) resolution 1024x768

Could you try to make each stream 512x384, so that the gpu does not need to scale

si458 commented 1 year ago

Do you still get the events_freezable mmc_rescan and mmc errors? Other than that

the streams are from dahua DVR, sub stream (352x288), the display is a DVI monitor (used hdmi to dvi cable) resolution 1024x768

Could you try to make each stream 512x384, so that the gpu does not need to scale

i will try changing the stream quality tomorrow (if it crashes again) as i have changed the camera from one camera to another to see if its maybe that perticular camera thats causing the issue as every time i check is appears to only a single VLC instance thats causing the memory leak and not all 4 of them?

also my monitor is actually 1280x1024 2022/11/07 11:49:37 - l_default - INFO - Auto detected display ENC-L461 at HDMI 0 with display number 2 1280 x 1024

SvenVD commented 1 year ago

640x512 then

si458 commented 1 year ago

@SvenVD, ok so it lasted longer this time (about 24 hours instead of 12hours) i removed the camera i throught was causing it to crash BUT instead it was a different camera that crashed it this time BUT this time the OOM kicked in and killed the VLC and it restarted as expected (so the OOM problem of crashing the PI must of been a bad memory card)

i cant change the quality of the resolution on the stream from the DVR unit sadly (its a BNC unit not IP based) i only have D1: 720 X 480, CIF: 352 X 240 or 960H 976 X 582 to pick from sadly the monitor is locked to 1280x1024 or lower (like 800x600)

could i set the GPU memory to 640 or 768? or any other suggestions?

pi@fthcctvpi:/var/log $ free -mh
              total        used        free      shared  buff/cache   available
Mem:          491Mi       115Mi       238Mi       5.0Mi       137Mi       320Mi
Swap:          99Mi        29Mi        70Mi

EDIT: sample output from ps aux of the vlc command

root      7429 22.6  4.2 175796 21432 ?        Ssl  13:22   0:00 /usr/bin/vlc -I dummy --aspect-ratio=640:512 --vout mmal_vout --network-caching 500 --no-video-title-show --mmal-display=hdmi-1 --input-timeshift-granularity=0 --repeat --mmal-vout-transparent --mmal-vout-window 640x512+640+512 --mmal-layer 2000000000 --no-audio rtsp://USER:PASSWORD@192.168.52.123:554/cam/realmonitor?channel=14&subtype=1
SvenVD commented 1 year ago

maybe set the GPU mem lower so the OS processes have more memory left?

si458 commented 1 year ago

maybe set the GPU mem lower so the OS processes have more memory left?

so i lowered the gpu memory but its still the same issue, 1 camera crashed and oom killed it then rpisurv detected the rtsp wasnt responding so killed all vlc's and restarted them, so im totally at a loss now, unless the rpi 2 b+ just isnt powerful enough 👎

ive just put a cronjob every 6 hours restart rpisurv for the moment,

but i will try make a script to watch the memory usage and if its above say 90% every 5 mins then kill the highest vlc memory process, as the oom is waiting before killing the vlc, so the pi does freeze the camera feeds, and i would rather one camera went off and on, than all of the cameras freeze for 5mins before they are restarted

Vattar commented 1 year ago

@si458 I believe we are having the same problem, however, I have no idea how I can solve it, nor where the problem is.

I've tried to reinstall using other versions of RaspiOS, as well as various combinations of system settings and rpisurv itself, but to no avail.

I can't say with 100% certainty, but I suspect this problem started when I updated vlc from version 3.0.16 to 3.0.17. I tried to downgrade to the old version of vlc but was unsuccessful.

jesjhoward commented 1 year ago

I was having this same issue, and it happened VERY quickly. I found this page referencing an application call earlyoom. Installing that seems to have fixed the issue, but time will have to tell. I will report back if this actually solves the issue in the end. Any other testers will be greatly appreciated

si458 commented 1 year ago

@SvenVD im still having this issue with VLC doing a memory leak, i think i may have identified the problem, it might be down to me connecting the pi to a DVR unit and the BNC cable being faulty for the camera on a certain channel because the VLC im having the memory run away issue with is actually a faulty camera(the cable is broken) and i cant replace it, so sometimes the image will just show a BLACK picture with a timestamp on it, BUt the monitor on the Pi shows the timestamp has frozen. BUT if i connect with VLC from my laptop with the same RTSP url, the timestamp is indeed moving just a black image so VLC doesnt like the fact the camera goes on and off due to a faulty BNC cable so the checker thats built into rpisurv to check is the stream is working, isnt working correctly to restart the VLC

EDIT: i have just tried @jesjhoward suggestion and the Pi is running out of memory very soon, so ill see if it KILLS the vlc process or not

EDIT2: i can confirm @jesjhoward suggestion worked!

Jul 29 09:29:49 fthcctvpi earlyoom[319]: mem avail:    93 of   490 MiB (19.02%), swap free:   12 of   99 MiB (12.25%)
Jul 29 10:29:59 fthcctvpi earlyoom[319]: mem avail:    78 of   490 MiB (15.91%), swap free:    2 of   99 MiB ( 2.45%)
Jul 29 11:30:09 fthcctvpi earlyoom[319]: mem avail:    70 of   490 MiB (14.35%), swap free:    0 of   99 MiB ( 0.01%)
Jul 29 11:35:48 fthcctvpi earlyoom[319]: mem avail:    49 of   490 MiB ( 9.99%), swap free:    0 of   99 MiB ( 0.00%)
Jul 29 11:35:48 fthcctvpi earlyoom[319]: low memory! at or below SIGTERM limits: mem 10.00%, swap 10.00%
Jul 29 11:35:48 fthcctvpi earlyoom[319]: memory situation has recovered while selecting victim
Jul 29 11:40:40 fthcctvpi earlyoom[319]: mem avail:    49 of   490 MiB ( 9.99%), swap free:    0 of   99 MiB ( 0.00%)
Jul 29 11:40:40 fthcctvpi earlyoom[319]: low memory! at or below SIGTERM limits: mem 10.00%, swap 10.00%
Jul 29 11:40:40 fthcctvpi earlyoom[319]: sending SIGTERM to process 627 uid 0 "vlc": badness 1050, VmRSS 305 MiB
Jul 29 11:40:43 fthcctvpi earlyoom[319]: process exited after 3.0 seconds

EDIT3: i also noticed the image has now resfreshed on the screen too, so while the image is still black the timestamp is moving! so it seems the rpisurv watcher isnt restarting VLC for some reason and VLC is just freezing with a memory leak?

SvenVD commented 1 year ago

If VLC does not crash ( or is killed) and the remote device is still responding then Rpisurv will not attempt to restart vlc. Using earlyoom to kill vlc ( after which rpisurv restarts it) or a cronjob to restart seems to be a good enough solution. Otherwise you can perhaps report the issue to the VLC developers.

SvenVD commented 11 months ago

stale issue