search

SvenVD / rpisurv

Raspberry Pi surveillance
https://community.rpisurv.net
GNU General Public License v2.0
621 stars 101 forks source link

Memory leak? #88

Closed kris6673 closed 5 years ago

kris6673 commented 5 years ago

Hi there Thanks for making a great program! I'm having some issues though. I'm running a raspi 3 setup with 1 Vivotek 1920 x 1080 15 fps camera in full screen. I'm experiencing stutter when the program recieves a keyframe/ intra frame it hangs for about 250-500 ms. I have tried lowering the resolution and framerate. Same problem. I tried running the stream directly with omxplayer, did that just fine, no stuttering/hanging at all. The stuttering isn't there for the first 10 sec from startup, then ramps up from there. I have tried reinstalling the program and Rasbian, issue persists.

I'm fairly certain that it's a memory usage issue, since it does not matter how much GPU memory I assign it in /boot/config.txt always throws GPU mem errors. From 64 to 512 , throws the same error. Currently i have 512 memory for the GPU. If you need anything else, please let me know. Thanks!

-Kris6673

These are the errors from /var/log/daemon.log:

Mar 22 08:37:13 raspberrypi rpisurv[2126]: 2019/03/22 08:37:13 - l_default - DEBUG - Free gpu memory value is 8K
Mar 22 08:37:13 raspberrypi rpisurv[2126]: 2019/03/22 08:37:13 - l_default - DEBUG - Free memory in bytes: 8192.0
Mar 22 08:37:13 raspberrypi rpisurv[2126]: 2019/03/22 08:37:13 - l_default - ERROR - Free gpu mem is 8192.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:13 raspberrypi rpisurv[2126]: 2019/03/22 08:37:13 - l_default - DEBUG - Screen: screen1 active_run_time: 4 / 60
Mar 22 08:37:14 raspberrypi rpisurv[2126]: 2019/03/22 08:37:14 - l_default - DEBUG - Free gpu memory value is 800
Mar 22 08:37:14 raspberrypi rpisurv[2126]: 2019/03/22 08:37:14 - l_default - DEBUG - Free memory in bytes: 800.0
Mar 22 08:37:14 raspberrypi rpisurv[2126]: 2019/03/22 08:37:14 - l_default - ERROR - Free gpu mem is 800.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:14 raspberrypi rpisurv[2126]: 2019/03/22 08:37:14 - l_default - DEBUG - Screen: screen1 active_run_time: 5 / 60
Mar 22 08:37:15 raspberrypi rpisurv[2126]: 2019/03/22 08:37:15 - l_default - DEBUG - Free gpu memory value is 3K
Mar 22 08:37:15 raspberrypi rpisurv[2126]: 2019/03/22 08:37:15 - l_default - DEBUG - Free memory in bytes: 3072.0
Mar 22 08:37:15 raspberrypi rpisurv[2126]: 2019/03/22 08:37:15 - l_default - ERROR - Free gpu mem is 3072.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:15 raspberrypi rpisurv[2126]: 2019/03/22 08:37:15 - l_default - DEBUG - Screen: screen1 active_run_time: 6 / 60
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - Screen: screen1 active_run_time: 7 / 60
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - MAIN: regular start update_active_screen (every 19 seconds since start of rpisurv)
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - ScreenManager: update_active_screen screen1
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - Screen: Start polling connectivity for the camera_streams part of screen: screen1
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - CameraStream: screen1_cam_stream1 rtsp://192.168.20.179/live.sdp Connectable
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - ScreenManager:  update_connectable_camera_streams, disable_probing_for_all_streams is off for this screen, so using probes screen1
Mar 22 08:37:16 raspberrypi rpisurv[2126]: 2019/03/22 08:37:16 - l_default - DEBUG - Screen: Connectable camera streams stayed the same, from 1 to 1, screen: screen1 does not need full redraw
Mar 22 08:37:17 raspberrypi rpisurv[2126]: 2019/03/22 08:37:17 - l_default - DEBUG - Free gpu memory value is 8K
Mar 22 08:37:17 raspberrypi rpisurv[2126]: 2019/03/22 08:37:17 - l_default - DEBUG - Free memory in bytes: 8192.0
Mar 22 08:37:17 raspberrypi rpisurv[2126]: 2019/03/22 08:37:17 - l_default - ERROR - Free gpu mem is 8192.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:17 raspberrypi rpisurv[2126]: 2019/03/22 08:37:17 - l_default - DEBUG - Screen: screen1 active_run_time: 9 / 60
Mar 22 08:37:18 raspberrypi rpisurv[2126]: 2019/03/22 08:37:18 - l_default - DEBUG - Free gpu memory value is 928
Mar 22 08:37:18 raspberrypi rpisurv[2126]: 2019/03/22 08:37:18 - l_default - DEBUG - Free memory in bytes: 928.0
Mar 22 08:37:18 raspberrypi rpisurv[2126]: 2019/03/22 08:37:18 - l_default - ERROR - Free gpu mem is 928.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:18 raspberrypi rpisurv[2126]: 2019/03/22 08:37:18 - l_default - DEBUG - Screen: screen1 active_run_time: 10 / 60
Mar 22 08:37:19 raspberrypi rpisurv[2126]: 2019/03/22 08:37:19 - l_default - DEBUG - Free gpu memory value is 546
Mar 22 08:37:19 raspberrypi rpisurv[2126]: 2019/03/22 08:37:19 - l_default - DEBUG - Free memory in bytes: 546.0
Mar 22 08:37:19 raspberrypi rpisurv[2126]: 2019/03/22 08:37:19 - l_default - ERROR - Free gpu mem is 546.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:19 raspberrypi rpisurv[2126]: 2019/03/22 08:37:19 - l_default - DEBUG - Screen: screen1 active_run_time: 11 / 60
Mar 22 08:37:20 raspberrypi rpisurv[2126]: 2019/03/22 08:37:20 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:20 raspberrypi rpisurv[2126]: 2019/03/22 08:37:20 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:20 raspberrypi rpisurv[2126]: 2019/03/22 08:37:20 - l_default - DEBUG - Screen: screen1 active_run_time: 12 / 60
Mar 22 08:37:21 raspberrypi rpisurv[2126]: 2019/03/22 08:37:21 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:21 raspberrypi rpisurv[2126]: 2019/03/22 08:37:21 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:21 raspberrypi rpisurv[2126]: 2019/03/22 08:37:21 - l_default - DEBUG - Screen: screen1 active_run_time: 13 / 60
Mar 22 08:37:22 raspberrypi rpisurv[2126]: 2019/03/22 08:37:22 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:22 raspberrypi rpisurv[2126]: 2019/03/22 08:37:22 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:22 raspberrypi rpisurv[2126]: 2019/03/22 08:37:22 - l_default - DEBUG - Screen: screen1 active_run_time: 14 / 60
Mar 22 08:37:23 raspberrypi rpisurv[2126]: 2019/03/22 08:37:23 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:23 raspberrypi rpisurv[2126]: 2019/03/22 08:37:23 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:23 raspberrypi rpisurv[2126]: 2019/03/22 08:37:23 - l_default - DEBUG - Screen: screen1 active_run_time: 15 / 60
Mar 22 08:37:24 raspberrypi rpisurv[2126]: 2019/03/22 08:37:24 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:24 raspberrypi rpisurv[2126]: 2019/03/22 08:37:24 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:24 raspberrypi rpisurv[2126]: 2019/03/22 08:37:24 - l_default - DEBUG - Screen: screen1 active_run_time: 16 / 60
Mar 22 08:37:26 raspberrypi rpisurv[2126]: 2019/03/22 08:37:26 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:26 raspberrypi rpisurv[2126]: 2019/03/22 08:37:26 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:26 raspberrypi rpisurv[2126]: 2019/03/22 08:37:26 - l_default - DEBUG - Screen: screen1 active_run_time: 17 / 60
Mar 22 08:37:27 raspberrypi rpisurv[2126]: 2019/03/22 08:37:27 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:27 raspberrypi rpisurv[2126]: 2019/03/22 08:37:27 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:27 raspberrypi rpisurv[2126]: 2019/03/22 08:37:27 - l_default - DEBUG - Screen: screen1 active_run_time: 18 / 60
Mar 22 08:37:28 raspberrypi rpisurv[2126]: 2019/03/22 08:37:28 - l_default - DEBUG - Free gpu memory value is 5K
Mar 22 08:37:28 raspberrypi rpisurv[2126]: 2019/03/22 08:37:28 - l_default - DEBUG - Free memory in bytes: 5120.0
Mar 22 08:37:28 raspberrypi rpisurv[2126]: 2019/03/22 08:37:28 - l_default - ERROR - Free gpu mem is 5120.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:28 raspberrypi rpisurv[2126]: 2019/03/22 08:37:28 - l_default - DEBUG - Screen: screen1 active_run_time: 19 / 60
Mar 22 08:37:29 raspberrypi rpisurv[2126]: 2019/03/22 08:37:29 - l_default - DEBUG - Free gpu memory value is 1K
Mar 22 08:37:29 raspberrypi rpisurv[2126]: 2019/03/22 08:37:29 - l_default - DEBUG - Free memory in bytes: 1024.0
Mar 22 08:37:29 raspberrypi rpisurv[2126]: 2019/03/22 08:37:29 - l_default - ERROR - Free gpu mem is 1024.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:29 raspberrypi rpisurv[2126]: 2019/03/22 08:37:29 - l_default - DEBUG - Screen: screen1 active_run_time: 20 / 60
Mar 22 08:37:30 raspberrypi rpisurv[2126]: 2019/03/22 08:37:30 - l_default - DEBUG - Free gpu memory value is 0K
Mar 22 08:37:30 raspberrypi rpisurv[2126]: 2019/03/22 08:37:30 - l_default - DEBUG - Free memory in bytes: 0.0
Mar 22 08:37:30 raspberrypi rpisurv[2126]: 2019/03/22 08:37:30 - l_default - ERROR - Free gpu mem is 0.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:30 raspberrypi rpisurv[2126]: 2019/03/22 08:37:30 - l_default - DEBUG - Screen: screen1 active_run_time: 21 / 60
Mar 22 08:37:31 raspberrypi rpisurv[2126]: 2019/03/22 08:37:31 - l_default - DEBUG - Free gpu memory value is 5K
Mar 22 08:37:31 raspberrypi rpisurv[2126]: 2019/03/22 08:37:31 - l_default - DEBUG - Free memory in bytes: 5120.0
Mar 22 08:37:31 raspberrypi rpisurv[2126]: 2019/03/22 08:37:31 - l_default - ERROR - Free gpu mem is 5120.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:31 raspberrypi rpisurv[2126]: 2019/03/22 08:37:31 - l_default - DEBUG - Screen: screen1 active_run_time: 22 / 60
Mar 22 08:37:32 raspberrypi rpisurv[2126]: 2019/03/22 08:37:32 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:32 raspberrypi rpisurv[2126]: 2019/03/22 08:37:32 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:32 raspberrypi rpisurv[2126]: 2019/03/22 08:37:32 - l_default - DEBUG - Screen: screen1 active_run_time: 23 / 60
Mar 22 08:37:33 raspberrypi rpisurv[2126]: 2019/03/22 08:37:33 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:33 raspberrypi rpisurv[2126]: 2019/03/22 08:37:33 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:33 raspberrypi rpisurv[2126]: 2019/03/22 08:37:33 - l_default - DEBUG - Screen: screen1 active_run_time: 25 / 60
Mar 22 08:37:34 raspberrypi rpisurv[2126]: 2019/03/22 08:37:34 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:34 raspberrypi rpisurv[2126]: 2019/03/22 08:37:34 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:34 raspberrypi rpisurv[2126]: 2019/03/22 08:37:34 - l_default - DEBUG - Screen: screen1 active_run_time: 26 / 60
Mar 22 08:37:35 raspberrypi rpisurv[2126]: 2019/03/22 08:37:35 - l_default - DEBUG - Free gpu memory value is 288
Mar 22 08:37:35 raspberrypi rpisurv[2126]: 2019/03/22 08:37:35 - l_default - DEBUG - Free memory in bytes: 288.0
Mar 22 08:37:35 raspberrypi rpisurv[2126]: 2019/03/22 08:37:35 - l_default - ERROR - Free gpu mem is 288.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
Mar 22 08:37:35 raspberrypi rpisurv[2126]: 2019/03/22 08:37:35 - l_default - DEBUG - Screen: screen1 active_run_time: 27 / 60
Mar 22 08:37:36 raspberrypi rpisurv[2126]: 2019/03/22 08:37:36 - l_default - DEBUG - Free gpu memory value is 448M
Mar 22 08:37:36 raspberrypi rpisurv[2126]: 2019/03/22 08:37:36 - l_default - DEBUG - Free memory in bytes: 469762048.0
Mar 22 08:37:36 raspberrypi rpisurv[2126]: 2019/03/22 08:37:36 - l_default - DEBUG - Screen: screen1 active_run_time: 28 / 60

Log from /usr/local/bin/rpisurv/logs/main.log:

2019/03/22 08:33:08 - l_default - INFO - Starting rpisurv 2.0.0
2019/03/22 08:33:08 - l_default - DEBUG - autodetected resolution of['1920', '1080']
2019/03/22 08:33:08 - l_default - INFO - Unique id of this installation is 8a11ed3c412ba134026d649e4f0f49ec6850b68fd0829ef378a4643b5839b068
2019/03/22 08:33:08 - l_default - DEBUG - Start_time is 1553243588.11
2019/03/22 08:33:09 - l_default - DEBUG - ScreenManager: Initialising screen with config {'duration': 60, 'nr_of_columns': '1', 'disable_probing_for_all_streams': False, 'camera_streams': [{'url': 'rtsp://192.168.20.179/live.sdp', 'rtsp_over_tcp': True}], 'autostretch': False}
2019/03/22 08:33:09 - l_default - DEBUG - Screen: screen1 duration from config is: 60
2019/03/22 08:33:09 - l_default - DEBUG - Screen: This screen: screen1 will be started in cache
2019/03/22 08:33:09 - l_default - DEBUG - Drawing placeholder with coordinates: 0, 0 and width: 1920 height: 1080 with image images/connecting.png
2019/03/22 08:33:09 - l_default - DEBUG - ScreenManager: only one screen configured, do not rotate
2019/03/22 08:33:09 - l_default - DEBUG - Screen: Make screen screen1 active
2019/03/22 08:33:09 - l_default - DEBUG - Screen: screen1 start_of_active_time: 1553243589.57
2019/03/22 08:33:09 - l_default - DEBUG - MAIN: bootstrap update_active_screen
2019/03/22 08:33:09 - l_default - DEBUG - ScreenManager: update_active_screen screen1
2019/03/22 08:33:09 - l_default - DEBUG - Screen: Start polling connectivity for the camera_streams part of screen: screen1
2019/03/22 08:33:09 - l_default - DEBUG - CameraStream: screen1_cam_stream1 rtsp://192.168.20.179/live.sdp Connectable
2019/03/22 08:33:09 - l_default - DEBUG - ScreenManager:  update_connectable_camera_streams, disable_probing_for_all_streams is off for this screen, so using probes screen1
2019/03/22 08:33:09 - l_default - DEBUG - Screen: screen1 Connectable camera streams changed from 0 to 1 or we change from previous_cached value: True to current cached value: False, screen: screen1 needs update/redraw
2019/03/22 08:33:09 - l_default - DEBUG - Screen: screen1 number of fields= 1
2019/03/22 08:33:09 - l_default - DEBUG - blanking the screen
2019/03/22 08:33:09 - l_default - DEBUG - Drawing placeholder with coordinates: 0, 0 and width: 1920 height: 1080 with image images/blank.png
2019/03/22 08:33:09 - l_default - DEBUG - Screen: cam stream name =screen1_cam_stream1
2019/03/22 08:33:09 - l_default - DEBUG - CameraStream: Start stream screen1_cam_stream1
2019/03/22 08:33:09 - l_default - DEBUG - BusFinder: initialised with path: None
2019/03/22 08:33:09 - l_default - DEBUG - CameraStream: screen1_cam_stream1 DBus connect attempt: 0
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Opening file at /tmp/omxplayerdbus.root
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Opened file at /tmp/omxplayerdbus.root
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Address 'unix:abstract=/tmp/dbus-Gfdi8Yd8aB,guid=9bad933147f6bc4e009e6cfe5c949dc5' parsed from file
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Opening file at /tmp/omxplayerdbus.root
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Opened file at /tmp/omxplayerdbus.root
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Address 'unix:abstract=/tmp/dbus-Gfdi8Yd8aB,guid=9bad933147f6bc4e009e6cfe5c949dc5' parsed from file
2019/03/22 08:33:10 - l_default - DEBUG - CameraStream: screen1_cam_stream1 Failed to connect to omxplayer at dbus address:unix:abstract=/tmp/dbus-Gfdi8Yd8aB,guid=9bad933147f6bc4e009e6cfe5c949dc5 with dbus name: org.mpris.MediaPlayer2.screen1_cam_stream1
2019/03/22 08:33:10 - l_default - DEBUG - CameraStream: screen1_cam_stream1 DBus connect attempt: 1
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Opening file at /tmp/omxplayerdbus.root
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Opened file at /tmp/omxplayerdbus.root
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Address 'unix:abstract=/tmp/dbus-Gfdi8Yd8aB,guid=9bad933147f6bc4e009e6cfe5c949dc5' parsed from file
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Opening file at /tmp/omxplayerdbus.root
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Opened file at /tmp/omxplayerdbus.root
2019/03/22 08:33:10 - l_default - DEBUG - BusFinder: Address 'unix:abstract=/tmp/dbus-Gfdi8Yd8aB,guid=9bad933147f6bc4e009e6cfe5c949dc5' parsed from file
2019/03/22 08:33:10 - l_default - DEBUG - CameraStream: screen1_cam_stream1 Connected to omxplayer at dbus address:unix:abstract=/tmp/dbus-Gfdi8Yd8aB,guid=9bad933147f6bc4e009e6cfe5c949dc5 with dbus name: org.mpris.MediaPlayer2.screen1_cam_stream1
2019/03/22 08:33:10 - l_default - DEBUG - CameraStream: screen1_cam_stream1 Set new position for screen1_cam_stream1 with new coordinates: + [0, 0, 1920, 1080] on dbus interface
2019/03/22 08:33:10 - l_default - DEBUG - DBusConnection: start TESTING if bus org.mpris.MediaPlayer2.screen1_cam_stream1 is ready to accept videopos commands
2019/03/22 08:33:12 - l_default - DEBUG - DBusConnection: bus: org.mpris.MediaPlayer2.screen1_cam_stream1 is ready to accept videopos commands
2019/03/22 08:33:12 - l_default - DEBUG - CameraStream: This stream screen1_cam_stream1 is not running in cache
2019/03/22 08:33:12 - l_default - DEBUG - Drawing placeholder with coordinates: 0, 0 and width: 1920 height: 1080 with image images/connecting.png
2019/03/22 08:33:12 - l_default - DEBUG - Current time detected is 1553243592.59 runtime calculated is 4
2019/03/22 08:33:12 - l_default - INFO - Sending stats is disabled, not sending stats
2019/03/22 08:33:12 - l_default - DEBUG - Free gpu memory value is 1K
2019/03/22 08:33:12 - l_default - DEBUG - Free memory in bytes: 1024.0
2019/03/22 08:33:12 - l_default - ERROR - Free gpu mem is 1024.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:12 - l_default - DEBUG - Screen: screen1 active_run_time: 3 / 60
2019/03/22 08:33:13 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:13 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:13 - l_default - DEBUG - Screen: screen1 active_run_time: 4 / 60
2019/03/22 08:33:14 - l_default - DEBUG - Free gpu memory value is 1K
2019/03/22 08:33:14 - l_default - DEBUG - Free memory in bytes: 1024.0
2019/03/22 08:33:14 - l_default - ERROR - Free gpu mem is 1024.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:14 - l_default - DEBUG - Screen: screen1 active_run_time: 5 / 60
2019/03/22 08:33:15 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:15 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:15 - l_default - DEBUG - Screen: screen1 active_run_time: 6 / 60
2019/03/22 08:33:16 - l_default - DEBUG - Free gpu memory value is 384
2019/03/22 08:33:16 - l_default - DEBUG - Free memory in bytes: 384.0
2019/03/22 08:33:16 - l_default - ERROR - Free gpu mem is 384.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:16 - l_default - DEBUG - Screen: screen1 active_run_time: 7 / 60
2019/03/22 08:33:17 - l_default - DEBUG - Free gpu memory value is 577
2019/03/22 08:33:17 - l_default - DEBUG - Free memory in bytes: 577.0
2019/03/22 08:33:17 - l_default - ERROR - Free gpu mem is 577.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:17 - l_default - DEBUG - Screen: screen1 active_run_time: 8 / 60
2019/03/22 08:33:19 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:19 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:19 - l_default - DEBUG - Screen: screen1 active_run_time: 9 / 60
2019/03/22 08:33:20 - l_default - DEBUG - Free gpu memory value is 2K
2019/03/22 08:33:20 - l_default - DEBUG - Free memory in bytes: 2048.0
2019/03/22 08:33:20 - l_default - ERROR - Free gpu mem is 2048.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:20 - l_default - DEBUG - Screen: screen1 active_run_time: 11 / 60
2019/03/22 08:33:21 - l_default - DEBUG - Free gpu memory value is 7K
2019/03/22 08:33:21 - l_default - DEBUG - Free memory in bytes: 7168.0
2019/03/22 08:33:21 - l_default - ERROR - Free gpu mem is 7168.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:21 - l_default - DEBUG - Screen: screen1 active_run_time: 12 / 60
2019/03/22 08:33:22 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:22 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:22 - l_default - DEBUG - Screen: screen1 active_run_time: 13 / 60
2019/03/22 08:33:23 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:23 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:23 - l_default - DEBUG - Screen: screen1 active_run_time: 14 / 60
2019/03/22 08:33:24 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:24 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:24 - l_default - DEBUG - Screen: screen1 active_run_time: 15 / 60
2019/03/22 08:33:25 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:25 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:25 - l_default - DEBUG - Screen: screen1 active_run_time: 16 / 60
2019/03/22 08:33:26 - l_default - DEBUG - Free gpu memory value is 2K
2019/03/22 08:33:26 - l_default - DEBUG - Free memory in bytes: 2048.0
2019/03/22 08:33:26 - l_default - ERROR - Free gpu mem is 2048.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:26 - l_default - DEBUG - Screen: screen1 active_run_time: 17 / 60
2019/03/22 08:33:27 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:27 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:27 - l_default - DEBUG - Screen: screen1 active_run_time: 18 / 60
2019/03/22 08:33:28 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:28 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:28 - l_default - DEBUG - Screen: screen1 active_run_time: 19 / 60
2019/03/22 08:33:29 - l_default - DEBUG - Free gpu memory value is 1K
2019/03/22 08:33:29 - l_default - DEBUG - Free memory in bytes: 1024.0
2019/03/22 08:33:29 - l_default - ERROR - Free gpu mem is 1024.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:29 - l_default - DEBUG - Screen: screen1 active_run_time: 20 / 60
2019/03/22 08:33:30 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:30 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:30 - l_default - DEBUG - Screen: screen1 active_run_time: 21 / 60
2019/03/22 08:33:31 - l_default - DEBUG - Free gpu memory value is 5K
2019/03/22 08:33:31 - l_default - DEBUG - Free memory in bytes: 5120.0
2019/03/22 08:33:31 - l_default - ERROR - Free gpu mem is 5120.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:31 - l_default - DEBUG - Screen: screen1 active_run_time: 22 / 60
2019/03/22 08:33:31 - l_default - DEBUG - MAIN: regular start update_active_screen (every 19 seconds since start of rpisurv)
2019/03/22 08:33:31 - l_default - DEBUG - ScreenManager: update_active_screen screen1
2019/03/22 08:33:31 - l_default - DEBUG - Screen: Start polling connectivity for the camera_streams part of screen: screen1
2019/03/22 08:33:31 - l_default - DEBUG - CameraStream: screen1_cam_stream1 rtsp://192.168.20.179/live.sdp Connectable
2019/03/22 08:33:31 - l_default - DEBUG - ScreenManager:  update_connectable_camera_streams, disable_probing_for_all_streams is off for this screen, so using probes screen1
2019/03/22 08:33:31 - l_default - DEBUG - Screen: Connectable camera streams stayed the same, from 1 to 1, screen: screen1 does not need full redraw
2019/03/22 08:33:32 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:32 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:32 - l_default - DEBUG - Screen: screen1 active_run_time: 23 / 60
2019/03/22 08:33:33 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:33 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:33 - l_default - DEBUG - Screen: screen1 active_run_time: 24 / 60
2019/03/22 08:33:35 - l_default - DEBUG - Free gpu memory value is 5K
2019/03/22 08:33:35 - l_default - DEBUG - Free memory in bytes: 5120.0
2019/03/22 08:33:35 - l_default - ERROR - Free gpu mem is 5120.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:35 - l_default - DEBUG - Screen: screen1 active_run_time: 25 / 60
2019/03/22 08:33:36 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:36 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:36 - l_default - DEBUG - Screen: screen1 active_run_time: 27 / 60
2019/03/22 08:33:37 - l_default - DEBUG - Free gpu memory value is 2K
2019/03/22 08:33:37 - l_default - DEBUG - Free memory in bytes: 2048.0
2019/03/22 08:33:37 - l_default - ERROR - Free gpu mem is 2048.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:37 - l_default - DEBUG - Screen: screen1 active_run_time: 28 / 60
2019/03/22 08:33:38 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:38 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:38 - l_default - DEBUG - Screen: screen1 active_run_time: 29 / 60
2019/03/22 08:33:39 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:39 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:39 - l_default - DEBUG - Screen: screen1 active_run_time: 30 / 60
2019/03/22 08:33:40 - l_default - DEBUG - Free gpu memory value is 576
2019/03/22 08:33:40 - l_default - DEBUG - Free memory in bytes: 576.0
2019/03/22 08:33:40 - l_default - ERROR - Free gpu mem is 576.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:40 - l_default - DEBUG - Screen: screen1 active_run_time: 31 / 60
2019/03/22 08:33:41 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:41 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:41 - l_default - DEBUG - Screen: screen1 active_run_time: 32 / 60
2019/03/22 08:33:42 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:42 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:42 - l_default - DEBUG - Screen: screen1 active_run_time: 33 / 60
2019/03/22 08:33:43 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:43 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:43 - l_default - DEBUG - Screen: screen1 active_run_time: 34 / 60
2019/03/22 08:33:44 - l_default - DEBUG - Free gpu memory value is 2K
2019/03/22 08:33:44 - l_default - DEBUG - Free memory in bytes: 2048.0
2019/03/22 08:33:44 - l_default - ERROR - Free gpu mem is 2048.0 bytes which is less than 80000000 bytes. Streams might fail to start. Consider assigning more memory to gpu in /boot/config.txt with the gpu_mem option
2019/03/22 08:33:44 - l_default - DEBUG - Screen: screen1 active_run_time: 35 / 60
2019/03/22 08:33:45 - l_default - DEBUG - Free gpu memory value is 448M
2019/03/22 08:33:45 - l_default - DEBUG - Free memory in bytes: 469762048.0
2019/03/22 08:33:45 - l_default - DEBUG - Screen: screen1 active_run_time: 36 / 60
SvenVD commented 5 years ago

On first sight I suspect omxplayer to be the cause. Did you try to run omxplayer standalone with exact same arguments as rpisurv uses?

--video_fifo 1 --video_queue 1 --live --timeout 60 --aidx -1 -o hdmi --threshold 0 --avdict rtsp_transport:tcp or without tcp --video_fifo 1 --video_queue 1 --live --timeout 60 --aidx -1 -o hdmi --threshold 0

What version of omxplayer are you using?

kris6673 commented 5 years ago

I did. When i ran it with the exact same arguments there was no stutter. Started rpisurv up and the stutter was there after about 10 seconds. Not sure, i don't have access the to the rpi atm. If it's any help, it's the default version that comes with a brand new install of Raspbian from 2 days ago.

SvenVD commented 5 years ago

Can you try to disable memory_usage_check: False. I had reports in the past that this was causing some issues.

SvenVD commented 5 years ago

It's really weird that the memory usage reported jumps around from almost nothing to almost everything in use. I never saw that before.

kris6673 commented 5 years ago

Is it a config setting? Yeah, its really weird. Just gobbles all the memory like it was cosplaying Chrome ^^

kris6673 commented 5 years ago

I turned memory_usage_check to false. Stuttering is still happening. This is what htop reports for memory usage when rpisurv runs, while the stuttering is still there. OMXplayer version is 061425a.

kris6673 commented 5 years ago

Small update: Even with no camera connected it still uses all the GPU memory.

lugaru90 commented 5 years ago

Same here... tested it outside of your Script - but i'm not sure, if not the vcdbg reloc command is the cause of the problem...

PS: I've just made a rpi-update and installed the latest version of Raspbian.

pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 29 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 31 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 30 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 30 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 33 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 32 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 29 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 33 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
7.8K free memory in 26 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 30 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 33 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 30 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 33 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 29 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
352 free memory in 3 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
352 free memory in 3 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 32 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 30 free block(s)
pi@raspberrypi:/usr/local/bin/rpisurv $ sudo /usr/bin/vcdbg reloc | grep 'free memory'
691M free memory in 32 free block(s)
SvenVD commented 5 years ago

Did you restart after setting memory_usage_check: False? I can not reproduce the errors you are describing.

I am running

root@raspberrypi:~# omxplayer -v omxplayer - Commandline multimedia player for the Raspberry Pi Build date: Mon, 05 Nov 2018 15:45:07 +0000 Version : 061425a [master] Repository: https://github.com/popcornmix/omxplayer.git root@raspberrypi:~# lsb_release -a No LSB modules are available. Distributor ID: Raspbian Description: Raspbian GNU/Linux 9.8 (stretch) Release: 9.8 Codename: stretch

So, it seems there is an issue with the /usr/bin/vcdbg reloc command sometimes using up all the memory, can both of you confirm by stopping rpisurv. systemctl stop rpisurv. And run /usr/bin/vcdbg reloc | grep 'free memory' some times to see if it uses up the memory.

Thanks

kris6673 commented 5 years ago

This This is the output of running sudo /usr/bin/vcdbg reloc | grep 'free memory' before and after the start of rpisurv.

SvenVD commented 5 years ago

Hi, I think we are running in circles here.

  1. Did you set the memory_usage_check: False and did you restart afterwards? Is the problem with the stuttering still there?
  2. What version of omxplayer and raspbian are you running
  3. What is your current GPU mem value set to
  4. Which codec is your stream using, is it h.264? If not try changing it to h.264
  5. Try to capture what is using up the memory with top or similar tools. In your last htop there wasn't any memory issue visible. This could mean that the weird bumps in your logs of memory usage are unrelated to your stuttering. By the way, rpisurv code is not using GPU memory part it is using the arm memory part. Only omxplayer uses the gpu memory.

I can not reproduce the issue you are describing, so the only way to potentially solve this is for you to give me plenty of information :-)

kris6673 commented 5 years ago

We might be.

  1. I set it to false and restarted, the stuttering was still there.
  2. OMXplayer version is 061425a. Raspbian is kernel version 4.14, release date 2018-11-13
  3. 512
  4. h.264
  5. No clue why it's not showing the memory as used. rpisurv is up and running. hotp_prisurv

Sorry. If you know a convenient way of sending a 16 gb file, I could image the SD card and send it to you?

SvenVD commented 5 years ago
  1. In the raspberry pi, you have 2 kind of memory. One that is dedicated for the GPU and one that is dedicated to the arm processor. htop is showing the latter. Rpisurv only uses arm memory, so there isn't a memory leak in rpisurv itself. Only omxplayer uses the GPU memory.

In your htop I see multiple streams?

Could you send me a video of the stuttering?

kris6673 commented 5 years ago

Alright. The multiple streams are child processes according to htop i think. image gif of the problem: https://imgur.com/a/s7wQeLn

SvenVD commented 5 years ago

I am really puzzled. I see no reason why omxplayer should behave differently when started by rpisurv as when started manually. According to the info you provided me I see no issue witht the rpisurv code itself. Are you sure you started omxplayer manually with exact the same arguments? It could be that the stuttering you are describing is just because you have a low framerate. 15 fps can look like image is stuttering

kris6673 commented 5 years ago

I'm puzzled myself, it worked perfectly for 2 weeks, sat around unpowered for 1 week and then the stuttering began. The pi has always been shutdown with a shutdown command, not just pulling the power. It's the exact same arguments on omxplayer, iv'e checked it far too many times. When i look at the stream in VLC it's very clear that it's not the stream being 15 fps that's the problem. No stuttering at all when playing from VLC. Not sure what to do now, got any other ideas or should we just close this issue and call it a day?

SvenVD commented 5 years ago

Are you also running that VLC on from the raspberry pi? But then again you are running omxplayer from the raspberry pi too. Maybe a hardware problem, do you have another pi to try, or even another camera? Or maybe try to substreams of your camera

I'm out of other options

kris6673 commented 5 years ago

Nope, from a Windows PC on the same network. Tried with other camera and that stuttered too. Tried 3 substreams, Still stuttered.. :( Since last time i have installed displaycameras on another SD card and that runs like a charm, not stuttering at all, even if the features leaves a bit to be desired comming from rpisurv. Where in the code do you determine if you need to make another stream/omxplayer window? Those "child processes" might actually be full on streams as you thought earlier. Grasping for straws here. Sadly i don't have another Pi to test on, we would know real fast if it was a hardware problem then.

SvenVD commented 5 years ago

The code managing the worker processes is here: https://github.com/SvenVD/rpisurv/blob/master/surveillance/core/worker.py

What is your config file? You are using one screen right? No rotation?

You are mentioning it did work for 2 weeks without stuttering? What was changed, besides the pi being unpowered for a while.

kris6673 commented 5 years ago

Thank you. Nothing out of the ordinary, i'll upload it when i get access to it on monday. 1 Full HD screen, no rotation. Nothing at all, we changed the url in the config but that's all. It just sat there and when we powered it up again the issue was there.

SvenVD commented 5 years ago

Just an idea, there aren't other sources pulling streams from the same camera are they? Cameras are limited to a certain amount of streams, more expensive cameras can handle more simultaneous streams.

kris6673 commented 5 years ago

Nope, both camera and rpi are only used for eachother.

kris6673 commented 5 years ago

Didnt find the solution and not gonna spend more time on it. Thanks for the help anyway Sven!

bashoogzaad commented 4 years ago

I have the exact same setup and problems. Unable to find the solution either.