Zoneminder slowly consuming RAM until 100%

[TheOneOgre]

Describe Your Environment

• Version: v1.34.23
• Unraid Docker Repo
• Ubuntu Focal Fossa

Describe the bug After running for almost 24 hours, zoneminder slowly consumes more and more RAM until the system runs out. This was not a problem in the previous release and nothing else has changed. This system is part of a multi-server but the main one having issue is the host system with only 5 low resolution cameras on it running at 6fps (between 25-50kbps per camera).

Upon starting zoneminder, everything works fine, but as time goes on, the RAM usage will slowly fill up until the system starts swapping. There doesn't seem to be any indication of what is happening, using "top" shows zmc %MEM slowly rising over time.

All 4 monitors are using H264 Passthrough

I have not posted a debug log simply because there is no useful information inside of it at level 1. All logs are simply information logs, Analyzing and Capturing as normal.

It should be noted, the ram usage drops back to normal as soon as the state is set to restart in the webUI and all monitors are refreshed.

[welcome[bot]]

Thanks for opening your first issue here! Just a reminder, this forum is for Bug Reports only. Be sure to follow the issue template!

[soreau]

I can reproduce this on an ODROID-HC4 ubuntu server 20.10 with 4GB ram and 2x640x480 cams, it was working fine for some 1.3x releases and then an update around the beginning of March or sometime before, made it slowly consume all RAM. I added 4GB swap but it chewed through all of that too, for a total of 8GB consumed by zmc process. Feels like a memory leak. It happens here after creating a few events and viewing them, watching zmc in htop consume all RAM until the system is no longer responsive.

[TheOneOgre]

What's very odd about the issue is, in a multi server config, only the instance running the database consumes the RAM. I switched out all but 1 camera to the other server and it's working just fine until that single 5mp camera's ZMC process works it way up to 12+GB filling the ram, and causing the system to be unresponsive.

I should note, the other server has 3 streams from the different, but same model cameras and does not experience the issue.

The only way to resolve the issue is to move every camera to the secondary server (configured using everything exactly the same as the primary)

Also happens outside of a multi-server configuration, just have tried so many things to resolve this issue to no avail.

[soreau]

For what it's worth, I'm trying iSpy/Agent. After installing dotnet, it was easy to run and configure. Email works with postfix gmail relay. It consumes much more cpu than zoneminder but much less RAM, so far so good, the system doesn't crash with 3x640x480 and sends emails happily while also running a wayland vnc display server.

[connortechnology]

This is all a bit of a problem: mainly because not much changed between 1.34.22 and 1.34.23. Certainly nothing that I could imagine causing a memleak.

Also, my systems running 1.34.23 (demo.zoneminder.com) don't do it.

So what you can do is: Install valgrind, sudo zmdc.pl stop zmwatch.pl sudo zmdc.pl stop zmc -m whateverthemonitoridis sudo su -s/bin/bash -c"valgrind -v --leak-check=full --show-leak-kinds=all zmc -m monitorid" www-data

Let it run until it runs out of ram and post the results here.

[TheOneOgre]

I've run the command and since this is only a single monitor that is consuming all the ram it will take roughly a week before it runs out but the data collected so far seems to show an issue related to iHD_drv_video.so which I am assuming is the QuickSync driver for hardware encoding/decoding.

I am indeed using vaapi for the monitor on this server to reduce the CPU load. Should I try disabling vaapi acceleration once the ZMC process runs out of memory? and run valgrind again?

Thanks a lot for the help! terminal.txt

Edit: I should also note that during my debugging of this problem on my own, the server is now only running a single 5MP camera (RLC-520) at 5fps, to minimize variables.

[TheOneOgre]

After submitting my last comment, valgrind reported ==1308== More than 10000000 total errors detected. I'm not reporting any more. ==1308== Final error counts will be inaccurate. Go fix your program! ==1308== Rerun with --error-limit=no to disable this cutoff. Note ==1308== that errors may occur in your program without prior warning from ==1308== Valgrind, because errors are no longer being displayed.

I then exited the command as it simply wouldn't log anymore.

If the data presented in the text file is not enough, I can re-run the command with the --error-limit=no if need be.

Another sidenote, the ImageBufferCount 25 is an issue I am only encountering while running the valgrind command, the normal system operation does not see this as an issue and the FPS is being reported correctly.

Thanks again! terminal2.txt

[connortechnology]

Well that certainly sheds some light. Gives me a place to look. I know I did some different stuff in 1.35 with hwaccel so maybe I need to back port that.

[fl0wm0ti0n]

i run also into this with version v1.34.20

[xipe-jringland]

I'm running into a similar issue on 1.36.1 under Ubuntu 20.04.02 LTS. Fresh install of OS and ZM from the PPA, under Valgrind, ZMC mostly behaves itself, topping out at about 2.5GB of memory. (PID 5387 in the attached GIF). Without Valgrind at startup, it maxes out the machine until Linux won't respond anymore. PID 5539 in the image after running 2m 24s of CPU time is at 52.1G resident, 75.7G V and climbing fast.

I also included a GIF of it running to MAX_MEMORY of 64G. Takes about 1min.

zmmem2.txt

[connortechnology]

Under valgrind, decoding takes too long. You have to set a max image buffer count or else yeah, it will use up all ram.

[tuxedo0801]

seeing the same or similar behavior with 1.36.7 ...

I think it worked with 1.34 ... now with 1.36 it's totally weird.

[tuxedo0801]

Back to 1.34.23 on a newly installed Debian 11 in a VM. With enough /dev/shm, it seems to run fine...

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.