search

ZoneMinder / zoneminder

ZoneMinder is a free, open source Closed-circuit television software application developed for Linux which supports IP, USB and Analog cameras.
http://www.zoneminder.com/
GNU General Public License v2.0
5.16k stars 1.23k forks source link

High CPU load (100%) and multiple zmc processes #3448

Closed tb205gti closed 1 year ago

tb205gti commented 2 years ago

Describe Your Environment

Describe the bug

Once Zonemidner is started it quickly rises to 100% CPU load. I have 7 cemeras all UDP rtsp streams. all running 1080. 'top' reveals that there are multiple zmc processes per camera, is this normal, I'd expect one capture process per camera, and not multiple. See screenshot: https://i.imgur.com/w3wupNR.png

To Reproduce Steps to reproduce the behavior:

  1. Start Zoneminder
  2. Enable Capturing on all cameras

Expected behavior lower CPU usage than 100%

Debug Logs

tb205gti commented 2 years ago

I'm a retard :) I was using htop - so if I sort by process, I only get one zmc process per camera.

however - I still have massive high cpu load. Historically load with all cameras enabled is about 80% max - but now I hammers up to 100%

anonimou0 commented 2 years ago

I have the same behavior as yours. I was on zoneminder-master respository and recently upgrade to 1.37.12 and zmc processes have been hammering the CPU.

The load average was way over 20, and the server became unresponsive. It even reaching 90+!!! .

Usually, the load average would hover around 4-7.

Downgrading to 1.36.12 (zoneminder-proposed repository) fixed it.

tb205gti commented 2 years ago

I cannot use 1.37.12 at all - it fails to start - and I have not had time to investigate what is going on. 1.37.11 is the latest version that is working for me - but with massive CPU load.

tb205gti commented 2 years ago

Downgraded to 1.36.12, and CPU load went down ~20%- thanks @anonimou0

I'm now back at 75-80% load with all cameras enabled and analysis active.

anonimou0 commented 2 years ago

The CPU usage was going crazy with zmc using ~most~ all CPU. At least, that's what top would show.

Before the issues started, streams would capture at a steady pace of ~20 frames. Below, we can see spikes to over 80 fps (???) and going down. Also, some errors. Hope logs for zmc on one stream helps:

zmc_m13.log

klaasvakie commented 2 years ago

This affects me as well - upgrade to 1.37.13 has zmc using massive amounts of cpu, and a load average of 70. Only difference I see in the logs are a bunch of errors w.r.t. shm

Mar 29 21:01:18 zm zmeventnotification[524]: ERR [Memory map file '/dev/shm/zm.mmap.7' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:18 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.5' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:18 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a5b0) Guest] Mar 29 21:01:18 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.6' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:19 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a610) Driveway-mon] Mar 29 21:01:19 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.7' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:19 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a670) Garage-mon] Mar 29 21:01:19 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.8' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:20 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a6d0) Garden-mon] Mar 29 21:01:20 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.9' does not exist in zmMemAttach. zmc might not be running.]

and some of these as well:

Mar 29 21:01:31 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf83d418) Driveway] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf83d4f0) Garage] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86ff78) Garden] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf51a8d8) Pool] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86ff90) Dogdoor] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a5b0) Guest] Mar 29 21:01:33 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a610) Driveway-mon] Mar 29 21:01:33 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a670) Garage-mon] Mar 29 21:01:33 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a6d0) Garden-mon] Mar 29 21:01:34 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a730) Pool-mon] Mar 29 21:01:34 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a790) Dogdoor-mon] Mar 29 21:01:35 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a7f0) Guest-mon]

klaasvakie commented 2 years ago

This affects me as well - upgrade to 1.37.13 has zmc using massive amounts of cpu, and a load average of 70. Only difference I see in the logs are a bunch of errors w.r.t. shm

So I found the issue on my side at least - the issue was caused by the change in how "DecodingEnabled" is handled. See this commit: https://github.com/ZoneMinder/zoneminder/commit/abf238a4696b6fd26c6d9b51a9a606f11cb37640

After changing the relevant channels to have "Decoding = None" everything seems to be back to normal. Apologies for the noise

anonimou0 commented 2 years ago

Thank you @klaasvakie . That seems to be the issue.

I have 16 cameras that record without decoding in H265 in high resolution (4MP or 6MP). I use the cameras sub-streams to view these cameras on lower resolutions, which, then, has a decoding being used.

After upgrading to 1.37.11+, the Decoding option (on Source tab) comes set to Always. Re-setting it to None made the system usable again.

Currently running 1.37.13

connortechnology commented 2 years ago

You may be interested in the new Decoding options : Keyframes Only and On Demand. KeyFrames Only offers almost all the benefits of None, but with thumbnails and depending on keyframe interval, a watchable live stream.

mfauvain commented 2 years ago

Had the exact same issue upgarding to 1.36.19 lately. multiple zmc processes on all monitors (all in Mocord) and couldn't figure out the issue as none of them have the option 'Decoding Enabled' 220628-153847 ' Checking the database/Monitors table, all of them had 'DecodingEnabled' in True... Setting them back to 0 directly in the database did the trick. I would have had (I guess) the same result switching them to Record, changing the Decoding Enabled to unticked and then back to Mocord but anyway... 220628-154106

Leaving it here as it might help others

CV8R commented 2 years ago

Same issue v1.36.19. Couldn't change Run States or stop and start Zoneminder from terminal. High CPU. Solution, change to Record for each Camera and untick Decoding Enabled for each camera. I then saved to a new run state. From that point I could change between my existing Run States.

Log was filled with Memory map file '/dev/shm/zm.mmap. does not exist in zmMemAttach.

connortechnology commented 1 year ago

@mfauvain you can't do motion detection without decoding, which is why you weren't seeing the option.

Closing since this is a simple issue of not enough cpu for the resolution.

tb205gti commented 1 year ago

quote: "Closing since this is a simple issue of not enough cpu for the resolution."

That is not the case - running an older (1.36.x) version with the exact same config gives me much lower CPU utilisation. So how you think this can be due to "not enough cpu" is a mystery to me. (Cpu is: Intel(R) Core(TM) i5-6300U CPU, running @ 3GHz, should be more than enough)

connortechnology commented 1 year ago

You'd think so, wouldn't you? The issue is less about the power of the cpu, and the design of the code.

The newer version is more correct, but less efficient (there is 1 more image copy than in the past).

The latest 1.37.33 has removed 1 image copy. You may find that it is enough for things to work.

Long term, we can't operate the way we used to. The code was designed for 640x480, not 4k res. The cameras keep getting higher res faster than we can keep up.

tb205gti commented 1 year ago

Ahh, I'll try out the 1.37.33+ then - see if it works.

So what needs to be changed for higher resolutions - is it a complete rewrite of the code?

connortechnology commented 1 year ago

Not a complete rewrite. Currently: Step 1 get h265 packet Step 2 decode to yuyv420p Step 3 convert to rgba Step 4 copy to /dev/shm

We need to eliminate Step 3, and do motion detection on the y-channel. The copy to /dev/shm is now smaller as well. We then need code to generate a jpeg from yuv420p for live view, etc.

Eventually maybe we can get rid of the copy to /dev/shm.