blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.78k stars 1.71k forks source link

Dynamically decrease motion / object detection / limit cpu usage to stop system failure #14491

Closed DReffects closed 5 hours ago

DReffects commented 6 hours ago

Describe what you are trying to accomplish and why in non technical terms I am running frigate on a intel N100 mini PC with 4 cameras and object detection with great success. Low CPU usage and good inference speeds. It's running within Home Assistant but that's not importan.

Twice in the past two days I had incredibly slow speeds in Home Assistent (which I check multiple times daily). System monitor was showing a linux load of 40 for over an hour. system was not really responsive but a reboot solved the issue.

Today it happended again so I decided to check the load history logs and addon behaviour more closely.

Turns out a spider has woven its web right in front of one of my cameras and having strong winds this triggered motion detection for each and every frame making the object detection run amok. This eventually congestated the entire system to a point where only a reboot was able to solve it.

Describe the solution you'd like It would be great to have some sort of failsave mechanism like "if system load is over X for Y Minutes and Motion / Object detection is active, pause motion and object detection for X minutes" Or a feature that hard-limits resources for motion and object detection in general.

Describe alternatives you've considered Well I've considered beefy hardware but frigate runs amazingly. Just not when spiders claim space in front of the cam.

NickM-27 commented 5 hours ago

You can set resource limits in docker itself, and even then if you are running in the addon you can do this yourself by monitoring the Frigate camera CPU usage sensors and disabling detection on that specific camera if that is what you want to do.

I don't think it really makes sense for Frigate itself to stop running detection based on system load because for one I have never seen this behavior myself even during times of extremely high activity, and also because Frigate is meant to be doing a job. If someone was able to stop detection for example by creating enough activity that Frigate stopped detecting motion / objects then that would defeat the purpose of having it in the first place.

DReffects commented 5 hours ago

Ah, interesting. How can I edit the resource limits from Home Assistant? Where can I find the CPU Usage for frigate within the home assistant devices?

I am with you on the "doing it's job" aspect but that's exeactly my strain of thought as well. If motion detection can overwhelm the system itself due to unusual events it stops completely. (my system more or less crashed, recordings did not take place anymore during high load). My priority would be:

I've seen this twice now over 4 hours without it being able to recover.

NickM-27 commented 5 hours ago

How can I edit the resource limits from Home Assistant?

if you are using addons then you can't

Where can I find the CPU Usage for frigate within the home assistant devices?

it is one of the sensors for every camera in the integration

DReffects commented 5 hours ago

Thanks. Just found then disabled :-) Is there a grand total CPU usage sensor for frigate as well? Frigate reports around ~30% CPU usage in it's status bar but HA Glance shows ~60-95% usage.

NickM-27 commented 5 hours ago

it is in the MQTT stats but it is not currently exposed as a sensor