[Bug/Degradation]: High CPU Usage With Motion Detection

[skrashevich]

Describe the problem you are having

after some latest commits, related to motion detection, it's works worse then before

Version

dev-7c1568f

Frigate config file

same as in #6802

Relevant log output

irrelevant, no errors/warnings etc

FFprobe output from your camera

irrelevant

Frigate stats

No response

Operating system

UNRAID

Install method

Docker CLI

Coral version

USB

Network connection

Wired

Camera make and model

netatmo welcome

Any other information that may be helpful

[kirsch33]

i am running this same dev build and have noticed a degradation in overall detection performance as well, with true positives specifically. seems to be related to one of these PRs https://github.com/blakeblackshear/frigate/pull/6516 https://github.com/blakeblackshear/frigate/pull/6741 since if i downgrade to a build prior to those, its back to normal

[NickM-27]

i am running this same dev build and have noticed a degradation in overall detection performance as well, with true positives specifically. seems to be related to one of these PRs https://github.com/blakeblackshear/frigate/pull/6516 https://github.com/blakeblackshear/frigate/pull/6741 since if i downgrade to a build prior to those, its back to normal

https://github.com/blakeblackshear/frigate/commit/7c1568fcb941ad17256ab73438cb997b23fb7f33 is known to cause issues with motion detection being overly sensitive leading to more false positive.

The motion PR before that made motion less sensitive in certain situations.

Obviously it's going to be worked on but for the time being simply adjusting motion settings in the config will result in fixing the issue.

[kirsch33]

i am running this same dev build and have noticed a degradation in overall detection performance as well, with true positives specifically. seems to be related to one of these PRs #6516 #6741 since if i downgrade to a build prior to those, its back to normal

7c1568f is known to cause issues with motion detection being overly sensitive leading to more false positive.

The motion PR before that made motion less sensitive in certain situations.

Obviously it's going to be worked on but for the time being simply adjusting motion settings in the config will result in fixing the issue.

for what it’s worth, here is my motion config, adjusted after reading a recent PR:

motion:
  threshold: 30
  contour_area: 50
  frame_height: 100

[skrashevich]

Obviously it's going to be worked on but for the time being simply adjusting motion settings in the config will result in fixing the issue.

Can you give an example of configuration? Honestly, the code of this functionality scares me :)

[NickM-27]

basically:

• contour_area is how large an area of motion needs to be to be considered motion. Lower is more sensitive.

• threshold is the difference in the pixel required to count as motion. Lower is more sensitive.

It really depends on the camera as for some of mine the default settings are perfect and for others it's not.

One thing to try is to disable improve_contrast and then make the threshold much more sensitive.

[ccutrer]

I just tried with the latest commit. It still seems too sensitive with default settings (I've never had to tweak motion settings before):

https://capture.dropbox.com/JsrDL15BdHDBenxu

There's only a tiny bit of wind right now, so motion on the edges of leaves would be fine, but there's no visible motion in the grass. It is interesting that all the suspected motion is at the edges of the shadow of the roof - very high contrast areas at the moment. There also wasn't any clouds around where it suddenly got bright, mistaking that as motion.

[ccutrer]

Another instance where high contrast edges are seeing lots of motion:

I noticed on another camera it's seeing motion around the timestamp, even though the timestamp itself is masked out. I can only guess that either the contrast improvement process is blooming out from that change, or it's detecting small changes from compression artifacts as movement. I'm going to try disabling improve_contrast for now.

[NickM-27]

@ccutrer what is the detect resolution on those camera(s)?

[ccutrer]

1280x720

[ccutrer]

I've turned off improve_contrast, and I'm not seeing nearly as many false positives. But my CPU usage is much higher than yesterday (with improve_contrast on by default, and before https://github.com/blakeblackshear/frigate/pull/6870 was merged). I'm now at 0% idle CPU, with the majority taken up by the frigate.process: process for cameras that have any motion (12-13% each). I do have latest snapshots for everything now, though. And on the system page, cameras without any motion have a capture FPS that matches their ffmpeg FPS (meaning it's keeping up, and not skipping any frames). Overall detector fps is now closer 30-40 (was 60-90 yesterday). Cameras with consistent motion have a capture FPS of 0.4-2.

[NickM-27]

Interesting. We are aware of a problem where the fast resize method is causing lots of noise on higher resolutions like 1080p, and setting a higher frame height would fix that.

All my cameras are 720p in detect and the default settings in the latest have been working quite well for me. It's supposed to rain today so will be curious how it works with the rain, but also don't have any bright sun right now so can't test to see if I am seeing the same there.

I think in general, the same set of motion settings won't work for every camera and some adjustments will need to be made to tune for certain cameras, and I don't really see a problem with that.

[NickM-27]

Sun peeked out for a couple minutes and I was seeing something similar on one of the cameras so I adjusted the threshold to 50 and things are working well now

[ccutrer]

:nod:. I definitely need to find some time to become familiar with the motion settings, and dial them in better. Are there in narrative-style docs for how best to tweak motion settings? Something like https://docs.frigate.video/guides/false_positives, https://docs.frigate.video/guides/stationary_objects, and https://docs.frigate.video/configuration/stationary_objects, which guide which settings to tweak first, and why. So far all I've found is the config file reference which tells what the tweakable knobs are, but no recommendations are where to start in which situations.

I looked a bit more at which cameras were using high CPU, even if they don't necessarily have high motion. I noticed my theater room cameras are both in that bucket. Dark room, cameras in night vision mode, with a lot of noise, but zero actual motion. Definitely seem like candidates for tweaking motion setting.

Other cameras I really have no idea why they would be in high CPU if it's not a systemic problem across all cameras. Two other indoor, but bright, currently unoccupied rooms, for example.

[NickM-27]

Something like https://docs.frigate.video/guides/false_positives, https://docs.frigate.video/guides/stationary_objects, and https://docs.frigate.video/configuration/stationary_objects, which guide which settings to tweak first, and why

I think a Tuning Motion Detection would be a good guide to write. The reference docs do say when to use what but I think you need to be familiar with the terms to understand what they mean.

Other cameras I really have no idea why they would be in high CPU if it's not a systemic problem across all cameras. Two other indoor, but bright, currently unoccupied rooms, for example.

That might be a better question for Blake, I don't know what exactly would lead to higher CPU usage, mine are mostly pretty low but without much activity right now.

[ccutrer]

The ones that concern me are say Entry and Family in this screenshot - 12.x% CPU, with 0 frames even sent for object detection, and no actual motion at the moment.

[NickM-27]

@ccutrer let's try something here:

For the cameras with high CPU usage but no motion

1. Using the HA integration or MQTT turn off object detect and motion detect (they are separate). Make note of what the baseline is.
2. Enable motion detect and let things settle for at least 1 minute, make note of what this is.
3. Toggle improve contrast (this is another switch in HA / MQTT) and let things settle for at least 1 minute, make note of what CPU usage is.
4. Finally, enable object detect and see how things change after that.

[ccutrer]

For gaming area camera:

Baseline CPU is 12-14% consistently, with no visible motion: Indeed, going to the debug view and watching it, no motion boxes are shown.

Disabling object detection, motion detection, and improve contrast: as expected CPU usage drops to ~0.3%. Enabling just motion: It rises back to 12-13% Enabling improve contrast: Stays steady at 12-13% Enabling object detection: Stays steady at 12-13% (stats report 0 fps on detect, so make sense - no frames were even sent to object detection, cause there is no motion).

[blakeblackshear]

@ccutrer what is the detect resolution on that camera?

Would you be able to install and run py-spy in the container to dig into where the CPU usage is actually coming from?

[ccutrer]

@ccutrer what is the detect resolution on that camera?

Would you be able to install and run py-spy in the container to dig into where the CPU usage is actually coming from?

1280x720

py-spy top --pid <pid> has 12.00% 12.00% 3.30s 3.37s detect (frigate/motion/improved_motion.py as the consistent top entry.

I'm not sure what format you would like, so I just did an SVG and let it run for about a minute. Let me know if you want me to run it some other way.

[blakeblackshear]

I'm most interested in which function is taking the most time in py-spy top --pid <pid>. You can sort by own time and total time.

[kirsch33]

i am experiencing the same increased CPU usage even on the dev-9e531b0 container with revised motion settings below:

motion:
  threshold: 30
  contour_area: 50
  frame_height: 100

this is on 2 cameras with a detect resolution of 1080p. if needed I can post my full config.

whats odd is that at first i noticed a lot of motion boxes being created from a close by tree fluttering in the wind, but even after masking that region the CPU usage hasnt tapered down.

this evening i can also run py-spy as mentioned and follow up.

[NickM-27]

This is what I am getting with py-spy on one of my cameras that is pretty active right now.

Sorted by own %

Sorted by own time

[ccutrer]

I ran an experiment this morning. I rolled back to 0.12.1 (copying the same config, but letting it start with a fresh database and no prior recordings). My overall CPU usage is 10-15% idle. Interestingly, active cameras are actually using from 12-30% CPU. I'm using a camera that my kids are currently play in front of, so lots of motion. Py-spy looks like this (OwnTime sorted):

TotalTime sorted:

A relatively idle camera, OwnTime:

TotalTime:

Now I repeat, with 0.13-9E531B0. Overall CPU is 4-5% idle, with the cameras using lots of processing time maxing at 12-25% (which is actually lower CPU usage than 0.12.1).

OwnTime of busy camera:

TotalTime:

Idle camera, OwnTime:

TotalTime:

The major difference seems to be that almost all cameras are using either ~12% CPU (movement) or ~6% CPU (idle), whereas with 0.12.1 idle cameras are using ~0.3% CPU (but busy cameras use as much as 30% CPU). And overall, I think I'm running closer to redline than I thought I was, with any version of Frigate.

[ccutrer]

So... one thing I found is that the default frame_height changed from 50 to 100 from 0.12.1 to current dev, which implies why CPU usage would be up on the image resize for motion detection. But changing it back to 50 doesn't seem to have had any effect on lowering my CPU usage, or explain why some cameras on 0.12.1 seem to need almost no CPU for motion detection.

[ccutrer]

NAILED IT. Well, at least where the CPU time is going. As to why, I have no idea. I added logging like so to both 0.12.1 and current dev: https://github.com/blakeblackshear/frigate/commit/64cab909a0ffc8e53abe0a13253c39fbdc0b9d1c.

In 0.12, the output looks like:

2023-06-22 21:13:39.038076722  [2023-06-22 21:13:39] frigate.video                  INFO    : took 0.0008468349999999902s to process a frame in sewing
2023-06-22 21:13:39.199178124  [2023-06-22 21:13:39] frigate.video                  INFO    : Took 0.00115266399999997s of CPU time to get a frame
2023-06-22 21:13:39.199182435  [2023-06-22 21:13:39] frigate.video                  INFO    : took 0.0006249140000000319s to process a frame in sewing
2023-06-22 21:13:39.366947193  [2023-06-22 21:13:39] frigate.video                  INFO    : Took 0.0009439820000000099s of CPU time to get a frame
2023-06-22 21:13:39.368941350  [2023-06-22 21:13:39] frigate.video                  INFO    : took 0.0010397639999999986s to process a frame in sewing
2023-06-22 21:13:39.591830061  [2023-06-22 21:13:39] frigate.video                  INFO    : Took 0.0013261360000000333s of CPU time to get a frame
2023-06-22 21:13:39.591918976  [2023-06-22 21:13:39] frigate.video                  INFO    : took 0.000709407999999967s to process a frame in sewing
2023-06-22 21:13:39.721831109  [2023-06-22 21:13:39] frigate.video                  INFO    : Took 0.0009642069999999947s of CPU time to get a frame
2023-06-22 21:13:39.721836222  [2023-06-22 21:13:39] frigate.video                  INFO    : took 0.0006821079999999591s to process a frame in sewing
2023-06-22 21:13:39.871252578  [2023-06-22 21:13:39] frigate.video                  INFO    : Took 0.0010561069999999728s of CPU time to get a frame
2023-06-22 21:13:39.871835558  [2023-06-22 21:13:39] frigate.video                  INFO    : took 0.0007609920000000159s to process a frame in sewing
2023-06-22 21:13:40.035793741  [2023-06-22 21:13:40] frigate.video                  INFO    : Took 0.0010022150000000285s of CPU time to get a frame
2023-06-22 21:13:40.039007570  [2023-06-22 21:13:40] frigate.video                  INFO    : took 0.0011003139999999911s to process a frame in sewing
2023-06-22 21:13:40.225747470  [2023-06-22 21:13:40] frigate.video                  INFO    : Took 0.001642054000000004s of CPU time to get a frame
2023-06-22 21:13:40.225753350  [2023-06-22 21:13:40] frigate.video                  INFO    : took 0.0009611479999999673s to process a frame in sewing
2023-06-22 21:13:40.353746081  [2023-06-22 21:13:40] frigate.video                  INFO    : Took 0.0013178469999999831s of CPU time to get a frame
2023-06-22 21:13:40.353972537  [2023-06-22 21:13:40] frigate.video                  INFO    : took 0.0007735070000000066s to process a frame in sewing
2023-06-22 21:13:40.652160811  [2023-06-22 21:13:40] frigate.video                  INFO    : Took 0.0011207189999999922s of CPU time to get a frame
2023-06-22 21:13:40.652872978  [2023-06-22 21:13:40] frigate.video                  INFO    : took 0.0008133060000000136s to process a frame in sewing
2023-06-22 21:13:40.783326671  [2023-06-22 21:13:40] frigate.video                  INFO    : Took 0.0011193760000000053s of CPU time to get a frame
2023-06-22 21:13:40.783331407  [2023-06-22 21:13:40] frigate.video                  INFO    : took 0.000818384999999977s to process a frame in sewing
2023-06-22 21:13:40.871037528  [2023-06-22 21:13:40] frigate.video                  INFO    : Took 0.0011317980000000172s of CPU time to get a frame
2023-06-22 21:13:40.871750462  [2023-06-22 21:13:40] frigate.video                  INFO    : took 0.0007742159999999942s to process a frame in sewing
2023-06-22 21:13:41.036792571  [2023-06-22 21:13:41] frigate.video                  INFO    : Took 0.0010781939999999768s of CPU time to get a frame
2023-06-22 21:13:41.037420576  [2023-06-22 21:13:41] frigate.video                  INFO    : took 0.0010032620000000048s to process a frame in sewing
2023-06-22 21:13:41.203382935  [2023-06-22 21:13:41] frigate.video                  INFO    : Took 0.001334771999999984s of CPU time to get a frame
2023-06-22 21:13:41.203387172  [2023-06-22 21:13:41] frigate.video                  INFO    : took 0.0008461980000000202s to process a frame in sewing
2023-06-22 21:13:41.352670805  [2023-06-22 21:13:41] frigate.video                  INFO    : Took 0.0013475099999999962s of CPU time to get a frame
2023-06-22 21:13:41.352675210  [2023-06-22 21:13:41] frigate.video                  INFO    : took 0.0006954860000000229s to process a frame in sewing
2023-06-22 21:13:41.562442167  [2023-06-22 21:13:41] frigate.video                  INFO    : Took 0.0010901170000000016s of CPU time to get a frame
2023-06-22 21:13:41.562447702  [2023-06-22 21:13:41] frigate.video                  INFO    : took 0.0007591860000000228s to process a frame in sewing
2023-06-22 21:13:41.719295082  [2023-06-22 21:13:41] frigate.video                  INFO    : Took 0.0011082159999999952s of CPU time to get a frame
2023-06-22 21:13:41.719299980  [2023-06-22 21:13:41] frigate.video                  INFO    : took 0.000649044000000043s to process a frame in sewing
2023-06-22 21:13:41.870255794  [2023-06-22 21:13:41] frigate.video                  INFO    : Took 0.0009396269999999984s of CPU time to get a frame
2023-06-22 21:13:41.871052217  [2023-06-22 21:13:41] frigate.video                  INFO    : took 0.000766570000000022s to process a frame in sewing
2023-06-22 21:13:42.047346288  [2023-06-22 21:13:42] frigate.video                  INFO    : Took 0.0010989180000000043s of CPU time to get a frame
2023-06-22 21:13:42.047521383  [2023-06-22 21:13:42] frigate.video                  INFO    : took 0.0009159920000000321s to process a frame in sewing
2023-06-22 21:13:42.194802287  [2023-06-22 21:13:42] frigate.video                  INFO    : Took 0.0015327500000000271s of CPU time to get a frame
2023-06-22 21:13:42.197586550  [2023-06-22 21:13:42] frigate.video                  INFO    : took 0.0011527370000000148s to process a frame in sewing
2023-06-22 21:13:42.366508694  [2023-06-22 21:13:42] frigate.video                  INFO    : Took 0.001724567999999982s of CPU time to get a frame
2023-06-22 21:13:42.367822884  [2023-06-22 21:13:42] frigate.video                  INFO    : took 0.000867765999999992s to process a frame in sewing
2023-06-22 21:13:42.617824046  [2023-06-22 21:13:42] frigate.video                  INFO    : Took 0.0011956780000000333s of CPU time to get a frame
2023-06-22 21:13:42.617830050  [2023-06-22 21:13:42] frigate.video                  INFO    : took 0.0007750359999999512s to process a frame in sewing
2023-06-22 21:13:42.718004084  [2023-06-22 21:13:42] frigate.video                  INFO    : Took 0.0012124239999999897s of CPU time to get a frame
2023-06-22 21:13:42.719380366  [2023-06-22 21:13:42] frigate.video                  INFO    : took 0.0014712879999999595s to process a frame in sewing
2023-06-22 21:13:42.877469581  [2023-06-22 21:13:42] frigate.video                  INFO    : Took 0.001998679000000003s of CPU time to get a frame
2023-06-22 21:13:42.882687128  [2023-06-22 21:13:42] frigate.video                  INFO    : took 0.0014282039999999885s to process a frame in sewing
2023-06-22 21:13:43.034180269  [2023-06-22 21:13:43] frigate.video                  INFO    : Took 0.001988683999999963s of CPU time to get a frame
2023-06-22 21:13:43.034185984  [2023-06-22 21:13:43] frigate.video                  INFO    : took 0.000594123000000002s to process a frame in sewing
2023-06-22 21:13:43.217970854  [2023-06-22 21:13:43] frigate.video                  INFO    : Took 0.0008914320000000253s of CPU time to get a frame
2023-06-22 21:13:43.217975743  [2023-06-22 21:13:43] frigate.video                  INFO    : took 0.0008995170000000163s to process a frame in sewing
2023-06-22 21:13:43.361897303  [2023-06-22 21:13:43] frigate.video                  INFO    : Took 0.0012474060000000065s of CPU time to get a frame
2023-06-22 21:13:43.362043707  [2023-06-22 21:13:43] frigate.video                  INFO    : took 0.0010529340000000054s to process a frame in sewing
2023-06-22 21:13:43.562377853  [2023-06-22 21:13:43] frigate.video                  INFO    : Took 0.001956979999999997s of CPU time to get a frame
2023-06-22 21:13:43.562382673  [2023-06-22 21:13:43] frigate.video                  INFO    : took 0.0006874940000000107s to process a frame in sewing
2023-06-22 21:13:43.715901892  [2023-06-22 21:13:43] frigate.video                  INFO    : Took 0.0010155259999999888s of CPU time to get a frame
2023-06-22 21:13:43.715906895  [2023-06-22 21:13:43] frigate.video                  INFO    : took 0.0006522529999999915s to process a frame in sewing
2023-06-22 21:13:43.875087434  [2023-06-22 21:13:43] frigate.video                  INFO    : Took 0.0011131389999999852s of CPU time to get a frame
2023-06-22 21:13:43.875093222  [2023-06-22 21:13:43] frigate.video                  INFO    : took 0.000813100000000011s to process a frame in sewing
2023-06-22 21:13:44.068225202  [2023-06-22 21:13:44] frigate.video                  INFO    : Took 0.0013299079999999908s of CPU time to get a frame
2023-06-22 21:13:44.068230342  [2023-06-22 21:13:44] frigate.video                  INFO    : took 0.0007638790000000228s to process a frame in sewing
2023-06-22 21:13:44.230986946  [2023-06-22 21:13:44] frigate.video                  INFO    : Took 0.0011208460000000087s of CPU time to get a frame
2023-06-22 21:13:44.231804794  [2023-06-22 21:13:44] frigate.video                  INFO    : took 0.0009961700000000184s to process a frame in sewing
2023-06-22 21:13:44.350373914  [2023-06-22 21:13:44] frigate.video                  INFO    : Took 0.001444470000000031s of CPU time to get a frame
2023-06-22 21:13:44.351458155  [2023-06-22 21:13:44] frigate.video                  INFO    : took 0.0013727080000000003s to process a frame in sewing
2023-06-22 21:13:44.571248974  [2023-06-22 21:13:44] frigate.video                  INFO    : Took 0.0017139509999999913s of CPU time to get a frame
2023-06-22 21:13:44.571254307  [2023-06-22 21:13:44] frigate.video                  INFO    : took 0.0007136400000000154s to process a frame in sewing
2023-06-22 21:13:44.761456231  [2023-06-22 21:13:44] frigate.video                  INFO    : Took 0.001065790000000011s of CPU time to get a frame
2023-06-22 21:13:44.762298054  [2023-06-22 21:13:44] frigate.video                  INFO    : took 0.0007186969999999904s to process a frame in sewing
2023-06-22 21:13:44.870884333  [2023-06-22 21:13:44] frigate.video                  INFO    : Took 0.0010022570000000064s of CPU time to get a frame
2023-06-22 21:13:44.871051043  [2023-06-22 21:13:44] frigate.video                  INFO    : took 0.0007123879999999527s to process a frame in sewing
2023-06-22 21:13:45.037199968  [2023-06-22 21:13:45] frigate.video                  INFO    : Took 0.0010852359999999894s of CPU time to get a frame
2023-06-22 21:13:45.037204378  [2023-06-22 21:13:45] frigate.video                  INFO    : took 0.0009966639999999805s to process a frame in sewing
2023-06-22 21:13:45.213293484  [2023-06-22 21:13:45] frigate.video                  INFO    : Took 0.001398153000000013s of CPU time to get a frame
2023-06-22 21:13:45.214166948  [2023-06-22 21:13:45] frigate.video                  INFO    : took 0.0014496709999999857s to process a frame in sewing
2023-06-22 21:13:45.377569791  [2023-06-22 21:13:45] frigate.video                  INFO    : Took 0.0018699499999999536s of CPU time to get a frame
2023-06-22 21:13:45.378681477  [2023-06-22 21:13:45] frigate.video                  INFO    : took 0.0013473680000000154s to process a frame in sewing
2023-06-22 21:13:45.556672072  [2023-06-22 21:13:45] frigate.video                  INFO    : Took 0.0017046099999999953s of CPU time to get a frame
2023-06-22 21:13:45.557278828  [2023-06-22 21:13:45] frigate.video                  INFO    : took 0.0008226200000000516s to process a frame in sewing
2023-06-22 21:13:45.716609079  [2023-06-22 21:13:45] frigate.video                  INFO    : Took 0.0011793780000000087s of CPU time to get a frame
2023-06-22 21:13:45.716653848  [2023-06-22 21:13:45] frigate.video                  INFO    : took 0.0014827410000000096s to process a frame in sewing
2023-06-22 21:13:45.903408360  [2023-06-22 21:13:45] frigate.video                  INFO    : Took 0.002046148999999997s of CPU time to get a frame
2023-06-22 21:13:45.903414831  [2023-06-22 21:13:45] frigate.video                  INFO    : took 0.0012030080000000054s to process a frame in sewing
2023-06-22 21:13:46.034719347  [2023-06-22 21:13:46] frigate.video                  INFO    : Took 0.0016427170000000157s of CPU time to get a frame
2023-06-22 21:13:46.034743028  [2023-06-22 21:13:46] frigate.video                  INFO    : took 0.0011984509999999893s to process a frame in sewing
2023-06-22 21:13:46.196542796  [2023-06-22 21:13:46] frigate.video                  INFO    : Took 0.0017903040000000203s of CPU time to get a frame
2023-06-22 21:13:46.198153475  [2023-06-22 21:13:46] frigate.video                  INFO    : took 0.0018881769999999909s to process a frame in sewing
2023-06-22 21:13:46.360417155  [2023-06-22 21:13:46] frigate.video                  INFO    : Took 0.00262463199999996s of CPU time to get a frame
2023-06-22 21:13:46.361899420  [2023-06-22 21:13:46] frigate.video                  INFO    : took 0.001120968s to process a frame in sewing
2023-06-22 21:13:46.567443975  [2023-06-22 21:13:46] frigate.video                  INFO    : Took 0.00149120700000005s of CPU time to get a frame
2023-06-22 21:13:46.567449221  [2023-06-22 21:13:46] frigate.video                  INFO    : took 0.0006917109999999838s to process a frame in sewing
2023-06-22 21:13:46.713540704  [2023-06-22 21:13:46] frigate.video                  INFO    : Took 0.0010836949999999956s of CPU time to get a frame
2023-06-22 21:13:46.714711480  [2023-06-22 21:13:46] frigate.video                  INFO    : took 0.0009756380000000009s to process a frame in sewing

On dev, it looks like:

2023-06-22 21:17:13.027203887  [2023-06-22 21:17:13] frigate.video                  INFO    : took 0.0013623089999992288s to process a frame in sewing
2023-06-22 21:17:13.066056366  [2023-06-22 21:17:13] frigate.video                  INFO    : Took 0.021322379999999086s of CPU time to get a frame
2023-06-22 21:17:13.066060232  [2023-06-22 21:17:13] frigate.video                  INFO    : took 0.0010076489999999438s to process a frame in sewing
2023-06-22 21:17:13.270283723  [2023-06-22 21:17:13] frigate.video                  INFO    : Took 0.021338412000000417s of CPU time to get a frame
2023-06-22 21:17:13.270288180  [2023-06-22 21:17:13] frigate.video                  INFO    : took 0.0014028239999994696s to process a frame in sewing
2023-06-22 21:17:13.642889931  [2023-06-22 21:17:13] frigate.video                  INFO    : Took 0.020646795999999412s of CPU time to get a frame
2023-06-22 21:17:13.642895004  [2023-06-22 21:17:13] frigate.video                  INFO    : took 0.0019041560000001567s to process a frame in sewing
2023-06-22 21:17:13.780373284  [2023-06-22 21:17:13] frigate.video                  INFO    : Took 0.020232702000001268s of CPU time to get a frame
2023-06-22 21:17:13.780378408  [2023-06-22 21:17:13] frigate.video                  INFO    : took 0.0014676710000003368s to process a frame in sewing
2023-06-22 21:17:14.013328932  [2023-06-22 21:17:14] frigate.video                  INFO    : Took 0.020040994999998674s of CPU time to get a frame
2023-06-22 21:17:14.205433440  [2023-06-22 21:17:14] frigate.video                  INFO    : Took 0.020562701000001127s of CPU time to get a frame
2023-06-22 21:17:14.235199679  [2023-06-22 21:17:14] frigate.video                  INFO    : Took 0.02141737899999896s of CPU time to get a frame
2023-06-22 21:17:14.235204243  [2023-06-22 21:17:14] frigate.video                  INFO    : took 0.0018564100000002526s to process a frame in sewing
2023-06-22 21:17:14.303876940  [2023-06-22 21:17:14] frigate.video                  INFO    : Took 0.02001092400000104s of CPU time to get a frame
2023-06-22 21:17:14.303881574  [2023-06-22 21:17:14] frigate.video                  INFO    : took 0.000848002999999764s to process a frame in sewing
2023-06-22 21:17:14.607078947  [2023-06-22 21:17:14] frigate.video                  INFO    : Took 0.02120484900000008s of CPU time to get a frame
2023-06-22 21:17:14.618862774  [2023-06-22 21:17:14] frigate.video                  INFO    : Took 0.013763854000000464s of CPU time to get a frame
2023-06-22 21:17:14.691951356  [2023-06-22 21:17:14] frigate.video                  INFO    : Took 0.02068784199999918s of CPU time to get a frame
2023-06-22 21:17:14.914103815  [2023-06-22 21:17:14] frigate.video                  INFO    : Took 0.021342326999999273s of CPU time to get a frame
2023-06-22 21:17:14.914109197  [2023-06-22 21:17:14] frigate.video                  INFO    : took 0.009063441000000338s to process a frame in sewing
2023-06-22 21:17:15.014876565  [2023-06-22 21:17:15] frigate.video                  INFO    : Took 0.02171150400000066s of CPU time to get a frame
2023-06-22 21:17:15.288921249  [2023-06-22 21:17:15] frigate.video                  INFO    : Took 0.01979613399999991s of CPU time to get a frame
2023-06-22 21:17:15.967072353  [2023-06-22 21:17:15] frigate.video                  INFO    : Took 0.020484568000000536s of CPU time to get a frame
2023-06-22 21:17:15.994648770  [2023-06-22 21:17:15] frigate.video                  INFO    : Took 0.02172303000000042s of CPU time to get a frame
2023-06-22 21:17:16.021275346  [2023-06-22 21:17:16] frigate.video                  INFO    : Took 0.02030572999999869s of CPU time to get a frame
2023-06-22 21:17:16.044816262  [2023-06-22 21:17:16] frigate.video                  INFO    : Took 0.019188919000001192s of CPU time to get a frame
2023-06-22 21:17:16.044821882  [2023-06-22 21:17:16] frigate.video                  INFO    : took 0.0008806919999990726s to process a frame in sewing
2023-06-22 21:17:16.317681348  [2023-06-22 21:17:16] frigate.video                  INFO    : Took 0.02036589099999908s of CPU time to get a frame
2023-06-22 21:17:16.317686405  [2023-06-22 21:17:16] frigate.video                  INFO    : took 0.0012094490000009728s to process a frame in sewing
2023-06-22 21:17:16.394565425  [2023-06-22 21:17:16] frigate.video                  INFO    : Took 0.021068671000000094s of CPU time to get a frame
2023-06-22 21:17:17.207380815  [2023-06-22 21:17:17] frigate.video                  INFO    : Took 0.019561240999999896s of CPU time to get a frame
2023-06-22 21:17:17.207385504  [2023-06-22 21:17:17] frigate.video                  INFO    : took 0.0010698829999995496s to process a frame in sewing
2023-06-22 21:17:17.269188179  [2023-06-22 21:17:17] frigate.video                  INFO    : Took 0.020485773000000762s of CPU time to get a frame
2023-06-22 21:17:17.273647107  [2023-06-22 21:17:17] frigate.video                  INFO    : took 0.0030357429999998686s to process a frame in sewing
2023-06-22 21:17:17.357049569  [2023-06-22 21:17:17] frigate.video                  INFO    : Took 0.02314155200000023s of CPU time to get a frame
2023-06-22 21:17:17.357055224  [2023-06-22 21:17:17] frigate.video                  INFO    : took 0.0010993670000001288s to process a frame in sewing
2023-06-22 21:17:17.435273705  [2023-06-22 21:17:17] frigate.video                  INFO    : Took 0.021785310999998586s of CPU time to get a frame
2023-06-22 21:17:17.435278426  [2023-06-22 21:17:17] frigate.video                  INFO    : took 0.0011619440000014691s to process a frame in sewing
2023-06-22 21:17:17.673971506  [2023-06-22 21:17:17] frigate.video                  INFO    : Took 0.02129070199999994s of CPU time to get a frame
2023-06-22 21:17:17.673975820  [2023-06-22 21:17:17] frigate.video                  INFO    : took 0.0014515790000011464s to process a frame in sewing
2023-06-22 21:17:17.977799096  [2023-06-22 21:17:17] frigate.video                  INFO    : Took 0.02298677600000154s of CPU time to get a frame
2023-06-22 21:17:17.977803595  [2023-06-22 21:17:17] frigate.video                  INFO    : took 0.001015596999998536s to process a frame in sewing
2023-06-22 21:17:18.039617993  [2023-06-22 21:17:18] frigate.video                  INFO    : Took 0.020503020999999677s of CPU time to get a frame
2023-06-22 21:17:18.045219203  [2023-06-22 21:17:18] frigate.video                  INFO    : took 0.001515691999999902s to process a frame in sewing
2023-06-22 21:17:18.343718739  [2023-06-22 21:17:18] frigate.video                  INFO    : Took 0.020028496000000118s of CPU time to get a frame
2023-06-22 21:17:18.343723670  [2023-06-22 21:17:18] frigate.video                  INFO    : took 0.0010590669999999136s to process a frame in sewing
2023-06-22 21:17:18.395291956  [2023-06-22 21:17:18] frigate.video                  INFO    : Took 0.01956371300000015s of CPU time to get a frame
2023-06-22 21:17:18.395296107  [2023-06-22 21:17:18] frigate.video                  INFO    : took 0.0012310719999995001s to process a frame in sewing
2023-06-22 21:17:18.604761740  [2023-06-22 21:17:18] frigate.video                  INFO    : Took 0.021760750999998635s of CPU time to get a frame
2023-06-22 21:17:18.604766762  [2023-06-22 21:17:18] frigate.video                  INFO    : took 0.0006970260000009887s to process a frame in sewing
2023-06-22 21:17:18.633702766  [2023-06-22 21:17:18] frigate.video                  INFO    : Took 0.02072146100000083s of CPU time to get a frame
2023-06-22 21:17:18.633708622  [2023-06-22 21:17:18] frigate.video                  INFO    : took 0.0007411279999995912s to process a frame in sewing
2023-06-22 21:17:18.845759402  [2023-06-22 21:17:18] frigate.video                  INFO    : Took 0.021105927999998997s of CPU time to get a frame
2023-06-22 21:17:18.845764645  [2023-06-22 21:17:18] frigate.video                  INFO    : took 0.0014717700000002054s to process a frame in sewing
2023-06-22 21:17:19.022716400  [2023-06-22 21:17:19] frigate.video                  INFO    : Took 0.023740829000001185s of CPU time to get a frame
2023-06-22 21:17:19.140279338  [2023-06-22 21:17:19] frigate.video                  INFO    : Took 0.022246037000000385s of CPU time to get a frame
2023-06-22 21:17:19.140283928  [2023-06-22 21:17:19] frigate.video                  INFO    : took 0.0008132539999987642s to process a frame in sewing
2023-06-22 21:17:19.229950962  [2023-06-22 21:17:19] frigate.video                  INFO    : Took 0.023757762999998988s of CPU time to get a frame
2023-06-22 21:17:19.361238986  [2023-06-22 21:17:19] frigate.video                  INFO    : Took 0.02435025599999996s of CPU time to get a frame
2023-06-22 21:17:19.502358876  [2023-06-22 21:17:19] frigate.video                  INFO    : Took 0.021803541000000592s of CPU time to get a frame

In other words, the line frame_time = frame_queue.get(True, 1) is taking, on average, about 0.02s on dev, but 0.0007s of actual CPU time on 0.12.1. That's a massive difference. I don't know enough (read: hardly anything) about Python multi-process primitives, so cannot say why it would be requiring so much more CPU time to transfer a frame timestamp through this queue on dev.

[NickM-27]

Very interesting, I'll look at this and I'm sure Blake will know more than me. One thing I wanted to point out though is that we don't really know that this increase in time has to do with the transferring of a frame id through the queue. The frame_queue.get(True, 1) means there is a timeout of 1 second, so the delay could also be a slowdown in the frame ids being put into the queue instead.

[ccutrer]

The frame_queue.get(True, 1) means there is a timeout of 1 second, so the delay could also be a slowdown in the frames being put into the queue instead.

No, that's why I used time.process_time, not time.perf_counter. It counts actual CPU time, not wall-clock time. So idle time spent waiting for a frame doesn't count, only actual processing time. I used perf_counter at first, and the wall-clock time was similar between 0.12 and 0.13 - about 0.2s. My frame rate is 6fps, which is slight more than 1s/0.2s. It's almost like for some reason 0.13 the frame_queue.get call is never entering an actual wait state, but instead is always spinning.

[NickM-27]

I used that code and I did track it down to d81dd60fefda1c05a968047b8296e62df65df9dc that increases my get frame time from 0.001 to 0.005

[ccutrer]

I used that code and I did track it down to d81dd60 that increases my get frame time from 0.001 to 0.005

:( but WHY??. That commit doesn't touch anything with the frame queue. It's almost like we're up against some sort boundary condition, and where slightly slowing down or speeding up the processing time changes how often it checks the queue, and that's make Queue.get decide to spin instead of enter an idle block.

[NickM-27]

Thankfully the original motion detector was left in the code, I simply changed the motion detector in video.py to use FrigateMotionDetector instead of ImprovedMotionDetector and on that commit it is back to 0.001. It seems there is something in that new motion detector, but I agree it is odd that the issue manifests itself there

[blakeblackshear]

@NickM-27 try turning off motion detection with the old motion detector and see what happens.

I'm not totally convinced that we are not just seeing increased wait times for the next frame because the newer motion detection actually does less work. It should be waiting longer for the next frame because it finishes the previous frame earlier.

[NickM-27]

@blakeblackshear Yes indeed, turning off motion detection and it immediately falls back down

[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.005485291000000059s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.004979210000000012s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.005536291999999943s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.005919828999999988s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0072390829999999795s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0006049149999999837s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.006045665000000033s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.00805941599999993s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.006102372999999939s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.006026333000000106s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.005916017999999967s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0007802920000000158s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.005774002000000111s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.006351564000000032s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.007430503000000033s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0004538760000000197s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0007422499999999443s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.005950585000000008s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0010318340000000648s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0016696659999999586s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.000742751000000097s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0006653739999999964s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0008575009999999272s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0008053350000000181s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0012079590000000362s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0009174979999999611s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0006456679999999881s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.005710542999999957s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0004921660000000161s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0009218770000000154s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0008889579999999953s of CPU time to get a frame
[2023-06-22 16:22:42] frigate.video                  INFO    : Took 0.0006090829999999547s of CPU time to get a frame

[ccutrer]

I do have a minor bug in https://github.com/blakeblackshear/frigate/commit/64cab909a0ffc8e53abe0a13253c39fbdc0b9d1c - start_pop needs to be reset at the end of the loop. but fixing that doesn't measurably change my results.

[ccutrer]

Okay, here's another counter-intuitive result. I have #6889 applied locally, so that I know when frames get skipped. On dev, when I use FrigateMotionDetector, my overall idle CPU is higher (~10%), but my skipped_fps are quite high - 4+ on many cameras. But using ImprovedMotionDetector, my overall idle CPU is 0.0% (completely pegged), but my skipped_fps is 0 for almost all cameras. So yes, I would agree that ImprovedMotionDetector itself is less CPU intensive given the same parameters. But somehow it takes my frame_queue.get times from 7e-5s (yes, it's even faster when I corrected my start_pop bug) with FrigateMotionDetector to the ~0.2s with ImprovedMotionDetector. I looked through multi_processing.Queue's source, and on through to multi_processing.Connection.wait, and I don't see where it might decide to spin instead of block (https://github.com/python/cpython/blob/1a79785e3e8fea80bcf6a800b45a04e06c787480/Lib/multiprocessing/connection.py#L922). My system CPU % is also ~10% higher (accounting for almost all of the previously idle % with FrigateMotionDetector) when using ImprovedMotionDetector.

[ccutrer]

I wonder if CPU affinity has something to do with it? I.e. when the capture process and the process process are running on the same CPU, the semaphore locking is very fast. But if they're not, there's a larger penalty for multiple cores accessing the same memory. And somehow something about ImprovedMotionDetector causes the process process to hop CPU cores more often. Using pidstat -u -p <pid>, I can see that the capture process does not hop CPU cores at all. The process process does, for both motion detectors, but seems like it does more often for the latter, but that's anecdotal at best. I don't know how to actually measure it. If this is the case, setting the cpu affinity for both capture and process processes to be the same per-camera might fix it.

[ccutrer]

Just a quick update... setting CPU affinity for at least a small handful of cameras did not help. I did notice that there are a lot of individual mp.Value variables that are accessed on every iteration of both capture_frames and process_frames. These all essentially require a lock to read and write, and may contribute to system CPU. I've been working on reducing those, but so far it seems to have only marginally improved my user CPU time, and system CPU time still sucks up the rest. I'm still hopeful that I'll find some... something... that will improve the IPC, and immediately drop my system cpu time significantly.

[blakeblackshear]

I spent some time trying to back out specific parts of the new motion detector to see what was driving the change in CPU usage.

I found that if I removed both the gaussian blur and contrast improvement calls, the CPU usage dropped back to very low levels.

What is interesting is that adding either of them increases CPU usage to the elevated levels. Adding a second one doesn't seem to increase levels further. It doesn't matter which one is enabled, just that at least one is.

I'm wondering if there is some underlying overhead associated with the opencv interface to python that initializes once for a given scope. I did try updating to the latest opencv and it reduced the CPU usage by half, but it's still elevated with either of those function calls enabled.

Just some weird python shit. One day I want to port this part of the code base to rust or something.

Another idea would be to look at doing the same operations in numpy and optimizing with numba, but there is a bunch of other opencv stuff in there.

[kirsch33]

so i had some time today to just watch the debug view, and things seem to be acting very..sporadic. see video below:

https://github.com/blakeblackshear/frigate/assets/37373320/290c471e-fb70-4f72-9df1-5ed802065119

this cam is outputting 1080p @ 10fps (detect fps set to same resolution but 5 fps). i have a motion mask over the closer tree branch that is moving alot.

motion settings:

motion:
  threshold: 30
  contour_area: 50
  frame_height: 100

is this just motion settings that need more tuning or something with the motion detector PRs being discussed here?

[NickM-27]

Seems like https://github.com/numpy/numpy/issues/6237 // https://stackoverflow.com/questions/40445983/why-does-just-importing-opencv-cause-massive-cpu-usage may be relevant

[ccutrer]

I'm not convinced it's the numpy/openblas thing. I've straced the motion detection processes, and it's very clearly spending all of its time in poll (or pselect6, when I temporarily forced multiprocessing.Connection to use select instead of poll; made no difference) not in gettimeofday. Not that I've also looked through the kernel's implementation of poll, and while there are provisions to use a busy-loop instead of waiting to be woken up, they shouldn't apply to the pipe vfs which is they type of fd we're polling for. One possibility is I noticed ~5% of my overall CPU time is spent in software IRQs, so I'm wondering if it's because I have so many different poll requests running, and when any one has a frame ready, it wakes up all of them in the kernel, which have to then loop and realize their particular FD still isn't ready. But I'm not sure on that - I haven't traced through how the wakeups actually happen in the kernel. And it's still weird how tweaking the motion detector -- all user code -- somehow seems to affect this quite drastically.

[ccutrer]

Okay, I think I give up, and am just resigned to the fact that I have too many cameras on one processor. I did a WIP (e32328863deba2a728e0057ade2f9fd4e3d159bc), combining the capture and motion detection processes into one, completely eliminating the queue between them. My CPU usage is about the same. I still don't understand why so much time is taken up in system CPU.

[blakeblackshear]

I haven't given up yet. Just been busy the past week.

[ccutrer]

I'll keep watching, but given what I know now, I don't think I was even keeping up under 0.12, I just was unaware I was falling behind. I'll hold off on ordering new hardware for a few days though, in case you pull out something I can't find :).

[nicoleise]

I'm willing to help with troubleshooting, but will need above average guidance due to infamiliarity with the tools used. So may not be worth much, up to you, but I suggest it because I see this issue reliably and use higher resolution cameras - it suggests to me that the issue may not be with the 30 some cameras being too much, but simply the "combined detect area".

I think the issue may be CPU related, but am also seeing what appears to be a memory leak.

Installation Method: Docker Compose Target: Debian 11 OS guest VM running on Proxmox 7.4 Assigned Ressources: 16 cores out of 2 Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz, 64 GB RAM out of 256 GB. 6x1TB HDDs in RAID10 ZFS for the recordings and 32G out of 1.6 TB SSD in RAID1 (mirror of two 1.6 TB SSDs). Currently only on 1x1G connection, planning for 2x10G, but there is nothing on the server that can saturate a 1G link in the slightest - host netin is under 2M. In other words, a fairly high "allowance" of ressources and more important; plenty of headroom on the host system.

Reported Metrics: Host/Guest; CPU usage total 9%/37%, RAM 85%/98%, IO delay on host 0-0,6 %. See below on Frigates reported metrics.

Cameras: 5 total, all Dahua. 3x 4K exterior cameras, 2x fullHD indoor cameras all streaming and detecting at native resolution.

Noteworthy: Hardware acceleration not enabled (yet). 2x USB corals installed, but only one configured (haven't been able to make both work reliably, haven't looked into it yet). Otherwise "normal" and simple config; properly applied motion masks for sky, timestamps, etc. and a few detection zones configured for each camera. Using experimental UI true and MQTT, cams configured with RTSP paths and roles detect and record. Birdseye in objects mode.

The setup was rock-steady on version 0.11, which was my initial installation. Never a wrong move, despite not having a Coral connected then, so using CPU detectors on all cameras.

Updating to 0.12.0-DA3E197 (current) as a fresh install, I noticed CPU usage being reported very high (fx 170 % per camera and 300+ % on the CPU detectors. Applying the Coral obviously removed the detector CPU usage, but ffmpeg continues to be around 170 % for the 4Ks and 43 %.

I thought this was a small UI issue that would just get fixed, but after running for some time, I see the behavior described here. No frames received, check logs on random cameras that work if I access them on eg. their built-in webserver or via app or VNC. Nothing in the log except "connection refused" (cameras have rate limits to them, I think frigate may cause them to drop the streams and reinitiate). The same cameras that report no frames in Cameras view will work in Birdseye, but the stream lags by many seconds (maybe 30s) or so, whereas they are normally near-instant, maybe a 1-2 s lag.

Restarting Frigate "fixes" things for a while. But typically not for long - maybe a day or two?

Observations:

1. I see similarly that the ffmpeg FPS are as specified while the capture and detect are very low (0.2). One camera hardly sees motion, this shows 5/5/0 (0 skipped).

2. No CPU usage reported for the Coral detector. When frigate is restarted, usage is reported.

As mentioned, there seems to be a CPU component to this issue but also a memory leak, in that the guestOS reports 98% memory usage of 64 GB. This also helps explain why a reset fixes things for some time.

As mentioned, I am writing this mainly because I have assigned an absurd amount of ressources to Frigate (planning to optimise later, but have plans to add many cameras, etc.) and currently don't really have many cameras, but do have some of higher resolution.

If I can help, please tag me and let me know specifically what you'd like. It's probably easiest if you simply ask for a method of output (fx pasting text, screenshots, etc.) and then write actions or commands, like "paste your config" or "please paste output of #>cat some command, cat some other command, etc. to avoid any confusion.

[NickM-27]

@nicoleise make your own issue, this issue has to do with 0.13 dev branch changes not 0.12.

also in general your setup sounds highly inefficient given that:

1. it is highly discouraged to run detect on the main stream, there are very few cases where there is any benefit and it vastly increases CPU usage
2. If you are not going to listen to 1 then hardware acceleration is very important

[NickM-27]

Closing this as the conversation in https://github.com/blakeblackshear/frigate/pull/6940 indicates the issue has been resolved.

[nicoleise]

@nicoleise make your own issue, this issue has to do with 0.13 dev branch changes not 0.12.

also in general your setup sounds highly inefficient given that:

1. it is highly discouraged to run detect on the main stream, there are very few cases where there is any benefit and it vastly increases CPU usage
2. If you are not going to listen to 1 then hardware acceleration is very important

Thanks, Nick. I understood that this is about the latest version, but also that the issue seemed to be introduced with 0.12 originally. Maybe my mistake. But wasn't "fishing for" support in someone elses thread, just meant to help - also the reason I didn't open a separate issue (would essentially just be spam when the problem is known and worked on).

Great that it seems resolved though.

I'll look into the docs for detecting on secondary streams, thanks. Only had time (since 0.12) to install it and have it "working", not to set it up properly yet. Sadly. Thanks for your advice.

[NickM-27]

Thanks, Nick. I understood that this is about the latest version, but also that the issue seemed to be introduced with 0.12 originally. Maybe my mistake.

Motion detection has remained unchanged for many versions until this current dev version (0.13) so it was not introduced in 0.12.

But wasn't "fishing for" support in someone elses thread, just meant to help - also the reason I didn't open a separate issue (would essentially just be spam when the problem is known and worked on).

No problem, generally if an issue is being reported on a version newer than you are using it is better to create a new issue.

[kirsch33]

@NickM-27 not sure if this should be re-opened or not but I pulled down the latest dev build dev-baf671b and seems a new set of issues are at hand. here is CPU usage before and after (before was running dev-c3b313a) not sure if related to https://github.com/blakeblackshear/frigate/pull/6986 or https://github.com/blakeblackshear/frigate/pull/6890 or maybe https://github.com/blakeblackshear/frigate/pull/7022

[NickM-27]

@kirsch33 unless you have reason to believe it is due to motion detection then it would definitely not be reopened.

also, my CPU usage has gone even lower after the recent PRs that have merged