search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.22k stars 1.76k forks source link

[Support]: Unable to keep up with recording segments in cache for <camera> in 0.13 dev #6808

Closed ccutrer closed 1 year ago

ccutrer commented 1 year ago

Describe the problem you are having

The maintainer is unable to keep up with moving recording segments to disk with a lot of cameras, even with only motion recording enabled.

Version

0.13.0-FF1B5C8

Frigate config file

record:
  enabled: true
  retain:
    days: 90
    mode: motion
  events:
    pre_capture: 2
    post_capture: 2

snapshots:
  enabled: true
  retain:
    default: 90

logger:
  logs:
    frigate.record: debug

<33 cameras>

Relevant log output

https://github.com/blakeblackshear/frigate/issues/6636#issuecomment-1592017428

FFprobe output from your camera

N/A

Frigate stats

https://gist.github.com/ccutrer/3d7663df79eefaf044d5e15cca796da8

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Mostly Hikvision

Any other information that may be helpful

33 cameras at the moment (all mentioned in that log as having dropped segments). Most running at 4Mbps. I may have just not been paying attention, but I don't remember seeing it with the early 0.12+ dev versions, before the maintainer was introduced. Setting to motion recording had been sufficient to keep things happy. I'd blame it on "it's raining outside today, so there's a lot more motion". But what's really odd to me is that theater and theater_front are in that list of having dropped (a lot of) recording segments. I know for a fact that that room is inside, has no windows, no lights have come on, and the door has not been opened all day (or anytime in the past 7 days). So it shouldn't have thought it needed to keep any of those segments. There are a few segments saved in the log for those cameras, but I assume that's just the startup "frames just started, everything is considered motion".

This was originally discussed at https://github.com/blakeblackshear/frigate/issues/6636#issuecomment-1592017428, but now splitting to a separate issue specifically for 0.13 dev.

NickM-27 commented 1 year ago

I created https://github.com/blakeblackshear/frigate/pull/6809 which I believe would improve this issue, but there may be a better way to go about it.

inspired27 commented 1 year ago

I created #6809 which I believe would improve this issue, but there may be a better way to go about it.

Thanks @NickM-27, has #6809 been approved as yet would like to try it. I am running two instances across different hardware to try things out. Like @ccutrer I have tried 0.13.0 dev-ff1b5c8 and dev-3efa77f both same result hopefully resolved soon.

NickM-27 commented 1 year ago

6809 is merged so a new image should be done in a bit, please update with any findings after running that

ccutrer commented 1 year ago

I'm getting errors from the maintainer:

2023-06-16 16:58:50.842034770  [2023-06-16 16:58:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for doorbell. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:50.849966727  [2023-06-16 16:58:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for fire_pit. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:50.871933255  [2023-06-16 16:58:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gaming. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:50.958341021  [2023-06-16 16:58:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for mud. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:50.960634999  [2023-06-16 16:58:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for playset. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:50.978210243  [2023-06-16 16:58:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:50.996287831  [2023-06-16 16:58:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for theater_back. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:51.014922623  [2023-06-16 16:58:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for theater_front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:51.024964089  [2023-06-16 16:58:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for trampoline. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-06-16 16:58:51.037648547  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f018e219500] moov atom not found
2023-06-16 16:58:51.037979516  [ERROR:0@8117.310] global /io/opencv/modules/videoio/src/cap.cpp (164) open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-06-16 16:58:51.037982524  
2023-06-16 16:58:51.037985040  OpenCV(4.5.5) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-06-16 16:58:51.037986357  
2023-06-16 16:58:51.037987652  
2023-06-16 16:58:51.041554299  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f018e1f23c0] moov atom not found
2023-06-16 16:58:51.042062962  [ERROR:0@8117.314] global /io/opencv/modules/videoio/src/cap.cpp (164) open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-06-16 16:58:51.042065538  
2023-06-16 16:58:51.042067709  OpenCV(4.5.5) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-06-16 16:58:51.042068800  
2023-06-16 16:58:51.042069928  
2023-06-16 16:58:51.045043432  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f018e1f23c0] moov atom not found
2023-06-16 16:58:51.045331001  [ERROR:0@8117.318] global /io/opencv/modules/videoio/src/cap.cpp (164) open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-06-16 16:58:51.045333255  
2023-06-16 16:58:51.045335514  OpenCV(4.5.5) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-06-16 16:58:51.045336863  
2023-06-16 16:58:51.045338410  
2023-06-16 16:58:51.052150530  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f018e1f23c0] moov atom not found
2023-06-16 16:58:51.052431502  [ERROR:0@8117.325] global /io/opencv/modules/videoio/src/cap.cpp (164) open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-06-16 16:58:51.052434282  
2023-06-16 16:58:51.052436404  OpenCV(4.5.5) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-06-16 16:58:51.052438138  
2023-06-16 16:58:51.052439402  
2023-06-16 16:58:51.055889101  [2023-06-16 16:58:51] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
2023-06-16 16:58:51.055892961  [2023-06-16 16:58:51] frigate.record.maintainer      ERROR   : 'NoneType' object is not subscriptable
NickM-27 commented 1 year ago

interesting... I will put up a PR to fix that

ccutrer commented 1 year ago

I'm still seeing a few discarded segments (usually about 5/7), but I'm just going to blame that on the super sensitive motion detector at the moment. Today so far (15:11 UTC) I have 136k recording segments in the database. My past average is 47k-60k per day (full 24 hours). So I would say that yes, significant progress has been made here, and it's appropriate to be closed, just wanted to report how much of an improvement it has been.

NickM-27 commented 1 year ago

Improvements to the motion detector have been merged this morning

ccutrer commented 1 year ago

Yup, and I'm already running them :)