search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18k stars 1.64k forks source link

[Support]: Unable to keep up with recording segments in cache (Proxmox LXC) #8323

Closed Barley194 closed 9 months ago

Barley194 commented 10 months ago

Describe the problem you are having

After moving frigate installation from proxmox vm (ubuntu server + docker) to proxmox lxc (privileged, as I couldn't make the coral usb pass-through work on unprivileged one) + docker in order to reduce the coral intereference speed from ~60ms to ~15ms

or

after started using go2rtc restream for recording and detection

(I'm not sure which of those 2 actions have started it)

I get the:

Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...

spam all the time.

I run this setup for a few weeks now and this spam is all day and night long everyday.

It seams like everything what should, is recorded. (I don't watch every single recording but the ones I watched didn't looked cropped at the end for e.q. - despite "post_capture: 60" the recordings ends in less than 10s after the motion is no longer visible, but maybe that's how it works?) Sometimes something is not detected or detected but not recorded (not sure) but I don't know if this is related.

My storage for media is a network samba drive which is located on the same proxmox device but inside a VM. At first I thought that's the issue so I pointed the save location of recordings to a local drive on the LXC, but the logs kept spamming. So this is not it.

I have only 1 camera for now, it's Reolink 410W and is connected via WiFi. My host for frigate is connected via ethernet. Camera sometimes loses wifi signal, but this is shown as ffmpeg error in frigate logs - and that's OK.

I feel like I tried everything, searched the web, red through bunch of past issues here on github and nothing has helped me.

PS: As I don't see a big issue caused by this log I could I just ignore that. But I'm afraid that there is something wrong and thus I'd love to fix it + my logs are cluttered now.

Frigate LXC resources: obraz

shm and cache: obraz

I'm willing to provide any necessary information which I did not include here.

Version

0.12.1-367d724

Frigate config file

mqtt:
  host: [host_id]
  topic_prefix: [frigate_prefix]
  client_id: [frigate_clientid]
  user: [user_name]
  password: [password]
database:
  path: /db/frigate.db                                                                                                                                                                                                          detectors:
 coral:
    type: edgetpu
    device: usb
logger:
  logs:
    frigate.record: debug                                                                                                                                                                                                       go2rtc:
  streams:
    reolink410WFirstFrigate:
      - "ffmpeg:http://[ip]/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=[user_name]&password=[password]#video=copy#audio=copy#audio=opus"                                                                                                                                                                                          reolink410WFirstFrigateLowRes:
      - "ffmpeg:http://[ip]/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=[user_name]&password=[password]#video=copy#audio=copy#audio=opus"                                                                                                                                                                                         webrtc:
    candidates:
      - [ip]:8555
      - stun:8554                                                                                                                                                                                                               ffmpeg:
  output_args:
    record: preset-record-generic-audio-aac                                                                                                                                                                                     cameras:
  reolink410WFirstFrigate:
    ffmpeg:
      inputs:
        - path: "rtsp://127.0.0.1:8554/reolink410WFirstFrigate?video=copy#audio=aac"
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: "rtsp://127.0.0.1:8554/reolink410WFirstFrigateLowRes?video=copy#audio=aac"
          input_args: preset-rtsp-restream
          roles:
            - detect

    detect:
      width: 640
      height: 360
      fps: 5
      max_disappeared: 25
      stationary:
        interval: 0
        threshold: 20
    motion:
      mask:
        - 526,33,593,35,591,11,511,13   #timestamp mask
        - 640,226,498,360,640,360 
        - 517,93,558,95,547,17,514,63 
        - 483,34,458,94,440,95,451,37 
    zones:
      zone1:
        coordinates: 438,360,118,360,126,129,342,166
      zone2:
        coordinates: 481,360,542,360,602,279,640,239,640,198,396,217
      zone3:
        coordinates: 354,72,398,190,640,174,640,0,130,0,124,46
        objects:
          - person
    objects:
      filters:
        person:
          max_area: 15000
          min_area: 2500
        motorcycle:
          max_area: 30000

objects:
 track:
  - person
  - car
  - motorcycle

record:
  enabled: True
  retain:
    days: 0
    mode: all
  events:
    pre_capture: 60
    post_capture: 60
    retain:
      default: 6
      mode: motion
    required_zones:
      - zone1
      - zone2
      - zone3

snapshots:
  enabled: True
  required_zones:
    - zone1
    - zone2
    - zone3

Relevant log output

2023-10-25 19:46:55.252252605  [2023-10-25 19:46:55] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-10-25 19:46:55.368179533  [2023-10-25 19:46:55] peewee_migrate                 INFO    : Starting migrations
2023-10-25 19:46:55.384509416  [2023-10-25 19:46:55] peewee_migrate                 INFO    : There is nothing to migrate
2023-10-25 19:46:55.453975554  [2023-10-25 19:46:55] frigate.app                    INFO    : Output process started: 293
2023-10-25 19:46:55.469127417  [2023-10-25 19:46:55] frigate.app                    INFO    : Camera processor started for reolink410WFirstFrigate: 298
2023-10-25 19:46:55.485958282  [2023-10-25 19:46:55] frigate.app                    INFO    : Capture process started for reolink410WFirstFrigate: 300
2023-10-25 19:46:58.130472582  [2023-10-25 19:46:55] detector.coral                 INFO    : Starting detection process: 291
2023-10-25 19:46:58.131096799  [2023-10-25 19:46:55] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-10-25 19:46:58.142057651  [2023-10-25 19:46:58] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-10-25 19:47:55.788833291  [2023-10-25 19:47:55] frigate.record                 DEBUG   : Start expire recordings (new).
2023-10-25 19:47:55.788844818  [2023-10-25 19:47:55] frigate.record                 DEBUG   : Start deleted cameras.
2023-10-25 19:47:55.796777656  [2023-10-25 19:47:55] frigate.record                 DEBUG   : Expiring 0 recordings
2023-10-25 19:47:55.804120421  [2023-10-25 19:47:55] frigate.record                 DEBUG   : End deleted cameras.
2023-10-25 19:47:55.804132355  [2023-10-25 19:47:55] frigate.record                 DEBUG   : Start all cameras.
2023-10-25 19:47:55.804137001  [2023-10-25 19:47:55] frigate.record                 DEBUG   : Start camera: reolink410WFirstFrigate.
2023-10-25 19:47:56.043758878  [2023-10-25 19:47:56] frigate.record                 DEBUG   : Expiring 0 recordings
2023-10-25 19:47:56.043770624  [2023-10-25 19:47:56] frigate.record                 DEBUG   : End camera: reolink410WFirstFrigate.
2023-10-25 19:47:56.043774858  [2023-10-25 19:47:56] frigate.record                 DEBUG   : End all cameras.
2023-10-25 19:47:56.043778975  [2023-10-25 19:47:56] frigate.record                 DEBUG   : End expire recordings (new).
2023-10-25 19:47:56.043828095  [2023-10-25 19:47:56] frigate.record                 DEBUG   : Start expire files (legacy).
2023-10-25 19:47:56.072492577  [2023-10-25 19:47:56] frigate.record                 DEBUG   : Oldest recording in the db: 1697796205.855539
2023-10-25 19:47:56.577552080  [2023-10-25 19:47:56] frigate.record                 DEBUG   : End expire files (legacy).
2023-10-25 19:48:00.807755952  [2023-10-25 19:48:00] frigate.record                 WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-25 19:48:10.807610671  [2023-10-25 19:48:10] frigate.record                 WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-25 19:48:20.811382970  [2023-10-25 19:48:20] frigate.record                 WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-25 19:48:30.812349556  [2023-10-25 19:48:30] frigate.record                 WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-25 19:48:40.812922250  [2023-10-25 19:48:40] frigate.record                 WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-25 19:48:50.817509459  [2023-10-25 19:48:50] frigate.record                 WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-25 19:49:00.810054797  [2023-10-25 19:49:00] frigate.record                 WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-25 19:49:10.810703197  [2023-10-25 19:49:10] frigate.record                 WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"0/0","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1440,"width":2560}]}},{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"0/0","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":512,"width":896}]}}]

Frigate stats

{"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"0.0"},"1":{"cpu":"0.0","mem":"0.0"},"109":{"cpu":"0.0","mem":"0.1"},"110":{"cpu":"3.3","mem":"3.4"},"119":{"cpu":"0.0","mem":"0.3"},"1245":{"cpu":"0.0","mem":"0.0"},"1256":{"cpu":"0.0","mem":"0.1"},"1260":{"cpu":"0.0","mem":"0.1"},"1264":{"cpu":"0.7","mem":"0.1"},"128":{"cpu":"0.0","mem":"0.1"},"130":{"cpu":"0.0","mem":"0.1"},"131":{"cpu":"0.0","mem":"0.1"},"132":{"cpu":"0.0","mem":"0.1"},"15":{"cpu":"0.0","mem":"0.0"},"17":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"279":{"cpu":"4.3","mem":"0.6"},"28":{"cpu":"0.0","mem":"0.0"},"280":{"cpu":"3.7","mem":"0.6"},"284":{"cpu":"0.0","mem":"1.7"},"29":{"cpu":"0.0","mem":"0.0"},"290":{"cpu":"0.0","mem":"0.2"},"291":{"cpu":"1.7","mem":"2.3"},"293":{"cpu":"1.0","mem":"1.9"},"298":{"cpu":"2.7","mem":"2.0"},"30":{"cpu":"0.0","mem":"0.0"},"300":{"cpu":"1.7","mem":"1.8"},"304":{"cpu":"0.0","mem":"0.4"},"305":{"cpu":"12.3","mem":"0.6"},"307":{"cpu":"0.0","mem":"0.4"},"309":{"cpu":"0.7","mem":"0.5"},"31":{"cpu":"0.0","mem":"0.0"},"32":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"90":{"cpu":"3.0","mem":"0.5"},"MiB":{"cpu":"524.8","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"detection_fps":2.4,"detectors":{"coral":{"detection_start":0.0,"inference_speed":13.58,"pid":291}},"reolink410WFirstFrigate":{"camera_fps":5.1,"capture_pid":300,"detection_enabled":1,"detection_fps":2.4,"ffmpeg_pid":305,"pid":298,"process_fps":5.1,"skipped_fps":0.0},"service":{"last_updated":1698263889,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":133.2,"mount_type":"tmpfs","total":134.2,"used":1.1},"/media/frigate/clips":{"free":31810.9,"mount_type":"cifs","total":36671.4,"used":4860.4},"/media/frigate/recordings":{"free":31810.9,"mount_type":"cifs","total":36671.4,"used":4860.4},"/tmp/cache":{"free":989.9,"mount_type":"tmpfs","total":1000.0,"used":10.1}},"temperatures":{},"uptime":674,"version":"0.12.1-367d724"}}

Operating system

Proxmox

Install method

Docker Compose

Coral version

USB

Network connection

Mixed

Camera make and model

Reolink 410W

Any other information that may be helpful

No response

NickM-27 commented 10 months ago

There have been a lot of optimizations to recordings in 0.13 (currently in beta) so in general this should be resolved but may be indicative of slow segment moving times. I'd suggest reading https://deploy-preview-6262--frigate-docs.netlify.app/troubleshooting/recordings

Barley194 commented 10 months ago

Moving times are fast:

2023-10-26 04:57:11.733593418 [2023-10-26 04:57:11] frigate.record WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-10-26 04:57:12.249325063 [2023-10-26 04:57:12] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/56.20.mp4 in 0.44483208656311035 seconds. 2023-10-26 04:57:12.565578536 [2023-10-26 04:57:12] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/56.29.mp4 in 0.2894909381866455 seconds. 2023-10-26 04:57:12.781152934 [2023-10-26 04:57:12] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/56.39.mp4 in 0.2014169692993164 seconds. 2023-10-26 04:57:12.996955421 [2023-10-26 04:57:12] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/56.49.mp4 in 0.19872617721557617 seconds. 2023-10-26 04:57:13.425259357 [2023-10-26 04:57:13] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/56.58.mp4 in 0.2891509532928467 seconds. 2023-10-26 04:57:22.070929600 [2023-10-26 04:57:22] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/57.10.mp4 in 0.19686198234558105 seconds. 2023-10-26 04:57:32.137783268 [2023-10-26 04:57:32] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/57.19.mp4 in 0.2512190341949463 seconds. 2023-10-26 04:57:42.077874152 [2023-10-26 04:57:42] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/57.29.mp4 in 0.19926190376281738 seconds. 2023-10-26 04:57:52.093959697 [2023-10-26 04:57:52] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/57.38.mp4 in 0.18979191780090332 seconds. 2023-10-26 04:58:02.123380471 [2023-10-26 04:58:02] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/57.48.mp4 in 0.21117496490478516 seconds. 2023-10-26 04:58:12.174261173 [2023-10-26 04:58:12] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/58.00.mp4 in 0.27132701873779297 seconds. 2023-10-26 04:58:22.068202075 [2023-10-26 04:58:22] frigate.record DEBUG : Copied /media/frigate/recordings/2023-10-26/04/reolink410WFirstFrigate/58.09.mp4 in 0.17260289192199707 seconds. 2023-10-26 04:59:21.733324824 [2023-10-26 04:59:21] frigate.record WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-10-26 04:59:31.747901673 [2023-10-26 04:59:31] frigate.record WARNING : Unable to keep up with recording segments in cache for reolink410WFirstFrigate. Keeping the 5 most recent segments out of 6 and discarding the rest...

And CPU usage is not high.

NickM-27 commented 10 months ago

Gotcha, then yeah this is something that'll require running 0.13 to fix

github-actions[bot] commented 9 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.