search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.2k stars 1.66k forks source link

[Support]: Various "Unable to..." errors in log #8110

Closed sshaikh closed 10 months ago

sshaikh commented 11 months ago

Describe the problem you are having

I see the following lines in my log - they seem to have started at the same time so I am assuming they are related (see later section for log entries).

The first happens across cameras, the others I can't tell which camera they relate to.

I suspect it's a resource issue but can confirm storage is available (a stable 72% in use as retain is enabled). CPU sits at 30-40%, 5m load is 2-2.5.

I've noticed the issue since installing a reolink doorbell camera which had gaps in recording until I reduced its framerate/bitrate and seems to be working fine now. There are no errors in Go2rtc.

Version

0.12.0-da3e197

Frigate config file

mqtt:
  host: mqtt
  user: frigate
  password: "{FRIGATE_MQTT_PASSWORD}"

ffmpeg:
  hwaccel_args: preset-vaapi

record:
  enabled: True
  retain:
    days: 7

snapshots:
  enabled: True

detect:
  enabled: True

objects:
  track:
    - person
    - bicycle
    - car
    - motocycle
    - bird
    - cat
    - dog
    - backpack
    - umbrella
    - suitcase

detectors:
  coral:
    type: edgetpu
    device: pci

go2rtc:
  streams:
    doorbell:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@doorbell-camera:554/h264Preview_01_main
    doorbell-sub:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@doorbell-camera:554/h264Preview_01_sub
    front-door:
      - rtsp://useruser:{FRIGATE_RTSP_PASSWORD}@front-door-camera:554/stream1
    front-door-sub:
      - rtsp://useruser:{FRIGATE_RTSP_PASSWORD}@front-door-camera:554/stream2
    driveway:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@driveway-camera:554/Streaming/Channels/101
    driveway-sub:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@driveway-camera:554/Streaming/Channels/102
    garden:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@garden-camera:554/Streaming/Channels/101
    garden-sub:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@garden-camera:554/Streaming/Channels/102
    decking:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@decking-camera:554/Streaming/Channels/101
    decking-sub:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@decking-camera:554/Streaming/Channels/102
    extension-roof:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@extension-roof-camera:554/Streaming/Channels/101
    extension-roof-sub:
      - rtsp://user:{FRIGATE_RTSP_PASSWORD}@extension-roof-camera:554/Streaming/Channels/102
    living-room:
      - rtsp://useruser:{FRIGATE_RTSP_PASSWORD}@living-room-camera:554/stream1
    living-room-sub:
      - rtsp://useruser:{FRIGATE_RTSP_PASSWORD}@living-room-camera:554/stream2

cameras:
  doorbell:
    enabled: true
    ui:
      order: 1
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/doorbell
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/doorbell-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 640
      height: 480
      fps: 15
    motion:
      mask:
        - 0,30,200,30,200,0,0,0
    zones:
      driveway-zone:
        coordinates: 0,480,640,480,640,319,467,306,0,305
        filters:
          person:
            min_area: 3600
    record:
      events:
        required_zones:
          - driveway-zone
    snapshots:
      required_zones:
        - driveway-zone
  front-door:
    ui:
      order: 2
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front-door
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front-door-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 15
    motion:
      mask:
        - 0,40,230,40,230,0,0,0
    zones:
      driveway-zone:
          coordinates: 0,360,640,360,640,250,275,225,0,235
          filters:
            person:
              min_area: 3600
    record:
      events:
        required_zones:
          - driveway-zone
    snapshots:
      required_zones:
        - driveway-zone
driveway:
    ui:
      order: 3
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/driveway
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/driveway-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 704
      height: 576
      fps: 6
    motion:
      mask:
        - 400,60,400,30,25,30,25,60
    zones:
      driveway-zone:
        coordinates: 704,0,704,576,0,576,0,140,230,0
    record:
      events:
        required_zones:
          - driveway-zone
    snapshots:
      required_zones:
        - driveway-zone
  garden:
    ui:
      order: 4
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garden
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garden-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 704
      height: 480
      fps: 6
    motion:
      mask:
        - 400,60,400,30,25,30,25,60
decking:
    ui:
      order: 5
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/decking
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/decking-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 704
      height: 480
      fps: 6
    motion:
      mask:
        - 400,60,400,30,25,30,25,60
  extension-roof:
    ui:
      order: 6
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/extension-roof
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/extension-roof-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 704
      height: 576
      fps: 6
    motion:
      mask:
        - 400,60,400,30,25,30,25,60
  lower-hallway:
    ui:
      order: 7
    ffmpeg:
      inputs:
        - path: rtsp://user:{FRIGATE_RTSP_PASSWORD}@lower-hallway-camera:554/unicast
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
      fps: 25
    motion:
      mask:
        - 0,0,380,0,380,70,0,70
decking:
    ui:
      order: 5
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/decking
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/decking-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 704
      height: 480
      fps: 6
    motion:
      mask:
        - 400,60,400,30,25,30,25,60
  extension-roof:
    ui:
      order: 6
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/extension-roof
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/extension-roof-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 704
      height: 576
      fps: 6
    motion:
      mask:
        - 400,60,400,30,25,30,25,60
  lower-hallway:
    ui:
      order: 7
    ffmpeg:
      inputs:
        - path: rtsp://user:{FRIGATE_RTSP_PASSWORD}@lower-hallway-camera:554/unicast
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
      fps: 25
    motion:
      mask:
        - 0,0,380,0,380,70,0,70
 living-room:
    ui:
      order: 8
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/living-room
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/living-room-sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 15
    motion:
      mask:
        - 0,40,230,40,230,0,0,0
  detect-test:
    enabled: False
    ui:
      dashboard: False
      order: -1
    ffmpeg:
      inputs:
        - path: /media/frigate/test-clips/test.mp4
          input_args: -re -stream_loop -1 -fflags +genpts
          roles:
            - detect
    record:
      enabled: False
    snapshots:
      enabled: False
    detect:
      width: 1920
      height: 1080
      fps: 5
    motion:
      mask:
        - 0,120,440,120,440,60,0,60
    zones:
      driveway-zone:
          coordinates: 438,1080,583,981,979,1080,1920,1080,1920,655,791,572,0,648,0,1080

Relevant log output

2023-10-07 00:06:47.994950326  [2023-10-07 00:06:47] frigate.record                 WARNING : Unable to keep up with recording segments in cache for doorbell. Keeping the 5 most recent segments out of 9 and discarding the rest...
...
2023-10-07 11:38:12.488890920  [2023-10-07 11:38:11] frigate.object_processing      WARNING : Unable to create jpg because frame 1696675088.187207 is not in the cache
...
2023-10-07 14:03:34.318744391  [2023-10-07 14:03:34] frigate.object_processing      WARNING : Unable to save snapshot for 1696683798.995412-v30x65.
...
2023-10-07 14:03:34.448693299  [2023-10-07 14:03:34] frigate.object_processing      WARNING : Unable to send mqtt snapshot for 1696683798.995412-v30x65.

### FFprobe output from your camera

```shell
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
width=2560
height=1920
coded_width=2560
coded_height=1920
closed_captions=0
film_grain=0
has_b_frames=0
sample_aspect_ratio=N/A
display_aspect_ratio=N/A
pix_fmt=yuv420p
level=51
color_range=unknown
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left
field_order=progressive
refs=1
is_avc=false
nal_length_size=0
id=N/A
r_frame_rate=20/1
avg_frame_rate=15/1
time_base=1/90000
start_pts=3000
start_time=0.033333
duration_ts=N/A
duration=N/A
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
extradata_size=34
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
DISPOSITION:captions=0
DISPOSITION:descriptions=0
DISPOSITION:metadata=0
DISPOSITION:dependent=0
DISPOSITION:still_image=0
[/STREAM]

Frigate stats

{"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"2.3"},"1":{"cpu":"0.0","mem":"0.0"},"116":{"cpu":"0.0","mem":"0.0"},"134":{"cpu":"0.0","mem":"0.0"},"135":{"cpu":"0.0","mem":"0.1"},"136":{"cpu":"0.0","mem":"0.0"},"137":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"283":{"cpu":"0.0","mem":"0.7"},"289":{"cpu":"1.7","mem":"0.1"},"29":{"cpu":"0.0","mem":"0.0"},"290":{"cpu":"20.7","mem":"1.0"},"292":{"cpu":"5.7","mem":"0.8"},"295":{"cpu":"0.0","mem":"0.0"},"296":{"cpu":"37.3","mem":"2.7"},"297":{"cpu":"22.3","mem":"1.6"},"298":{"cpu":"0.7","mem":"1.0"},"299":{"cpu":"1.0","mem":"1.2"},"30":{"cpu":"0.0","mem":"0.0"},"300":{"cpu":"1.0","mem":"1.1"},"301":{"cpu":"0.7","mem":"0.9"},"302":{"cpu":"5.3","mem":"2.0"},"303":{"cpu":"1.7","mem":"1.0"},"304":{"cpu":"2.3","mem":"0.9"},"307":{"cpu":"2.3","mem":"0.8"},"31":{"cpu":"0.0","mem":"0.0"},"311":{"cpu":"1.0","mem":"0.8"},"312":{"cpu":"2.0","mem":"0.4"},"313":{"cpu":"1.0","mem":"0.8"},"316":{"cpu":"0.7","mem":"0.8"},"318":{"cpu":"1.3","mem":"0.6"},"32":{"cpu":"0.0","mem":"0.0"},"321":{"cpu":"1.0","mem":"0.8"},"323":{"cpu":"1.3","mem":"0.7"},"324":{"cpu":"15.7","mem":"1.0"},"325":{"cpu":"1.7","mem":"0.6"},"332":{"cpu":"0.0","mem":"0.0"},"335":{"cpu":"13.3","mem":"1.7"},"336":{"cpu":"1.7","mem":"0.8"},"339":{"cpu":"0.0","mem":"0.0"},"341":{"cpu":"0.7","mem":"0.1"},"344":{"cpu":"0.7","mem":"0.1"},"347":{"cpu":"0.7","mem":"0.2"},"349":{"cpu":"0.7","mem":"0.1"},"352":{"cpu":"1.7","mem":"0.6"},"354":{"cpu":"0.0","mem":"0.0"},"355":{"cpu":"2.0","mem":"0.5"},"356":{"cpu":"0.0","mem":"0.0"},"360":{"cpu":"0.0","mem":"0.0"},"361":{"cpu":"0.0","mem":"0.0"},"362":{"cpu":"0.0","mem":"0.0"},"364":{"cpu":"0.0","mem":"0.0"},"365":{"cpu":"0.3","mem":"0.1"},"366":{"cpu":"0.0","mem":"0.0"},"378":{"cpu":"1.0","mem":"0.2"},"41":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"565481":{"cpu":"2.7","mem":"1.0"},"565627":{"cpu":"0.3","mem":"0.6"},"567970":{"cpu":"0.0","mem":"0.0"},"568019":{"cpu":"0.0","mem":"0.1"},"568020":{"cpu":"40.0","mem":"0.7"},"568030":{"cpu":"2.0","mem":"0.1"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"86":{"cpu":"13.7","mem":"0.7"},"98":{"cpu":"0.0","mem":"0.0"},"99":{"cpu":"47.0","mem":"25.7"},"MiB":{"cpu":"1595.8","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"decking":{"camera_fps":6.1,"capture_pid":316,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":325,"pid":300,"process_fps":6.1,"skipped_fps":0.0},"detect-test":{"camera_fps":0.0,"capture_pid":null,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":0,"pid":null,"process_fps":0.0,"skipped_fps":0.0},"detection_fps":81.0,"detectors":{"coral":{"detection_start":1696868350.470541,"inference_speed":7.32,"pid":290}},"doorbell":{"camera_fps":15.1,"capture_pid":304,"detection_enabled":1,"detection_fps":50.7,"ffmpeg_pid":565481,"pid":296,"process_fps":13.2,"skipped_fps":0.0},"driveway":{"camera_fps":6.1,"capture_pid":311,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":323,"pid":298,"process_fps":6.1,"skipped_fps":0.0},"extension-roof":{"camera_fps":6.0,"capture_pid":321,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":352,"pid":301,"process_fps":6.0,"skipped_fps":0.0},"front-door":{"camera_fps":15.0,"capture_pid":307,"detection_enabled":1,"detection_fps":30.0,"ffmpeg_pid":312,"pid":297,"process_fps":14.9,"skipped_fps":0.0},"garden":{"camera_fps":6.0,"capture_pid":313,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":318,"pid":299,"process_fps":6.0,"skipped_fps":0.0},"gpu_usages":{"intel-vaapi":{"gpu":"1.89 %","mem":"- %"}},"living-room":{"camera_fps":15.0,"capture_pid":336,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":355,"pid":303,"process_fps":15.0,"skipped_fps":0.0},"lower-hallway":{"camera_fps":25.1,"capture_pid":324,"detection_enabled":1,"detection_fps":0.3,"ffmpeg_pid":335,"pid":302,"process_fps":25.1,"skipped_fps":0.0},"service":{"last_updated":1696868354,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":139.2,"mount_type":"tmpfs","total":157.3,"used":18.1},"/media/frigate/clips":{"free":444733.7,"mount_type":"ext4","total":1678394.6,"used":1148327.6},"/media/frigate/recordings":{"free":444733.7,"mount_type":"ext4","total":1678394.6,"used":1148327.6},"/tmp/cache":{"free":983.2,"mount_type":"tmpfs","total":1000.0,"used":16.8}},"temperatures":{"apex_0":63.55},"uptime":234859,"version":"0.12.0-da3e197"}}

Operating system

Debian

Install method

Docker Compose

Coral version

PCIe

Network connection

Wired

Camera make and model

Reolink Doorbell

Any other information that may be helpful

No response

NickM-27 commented 11 months ago

What is your docker compose?

sshaikh commented 11 months ago 
version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "150mb" # update for your cameras based on calculation above
    devices:
      #      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      #- /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/cctv/frigate/config.yml:/config/config.yml:rw
      - /home/cctv/go2rtc/go2rtc:/config/go2rtc:ro
      - /cctv-storage/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # webrtc over tcp
      - "8555:8555/udp" # webrtc over udp
      - "1935:1935" # RTMP feeds
    environment:
       FRIGATE_RTSP_PASSWORD: "xxx"
       FRIGATE_MQTT_PASSWORD: "yyy"
NickM-27 commented 11 months ago

it is showing signs of CPU overwork, easiest to diagnose will be if you enable debug logs for recordings

logger:
  logs:
    frigate.record: debug

and then paste all logs here after you see one that says something like 2023-10-07 00:06:47.994950326 [2023-10-07 00:06:47] frigate.record WARNING : Unable to keep up with recording segments in cache for doorbell. Keeping the 5 most recent segments out of 9 and discarding the rest...

sshaikh commented 11 months ago

I've had this for a few days now and no longer see Unable to keep up log lines. I am still seeing Unable to create jpg because frame 1697110156.642571 is not in the cache though.

NickM-27 commented 11 months ago

that error has been fixed in 0.13 (currently in beta)

sshaikh commented 11 months ago

Ok - happy to move to beta or wait till release before revisiting this. Let me know which is more useful for the project.

github-actions[bot] commented 10 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.