search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.94k stars 1.73k forks source link

[Support]: CPU Pegged at 100% by Frigate with TPU Installed #3438

Closed carlos-sarmiento closed 2 years ago

carlos-sarmiento commented 2 years ago

Describe the problem you are having

I am running 8 cameras on Frigate on a computer with a coral module installed and I'm seeing my CPU pegged at 100% all the time. I haven't checked in a while, but I don't remember ever having the CPU usage so high when running frigate.

Here is a screenshot from HTOP running on the host

Screenshot 2022-07-07 10 22 53 AM

Version

0.10.1-83481AF

Frigate config file

database:
  path: /config/frigate.db

mqtt:
  host: mqtt_host.local

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p

snapshots:
  enabled: yes

record:
  enabled: True
  retain:
    days: 2
  events:
    pre_capture: 60
    post_capture: 60
    retain:
      default: 5

cameras:
  woodshop:
    ffmpeg:
      inputs:
        - path: rtsp://1080_camera_url/s0
          roles:
            - detect
            - rtmp
    detect:
      width: 1920
      height: 1080
    record:
      events:
        objects:
          - person

    objects:
      track:
        - person

  right_backyard:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  covered_patio:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  2car_garage_left:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  driveway_right:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

    motion:
      mask:
        - 2318,527,2278,407,1193,558,71,892,80,1059,548,2160,687,2160,979,1565,1525,1167,2193,1358,2664,2160,0,2160,0,1122,0,0,3840,0,3840,2160,2739,2160,2913,1838,3146,480
    objects:
      track:
        - person
        - car
      filters:
        person:
          mask:
            - 1002,0,3139,0,3151,525,2407,609,1096,904,527,1073,532,1779,539,2160,0,2160,0,0
        car:
          mask:
            - 2407,602,1104,894,558,1061,0,1249,0,0,3840,0,3840,2160,2739,2160,2913,1838,3078,894,3113,525

  main_door:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  front_yard_left:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://1080_camera_url:554/cam/realmonitor?channel=1&subtype=0
          roles:
            # - detect
            - rtmp
    detect:
      width: 1920
      height: 1080

Relevant log output

[services.d] starting services
[services.d] done.
[2022-07-07 10:24:22] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
Starting migrations
[2022-07-07 10:24:22] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-07-07 10:24:22] peewee_migrate                 INFO    : There is nothing to migrate
[2022-07-07 10:24:22] detector.coral                 INFO    : Starting detection process: 216
[2022-07-07 10:24:22] frigate.app                    INFO    : Output process started: 218
[2022-07-07 10:24:22] ws4py                          INFO    : Using epoll
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for woodshop: 225
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for right_backyard: 227
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for covered_patio: 229
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for 2car_garage_left: 230
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for driveway_right: 232
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for main_door: 234
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for front_yard_left: 235
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for doorbell: 236
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for woodshop: 238
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for right_backyard: 242
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for covered_patio: 246
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for 2car_garage_left: 249
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for driveway_right: 251
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for main_door: 254
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for front_yard_left: 260
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for doorbell: 264
[2022-07-07 10:24:23] ws4py                          INFO    : Using epoll
[2022-07-07 10:24:22] frigate.edgetpu                INFO    : Attempting to load TPU as usb
[2022-07-07 10:24:25] frigate.edgetpu                INFO    : TPU found

FFprobe output from your camera

Most Cameras:
root@cf50164e5945:/opt/frigate# ffprobe "rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64"
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[rtsp @ 0x55b1019a0b80] max delay reached. need to consume packet
[rtsp @ 0x55b1019a0b80] RTP: missed 130 packets
[h264 @ 0x55b1019a67c0] error while decoding MB 85 108, bytestream -5
[h264 @ 0x55b1019a67c0] concealing 6684 DC, 6684 AC, 6684 MV errors in P frame
[rtsp @ 0x55b1019a0b80] max delay reached. need to consume packet
[rtsp @ 0x55b1019a0b80] RTP: missed 109 packets
[h264 @ 0x55b1019a67c0] error while decoding MB 195 35, bytestream -13
[h264 @ 0x55b1019a67c0] concealing 24094 DC, 24094 AC, 24094 MV errors in P frame
[rtsp @ 0x55b1019a0b80] max delay reached. need to consume packet
[rtsp @ 0x55b1019a0b80] RTP: missed 67 packets
[rtsp @ 0x55b1019a0b80] max delay reached. need to consume packet
[rtsp @ 0x55b1019a0b80] RTP: missed 1010 packets
Input #0, rtsp, from 'rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.200000, bitrate: N/A
    Stream #0:0: Video: h264, yuv420p(progressive), 3840x2160, 100 tbr, 90k tbn, 180k tbc
root@cf50164e5945:/opt/frigate# ^C

Different Camera Model:

root@cf50164e5945:/opt/frigate# ffprobe "rtsp://camera_url/cam/realmonitor?channel=1&subtype=0"
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://camera_url/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.040000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080, 100 tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: aac, 16000 Hz, mono, fltp

Frigate stats

{
  "2car_garage_left": {
    "camera_fps": 5.1,
    "capture_pid": 249,
    "detection_fps": 3.7,
    "pid": 230,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "covered_patio": {
    "camera_fps": 5.1,
    "capture_pid": 246,
    "detection_fps": 4.6,
    "pid": 229,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "detection_fps": 30.6,
  "detectors": {
    "coral": {
      "detection_start": 1657207629.796821,
      "inference_speed": 9,
      "pid": 216
    }
  },
  "doorbell": {
    "camera_fps": 5.1,
    "capture_pid": 264,
    "detection_fps": 0,
    "pid": 236,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "driveway_right": {
    "camera_fps": 5.1,
    "capture_pid": 251,
    "detection_fps": 7.7,
    "pid": 232,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "front_yard_left": {
    "camera_fps": 5.1,
    "capture_pid": 260,
    "detection_fps": 2.9,
    "pid": 235,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "main_door": {
    "camera_fps": 4.9,
    "capture_pid": 254,
    "detection_fps": 0,
    "pid": 234,
    "process_fps": 4.9,
    "skipped_fps": 0
  },
  "right_backyard": {
    "camera_fps": 5,
    "capture_pid": 242,
    "detection_fps": 11.7,
    "pid": 227,
    "process_fps": 5,
    "skipped_fps": 0
  },
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 976.8,
        "mount_type": "tmpfs",
        "total": 1073.7,
        "used": 96.9
      },
      "/media/frigate/clips": {
        "free": 3053872.6,
        "mount_type": "zfs",
        "total": 3964644.2,
        "used": 910771.6
      },
      "/media/frigate/recordings": {
        "free": 3053872.6,
        "mount_type": "zfs",
        "total": 3964644.2,
        "used": 910771.6
      },
      "/tmp/cache": {
        "free": 1764.9,
        "mount_type": "tmpfs",
        "total": 2000,
        "used": 235.1
      }
    },
    "temperatures": {},
    "uptime": 167,
    "version": "0.10.1-83481af"
  },
  "woodshop": {
    "camera_fps": 5,
    "capture_pid": 238,
    "detection_fps": 0,
    "pid": 225,
    "process_fps": 5,
    "skipped_fps": 0
  }
}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

All Amcrest

Any other information that may be helpful

Docker container config (using Ansible)

 docker_container:
    name: frigate
    image: blakeblackshear/frigate:stable-amd64
    state: started
    restart_policy: unless-stopped
    shm_size: 1024M
    mounts:
      - type: tmpfs
        target: /tmp/cache
        tmpfs_size: 2000000000

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ~/frigate/config.yml:/config/config.yml:ro
      - ~/frigate/frigate.db:/config/frigate.db
      - ~/frigate:/media/frigate
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/dri/renderD128:/dev/dri/renderD128

    ports:
      - "1935:1935" # RTMP feeds
    env:
      FRIGATE_RTSP_PASSWORD: "**********"

    labels:
      traefik.enable: "true"
NickM-27 commented 2 years ago

You're running detect on full 4k frames. 

detect: 
  width: 3840 
  height: 2160

This is discouraged in the docs https://docs.frigate.video/guides/camera_setup#choosing-a-detect-resolution and 100% expected.

Running motion detection on 4K is computationally expensive and shown to not be helpful at all unless detecting very small objects (all regions are resized to 320x320 to do object detection, so resizing a higher resolution object means using a higher resolution is just a waste of CPU resources)

carlos-sarmiento commented 2 years ago

Thanks for the feedback. I updated my config to the following and the CPU problem improved but it is still pretty high. Would this be the correct configuration if I want to view and store the 4k footage but detect on a smaller resolution? 

  ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            - record
            - rtmp
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=1&authbasic=64
          roles:
            - detect
    detect: # this is the resolution of the second stream according to ffprobe
      width: 704
      height: 480
NickM-27 commented 2 years ago

Yes that looks correct. A new htop would be helpful. What CPU do you have and have you verified hwaccel is working?

carlos-sarmiento commented 2 years ago

New HTOP:

image

How can I verify hwaccel is working? Looked for instructions on the hw acceleration docs but couldn't find anything.

Also, my cameras are now being rendered on the UI using the lower resolution: image

Before updating the config, it would show like this (screenshot taken using the old config) image

NickM-27 commented 2 years ago

The ffmpeg processes look better now actually so I don't think it's hwaccel related. You would verify with intel_gpu_top if intel

As far as the UI goes, you'll want to adjust camera -> live -> height to change that.

I'm not sure what more can be done to reduce CPU usage more, perhaps that number of cameras is taxing for the CPU. @blakeblackshear may have a better idea on the CPU usage

carlos-sarmiento commented 2 years ago

This is a screenshot of intel_gpu_top. Since this computer has no UI installed, I assume the fact the GPU is seeing usage to mean frigate is using it.

image

Also, regarding the live view, the documentation says This must be less than or equal to the height of the detect stream. which would mean frigate is using the lower resolution stream even though I've configured both record and rtmp to use the high resolution one. This intepretation seems right because both the stream on the frigate UI and on Home Assistant now show the lower resolution stream instead of the 4K. Am I understanding this behavior correctly?

NickM-27 commented 2 years ago

The frigate live view uses the detect stream.

The home assistant camera.your_camera entity uses the rtmp stream

carlos-sarmiento commented 2 years ago

I understand what you are saying, but it doesn't seem to be what I'm seeing in HA:

With the new config this is what I see on my dashboard: image

This is how the same camera renders using the old config: image

Should I file this as a separate issue?

blakeblackshear commented 2 years ago

Is this a live camera entity or the zone/object specific cameras that are a static image that updates? Snapshots including these would be from the detect stream.

blakeblackshear commented 2 years ago

The best way to understand what is driving CPU usage is to turn off features to see what is driving it up. You can disable detect for each camera to get a sense of the baseline CPU to just decode the stream and look for motion.

carlos-sarmiento commented 2 years ago

This is the live camera entity. Not the cameras that get created for detection (person, dog, etc.)

I tried disabling all my cameras and CPU usage went down, so I assume that the sheer number of cameras is what is driving the CPU.

blakeblackshear commented 2 years ago

I would suggest watching the debug view of each camera with the motion boxes overlays and seeing what is triggering motion and object detection. Your best bet is making sure you use motion masks to avoid running unnecessary detections from things like camera timestamps.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.