search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.52k stars 1.8k forks source link

[Config Support: Nvidia hardware acceleration not working correctly with YUV444 stream from Raspberry Pi #3552

Closed BrodyStone21 closed 2 years ago

BrodyStone21 commented 2 years ago

Describe the problem you are having

Hey guys, I've recently set up frigate and gotten hardware transcoding working. However, it's only working on one of my cameras.

I have two cameras, one is a Google Pixel 3 running IP Webcam downloaded from the Google Play Store. It's streaming its contents using mjpeg over http.

The second camera is a Raspberry Pi 4 running 32-bit Bullseye with the official v2 camera module. It is configured to use libcamera (what Raspberry Pi OS has started migrating to) instead of the legacy camera option, and I'm using a modified version of raspberrypi/picamera2/examples/mjpeg_server.py script to stream the camera feed over http.

I'm on Ubuntu 21.04 LTS headless with the latest version of nvidia-driver-515 installed.

Hardware acceleration for the first camera works just fine. However, I cannot get hardware acclerationf for the second camera to work. I must be misunderstanding something, because I thought since both cameras were running mjpeg streams, it would be the same process.

I'm trying to get this set up correctly as a test, before I commit to buying more raspberry pis and starting a home project. 

# nvidia-smi
Thu Jul 28 09:46:41 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:06:00.0 Off |                  N/A |
| 21%   67C    P2    29W / 120W |    266MiB /  3072MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1117      G   /usr/lib/xorg/Xorg                  8MiB |
|    0   N/A  N/A      1198      G   /usr/bin/gnome-shell                2MiB |
|    0   N/A  N/A     27044      C   ffmpeg                            193MiB |
|    0   N/A  N/A     30929      C   ffmpeg                             57MiB |
+-----------------------------------------------------------------------------+
docker exec -it frigate nvidia-smi
Thu Jul 28 09:47:30 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:06:00.0 Off |                  N/A |
| 22%   67C    P2    29W / 120W |    209MiB /  3072MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Below is my docker run command, but I've had issues with both docker-compose and docker run. 

docker run -d \
  --gpus all \
  --name=frigate \
  --runtime=nvidia \
  --restart=unless-stopped \
  --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \
  --device /dev/bus/usb:/dev/bus/usb \
  --device /dev/dri/card0 \
  --shm-size=128m \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e NVIDIA_DRIVER_CAPABILITIES=all \
  -v /mnt/WD_White/Media/Surveiliance_Footage:/media/frigate \
  -v /home/ubuntu/.config/frigate/config.yml:/config/config.yml:ro \
  -v /etc/localtime:/etc/localtime:ro \
  -e FRIGATE_RTSP_PASSWORD="pwd" \
  -p 5000:5000 \
  -p 1935:1935 \
  blakeblackshear/frigate:0.11.0-rc1

Version

0.11.0-rc1

Frigate config file

mqtt:
  host: 192.168.1.2
  port: 1883

ffmpeg:
  hwaccel_args:
    - -c:v
    - mjpeg_cuvid

record:
  enabled: True
  expire_interval: 60
  retain:
    days: 1
    mode: all
  events:
    pre_capture: 5
    post_capture: 5
    objects:
      - person
    retain:
      default: 10
      mode: motion
      objects:
        person: 15

snapshots:
  enabled: True
  timestamp: True
  bounding_box: False
  crop: False
  retain:
    default: 10
    objects:
      person: 15

rtmp:
  enabled: True

live:
  height: 720
  quality: 8

timestamp_style:
  position: "bl"
  format: "%m/%d/%Y %H:%M:%S"
  color:
    red: 255
    green: 255
    blue: 255
  thickness: 2
  effect: solid

detect:
  width: 1280
  height: 720
  fps: 5
  enabled: True
  max_disappeared: 25
  stationary:
    interval: 0
    threshold: 50
    max_frames:
      default: 3000
      objects:
        person: 1000

objects:
  track:
    - person
  filters:
    person:
      min_area: 5000
      max_area: 100000
      min_score: 0.5
      threshold: 0.7

motion:
  threshold: 25
  contour_area: 30
  delta_alpha: 0.2
  frame_alpha: 0.2
  frame_height: 50
  improve_contrast: False

cameras:
  living_room:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
      inputs:
        - path: http://192.168.1.100:8000/stream.mjpg # <-- Raspberry Pi 
          roles:
            - detect
            - record
            - stmp
            - clips
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
        rtmp: -c:v libx264 -an -f flv
    best_image_timeout: 60
    zones:
      bedroom_door:
        coordinates: 479,137,479,259,346,261,351,141
  office:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
      inputs:
        - path: http://192.168.1.101:8000/video # <-- Google Pixel 3
          roles:
            - detect
            - record
            - stmp
            - clips
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
        rtmp: -c:v libx264 -an -f flv
    best_image_timeout: 60
    zones:
      office_door:
        coordinates: 69,720,375,720,390,49,115,42

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-07-28 09:38:58] frigate.app                    INFO    : Starting Frigate (0.11.0-37325c7)
Starting migrations
[2022-07-28 09:38:58] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-07-28 09:38:58] peewee_migrate                 INFO    : There is nothing to migrate
[2022-07-28 09:38:58] detector.cpu                   INFO    : Starting detection process: 223
[2022-07-28 09:38:58] frigate.app                    INFO    : Output process started: 225
[2022-07-28 09:38:58] ws4py                          INFO    : Using epoll
[2022-07-28 09:38:58] frigate.edgetpu                WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
[2022-07-28 09:38:58] frigate.app                    INFO    : Camera processor started for living_room: 227
[2022-07-28 09:38:58] frigate.app                    INFO    : Camera processor started for office: 228
[2022-07-28 09:38:58] frigate.app                    INFO    : Capture process started for living_room: 229
[2022-07-28 09:38:58] frigate.app                    INFO    : Capture process started for office: 232
[2022-07-28 09:38:59] ws4py                          INFO    : Using epoll
[2022-07-28 09:39:18] watchdog.living_room           INFO    : No frames received from living_room in 20 seconds. Exiting ffmpeg...
[2022-07-28 09:39:18] watchdog.living_room           INFO    : Waiting for ffmpeg to exit gracefully...
[2022-07-28 09:39:19] frigate.video                  ERROR   : living_room: Unable to read frames from ffmpeg process.
[2022-07-28 09:39:19] frigate.video                  ERROR   : living_room: Unable to read frames from ffmpeg process.
[2022-07-28 09:39:19] frigate.video                  ERROR   : living_room: Unable to read frames from ffmpeg process.
[2022-07-28 09:39:19] frigate.video                  ERROR   : living_room: ffmpeg process is not running. exiting capture thread...
[2022-07-28 09:39:29] watchdog.living_room           ERROR   : Ffmpeg process crashed unexpectedly for living_room.
[2022-07-28 09:39:29] watchdog.living_room           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-07-28 09:39:29] ffmpeg.living_room.detect      ERROR   : [mjpeg_cuvid @ 0x5613a0724c40] ctx->cvdl->cuvidDecodePicture(ctx->cudecoder, picparams) failed -> CUDA_ERROR_INVALID_HANDLE: invalid resource handle
[2022-07-28 09:39:29] ffmpeg.living_room.detect      ERROR   : [mjpeg_cuvid @ 0x5613a0724c40] cuvid decode callback error
[2022-07-28 09:39:29] ffmpeg.living_room.detect      ERROR   : Error while decoding stream #0:0: Generic error in an external library
[2022-07-28 09:39:29] ffmpeg.living_room.detect      ERROR   : [mjpeg_cuvid @ 0x5613a0724c40] ctx->cvdl->cuvidDecodePicture(ctx->cudecoder, picparams) failed -> CUDA_ERROR_INVALID_HANDLE: invalid resource handle

FFprobe output from your camera

ffprobe version 5.0.1-Jellyfin Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-shared --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-static --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100
Input #0, mpjpeg, from 'http://192.168.1.100:8000/stream.mjpg':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 1920x1080 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn

Frigate stats

{"detection_fps":0.0,"detectors":{"cpu":{"detection_start":0.0,"inference_speed":198.63,"pid":223}},"living_room":{"camera_fps":0.0,"capture_pid":229,"detection_fps":0.0,"pid":227,"process_fps":0.0,"skipped_fps":0.0},"office":{"camera_fps":3.1,"capture_pid":232,"detection_fps":0.0,"pid":228,"process_fps":3.1,"skipped_fps":0.0},"service":{"latest_version":"0.10.1","storage":{"/dev/shm":{"free":132.1,"mount_type":"tmpfs","total":134.2,"used":2.1},"/media/frigate/clips":{"free":17454812.6,"mount_type":"fuse.mergerfs","total":27988259.2,"used":9122607.1},"/media/frigate/recordings":{"free":17454812.6,"mount_type":"fuse.mergerfs","total":27988259.2,"used":9122607.1},"/tmp/cache":{"free":997.1,"mount_type":"tmpfs","total":1000.0,"used":2.9}},"temperatures":{},"uptime":815,"version":"0.11.0-37325c7"}}

Operating system

Debian

Install method

Docker CLI

Coral version

CPU (no coral)

Network connection

Mixed

Camera make and model

Raspberry Pi 4 (32-bit Bullseye) v2 Camera Module

Any other information that may be helpful

I can open my web interface endpoint, and it works fine. I see both cameras. However, the Raspberry Pi stream is green. After clicking on it, it shows the camera as white.

NickM-27 commented 2 years ago

because I thought since both cameras were running mjpeg streams, it would be the same process.

There is a separate process created for each path defined for each camera.

For the RPi camera I would try using the recommended mjpeg args in the docs: https://docs.frigate.video/configuration/camera_specific#mjpeg-cameras

NickM-27 commented 2 years ago

I'm not sure how relevant this is, but if the above doesn't work then https://patchwork.ffmpeg.org/project/ffmpeg/patch/1485019650-30152-1-git-send-email-pkoshevoy@gmail.com/ could give some clues. Seems like it doesn't support 444 which your ffprobe indicated yuvj444p

Maybe play around with the RPi camera streaming settings and see if something there makes a difference

BrodyStone21 commented 2 years ago

because I thought since both cameras were running mjpeg streams, it would be the same process.

There is a separate process created for each path defined for each camera.

For the RPi camera I would try using the recommended mjpeg args in the docs: https://docs.frigate.video/configuration/camera_specific#mjpeg-cameras

Thanks for the reply. Unfortunately, I'm already using the recommended input/output args for playing mjpeg streams; see my config.

I'm not sure how relevant this is, but if the above doesn't work then https://patchwork.ffmpeg.org/project/ffmpeg/patch/1485019650-30152-1-git-send-email-pkoshevoy@gmail.com/ could give some clues. Seems like it doesn't support 444 which your ffprobe indicated yuvj444p

Maybe play around with the RPi camera streaming settings and see if something there makes a difference

I'll try messing around some more with it. My original plan was to try and use rtsp with the pi instead of mjpeg, but finding a solution proved to be pretty difficult with the migration from the legacy cam to libcamera.

NickM-27 commented 2 years ago

Thanks for the reply. Unfortunately, I'm already using the recommended input/output args for playing mjpeg streams; see my config.

You do not have -use_wallclock_as_timestamps 1 set, which is recommended

BrodyStone21 commented 2 years ago

Thanks for the reply. Unfortunately, I'm already using the recommended input/output args for playing mjpeg streams; see my config.

You do not have -use_wallclock_as_timestamps 1 set, which is recommended

Oops, forgot to mention I removed that for testing. I obviously forgot to put it back in.

I put it back in and the issue persists. Any other ideas?

NickM-27 commented 2 years ago

Can you run ffprobe on the camera that works? Would be helpful to maybe confirm for sure what the differences between them are

BrodyStone21 commented 2 years ago

Can you run ffprobe on the camera that works? Would be helpful to maybe confirm for sure what the differences between them are

Sure thing

ffprobe version 5.0.1-Jellyfin Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-shared --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-static --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100
Input #0, mpjpeg, from 'http://192.168.1.101:8000/video':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 1920x1080 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn
NickM-27 commented 2 years ago

yeah looks like this one is yuv420 and not yuv444, seems that is the issue.

Something interesting here: https://github.com/raspberrypi/picamera2/pull/161

If unable to set it there, may need to look into cuvid args and see if you can set the pixel format to yuv444 on nvidia side

BrodyStone21 commented 2 years ago

If unable to set it there, may need to look into cuvid args and see if you can set the pixel format to yuv444 on nvidia side

I'll ask there, thank you. This problem is proving to be much more difficult than I thought haha.

I'm sure that once I get it figured out, it'll be able to help a lot of people using frigate with Raspberry Pi's.

BrodyStone21 commented 2 years ago

UPDATE!!

Finally got the Raspberry Pi to encode it's stream in YUV420 instead of YUV444. I asked on the raspberrypi/picamera2 repo and was helped there. The main developer has now created a pull request to make the JpegEncoder module use YUV420 by default, but it can also be manually specified.

This request will go into the "next" branch for now, but will eventually make it's way into the final release. For the meantime, you can reference this PR. Or just use the JpegEncoder module from the next branch.

NickM-27 commented 2 years ago

Cool, glad it is working now 👍