0.8.0rc3 >=Gen10 Hardware support

davidvf commented 3 years ago

I am trying to get the hardware acceleration running on my Gen10 QSV hardware. Issue: https://community.home-assistant.io/t/local-realtime-person-detection-for-rtsp-cameras/103107/3212?u=davidvf

Version of frigate 0.8.0 rc3

I am running the basic ffmpeg command to get started. From the OS, if i run this, i see the intel gpu start working

MAIN OS

[david@owl:~ $] ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i rtsp://192.168.0.5:7447/16gU54OAVReFohX3 -c:v h264_qsv hwaccel-test.mp4
ffmpeg version N-100411-g32586a42da Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --enable-libmfx
  libavutil      56. 62.100 / 56. 62.100
  libavcodec     58.115.102 / 58.115.102
  libavformat    58. 65.100 / 58. 65.100
  libavdevice    58. 11.103 / 58. 11.103
  libavfilter     7. 93.100 /  7. 93.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
Input #0, rtsp, from 'rtsp://192.168.0.5:7447/16gU54OAVReFohX3':
  Metadata:
    title           : B4FBE4FF7EDE_0
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 11025 Hz, mono, fltp
    Stream #0:1: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], 5 fps, 5 tbr, 90k tbn, 10 tbc
File 'hwaccel-test.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (h264_qsv) -> h264 (h264_qsv))
  Stream #0:0 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[aac @ 0x55f5818c25c0] Too many bits 6408.707483 > 6144 per frame requested, clamping to max
Output #0, mp4, to 'hwaccel-test.mp4':
  Metadata:
    title           : B4FBE4FF7EDE_0
    encoder         : Lavf58.65.100
    Stream #0:0: Video: h264 (avc1 / 0x31637661), qsv(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], q=2-31, 1000 kb/s, 5 fps, 10240 tbn
    Metadata:
      encoder         : Lavc58.115.102 h264_qsv
    Side data:
      cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 0 vbv_delay: N/A
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 11025 Hz, mono, fltp, 66 kb/s
    Metadata:
      encoder         : Lavc58.115.102 aac
frame=   67 fps= 11 q=24.0 Lsize=    1566kB time=00:00:13.37 bitrate= 959.0kbits/s dup=6 drop=1 speed=2.28x
video:1466kB audio:96kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.234253%
[aac @ 0x55f5818c25c0] Qavg: 65536.000
Exiting normally, received signal 2.

[david@owl:~ $]

intel_gpu_top output

[david@owl:~ $] sudo intel_gpu_top -l
 Freq MHz      IRQ RC6 Power           RCS/0           BCS/0           VCS/0          VECS/0
 req  act       /s   %     W       %  se  wa       %  se  wa       %  se  wa       %  se  wa
  59   59       67  95  0.12    1.61   0   0    0.00   0   0    2.83   1   0    0.00   0   0
  59   59       66  95  0.12    1.59   0   0    0.00   0   0    2.76   1   0    0.00   0   0
  68   68       72  94  0.12    1.66   0   0    0.00   0   0    3.04   0   0    0.00   0   0
  71   71       68  94  0.13    1.63   0   0    0.00   0   0    2.99   0   0    0.00   0   0
  65   65       66  94  0.12    1.72   0   0    0.00   0   0    3.04   1   0    0.00   0   0
  60   60       70  94  0.12    1.77   0   0    0.00   0   0    2.97   0   0    0.00   0   0
  59   59       70  94  0.13    1.74   0   0    0.00   0   0    2.86   0   0    0.00   0   0
  69   68       69  94  0.12    1.53   0   0    0.00   0   0    3.16   0   0    0.00   0   0
  59   60       68  94  0.12    1.77   0   0    0.00   0   0    2.98   0   0    0.00   0   0
  57   57       70  95  0.13    2.26   1   0    0.00   0   0    3.16   0   0    0.00   0   0
   0    0        0 100  0.00    0.00   0   0    0.00   0   0    0.00   0   0    0.00   0   0

If I run the same command from the docker container

root@81b0d26d6fc1:/opt/frigate# ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i rtsp://192.168.0.5:7447/16gU54OAVReFohX3 -c:v h264_qsv hwaccel-test.mp4
ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://192.168.0.5:7447/16gU54OAVReFohX3':
  Metadata:
    title           : B4FBE4FF7EDE_0
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac, 11025 Hz, mono, fltp
    Stream #0:1: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], 5 fps, 5 tbr, 90k tbn, 10 tbc
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (h264_qsv) -> h264 (h264_qsv))
  Stream #0:0 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[aac @ 0x5612d4186ac0] Too many bits 6408.707483 > 6144 per frame requested, clamping to max
[h264_qsv @ 0x5612d41877c0] Selected ratecontrol mode is unsupported
[h264_qsv @ 0x5612d41877c0] Low power mode is unsupported
[h264_qsv @ 0x5612d41877c0] Current frame rate is unsupported
[h264_qsv @ 0x5612d41877c0] Current picture structure is unsupported
[h264_qsv @ 0x5612d41877c0] Current resolution is unsupported
[h264_qsv @ 0x5612d41877c0] Current pixel format is unsupported
[h264_qsv @ 0x5612d41877c0] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x5612d4186ac0] Qavg: 65536.000
[aac @ 0x5612d4186ac0] 2 frames left in the queue on closing
Conversion failed!
root@81b0d26d6fc1:/opt/frigate#

FFprobe from your camera

root@81b0d26d6fc1:/opt/frigate# ffprobe rtsp://192.168.0.5:7447/16gU54OAVReFohX3
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://192.168.0.5:7447/16gU54OAVReFohX3':
  Metadata:
    title           : B4FBE4FF7EDE_0
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac, 11025 Hz, mono, fltp
    Stream #0:1: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], 5 fps, 5 tbr, 90k tbn, 10 tbc

Computer Hardware

OS:

cat /etc/os-release
NAME="Ubuntu"
VERSION="20.10 (Groovy Gorilla)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.10"
VERSION_ID="20.10"

Virtualization: Docker
Coral Version: USB
Network Setup: Wired

Camera Info:

Manufacturer: Unifi
Model: unifi g3
Resolution: 1920x1080
FPS: 5

ekos2001 commented 3 years ago

What if you try to run ffmpeg version 4.3.1 on the host? Also you can try to add -rtsp_transport tcp

blakeblackshear commented 3 years ago

post the output of ffmpeg -buildconf from the host and compare that with in the container

davidvf commented 3 years ago

HOST CUSTOMIZED FFMPEG

[david@owl:~ $] ffmpeg -buildconf
ffmpeg version N-100411-g32586a42da Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --enable-libmfx
  libavutil      56. 62.100 / 56. 62.100
  libavcodec     58.115.102 / 58.115.102
  libavformat    58. 65.100 / 58. 65.100
  libavdevice    58. 11.103 / 58. 11.103
  libavfilter     7. 93.100 /  7. 93.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100

  configuration:
    --enable-libmfx

[david@owl:~ $]

HOST ORIGINAL FFMPEG If i run the command i mentioned in the first post, this version is also using the hwacceleration. I had build a custom ffmpeg because in the beginning, for some reason it was not compiled with libmfx support. Now it does apparently :)

[david@owl:~ $] /usr/bin/ffmpeg -buildconf
ffmpeg version 4.3.1-4ubuntu1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 10 (Ubuntu 10.2.0-9ubuntu2)
  configuration: --prefix=/usr --extra-version=4ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100

  configuration:
    --prefix=/usr
    --extra-version=4ubuntu1
    --toolchain=hardened
    --libdir=/usr/lib/x86_64-linux-gnu
    --incdir=/usr/include/x86_64-linux-gnu
    --arch=amd64
    --enable-gpl
    --disable-stripping
    --enable-avresample
    --disable-filter=resample
    --enable-gnutls
    --enable-ladspa
    --enable-libaom
    --enable-libass
    --enable-libbluray
    --enable-libbs2b
    --enable-libcaca
    --enable-libcdio
    --enable-libcodec2
    --enable-libdav1d
    --enable-libflite
    --enable-libfontconfig
    --enable-libfreetype
    --enable-libfribidi
    --enable-libgme
    --enable-libgsm
    --enable-libjack
    --enable-libmp3lame
    --enable-libmysofa
    --enable-libopenjpeg
    --enable-libopenmpt
    --enable-libopus
    --enable-libpulse
    --enable-librabbitmq
    --enable-librsvg
    --enable-librubberband
    --enable-libshine
    --enable-libsnappy
    --enable-libsoxr
    --enable-libspeex
    --enable-libsrt
    --enable-libssh
    --enable-libtheora
    --enable-libtwolame
    --enable-libvidstab
    --enable-libvorbis
    --enable-libvpx
    --enable-libwavpack
    --enable-libwebp
    --enable-libx265
    --enable-libxml2
    --enable-libxvid
    --enable-libzmq
    --enable-libzvbi
    --enable-lv2
    --enable-omx
    --enable-openal
    --enable-opencl
    --enable-opengl
    --enable-sdl2
    --enable-pocketsphinx
    --enable-libmfx
    --enable-libdc1394
    --enable-libdrm
    --enable-libiec61883
    --enable-nvenc
    --enable-chromaprint
    --enable-frei0r
    --enable-libx264
    --enable-shared

[david@owl:~ $]

CONTAINER

root@0694fb90bba1:/opt/frigate# ffmpeg -buildconf
ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100

  configuration:
    --disable-debug
    --disable-doc
    --disable-ffplay
    --enable-shared
    --enable-avresample
    --enable-libopencore-amrnb
    --enable-libopencore-amrwb
    --enable-gpl
    --enable-libfreetype
    --enable-libvidstab
    --enable-libmfx
    --enable-libmp3lame
    --enable-libopus
    --enable-libtheora
    --enable-libvorbis
    --enable-libvpx
    --enable-libwebp
    --enable-libxcb
    --enable-libx265
    --enable-libxvid
    --enable-libx264
    --enable-nonfree
    --enable-openssl
    --enable-libfdk_aac
    --enable-postproc
    --enable-small
    --enable-version3
    --enable-libzmq
    --extra-libs=-ldl
    --prefix=/opt/ffmpeg
    --enable-libopenjpeg
    --enable-libkvazaar
    --enable-libaom
    --extra-libs=-lpthread
    --enable-vaapi
    --extra-cflags=-I/opt/ffmpeg/include
    --extra-ldflags=-L/opt/ffmpeg/lib
root@0694fb90bba1:/opt/frigate#

blakeblackshear commented 3 years ago

I have seen some strange issues like this in the past related to the OS on the host and the container having compatibility issues. I think its because the container and the OS do share some low level components. Is there any way you can run 20.04 instead of 20.10 on your host?

You could also try a custom build of the container with 20.10 by modifying this line and running make amd64_frigate. It built for me, but I'm not sure if it will make a difference.

davidvf commented 3 years ago

I just gave it a try but the build fails

Step 1/27 : ARG ARCH=amd64

Step 2/27 : ARG WHEELS_VERSION=1.0.1

Step 3/27 : ARG FFMPEG_VERSION=1.1.0

Step 4/27 : FROM blakeblackshear/frigate-wheels:${WHEELS_VERSION}-${ARCH} as wheels

 ---> c99273f03f6c

Step 5/27 : FROM blakeblackshear/frigate-ffmpeg:${FFMPEG_VERSION}-${ARCH} as ffmpeg

 ---> 2d08f39f2c07

Step 6/27 : FROM frigate-web as web

pull access denied for frigate-web, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

blakeblackshear commented 3 years ago

Did you run make amd64_frigate or something else?

davidvf commented 3 years ago

I copied the contents of the file you linked to, in my portainer, modified the line

FROM ubuntu:20.04

to

FROM ubuntu:20.10

and clicked build container

blakeblackshear commented 3 years ago

The error message is because the makefile builds the WebUI container first. Not sure if you can build it in portainer like that.

davidvf commented 3 years ago

In the meantime i've been trying a lot of things. Currently i ended up again with the VAAPI hwaccel method

ffmpeg:
  global_args: -hide_banner -loglevel fatal
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128

That at least let's the intel_gpu_top graphs start moving ;)

intel-gpu-top -   15/  15 MHz;   97% RC6;  0.01 Watts;       20 irqs/s

      IMC reads:   ------ (null)/s
     IMC writes:   ------ (null)/s

          ENGINE      BUSY                                                                                                                                                                              MI_SEMA MI_WAIT
     Render/3D/0    0.00% |                                                                                                                                                                           |      0%      0%
       Blitter/0    0.00% |                                                                                                                                                                           |      0%      0%
         Video/0    1.67% |██▊                                                                                                                                                                        |      0%      0%
  VideoEnhance/0    0.00% |                                                                                                                                                                           |      0%      0%

ekos2001 commented 3 years ago

Take a look into this page from another project. https://jellyfin.org/docs/general/administration/hardware-acceleration.html#configuring-intel-quicksyncqsv-on-debianubuntu There are some drivers that have to be installed for QSV acceleration. And they use custom built ffmpeg. Also I found the docker image, which includes all required drivers. https://hub.docker.com/r/linuxserver/jellyfin I was able to run their ffmpeg in their container with QSV acceleration using following command:

/usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel qsv -c:v h264_qsv -i 'rtsp://192.168.0.73/H264?ch=1&subtype=0' -vf hwdownload,format=nv12 -pix_fmt yuv420p -f rawvideo -y /dev/null

What is strange that adding -c:v h264_qsv to the output makes things worse - CPU and GPU usage is higher. Now I'm trying to figure out what changes should I apply to the host or frigate container to make QSV acceleration work with frigate.

davidvf commented 3 years ago

I succeeded in building a custom container as per your instructions @blakeblackshear I've changed to line to have ubuntu 20.10 but that did not make a difference.

But i also made a change in this file: Dockerfile.amd64

I replaced intel-media-va-driver with intel-media-va-driver-non-free That build succeeded as well.

Then i modified my config.yml with the input from @ekos2001

ffmpeg:
  global_args: -hide_banner -loglevel fatal
  hwaccel_args:
    - -hwaccel
    - qsv
    - -qsv_device
    - /dev/dri/renderD128
    - -c:v
    - h264_qsv
  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1
  output_args:
    detect: -vf hwdownload,format=nv12 -f rawvideo -pix_fmt yuv420p
    record: -vf hwdownload,format=nv12 -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v h264_qsv -an
    clips: -vf hwdownload,format=nv12 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v h264_qsv -an
    rtmp: -vf hwdownload,format=nv12 -c:v h264_qsv -f flv

That seems to work from a hwaccel perspective:

intel-gpu-top -  116/ 116 MHz;   91% RC6;  0.21 Watts;      110 irqs/s

      IMC reads:   ------ (null)/s
     IMC writes:   ------ (null)/s

          ENGINE      BUSY                                                                                                                                                                  MI_SEMA MI_WAIT
     Render/3D/0    3.26% |█████▏                                                                                                                                                         |      0%      0%
       Blitter/0    0.00% |                                                                                                                                                               |      0%      0%
         Video/0    4.72% |███████▌                                                                                                                                                       |      2%      0%
  VideoEnhance/0    0.00% |                                                                                                                                                               |      0%      0%

So far no errors in the container log. Everything seems to work fine for now. CPU usage seems to be less, but i have to monitor it for some more time to be sure that it really makes a difference.

blakeblackshear commented 3 years ago

Note that unless your video feeds are not already h264, there is no need to specify anything related to hardware acceleration for rtmp, clips, or record args. Those should just be direct copy operations because they don't need to decode the video stream. Your CPU usage for those should be almost nothing. Make sure you aren't accidentally decoding and reencoding those streams. The best way to test it would be to put detect on a dedicated stream and assign the others to something else (you can duplicate the input entry when testing).

davidvf commented 3 years ago

Here is the output of ffprobe on the video stream from the camera

[david@owl:~ $] /usr/bin/ffprobe rtsp://192.168.0.5:7447/16gU54OAVReFohX3
ffprobe version 4.3.1-4ubuntu1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 10 (Ubuntu 10.2.0-9ubuntu2)
  configuration: --prefix=/usr --extra-version=4ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[rtsp @ 0x564f71846880] max delay reached. need to consume packet
[rtsp @ 0x564f71846880] RTP: missed 124 packets
[h264 @ 0x564f7184d9c0] error while decoding MB 94 40, bytestream -25
[h264 @ 0x564f7184d9c0] concealing 3315 DC, 3315 AC, 3315 MV errors in I frame
Input #0, rtsp, from 'rtsp://192.168.0.5:7447/16gU54OAVReFohX3':
  Metadata:
    title           : B4FBE4FF7EDE_0
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 11025 Hz, mono, fltp
    Stream #0:1: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], 5 fps, 5 tbr, 90k tbn, 10 tbc

So that means i can remove -c:v h264_qsv for detect, clips, rtmp and record? Or should i leave it for detect, because you did not mention that one :) (just noticed that i did not even have it for that one apparently)

blakeblackshear commented 3 years ago

The only role that requires decoding the video stream is detect. The others should just copy that video feed as is. Can you post your entire config?

davidvf commented 3 years ago

detectors:
  coral:
    type: edgetpu

mqtt:
  host: owl.home
  topic_prefix: frigate
  client_id: frigate

clips:
  max_seconds: 300
  retain:
    default: 5

snapshots:
  retain:
    default: 5

ffmpeg:
  global_args: -hide_banner -loglevel fatal
  hwaccel_args:
    - -hwaccel
    - qsv
    - -qsv_device
    - /dev/dri/renderD128
    - -c:v
    - h264_qsv
  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1
  output_args:
    detect: -vf hwdownload,format=nv12 -f rawvideo -pix_fmt yuv420p
    record: -vf hwdownload,format=nv12 -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v h264_qsv -an
    clips: -vf hwdownload,format=nv12 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v h264_qsv -an
    rtmp: -vf hwdownload,format=nv12 -c:v h264_qsv -f flv

objects:
  track:
    - person
    - car

cameras:
  oprit:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.5:7447/16gU54OAVReFohX3
          roles:
            - detect
            - rtmp
            - clips
    height: 1080
    width: 1920
    fps: 5

    motion:
      mask: 0,1080,0,0,345,0,380,1080

    best_image_timeout: 60

    clips:
      enabled: True
      pre_capture: 10
      objects:
        - person
        - car

    rtmp:
      enabled: True

    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
      crop: False

    objects:
      track:
        - person
        - car

blakeblackshear commented 3 years ago

You can remove the custom output args for everything except detect. While testing CPU usage, I would either remove those roles from the input or move them to a duplicate (shown below). That way the detect stream will be isolated to a single ffmpeg process and you can make sure you aren't decoding for rtmp and clips.

cameras:
  oprit:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.5:7447/16gU54OAVReFohX3
          roles:
            - detect
        - path: rtsp://192.168.0.5:7447/16gU54OAVReFohX3
          roles:
            - rtmp
            - clips

davidvf commented 3 years ago

Ok that seems to work fine. So to summarize, the container where i changed the driver from intel-media-va-driver to intel-media-va-driver-non-free runs fine.

If I now start the original container based on 0.8.0-RC4 with the same config.yml, i have the following output:

 * Starting nginx nginx
   ...done.
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.mqtt                   INFO    : MQTT connected
detector.coral                 INFO    : Starting detection process: 33
frigate.app                    INFO    : Camera processor started for oprit: 36
frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.app                    INFO    : Capture process started for oprit: 37
frigate.edgetpu                INFO    : TPU found
ffmpeg.oprit.detect            ERROR   : ffmpeg: /build/intel-mediasdk-UBsAHl/intel-mediasdk-20.1.0/_studio/shared/src/mfx_vpp_vaapi.cpp:239: mfxStatus MfxHwVideoProcessing::VAAPIVideoProcessing::Init(void**, mfxVideoParam*): Assertion `0x00000000 == vaSts' failed.
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg process is not running. exiting capture thread...

blakeblackshear commented 3 years ago

You also updated the base to be 20.10 though right? So your container had the non-free version and the base changed to 20.10. See if just updating the base to 20.10 still results in the same error.

davidvf commented 3 years ago

As requested, this is the output if i use the standard intel intel-media-va-driver driver and base set to 20.10

 * Starting nginx nginx
   ...done.
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.mqtt                   INFO    : MQTT connected
detector.coral                 INFO    : Starting detection process: 36
frigate.app                    INFO    : Camera processor started for oprit: 39
frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.app                    INFO    : Capture process started for oprit: 41
frigate.edgetpu                INFO    : TPU found
ffmpeg.oprit.detect            ERROR   : ffmpeg: /build/intel-mediasdk-UBsAHl/intel-mediasdk-20.1.0/_studio/shared/src/mfx_vpp_vaapi.cpp:239: mfxStatus MfxHwVideoProcessing::VAAPIVideoProcessing::Init(void**, mfxVideoParam*): Assertion `0x00000000 == vaSts' failed.
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg process is not running. exiting capture thread...
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55be6b752140] moov atom not found
/tmp/cache/oprit-20210125154021.mp4: Invalid data found when processing input
frigate.events                 INFO    : bad file: oprit-20210125154021.mp4
ffmpeg.oprit.detect            ERROR   : ffmpeg: /build/intel-mediasdk-UBsAHl/intel-mediasdk-20.1.0/_studio/shared/src/mfx_vpp_vaapi.cpp:239: mfxStatus MfxHwVideoProcessing::VAAPIVideoProcessing::Init(void**, mfxVideoParam*): Assertion `0x00000000 == vaSts' failed.
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : oprit: ffmpeg process is not running. exiting capture thread...

blakeblackshear commented 3 years ago

I'm not sure why that happens. A few things came up with a google search for /build/intel-mediasdk-UBsAHl/intel-mediasdk-20.1.0/_studio/shared/src/mfx_vpp_vaapi.cpp:239: mfxStatus MfxHwVideoProcessing::VAAPIVideoProcessing::Init(void**, mfxVideoParam*): Assertion '0x00000000 == vaSts' failed. Maybe I need to pin a specific version of the driver. I don't believe I can package the non-free driver in a container and distribute.

ekos2001 commented 3 years ago

Why do you think that non-free driver can not be included? As I understand, non-free means that there are proprietary binaries included into it, but didn't find any restrictions for distribution. Also you can try to search on github - a lot of people are using this driver.

davidvf commented 3 years ago

I have no idea how that works from a "legal" perspective. @ekos2001 did you try to rebuild your container as well using the non-free driver? Is that for you the solution as well?

For me it works flawless... My CPU usage is around 4~5% in the last 24hours, but there are also other containers running...

ekos2001 commented 3 years ago

Yes, I'm also using non-free driver. I made small docker file Dockerfile.non-free on top of frigate for amd64:

FROM blakeblackshear/frigate:0.8.0-rc3-amd64
RUN apt-get -qq update \
    && apt-get -qq remove -y intel-media-va-driver \
    && apt-get -qq install --no-install-recommends -y \
        intel-media-va-driver-non-free \
    && (apt-get autoremove -y; apt-get autoclean -y)

Then I built it docker build -f Dockerfile.non-free -t frigate-non-free .

By the way, to use HW acceleration with H.265 cameras, -c:v h264_qsv should be replaced with -c:v hevc_qsv

blakeblackshear commented 3 years ago

Why do you think that non-free driver can not be included?

I can't remember. I can update to just use the non-free driver instead.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ekos2001 commented 3 years ago

Will this non-free driver be added?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tungmeister commented 3 years ago

Any solution for this on unraid?

jokerigno commented 3 years ago

Waiting for this issue to be fixed (log full of error in my unraid container).

blakeblackshear / frigate

0.8.0rc3 >=Gen10 Hardware support #581