search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.77k stars 1.7k forks source link

Best way to enable and verify Intel 10th gen hwaccell in proxmox/lxc #930

Closed erikarenhill closed 3 years ago

erikarenhill commented 3 years ago

Describe the bug Problems enabling and verifying hwaccel on Intel NUC 10th gen (QuickSync) running frigate in lxc in proxmox

Version of frigate 0.8.4-5043040

Config file Include your full config file wrapped in triple back ticks.

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - qsv
    - -qsv_device
    - /dev/dri/renderD128

Computer Hardware

Camera Info:

ffprobe from a reolink 810 (4k/8MP) camera with detect role

  Metadata:
    title           : Session streamed by "preview"
    comment         : h265Preview_01_main
  Duration: N/A, start: 0.000250, bitrate: N/A
    Stream #0:0: Video: hevc, yuv420p(tv), 3840x2160, 90k tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac, 16000 Hz, mono, fltp

ffprobe from a hikvision DS-2CD2143G0-IS (4MP) camera with detect role

  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 2688x1520 [SAR 1:1 DAR 168:95], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1: Audio: pcm_mulaw, 8000 Hz, 1 channels, s16, 64 kb/s

Additional context Currently my frigate instance is heavily hogging my CPU and I want to enable QuickSync on proxmox to frigate running on a debian 10 LXC

Starting the docker-compose with following parts of config:

    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    environment:
       LIBVA_DRIVER_NAME: iHD

vainfo output from lxc:

root@frigatelxc:~# vainfo
error: can't connect to X server!
libva info: VA-API version 1.11.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_11
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.11 (libva 2.4.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 21.1.3 (d2d3542c)
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointFEI
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD

vainfo from docker-container:

root@frigatelxc:~# docker exec -it frigate vainfo
error: XDG_RUNTIME_DIR not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.7.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD

lxc config

arch: amd64
cores: 4
cpulimit: 50
features: keyctl=1,nesting=1
hostname: frigatelxc
memory: 6144
mp0: local-lvm:vm-105-disk-1,mp=/mnt/frigate,backup=1,size=100G
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=46:B2:28:DE:C8:79,ip=192.168.1.15/24,type=veth
onboot: 1
ostype: debian
parent: standalone
rootfs: local-lvm:vm-105-disk-0,size=52G
swap: 8192
lxc.cgroup.devices.allow: c 189:385 rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/bus/usb/002/002 dev/bus/usb/002/002 none bind,optional,create=file
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:

I have installed intel-gpu-tools in proxmox os/host to try and verify if it's using the gpu but it shows 0% meanwhile top show high cpu usage for ffmpeg processes ( 50-100% )

Found a guide how to enable vfio passthrough by adding the pci device to modprobe but not sure how to proceed here (after doing this my /dev/dri/renderD128 device disappeared and intel_gpu_top showed 100%), I enabled intel_iommu=on at /etc/default/grub and rebooted but now I just feel lost.

Any pointers how to proceed would be very appreciated because currently my CPU runs at 70+ degrees celsius because of this

blakeblackshear commented 3 years ago

Without being able to reproduce your hardware, I'm not sure what to suggest over what you can find on stack overflow for ffmpeg. Maybe someone with similar hardware can chime in.

Is it really necessary to run detection on high resolution, high fps streams for you? It's always going to take a lot of processing power to decode that stream even if you weren't running anything except ffmpeg. Unless you are trying detect very small fast moving objects you shouldn't see any benefit over a lower resolution feed at a lower fps.

erikarenhill commented 3 years ago

Hi @blakeblackshear and thanks for the quick response.

The reason for running hi-res streams is because I want the snapshot images to be of high resolution to run face recognition on them through another software, and based on the full image a face could potentially be quite small. Is it possible to create the snapshots from another stream than detect/rtmp?

blakeblackshear commented 3 years ago

No. Only the stream used for detection is decoded. You need to decode the stream to grab an image. Can you lower the fps to something like 5 on the camera?

erikarenhill commented 3 years ago

I have FPS: 5 in my frigate config, but lowering the FPS in the cameras own configuration made the cpu go down quite a bit. Thanks for the tip! Optimally I'd like to run detection on low fps but record clips with higher fps, but I suppose that might be hard to achieve.

I hope someone with similiar hardware jumps in and gives some advice on how to enable and verify the hwaccel

erikarenhill commented 3 years ago

I managed to passthrough the GPU by using vfio to an Ubuntu VM (not lxc), so I will setup frigate there and see how it works. The monitor attached to the physical hdmi on the NUC displayed the screen from the VM so I suppose its working that far. Will add updates here if anyone else with similiar hardware needs to do the same if the rest succeeds.

HIT2022 commented 3 years ago

I'm also passing the Iris GPU through to an Ubuntu VM on a NUC8i7BEH. I believe vfio as well, but I can't recall exactly what steps/drivers were involved. I remember it being a bit of a pain, but not near as much of a pain as it was to pass the PCIE coral device through to the VM. At any rate, just wanted to chime in with the args that work for me -

ffmpeg:
hwaccel_args:
   - '-hwaccel'
   - vaapi
   - '-qsv_device'
   - /dev/dri/renderD128`
erikarenhill commented 3 years ago

I'm also passing the Iris GPU through to an Ubuntu VM on a NUC8i7BEH. I believe vfio as well, but I can't recall exactly what steps/drivers were involved. I remember it being a bit of a pain, but not near as much of a pain as it was to pass the PCIE coral device through to the VM. At any rate, just wanted to chime in with the args that work for me -

ffmpeg:
hwaccel_args:
   - '-hwaccel'
   - vaapi
   - '-qsv_device'
   - /dev/dri/renderD128`

Thanks for your feedback, I tested with vaapi instead of qsv but that increased the CPU usage of my ffmpeg processes with about 2.5x but I'm on 10th gen so it might differ. From what I understand 10th gen and newer has another type of driver needed for QuickSync.

These two links helped me with vfio and intel-media-driver installation:

https://forum.proxmox.com/threads/guide-intel-intergrated-graphic-passthrough.30451/

https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-focal.html