search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.1k stars 1.74k forks source link

[HW Accel Support]: NVIDIA HWAccel setup Proxmox --> LXC --> Docker #8652

Closed terrelsa13 closed 11 months ago

terrelsa13 commented 11 months ago

Describe the problem you are having

Anyone able to help get nvidia HWAccel setup in a LXC? I have followed the NVIDIA GPUs section, but cannot get it to work for me.

Yes I know running Docker in a Proxmox LXC is not recommended. But my significant other also recommends I do not setup any more machines running 24hrs. The power bill is getting ridiculous.

Both of these video cards are being used in LXCs with Emby and Jellyfin.

Proxmox Info:

root@hieroglyph:~# pveversion
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-19-pve)

LXC Info:

root@frigatelxc:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:    11
Codename:   bullseye

NVIDIA-Linux-x86_64-535.129.03.run installed on Proxmox.

root@hieroglyph:~# nvidia-smi 
Fri Nov 17 18:28:58 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P2000                   Off | 00000000:05:00.0 Off |                  N/A |
| 45%   32C    P8               7W /  75W |      6MiB /  5120MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4060        Off | 00000000:82:00.0 Off |                  N/A |
|  0%   30C    P8              N/A / 115W |      9MiB /  8188MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     51312      G   /usr/lib/xorg/Xorg                            4MiB |
|    1   N/A  N/A     51312      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+

NVIDIA-Linux-x86_64-535.129.03.run installed on frigatelxc.

root@frigatelxc:~# nvidia-smi 
Fri Nov 17 18:32:19 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P2000                   Off | 00000000:05:00.0 Off |                  N/A |
| 45%   33C    P8               5W /  75W |      6MiB /  5120MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4060        Off | 00000000:82:00.0 Off |                  N/A |
|  0%   30C    P8              N/A / 115W |      9MiB /  8188MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

NVIDIA-Linux-x86_64-535.129.03.run installed on Frigate Docker container.

root@21ffee2df543:/opt/frigate# nvidia-smi
Fri Nov 17 18:34:51 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P2000                   Off | 00000000:05:00.0 Off |                  N/A |
| 45%   33C    P8               5W /  75W |      6MiB /  5120MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4060        Off | 00000000:82:00.0 Off |                  N/A |
|  0%   30C    P8              N/A / 115W |      9MiB /  8188MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

nvidia-container-toolkit installed on frigatelxc.

root@frigatelxc:~# nvidia-ctk -v
NVIDIA Container Toolkit CLI version 1.14.3
commit: 53b24618a542025b108239fe602e66e912b7d6e2

nvidia-container-toolkit installed on Frigate Docker container.

root@21ffee2df543:/opt/frigate# nvidia-ctk -v
NVIDIA Container Toolkit CLI version 1.14.3
commit: 53b24618a542025b108239fe602e66e912b7d6e2

Docker daemon.json on frigatelxc.

root@frigatelxc:~# cat /etc/docker/daemon.json 
{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    },
    "storage-driver": "fuse-overlayfs"

Video card devices as seen from Proxmox.

root@hieroglyph:~# ls -al /dev/*nvidia*
crw-rw-rw- 1 root root 195,   0 Nov 17 08:19 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Nov 17 08:19 /dev/nvidia1
crw-rw-rw- 1 root root 195, 255 Nov 17 08:19 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Nov 17 08:19 /dev/nvidia-modeset
crw-rw-rw- 1 root root 238,   0 Nov 17 08:19 /dev/nvidia-uvm
crw-rw-rw- 1 root root 238,   1 Nov 17 08:19 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
drw-rw-rw-  2 root root     80 Nov 17 08:19 .
drwxr-xr-x 19 root root   6640 Nov 17 08:19 ..
cr--------  1 root root 241, 1 Nov 17 08:19 nvidia-cap1
cr--r--r--  1 root root 241, 2 Nov 17 08:19 nvidia-cap2

Video card devices as seen from frigatelxc.

root@frigatelxc:~# ls -al /dev/*nvidia*
crw-rw-rw- 1 root root 195, 254 Nov 10 21:18 /dev/nvidia-modeset
crw-rw-rw- 1 root root 235,   0 Nov 12 21:22 /dev/nvidia-uvm
crw-rw-rw- 1 root root 235,   1 Nov 16 18:58 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Nov 10 20:59 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Nov 10 20:59 /dev/nvidia1
crw-rw-rw- 1 root root 195, 255 Nov 10 21:18 /dev/nvidiactl

/dev/nvidia-caps:
total 1
drwxr-xr-x 2 root root     80 Nov 17 16:09 .
drwxr-xr-x 8 root root    700 Nov 17 16:09 ..
cr-------- 1 root root 240, 1 Nov 16 18:57 nvidia-cap1
cr--r--r-- 1 root root 240, 2 Nov 16 18:57 nvidia-cap2

Video card devices as seen from Frigate Docker.

root@21ffee2df543:/opt/frigate# ls -al /dev/*nvidia*
crw-rw-rw- 1 root root 195, 254 Nov 10 21:18 /dev/nvidia-modeset
crw-rw-rw- 1 root root 235,   0 Nov 12 21:22 /dev/nvidia-uvm
crw-rw-rw- 1 root root 235,   1 Nov 16 18:58 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Nov 10 20:59 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Nov 10 20:59 /dev/nvidia1
crw-rw-rw- 1 root root 195, 255 Nov 10 21:18 /dev/nvidiactl

/dev/nvidia-caps:
total 1
drwxr-xr-x 2 root root     80 Nov 17 18:48 .
drwxr-xr-x 6 root root    520 Nov 17 18:48 ..
cr-------- 1 root root 240, 1 Nov 16 18:57 nvidia-cap1
cr--r--r-- 1 root root 240, 2 Nov 16 18:57 nvidia-cap2

Configuration of frigatelxc.

root@hieroglyph:~# cat /etc/pve/lxc/2051.conf 
arch: amd64
cores: 16
features: fuse=1,keyctl=1,nesting=1
hostname: frigatelxc
memory: 16384
mp100: /media/osiris/backups/frigate/app,mp=/media/osiris/backups/frigate/app
mp101: /media/osiris/backups/frigate/rsync,mp=/media/osiris/backups/frigate/rsync
mp102: /media/osiris/nvr/frigate,mp=/media/osiris/nvr/frigate
mp103: /media/amunra,mp=/media/amunra
net0: name=eth0.16,bridge=vmbr0,firewall=1,hwaddr=0e:99:87:1f:31:21,ip=dhcp,tag=16,type=veth
onboot: 1
ostype: debian
rootfs: nefertiti:subvol-2051-disk-0,size=16G
startup: order=1,up=5
swap: 0
unprivileged: 1
lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 2000 1
lxc.idmap: g 1000 2000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535
lxc.mount.auto: cgroup:rw
lxc.cgroup2.devices.allow: c 120:0 rwm
lxc.cgroup2.devices.allow: c 120:1 rwm
lxc.mount.entry: /lxc/102010/devices/apex_0 dev/apex_0 none bind,optional,create=file
lxc.mount.entry: /lxc/102010/devices/apex_1 dev/apex_1 none bind,optional,create=file
lxc.cgroup2.devices.allow: c 195:0 rwm
lxc.cgroup2.devices.allow: c 195:1 rwm
lxc.mount.entry: /lxc/102010/devices/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /lxc/102010/devices/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.cgroup2.devices.allow: c 195:255 rwm
lxc.cgroup2.devices.allow: c 195:254 rwm
lxc.mount.entry: /lxc/102010/devices/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /lxc/102010/devices/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.cgroup2.devices.allow: c 235:0 rwm
lxc.cgroup2.devices.allow: c 235:1 rwm
lxc.mount.entry: /lxc/102010/devices/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /lxc/102010/devices/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.cgroup2.devices.allow: c 240:1 rwm
lxc.cgroup2.devices.allow: c 240:2 rwm
lxc.mount.entry: /lxc/102010/devices/nvidia-cap1 dev/nvidia-caps/nvidia-cap1 none bind,optional,create=file
lxc.mount.entry: /lxc/102010/devices/nvidia-cap2 dev/nvidia-caps/nvidia-cap2 none bind,optional,create=file
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:1 rwm
lxc.cgroup2.devices.allow: c 226:2 rwm
lxc.mount.entry: /lxc/102010/devices/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /lxc/102010/devices/card1 dev/dri/card1 none bind,optional,create=file
lxc.mount.entry: /lxc/102010/devices/card2 dev/dri/card2 none bind,optional,create=file
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 226:129 rwm
lxc.mount.entry: /lxc/102010/devices/renderD129 dev/dri/renderD129 none bind,optional,create=file
lxc.mount.entry: /lxc/102010/devices/renderD128 dev/dri/renderD128 none bind,optional,create=file

Version

0.12.1-367d724

Frigate config file

mqtt:
  enabled: false

#ffmpeg:
  #hwaccel_args: preset-nvidia-h264

detectors:
  coral1:
    type: edgetpu
    device: pci:0
  coral2:
    type: edgetpu
    device: pci:1

go2rtc:
  streams:
    cam0: 
      - rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@172.31.16.120:554/cam/realmonitor?channel=1&subtype=0
    cam0sub:
      - rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@172.31.16.120:554/cam/realmonitor?channel=1&subtype=2
    cam1: 
      - rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@172.31.16.121:554/cam/realmonitor?channel=1&subtype=0
    cam1sub:
      - rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@172.31.16.121:554/cam/realmonitor?channel=1&subtype=2

cameras:
  cam0:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam0sub
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/cam0
          roles:
            - record
    detect:
      # Optional: enables detection for the camera (default: True)
      enabled: True
      # Optional: desired fps for your camera for the input with the detect role (default: shown below)
      # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
      fps: 6
      # Optional: Number of frames without a detection before Frigate considers an object to be gone. (default: 5x the frame rate)
      max_disappeared: 30

    # Optional: birdseye configuration
    # NOTE: Can (enabled, mode) be overridden at the camera level
    birdseye:
      # Optional: Enable birdseye view (default: shown below)
      enabled: true
      # Optional: Restream birdseye via RTSP (default: shown below)
      # NOTE: Enabling this will set birdseye to run 24/7 which may increase CPU usage somewhat.
      #restream: True
      restream: false
      # Optional: Mode of the view. Available options are: objects, motion, and continuous
      #   objects - cameras are included if they have had a tracked object within the last 30 seconds
      #   motion - cameras are included if motion was detected in the last 30 seconds
      #   continuous - all cameras are included always
      mode: continuous

  cam1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cam1sub
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/cam1
          roles:
            - record
    detect:
      # Optional: enables detection for the camera (default: True)
      enabled: True
      # Optional: desired fps for your camera for the input with the detect role (default: shown below)
      # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
      fps: 6
      # Optional: Number of frames without a detection before Frigate considers an object to be gone. (default: 5x the frame rate)
      max_disappeared: 30

    # Optional: birdseye configuration
    # NOTE: Can (enabled, mode) be overridden at the camera level
    birdseye:
      # Optional: Enable birdseye view (default: shown below)
      enabled: true
      # Optional: Restream birdseye via RTSP (default: shown below)
      # NOTE: Enabling this will set birdseye to run 24/7 which may increase CPU usage somewhat.
      #restream: True
      restream: false
      # Optional: Mode of the view. Available options are: objects, motion, and continuous
      #   objects - cameras are included if they have had a tracked object within the last 30 seconds
      #   motion - cameras are included if motion was detected in the last 30 seconds
      #   continuous - all cameras are included always
      mode: continuous

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: false # this may not be necessary for all setups
    restart: unless-stopped
    #image: ghcr.io/blakeblackshear/frigate:stable
    image: frigate_nvidia_drivers_toolkit #<---Latest stable Frigate image with nvidia drivers and toolkit
    shm_size: "1024mb" # update for your cameras based on calculation above
    devices:
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/apex_1:/dev/apex_1 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/nvidia0:/dev/nvidia0
      - /dev/nvidia1:/dev/nvidia1
      - /dev/nvidia-modeset
      - /dev/nvidiactl:/dev/nvidiactl
      - /dev/nvidia-uvm
      - /dev/nvidia-uvm-tools
      - /dev/nvidia-caps/nvidia-cap1
      - /dev/nvidia-caps/nvidia-cap2
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/config/config.yaml:/config/config.yml
      - /media/osiris/nvr/frigate:/media/frigate
      - /media/amunra:/media/amunra
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
    environment:
      - FRIGATE_RTSP_USER=admin
      - FRIGATE_RTSP_PASSWORD=abcd1234
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
    #runtime: nvidia
    #deploy:    # <------------- Add this section
      #resources:
        #reservations:
          #devices:
            #- driver: nvidia
              #device_ids: ['0','1'] # this is only needed when using multiple GPUs
              #capabilities: [gpu]

Relevant log output

See other information.

FFprobe output from your camera

root@8ade19e6f9a3:/opt/frigate# ffprobe rtsp://admin:abcd1234@172.31.16.120:554
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://admin:abcd1234@172.31.16.120:554':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.001000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 3840x2160 [SAR 7713:7712 DAR 857:482], 15 fps, 100 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
root@8ade19e6f9a3:/opt/frigate# ffprobe rtsp://admin:abcd1234@172.31.16.121:554
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://admin:abcd1234@172.31.16.121:554':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.130000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 3840x2160 [SAR 7713:7712 DAR 857:482], 15 fps, 100 tbr, 90k tbn

Operating system

Proxmox

Install method

Docker Compose

Network connection

Wired

Camera make and model

Amcrest IP8M-T2599EW, Amcrest IP8M-2496EB-28MM

Any other information that may be helpful

With both of the sections below commented out, the Frigate Docker container starts perfectly fine in Portainer.

    #runtime: nvidia
    #deploy:    # <------------- Add this section
      #resources:
        #reservations:
          #devices:
            #- driver: nvidia
              #device_ids: ['0','1'] # this is only needed when using multiple GPUs
              #capabilities: [gpu]

With either or both of the sections un-commented...

    runtime: nvidia
    deploy:    # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0','1'] # this is only needed when using multiple GPUs
              capabilities: [gpu]

I get this error in Portainer.

Failure failed to deploy a stack: Container frigate Recreate Container frigate Recreated Container frigate Starting Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown

I have also tried using Docker CLI...

root@frigatelxc:~# docker run -d --name frigate --restart=unless-stopped --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 --gpus=all --device /dev/apex_0:/dev/apex_0 --device /dev/apex_1:/dev/apex_1 --device /dev/nvidia0:/dev/nvidia0 --device /dev/nvidia1:/dev/nvidia1 --device /dev/nvidia-modeset --device /dev/nvidiactl:/dev/nvidiactl --device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools --device /dev/nvidia-caps/nvidia-cap1 --device /dev/nvidia-caps/nvidia-cap2 --shm-size=1024m -v /media/amunra:/media/amumra -v /media/osiris/nvr/frigate:/media/frigate -v /opt/frigate/config/config.yaml:/config/config.yml -v /etc/localtime:/etc/localtime:ro -e FRIGATE_RTSP_USER='admin' -e FRIGATE_RTSP_PASSWORD='abcd1234' -e NVIDIA_VISIBLE_DEVICES='all' -e NVIDIA_DRIVER_CAPABILITIES='compute,utility,video' -p 5000:5000 -p 8554:8554 frigate_nvidia_drivers_toolkit
235b0e5e9d74fed9603cce8c2fc70e05490564d77334f5bb8a3905dacec66acb

And get a similar error.

Error docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown.

Cannot get past starting the Frigate Docker container with nvidia gpu options enabled.

terrelsa13 commented 11 months ago

Nevermind, found an answer here that seems to work.

In the frigatelxc, I set no-cgroups=true in the /etc/nvidia-container-runtime/config.toml. Then ran nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml.