search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.4k stars 1.78k forks source link

[HW Accel Support]: Host machine/container cuda version mismatch. #6092

Closed csreynolds closed 1 year ago

csreynolds commented 1 year ago

Describe the problem you are having

My host machine has cuda12 installed and the container has 11.7 listed in requirements-tensorrt.txt

Pretty sure cuda is backwards compatible so a bump shouldn't break anything.

Produces the log below:

2023-04-15 09:35:28.398163981  [2023-04-15 09:35:28] frigate.detectors.plugins.tensorrt ERROR   : ERROR: failed to load libraries. libcudart.so.12: cannot open shared object file: No such file or directory

this is the file present inside the container: 

root@160c9873dfa9:/opt/frigate# find / | grep libcudart
/usr/local/lib/python3.9/dist-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0

my system info

[creynolds@masterblaster frigate]$ uname -a
Linux masterblaster 6.2.9-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 30 22:31:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[creynolds@masterblaster frigate]$ cat /etc/redhat-release
Fedora release 37 (Thirty Seven)
[creynolds@masterblaster frigate]$ nvidia-smi
Sat Apr 15 09:48:37 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 530.41.03    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P2000                    Off| 00000000:01:00.0 Off |                  N/A |
| 45%   31C    P8                4W /  75W|      2MiB /  5120MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

### Version

0.12.0-da3e197

### Frigate config file

```yaml
mqtt:
  host: mqtt
database:
  path: /db/frigate.db

detectors:
  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /trt-models/yolov7x-640.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 640
  height: 480

birdseye:
  enabled: True
  restream: False
  width: 1280
  height: 720
  quality: 8
  mode: continuous

detect:
  width: 1280
  height: 720
  fps: 15

record:
  enabled: True
  retain:
    days: 0
  events:
    pre_capture: 8
    post_capture: 8
    retain:
      default: 30

snapshots:
  enabled: True

go2rtc:
  streams:
    camera1:
      - rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:camera1#video=h264"
    camera1_sub:
      - rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=1
    camera2:
      - rtsp://viewonly:viewonly12@192.168.7.182:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:camera2#video=h264"
    camera2_sub:
      - rtsp://viewonly:viewonly12@192.168.7.182:554/cam/realmonitor?channel=1&subtype=1
    camera3:
      - rtsp://viewonly:viewonly12@192.168.7.183:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:camera3#video=h264"
    camera3_sub:
      - rtsp://viewonly:viewonly12@192.168.7.183:554/cam/realmonitor?channel=1&subtype=1
    camera4:
      - rtsp://viewonly:viewonly12@192.168.7.184:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:camera4#video=h264"
    camera4_sub:
      - rtsp://viewonly:viewonly12@192.168.7.184:554/cam/realmonitor?channel=1&subtype=1
  webrtc:
    candidates:
      - 192.168.7.201:8555
      - stun:8555

cameras:
  camera1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera1
          input_args: preset-rtsp-restream
          roles:
            - record
            #- detect
        - path: rtsp://127.0.0.1:8554/camera1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
  camera2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera2
          input_args: preset-rtsp-restream
          roles:
            - record
            #- detect
        - path: rtsp://127.0.0.1:8554/camera2_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
  camera3:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera3
          input_args: preset-rtsp-restream
          roles:
            - record
            #- detect
        - path: rtsp://127.0.0.1:8554/camera3_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
  camera4:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera4
          input_args: preset-rtsp-restream
          roles:
            - record
            #- detect
        - path: rtsp://127.0.0.1:8554/camera4_sub
          input_args: preset-rtsp-restream
          roles:
            - detect

docker-compose file or Docker CLI command

version: "3"
services:
  frigate:
    container_name: frigate
    # add groups from host for render, plugdev, video
    group_add:
      - "998" # render
      - "39"  # video
    shm_size: "1024mb"
    image: ghcr.io/blakeblackshear/frigate:0.12.0-tensorrt
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=all
    devices:
      #      - /dev/dri:/dev/dri
      - /dev/nvidia0:/dev/nvidia0
      - /dev/nvidiactl:/dev/nvidiactl
      - /dev/nvidia-modeset:/dev/nvidia-modeset
      - /dev/nvidia-uvm:/dev/nvidia-uvm
      - /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri:/dev/dri
    deploy:
          resources:
              reservations:
                  devices:
                      - driver: nvidia
                        count: 1
                        capabilities: [gpu]
    privileged: true
    ports:
       - '5000:5000/tcp'
       - '5001:5001/tcp'
    volumes:
      - .:/workspace/frigate:cached
      - ./web/dist:/opt/frigate/web:cached
      - /etc/localtime:/etc/localtime:ro
      - ./config/:/config/:ro
      - ./debug:/media/frigate
      - ./db/:/db/
      - ./trt-models:/trt-models
      - /dev/bus/usb:/dev/bus/usb
  mqtt:
    container_name: mqtt
    image: eclipse-mosquitto:1.6
    ports:
      - "1883:1883"

Relevant log output

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service go2rtc-healthcheck successfully started
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2023-04-15 09:35:24.474028731  [INFO] Preparing go2rtc config...
2023-04-15 09:35:24.474069947  [INFO] Starting Frigate...
2023-04-15 09:35:24.474677048  [INFO] Starting NGINX...
s6-rc: info: service legacy-services successfully started
2023-04-15 09:35:25.421057805  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2023-04-15 09:35:25.466178842  [INFO] Starting go2rtc...
2023-04-15 09:35:25.585350853  09:35:25.584 INF go2rtc version 1.2.0 linux/amd64
2023-04-15 09:35:25.585573528  09:35:25.585 INF [api] listen addr=:1984
2023-04-15 09:35:25.586033460  09:35:25.585 INF [rtsp] listen addr=:8554
2023-04-15 09:35:25.586615313  09:35:25.586 INF [srtp] listen addr=:8443
2023-04-15 09:35:25.586987129  09:35:25.586 INF [webrtc] listen addr=:8555
2023-04-15 09:35:27.948783263  [2023-04-15 09:35:27] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-04-15 09:35:28.016844500  [2023-04-15 09:35:28] peewee_migrate                 INFO    : Starting migrations
2023-04-15 09:35:28.069571516  [2023-04-15 09:35:28] peewee_migrate                 INFO    : There is nothing to migrate
2023-04-15 09:35:28.120357245  [2023-04-15 09:35:28] frigate.app                    INFO    : Output process started: 441
2023-04-15 09:35:28.130203777  [2023-04-15 09:35:28] frigate.app                    INFO    : Camera processor started for camera1: 444
2023-04-15 09:35:28.140065643  [2023-04-15 09:35:28] frigate.app                    INFO    : Camera processor started for camera2: 445
2023-04-15 09:35:28.151050113  [2023-04-15 09:35:28] frigate.app                    INFO    : Camera processor started for camera3: 447
2023-04-15 09:35:28.152124504  [2023-04-15 09:35:28] detector.tensorrt              INFO    : Starting detection process: 439
2023-04-15 09:35:28.161122272  [2023-04-15 09:35:28] frigate.app                    INFO    : Camera processor started for camera4: 449
2023-04-15 09:35:28.176210619  [2023-04-15 09:35:28] frigate.app                    INFO    : Capture process started for camera1: 450
2023-04-15 09:35:28.187715454  [2023-04-15 09:35:28] frigate.app                    INFO    : Capture process started for camera2: 454
2023-04-15 09:35:28.224273371  [2023-04-15 09:35:28] frigate.app                    INFO    : Capture process started for camera3: 459
2023-04-15 09:35:28.257211216  [2023-04-15 09:35:28] frigate.app                    INFO    : Capture process started for camera4: 467
2023-04-15 09:35:28.398163981  [2023-04-15 09:35:28] frigate.detectors.plugins.tensorrt ERROR   : ERROR: failed to load libraries. libcudart.so.12: cannot open shared object file: No such file or directory
2023-04-15 09:35:29.565415398  [2023-04-15 09:35:29] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 132 (MiB)
2023-04-15 09:35:30.175258220  Process camera_processor:camera2:
2023-04-15 09:35:30.183258956  Traceback (most recent call last):
2023-04-15 09:35:30.183265741    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 09:35:30.183267707      self.run()
2023-04-15 09:35:30.183269993    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 09:35:30.183275489      self._target(*self._args, **self._kwargs)
2023-04-15 09:35:30.183277555    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-04-15 09:35:30.183304188      process_frames(
2023-04-15 09:35:30.183306224    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-04-15 09:35:30.183308433      detect(
2023-04-15 09:35:30.183310221    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-04-15 09:35:30.183312136      region_detections = object_detector.detect(tensor_input)
2023-04-15 09:35:30.183313963    File "/opt/frigate/frigate/object_detection.py", line 208, in detect
2023-04-15 09:35:30.183343166      self.np_shm[:] = tensor_input[:]
2023-04-15 09:35:30.183345564  ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
2023-04-15 09:35:30.200231636  Process camera_processor:camera4:
2023-04-15 09:35:30.201536917  Traceback (most recent call last):
2023-04-15 09:35:30.201779862    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 09:35:30.201782797      self.run()
2023-04-15 09:35:30.201923154    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 09:35:30.201925922      self._target(*self._args, **self._kwargs)
2023-04-15 09:35:30.202043936    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-04-15 09:35:30.202046225      process_frames(
2023-04-15 09:35:30.202174690    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-04-15 09:35:30.202176791      detect(
2023-04-15 09:35:30.202306790    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-04-15 09:35:30.202309232      region_detections = object_detector.detect(tensor_input)
2023-04-15 09:35:30.202449917    File "/opt/frigate/frigate/object_detection.py", line 208, in detect
2023-04-15 09:35:30.202452628      self.np_shm[:] = tensor_input[:]
2023-04-15 09:35:30.202609751  ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
2023-04-15 09:35:30.287129708  Process camera_processor:camera1:
2023-04-15 09:35:30.289133575  Traceback (most recent call last):
2023-04-15 09:35:30.289495419    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 09:35:30.289498810      self.run()
2023-04-15 09:35:30.289677821    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 09:35:30.289680734      self._target(*self._args, **self._kwargs)
2023-04-15 09:35:30.289816025    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-04-15 09:35:30.289818605      process_frames(
2023-04-15 09:35:30.289955394    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-04-15 09:35:30.289957788      detect(
2023-04-15 09:35:30.290086416    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-04-15 09:35:30.290089061      region_detections = object_detector.detect(tensor_input)
2023-04-15 09:35:30.290232236    File "/opt/frigate/frigate/object_detection.py", line 208, in detect
2023-04-15 09:35:30.290234801      self.np_shm[:] = tensor_input[:]
2023-04-15 09:35:30.290401080  ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
2023-04-15 09:35:30.369906527  Process camera_processor:camera3:
2023-04-15 09:35:30.371179709  Traceback (most recent call last):
2023-04-15 09:35:30.371332794    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 09:35:30.371335671      self.run()
2023-04-15 09:35:30.371394953    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 09:35:30.371397665      self._target(*self._args, **self._kwargs)
2023-04-15 09:35:30.371449085    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-04-15 09:35:30.371451429      process_frames(
2023-04-15 09:35:30.371504476    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-04-15 09:35:30.371506496      detect(
2023-04-15 09:35:30.371549675    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-04-15 09:35:30.371552363      region_detections = object_detector.detect(tensor_input)
2023-04-15 09:35:30.371602921    File "/opt/frigate/frigate/object_detection.py", line 208, in detect
2023-04-15 09:35:30.371605359      self.np_shm[:] = tensor_input[:]
2023-04-15 09:35:30.371678946  ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
2023-04-15 09:35:32.459394714  [2023-04-15 09:35:32] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 328 MiB

FFprobe output from your camera

root@160c9873dfa9:/opt/frigate# ffprobe "rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=0"
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.138000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuvj420p(pc), 3840x2160, 15 fps, 100 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 8000 Hz, mono, fltp
root@160c9873dfa9:/opt/frigate# ffprobe "rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=1"
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=1':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.128000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuvj420p(pc), 704x480, 15 fps, 100 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 8000 Hz, mono, fltp
root@160c9873dfa9:/opt/frigate#

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Amcrest https://www.amazon.com/dp/B07RHQGS8V/

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

Cc @NateMeyer

NateMeyer commented 1 year ago

What is your config file? Looks like the model dimensions are off.

Nevermind, I am blind.

csreynolds commented 1 year ago

What is your config file? Looks like the model dimensions are off.

Nevermind, I am blind.

All good. I should have added that if I change detectors to cpu it runs fine.

It's really just the cuda version present/expected mismatch that is stopping the gpu from being used.

edit: forgot the word cuda

NickM-27 commented 1 year ago

The model size is off though, it should be a square (640x640) not 640x480

csreynolds commented 1 year ago

ooo Hot tip! This is my first stab at getting this going, forgive my ignorance config side.

I've changed it to the following but still errors out before it even loads the model because of the cannot find libcudart.so.12 

detectors:
  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /trt-models/yolov7x-640.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 640
  height: 640
NateMeyer commented 1 year ago

I've run this with with Cuda 12.0 on my machine. I'll update to 12.1 and confirm it still works.

NateMeyer commented 1 year ago

Works on my machine with Cuda 12.1

Sat Apr 15 11:23:19 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.46                 Driver Version: 531.61       CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1080 Ti      On | 00000000:26:00.0  On |                  N/A |
| 37%   42C    P0               66W / 300W|   1033MiB / 11264MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        20      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        20      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        22      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

And the tensorrt detector seems to load without issue.


frigate-tensor-frigate-1  | 2023-04-15 11:18:13.207724731  [2023-04-15 11:18:13] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
frigate-tensor-frigate-1  | 2023-04-15 11:18:13.223706414  [2023-04-15 11:18:13] frigate.app                    INFO    : Creating directory: /tmp/cache
frigate-tensor-frigate-1  | 2023-04-15 11:18:13.225429917  [2023-04-15 11:18:13] peewee_migrate                 INFO    : Starting migrations
frigate-tensor-frigate-1  | 2023-04-15 11:18:13.242344340  [2023-04-15 11:18:13] peewee_migrate                 INFO    : There is nothing to migrate
frigate-tensor-frigate-1  | 2023-04-15 11:18:13.252592370  [2023-04-15 11:18:13] frigate.app                    INFO    : Output process started: 1012
frigate-tensor-frigate-1  | 2023-04-15 11:18:13.255297902  [2023-04-15 11:18:13] frigate.app                    INFO    : Camera processor started for front_door: 1016
frigate-tensor-frigate-1  | 2023-04-15 11:18:13.259829838  [2023-04-15 11:18:13] frigate.app                    INFO    : Capture process started for front_door: 1017
frigate-tensor-frigate-1  | 2023-04-15 11:18:13.675196273  [2023-04-15 11:18:13] detector.tensorrt              INFO    : Starting detection process: 1010
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.050631486  [2023-04-15 11:18:13] frigate.detectors.plugins.tensorrt DEBUG   : Num Available Devices: 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.240346821  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchor_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.264781621  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchorRect_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.264968439  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMS_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265023879  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Reorg_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265088342  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Region_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265117792  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Clip_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265167981  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LReLU_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265229074  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PriorBox_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265282080  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Normalize_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265330376  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ScatterND version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265386113  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::RPROI_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265409563  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMS_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265458414  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265513928  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchTilePlugin_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265571943  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::FlattenConcat_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265627332  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResize version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265667431  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResizeDynamic version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265724901  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DetectionLayer_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265772981  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265828185  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265876289  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.265927539  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266012016  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalDynamic version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266067606  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Proposal version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266111787  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalLayer_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266163947  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PyramidROIAlign_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266234391  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ResizeNearest_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266257260  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Split version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266317320  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SpecialSlice_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266400034  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266418884  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 2
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266471782  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CoordConvAC version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266499212  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DecodeBbox3DPlugin version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266549594  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GenerateDetection_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266602132  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266657952  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelProposeROI_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266727919  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMSDynamic_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266791470  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PillarScatterPlugin version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266867739  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::VoxelGeneratorPlugin version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266939599  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.266985218  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 1140 (MiB)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.284277268  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.539384998  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as a tactic source
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.539468007  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +194, GPU +74, now: CPU 501, GPU 1250 (MiB)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.539698935  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as a tactic source
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.646830924  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +111, GPU +44, now: CPU 612, GPU 1294 (MiB)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.653447819  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Deserialization required 369097 microseconds.
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.659505728  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660080924  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as a tactic source
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660146422  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 1286 (MiB)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660203452  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as a tactic source
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660256341  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 1294 (MiB)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660310521  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner device persistent memory is 570880
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660361301  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner host persistent memory is 102816
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660416590  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Allocated activation device memory of size 12979200
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660476899  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660533488  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding input Memory 2076672 Bytes (519168 * DataType.FLOAT)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660612578  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Input has Shape (1, 3, 416, 416)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660663787  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding detections Memory 298116 Bytes (74529 * DataType.FLOAT)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660716047  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : Output has Shape (1, 74529, 1, 1)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660784456  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT loaded. Input shape is ((416, 416), <class 'numpy.float32'>)
frigate-tensor-frigate-1  | 2023-04-15 11:18:14.660804616  [2023-04-15 11:18:14] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT version is 8

What version of the NVidia Container runtime do you have installed?

$ nvidia-container-runtime-hook --version
NVIDIA Container Runtime Hook version 1.13.0
commit: b7079454b5b8fed1390ce78ca5a3343748f62657
NateMeyer commented 1 year ago

Do you have the libcudart installed on your host?

csreynolds commented 1 year ago

My nvidia-container-runtime-hook was way out of date(1.3.0). I've updated it to the following version which did not fix the problem. Would have been a nice easy win there. 

[creynolds@masterblaster docker-compose]$ nvidia-container-runtime-hook --version
NVIDIA Container Runtime Hook version 1.13.0
commit: b7079454b5b8fed1390ce78ca5a3343748f62657

I do not see a libcudart package available but I do have these installed

[creynolds@masterblaster docker-compose]$ rpm -qa | grep cudart
cuda-cudart-12.1.55-1.fc37.x86_64
cuda-cudart-devel-12.1.55-1.fc37.x86_64
cuda-cudart-static-12.1.55-1.fc37.x86_64

edit: add contents of cuda-cudart package

[creynolds@masterblaster docker-compose]$ rpm -qlv cuda-cudart
lrwxrwxrwx    1 root     root                       20 Apr 11 07:09 /usr/lib64/libcudart.so.12 -> libcudart.so.12.1.55
-rwxr-xr-x    1 root     root                   679264 Apr 11 07:09 /usr/lib64/libcudart.so.12.1.55
drwxr-xr-x    2 root     root                        0 Apr 11 07:09 /usr/share/licenses/cuda-cudart
-rw-r--r--    1 root     root                    61498 Jan 24 23:09 /usr/share/licenses/cuda-cudart/LICENSE
NateMeyer commented 1 year ago

Not sure if it matters, but maybe try changing the "capabilities" in your docker-compose.yml? Here is what I use.

deploy:
      resources:
          reservations:
              devices:
                  - driver: nvidia
                    count: 1
                    capabilities: [video,compute,utility]
NateMeyer commented 1 year ago

Also, can you try to run with the debug logging enabled? Maybe we're going down the wrong path.

Add this to your config.yml

 logger:
     logs:
         frigate.detectors.plugins.tensorrt: debug
csreynolds commented 1 year ago

Ok, i've updated my compose file to match the suggested deploy section and added the logger line to the config.yml. this is the resulting log.

Thanks for taking time to look at this. I will try directly mapping in the libcudart.12.so from the host machine into the container and see what happens later tonight when I can sit at the computer again. Not sure why frigate would be looking for .12 when it has .11 available.

edit: this is the system working when I change the detector to cpu: https://i.imgur.com/kh2qjvF.jpg

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service go2rtc-healthcheck successfully started
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2023-04-15 15:31:56.037356878  [INFO] Starting NGINX...
2023-04-15 15:31:56.037359844  [INFO] Preparing go2rtc config...
2023-04-15 15:31:56.037429924  [INFO] Starting Frigate...
s6-rc: info: service legacy-services successfully started
2023-04-15 15:31:56.283015169  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2023-04-15 15:31:56.325714165  [INFO] Starting go2rtc...
2023-04-15 15:31:56.452139485  15:31:56.452 INF go2rtc version 1.2.0 linux/amd64
2023-04-15 15:31:56.452405553  15:31:56.452 INF [api] listen addr=:1984
2023-04-15 15:31:56.452753543  15:31:56.452 INF [rtsp] listen addr=:8554
2023-04-15 15:31:56.453104726  15:31:56.453 INF [srtp] listen addr=:8443
2023-04-15 15:31:56.453389402  15:31:56.453 INF [webrtc] listen addr=:8555
2023-04-15 15:31:57.415857970  [2023-04-15 15:31:57] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-04-15 15:31:57.452201170  [2023-04-15 15:31:57] frigate.app                    INFO    : Creating directory: /tmp/cache
2023-04-15 15:31:57.455214384  [2023-04-15 15:31:57] peewee_migrate                 INFO    : Starting migrations
2023-04-15 15:31:57.495345149  [2023-04-15 15:31:57] peewee_migrate                 INFO    : There is nothing to migrate
2023-04-15 15:31:57.520735800  [2023-04-15 15:31:57] detector.tensorrt              INFO    : Starting detection process: 437
2023-04-15 15:31:57.526496251  [2023-04-15 15:31:57] frigate.app                    INFO    : Output process started: 439
2023-04-15 15:31:57.536403212  [2023-04-15 15:31:57] frigate.app                    INFO    : Camera processor started for camera1: 442
2023-04-15 15:31:57.545745059  [2023-04-15 15:31:57] frigate.app                    INFO    : Camera processor started for camera2: 444
2023-04-15 15:31:57.563477730  [2023-04-15 15:31:57] frigate.app                    INFO    : Camera processor started for camera3: 447
2023-04-15 15:31:57.571649549  [2023-04-15 15:31:57] frigate.app                    INFO    : Camera processor started for camera4: 448
2023-04-15 15:31:57.581104295  [2023-04-15 15:31:57] frigate.app                    INFO    : Capture process started for camera1: 449
2023-04-15 15:31:57.594334660  [2023-04-15 15:31:57] frigate.app                    INFO    : Capture process started for camera2: 452
2023-04-15 15:31:57.622412532  [2023-04-15 15:31:57] frigate.app                    INFO    : Capture process started for camera3: 459
2023-04-15 15:31:57.632663988  [2023-04-15 15:31:57] frigate.app                    INFO    : Capture process started for camera4: 464
2023-04-15 15:31:57.757909450  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Num Available Devices: 1
2023-04-15 15:31:57.787278351  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchor_TRT version 1
2023-04-15 15:31:57.787984649  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchorRect_TRT version 1
2023-04-15 15:31:57.788116230  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMS_TRT version 1
2023-04-15 15:31:57.788221827  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Reorg_TRT version 1
2023-04-15 15:31:57.788306192  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Region_TRT version 1
2023-04-15 15:31:57.788389375  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Clip_TRT version 1
2023-04-15 15:31:57.788472115  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LReLU_TRT version 1
2023-04-15 15:31:57.788589062  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PriorBox_TRT version 1
2023-04-15 15:31:57.788681228  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Normalize_TRT version 1
2023-04-15 15:31:57.788761026  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ScatterND version 1
2023-04-15 15:31:57.788839473  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::RPROI_TRT version 1
2023-04-15 15:31:57.788922747  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMS_TRT version 1
2023-04-15 15:31:57.789011227  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
2023-04-15 15:31:57.789099144  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchTilePlugin_TRT version 1
2023-04-15 15:31:57.789180866  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::FlattenConcat_TRT version 1
2023-04-15 15:31:57.789264893  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResize version 1
2023-04-15 15:31:57.789345149  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResizeDynamic version 1
2023-04-15 15:31:57.789418270  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DetectionLayer_TRT version 1
2023-04-15 15:31:57.789500964  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_TRT version 1
2023-04-15 15:31:57.789576647  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
2023-04-15 15:31:57.789658289  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
2023-04-15 15:31:57.789734360  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
2023-04-15 15:31:57.789802654  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalDynamic version 1
2023-04-15 15:31:57.789873588  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Proposal version 1
2023-04-15 15:31:57.789946279  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalLayer_TRT version 1
2023-04-15 15:31:57.790017899  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PyramidROIAlign_TRT version 1
2023-04-15 15:31:57.790097294  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ResizeNearest_TRT version 1
2023-04-15 15:31:57.790170597  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Split version 1
2023-04-15 15:31:57.790244785  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SpecialSlice_TRT version 1
2023-04-15 15:31:57.790313610  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 1
2023-04-15 15:31:57.790389214  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 2
2023-04-15 15:31:57.790459447  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CoordConvAC version 1
2023-04-15 15:31:57.790547208  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DecodeBbox3DPlugin version 1
2023-04-15 15:31:57.790618450  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GenerateDetection_TRT version 1
2023-04-15 15:31:57.790687805  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
2023-04-15 15:31:57.790757960  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelProposeROI_TRT version 1
2023-04-15 15:31:57.790828645  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMSDynamic_TRT version 1
2023-04-15 15:31:58.681534419  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PillarScatterPlugin version 1
2023-04-15 15:31:58.681717411  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::VoxelGeneratorPlugin version 1
2023-04-15 15:31:58.681822006  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
2023-04-15 15:31:58.681926540  [2023-04-15 15:31:57] frigate.detectors.plugins.tensorrt ERROR   : ERROR: failed to load libraries. libcudart.so.12: cannot open shared object file: No such file or directory
2023-04-15 15:31:58.730965889  [2023-04-15 15:31:58] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 132 (MiB)
2023-04-15 15:32:01.540110391  [2023-04-15 15:32:01] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 328 MiB
2023-04-15 15:32:01.590784123  [2023-04-15 15:32:01] frigate.detectors.plugins.tensorrt ERROR   : 1: [stdArchiveReader.cpp::StdArchiveReader::40] Error Code 1: Serialization (Serialization assertion stdVersionRead == serializationVersion failed.Version tag does not match. Note: Current Version: 213, Serialized Engine Version: 232)
2023-04-15 15:32:01.604333330  [2023-04-15 15:32:01] frigate.detectors.plugins.tensorrt ERROR   : 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)
2023-04-15 15:32:01.635978083  Process detector:tensorrt:
2023-04-15 15:32:01.639431130  Traceback (most recent call last):
2023-04-15 15:32:01.639449519    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 15:32:01.639451626      self.run()
2023-04-15 15:32:01.639453358    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 15:32:01.639454882      self._target(*self._args, **self._kwargs)
2023-04-15 15:32:01.639456481    File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector
2023-04-15 15:32:01.639493940      object_detector = LocalObjectDetector(detector_config=detector_config)
2023-04-15 15:32:01.639496396    File "/opt/frigate/frigate/object_detection.py", line 52, in __init__
2023-04-15 15:32:01.639498696      self.detect_api = create_detector(detector_config)
2023-04-15 15:32:01.639500353    File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector
2023-04-15 15:32:01.639501918      return api(detector_config)
2023-04-15 15:32:01.639529584    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 220, in __init__
2023-04-15 15:32:01.639532554      self.input_shape = self._get_input_shape()
2023-04-15 15:32:01.639534878    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 92, in _get_input_shape
2023-04-15 15:32:01.639536341      binding = self.engine[0]
2023-04-15 15:32:01.639537811  TypeError: 'NoneType' object is not subscriptable
2023-04-15 15:32:01.639548242  Exception ignored in: <function TensorRtDetector.__del__ at 0x7f516c1c65e0>
2023-04-15 15:32:01.639549843  Traceback (most recent call last):
2023-04-15 15:32:01.639551522    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 238, in __del__
2023-04-15 15:32:01.639721254      if self.outputs is not None:
2023-04-15 15:32:01.639725996  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'
2023-04-15 15:32:06.029796100  [INFO] Starting go2rtc healthcheck service...
2023-04-15 15:32:17.962106657  [2023-04-15 15:32:17] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service go2rtc-healthcheck: stopping
2023-04-15 15:32:17.979411028  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
s6-rc: info: service go2rtc-healthcheck successfully stopped
2023-04-15 15:32:18.010487087  [INFO] Service NGINX exited with code 0 (by signal 0)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service nginx-log: stopping
s6-rc: info: service frigate: stopping
2023-04-15 15:32:18.013679405  [2023-04-15 15:32:18] frigate.app                    INFO    : Stopping...
s6-rc: info: service nginx-log successfully stopped
2023-04-15 15:32:18.014861764  [2023-04-15 15:32:18] frigate.watchdog               INFO    : Exiting watchdog...
2023-04-15 15:32:18.020580526  [2023-04-15 15:32:18] frigate.record                 INFO    : Exiting recording cleanup...
2023-04-15 15:32:18.021036923  [2023-04-15 15:32:18] frigate.stats                  INFO    : Exiting stats emitter...
2023-04-15 15:32:18.021040558  [2023-04-15 15:32:18] frigate.record                 INFO    : Exiting recording maintenance...
2023-04-15 15:32:18.021042921  [2023-04-15 15:32:18] frigate.storage                INFO    : Exiting storage maintainer...
2023-04-15 15:32:18.021045270  [2023-04-15 15:32:18] frigate.events                 INFO    : Exiting event cleanup...
2023-04-15 15:32:18.021213153  [2023-04-15 15:32:18] frigate.object_processing      INFO    : Exiting object processor...
2023-04-15 15:32:18.521274640  [2023-04-15 15:32:18] frigate.comms.ws               INFO    : Exiting websocket client...
2023-04-15 15:32:18.985641189  [2023-04-15 15:32:18] frigate.events                 INFO    : Exiting event processor...
2023-04-15 15:32:18.986090731  [2023-04-15 15:32:18] peewee.sqliteq                 INFO    : writer received shutdown request, exiting.
2023-04-15 15:32:18.993085671  [2023-04-15 15:32:18] watchdog.camera3               INFO    : Terminating the existing ffmpeg process...
2023-04-15 15:32:18.993453008  [2023-04-15 15:32:18] watchdog.camera1               INFO    : Terminating the existing ffmpeg process...
2023-04-15 15:32:18.993798438  [2023-04-15 15:32:18] watchdog.camera4               INFO    : Terminating the existing ffmpeg process...
2023-04-15 15:32:18.993958922  [2023-04-15 15:32:18] watchdog.camera3               INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-15 15:32:18.994127913  [2023-04-15 15:32:18] watchdog.camera4               INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-15 15:32:18.994268860  [2023-04-15 15:32:18] watchdog.camera2               INFO    : Terminating the existing ffmpeg process...
2023-04-15 15:32:18.994386100  [2023-04-15 15:32:18] watchdog.camera2               INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-15 15:32:18.994528493  [2023-04-15 15:32:18] watchdog.camera1               INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-15 15:32:19.010167283  [2023-04-15 15:32:19] watchdog.camera2               INFO    : Terminating the existing ffmpeg process...
2023-04-15 15:32:19.010331970  [2023-04-15 15:32:19] watchdog.camera2               INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-15 15:32:19.020597265  [2023-04-15 15:32:19] watchdog.camera3               INFO    : Terminating the existing ffmpeg process...
2023-04-15 15:32:19.020764297  [2023-04-15 15:32:19] watchdog.camera3               INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-15 15:32:19.023953382  [2023-04-15 15:32:19] watchdog.camera4               INFO    : Terminating the existing ffmpeg process...
2023-04-15 15:32:19.024041925  [2023-04-15 15:32:19] watchdog.camera4               INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-15 15:32:19.039158232  [2023-04-15 15:32:19] watchdog.camera1               INFO    : Terminating the existing ffmpeg process...
2023-04-15 15:32:19.039246568  [2023-04-15 15:32:19] watchdog.camera1               INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-15 15:32:19.175721971  [2023-04-15 15:32:19] frigate.output                 INFO    : exiting output process...
2023-04-15 15:32:23.242191248  /usr/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 23 leaked shared_memory objects to clean up at shutdown
2023-04-15 15:32:23.242197533    warnings.warn('resource_tracker: There appear to be %d '
2023-04-15 15:32:23.245059898  [INFO] Service Frigate exited with code 0 (by signal 0)
s6-rc: info: service frigate successfully stopped
s6-rc: info: service go2rtc: stopping
s6-rc: info: service frigate-log: stopping
2023-04-15 15:32:23.247983790  exit OK
s6-rc: info: service frigate-log successfully stopped
2023-04-15 15:32:23.254215098  [INFO] The go2rtc service exited with code 0 (by signal 0)
s6-rc: info: service go2rtc successfully stopped
s6-rc: info: service go2rtc-log: stopping
s6-rc: info: service go2rtc-log successfully stopped
s6-rc: info: service log-prepare: stopping
s6-rc: info: service s6rc-fdholder: stopping
s6-rc: info: service log-prepare successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service s6rc-fdholder successfully stopped
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
NateMeyer commented 1 year ago

If I remember correctly, this looks like it could be complaining about the version of tensorrt libraries used to generate the model?

2023-04-15 15:32:01.590784123  [2023-04-15 15:32:01] frigate.detectors.plugins.tensorrt ERROR   : 1: [stdArchiveReader.cpp::StdArchiveReader::40] Error Code 1: Serialization (Serialization assertion stdVersionRead == serializationVersion failed.Version tag does not match. Note: Current Version: 213, Serialized Engine Version: 232)

What version of the tensorrt container did you use to generate the models? From the docs: 

docker run --gpus=all --rm -it -v `pwd`/trt-models:/tensorrt_models -v `pwd`/tensorrt_models.sh:/tensorrt_models.sh nvcr.io/nvidia/tensorrt:22.07-py3 /tensorrt_models.sh

This specifically uses the 22.07-py3 version to produce trt models compatible with frigate's included libraries.

csreynolds commented 1 year ago

Wasn't sure if I pasted the command from the docs or not so I regen-d the trt-models with the line and now it works.

Sorry for the goat rope!

doctorttt commented 1 year ago

I'm facing the same issue and I need some help.

ERROR: failed to load libraries. libcudart.so.12: cannot open shared object file: No such file or directory
frigate.detectors.plugins.tensorrt ERROR   : 1: [stdArchiveReader.cpp::StdArchiveReader::30] Error Code 1: Serialization (Serialization assertion magicTagRead == kMAGIC_TAG failed.Magic tag does not match)
2023-05-31 21:21:44.021000447  [2023-05-31 21:21:44] frigate.detectors.plugins.tensorrt ERROR   : 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)

I'm on Unraid and nvidia-smi shows Cuda version 12.1. I'm not sure what to do. Do I need to manually install the cuda toolkit?

I compiled the trt models using a version higher than 22.07-py3. 22.07-py3 didn't work for me. It says Nvidia gpu is detected but not supported.

doctorttt commented 1 year ago

I narrowed down the issue. 

WARNING: Detected NVIDIA NVIDIA Graphics Device GPU, which is not yet supported in this version of the container
ERROR: No supported GPU(s) detected to run this container

Tensorrt 22.07-py3 doesn't support rtx 4060ti. I tried 22.08-py3 and no luck. I could compile the trt models using 22.09-py3 but frigate threw out a lot of errors related to magic tag does not match. Perhaps the entire rtx 4000 series won't work in frigate's current tensorrt build? Any chance for the frigate team to consider supporting a higher version of tensorrt that works with rtx 4000 series?