search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.74k stars 1.7k forks source link

[Support]: Self built TensorRT YOLOv8 doesn't work #8236

Closed qubex22 closed 10 months ago

qubex22 commented 1 year ago

Describe the problem you are having

I have built yolov8 for tensorrt 8.4.1 (same version as it is in frigate 0.12.1)

The GPU is a NVIDIA Quadro K620

trtexec --onnx=yolov8s.onnx --saveEngine=yolov8s.engine --workspace=2000

It shows these warnings when building:

[W] [TRT] parsers/onnx/onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[W] [TRT] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
[W] [TRT] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.

When I map the model in the config and start frigate the detection crashes and shows this error:

2023-10-19 18:43:57.484444651  Process camera_processor:dahua:
2023-10-19 18:43:57.487139413  Traceback (most recent call last):
2023-10-19 18:43:57.487160563    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 18:43:57.487161863      self.run()
2023-10-19 18:43:57.487163183    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 18:43:57.487164313      self._target(*self._args, **self._kwargs)
2023-10-19 18:43:57.487172412    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 18:43:57.487173712      process_frames(
2023-10-19 18:43:57.487175082    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 18:43:57.487176282      detect(
2023-10-19 18:43:57.487177392    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 18:43:57.487178572      region_detections = object_detector.detect(tensor_input)
2023-10-19 18:43:57.487182792    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 18:43:57.487184752      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 18:43:57.487225042  KeyError: 638
2023-10-19 18:43:57.564795557  Process camera_processor:yi_pasillo:
2023-10-19 18:43:57.565902386  Traceback (most recent call last):
2023-10-19 18:43:57.565933915    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 18:43:57.565936065      self.run()
2023-10-19 18:43:57.565937725    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 18:43:57.565939235      self._target(*self._args, **self._kwargs)
2023-10-19 18:43:57.565940655    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 18:43:57.565941925      process_frames(
2023-10-19 18:43:57.565943365    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 18:43:57.565944525      detect(
2023-10-19 18:43:57.565955995    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 18:43:57.565957575      region_detections = object_detector.detect(tensor_input)
2023-10-19 18:43:57.565958995    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 18:43:57.565960465      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 18:43:57.565969735  KeyError: 636
2023-10-19 18:43:57.993724632  Process camera_processor:yi_salon:
2023-10-19 18:43:57.994535773  Traceback (most recent call last):
2023-10-19 18:43:57.994548723    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 18:43:57.994550083      self.run()
2023-10-19 18:43:57.994551423    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 18:43:57.994552553      self._target(*self._args, **self._kwargs)
2023-10-19 18:43:57.994553653    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 18:43:57.994554553      process_frames(
2023-10-19 18:43:57.994555613    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 18:43:57.994559363      detect(
2023-10-19 18:43:57.994566083    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 18:43:57.994567143      region_detections = object_detector.detect(tensor_input)
2023-10-19 18:43:57.994568243    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 18:43:57.994569303      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 18:43:57.994570143  KeyError: 636

I'm a bit lost now, I hope someone can help.

Version

0.12.1-367d724

Frigate config file

mqtt:
  # Optional: Enable mqtt server (default: shown below)
  enabled: True
  # Required: host name
  host: mosquitto-1
  # Optional: port (default: shown below)
  port: 1883
  # Optional: topic prefix (default: shown below)
  # NOTE: must be unique if you are running multiple instances
  topic_prefix: frigate
  # Optional: client id (default: shown below)
  # NOTE: must be unique if you are running multiple instances
  client_id: frigate
  # Optional: user
  # NOTE: MQTT user can be specified with an environment variables that must begin with 'FRIGATE_'.
  #       e.g. user: '{FRIGATE_MQTT_USER}'

cameras:
  yi_pasillo: # <------ Name the camera
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://172.16.1.201:8554/yi_pasillo?mp4 # <----- The stream you want to use for detection
          roles:
            - detect
    detect:
      enabled: True # <---- disable detection until you have a working camera feed
      width: 1920 # <---- update for your camera's resolution
      height: 1080 # <---- update for your camera's resolution
  yi_salon: # <------ Name the camera
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://172.16.1.201:8554/yi_salon?mp4 # <----- The stream you want to use for detection
          roles:
            - detect
    detect:
      enabled: True # <---- disable detection until you have a working camera feed
      width: 1920 # <---- update for your camera's resolution
      height: 1080 # <---- update for your camera's resolution
  dahua: # <------ Name the camera
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://172.16.1.201:8554/dahua?mp4 # <----- The stream you want to use for detection
          roles:
            - detect
    detect:
      enabled: True # <---- disable detection until you have a working camera feed
      width: 2336 # <---- update for your camera's resolution
      height: 1752 # <---- update for your camera's resolution
objects:
  track:
    - cat
    - person
detectors:
  tensorrt:
    type: tensorrt
    device: 0
model:
  #path: /trt-models/yolov7-tiny-416.trt
  path: /trt-models/yolov8s.engine
  input_tensor: nchw
  input_pixel_format: rgb
  width: 640
  height: 640
ui:
  # Optional: Set the default live mode for cameras in the UI (default: shown below)
  live_mode: mse
motion:
  # Optional: The threshold passed to cv2.threshold to determine if a pixel is different enough to be counted as motion. (default: shown below)
  # Increasing this value will make motion detection less sensitive and decreasing it will make motion detection more sensitive.
  # The value should be between 1 and 255.
  threshold: 30
  # Optional: Minimum size in pixels in the resized motion image that counts as motion (default: 30)
  # Increasing this value will prevent smaller areas of motion from being detected. Decreasing will
  # make motion detection more sensitive to smaller moving objects.
  # As a rule of thumb:
  #  - 15 - high sensitivity
  #  - 30 - medium sensitivity
  #  - 50 - low sensitivity
  contour_area: 30
  # Optional: Alpha value passed to cv2.accumulateWeighted when averaging the motion delta across multiple frames (default: shown below)
  # Higher values mean the current frame impacts the delta a lot, and a single raindrop may register as motion.
  # Too low and a fast moving person wont be detected as motion.
  delta_alpha: 0.2
  # Optional: Alpha value passed to cv2.accumulateWeighted when averaging frames to determine the background (default: shown below)
  # Higher values mean the current frame impacts the average a lot, and a new object will be averaged into the background faster.
  # Low values will cause things like moving shadows to be detected as motion for longer.
  # https://www.geeksforgeeks.org/background-subtraction-in-an-image-using-concept-of-running-average/
  frame_alpha: 0.2
  # Optional: Height of the resized motion frame  (default: 50)
  # This operates as an efficient blur alternative. Higher values will result in more granular motion detection at the expense
  # of higher CPU usage. Lower values result in less CPU, but small changes may not register as motion.
  frame_height: 50
  # Optional: motion mask
  # NOTE: see docs for more detailed info on creating masks
  #mask: 0,900,0,0,1280,0,1280,720,839,720,888,34,660,37,475,127,516,591,421,720,285,238,0,429,123,720,1080,900,1080,1920,0,1920
  # Optional: improve contrast (default: shown below)
  # Enables dynamic contrast improvement. This should help improve night detections at the cost of making motion detection more sensitive
  # for daytime.
  improve_contrast: False

timestamp_style:
  # Optional: Position of the timestamp (default: shown below)
  #           "tl" (top left), "tr" (top right), "bl" (bottom left), "br" (bottom right)
  position: "tr"
  # Optional: Format specifier conform to the Python package "datetime" (default: shown below)
  #           Additional Examples:
  #             german: "%d.%m.%Y %H:%M:%S"
  format: "%m/%d/%Y %H:%M:%S"
  # Optional: Color of font
  color:
    # All Required when color is specified (default: shown below)
    red: 255
    green: 255
    blue: 255
  # Optional: Line thickness of font (default: shown below)
  thickness: 2

snapshots:
  # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
  enabled: True
  # Optional: save a clean PNG copy of the snapshot image (default: shown below)
  clean_copy: True
  # Optional: print a timestamp on the snapshots (default: shown below)
  timestamp: False
  # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box: False
  # Optional: crop the snapshot (default: shown below)
  crop: False
  # Optional: height to resize the snapshot to (default: original size)
  height: 175
  # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
  required_zones: []
  # Optional: Camera override for retention settings (default: global values)
  retain:
    # Required: Default retention days (default: shown below)
    default: 10

record:
  # Optional: Enable recording (default: shown below)
  # WARNING: If recording is disabled in the config, turning it on via
  #          the UI or MQTT later will have no effect.
  enabled: True
  # Optional: Number of minutes to wait between cleanup runs (default: shown below)
  # This can be used to reduce the frequency of deleting recording segments from disk if you want to minimize i/o
  expire_interval: 60
  # Optional: Retention settings for recording
  retain:
    # Optional: Number of days to retain recordings regardless of events (default: shown below)
    # NOTE: This should be set to 0 and retention should be defined in events section below
    #       if you only want to retain recordings of events.
    days: 0
    # Optional: Mode for retention. Available options are: all, motion, and active_objects
    #   all - save all recording segments regardless of activity
    #   motion - save all recordings segments with any detected motion
    #   active_objects - save all recording segments with active/moving objects
    # NOTE: this mode only applies when the days setting above is greater than 0
    mode: all
  # Optional: Event recording settings
  events:
    # Optional: Number of seconds before the event to include (default: shown below)
    pre_capture: 5
    # Optional: Number of seconds after the event to include (default: shown below)
    post_capture: 5
    # Optional: Objects to save recordings for. (default: all tracked objects)
    objects:
      - cat
    # Optional: Restrict recordings to objects that entered any of the listed zones (default: no required zones)
    required_zones: []
    # Optional: Retention settings for recordings of events
    retain:
      # Required: Default retention days (default: shown below)
      default: 10
      # Optional: Mode for retention. (default: shown below)
      #   all - save all recording segments for events regardless of activity
      #   motion - save all recordings segments for events with any detected motion
      #   active_objects - save all recording segments for event with active/moving objects
      #
      # NOTE: If the retain mode for the camera is more restrictive than the mode configured
      #       here, the segments will already be gone by the time this mode is applied.
      #       For example, if the camera retain mode is "motion", the segments without motion are
      #       never stored, so setting the mode to "all" here won't bring them back.
      mode: motion
      # Optional: Per object retention days
go2rtc:
  candidates:
    - 172.16.0.250:8555
  ice_servers:
    - urls: [stun:stun.l.google.com:19302]

  streams:
    yi_pasillo:
    - rtsp://172.16.1.201:8554/yi_pasillo?mp4
    yi_salon:
    - rtsp://172.16.1.201:8554/yi_salon?mp4
    dahua:
    - rtsp://172.16.1.201:8554/dahua?mp4

live:
  # Optional: Set the name of the stream that should be used for live view
  # in frigate WebUI. (default: name of camera)
  #stream_name: yi_pasillo
  # Optional: Set the height of the jsmpeg stream. (default: 720)
  # This must be less than or equal to the height of the detect stream. Lower resolutions
  # reduce bandwidth required for viewing the jsmpeg stream. Width is computed to match known aspect ratio.
  #height: 720
  # Optional: Set the encode quality of the jsmpeg stream (default: shown below)
  # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
  quality: 8

logger:
  # Optional: Default log verbosity (default: shown below)
  default: info
  # Optional: Component specific logger overrides
  logs:
    frigate.event: debug

Relevant log output

2023-10-19 18:43:57.263156535  [2023-10-19 18:43:57] frigate.detectors.plugins.tensorrt WARNING : The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
2023-10-19 18:43:57.484444651  Process camera_processor:dahua:
2023-10-19 18:43:57.487139413  Traceback (most recent call last):
2023-10-19 18:43:57.487160563    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 18:43:57.487161863      self.run()
2023-10-19 18:43:57.487163183    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 18:43:57.487164313      self._target(*self._args, **self._kwargs)
2023-10-19 18:43:57.487172412    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 18:43:57.487173712      process_frames(
2023-10-19 18:43:57.487175082    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 18:43:57.487176282      detect(
2023-10-19 18:43:57.487177392    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 18:43:57.487178572      region_detections = object_detector.detect(tensor_input)
2023-10-19 18:43:57.487182792    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 18:43:57.487184752      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 18:43:57.487225042  KeyError: 638
2023-10-19 18:43:57.564795557  Process camera_processor:yi_pasillo:
2023-10-19 18:43:57.565902386  Traceback (most recent call last):
2023-10-19 18:43:57.565933915    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 18:43:57.565936065      self.run()
2023-10-19 18:43:57.565937725    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 18:43:57.565939235      self._target(*self._args, **self._kwargs)
2023-10-19 18:43:57.565940655    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 18:43:57.565941925      process_frames(
2023-10-19 18:43:57.565943365    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 18:43:57.565944525      detect(
2023-10-19 18:43:57.565955995    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 18:43:57.565957575      region_detections = object_detector.detect(tensor_input)
2023-10-19 18:43:57.565958995    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 18:43:57.565960465      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 18:43:57.565969735  KeyError: 636
2023-10-19 18:43:57.993724632  Process camera_processor:yi_salon:
2023-10-19 18:43:57.994535773  Traceback (most recent call last):
2023-10-19 18:43:57.994548723    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 18:43:57.994550083      self.run()
2023-10-19 18:43:57.994551423    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 18:43:57.994552553      self._target(*self._args, **self._kwargs)
2023-10-19 18:43:57.994553653    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 18:43:57.994554553      process_frames(
2023-10-19 18:43:57.994555613    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 18:43:57.994559363      detect(
2023-10-19 18:43:57.994566083    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 18:43:57.994567143      region_detections = object_detector.detect(tensor_input)
2023-10-19 18:43:57.994568243    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 18:43:57.994569303      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 18:43:57.994570143  KeyError: 636

FFprobe output from your camera

root@c80cf4aafae0:/opt/frigate# ffprobe rtsp://172.16.1.201:8554/dahua?mp4
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[rtsp @ 0x55cf57305140] method SETUP failed: 461 Unsupported transport
Input #0, rtsp, from 'rtsp://172.16.1.201:8554/dahua?mp4':
  Metadata:
    title           : go2rtc/1.8.1
  Duration: N/A, start: 0.040000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 2336x1752 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 90k tbn

Frigate stats

{"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"0.1"},"1":{"cpu":"0.0","mem":"0.0"},"109":{"cpu":"0.0","mem":"0.0"},"1174":{"cpu":"0.0","mem":"0.0"},"1233":{"cpu":"0.0","mem":"0.0"},"1261":{"cpu":"0.0","mem":"0.0"},"134":{"cpu":"0.0","mem":"0.0"},"135":{"cpu":"0.0","mem":"0.0"},"136":{"cpu":"0.0","mem":"0.0"},"146":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"165":{"cpu":"0.0","mem":"0.0"},"187":{"cpu":"0.0","mem":"0.0"},"222":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"258":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"289":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"32":{"cpu":"0.0","mem":"0.0"},"323":{"cpu":"0.0","mem":"0.0"},"356":{"cpu":"0.0","mem":"0.0"},"379":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"426":{"cpu":"0.0","mem":"0.0"},"468":{"cpu":"0.0","mem":"0.0"},"510":{"cpu":"0.0","mem":"0.0"},"535":{"cpu":"0.0","mem":"0.0"},"714":{"cpu":"0.0","mem":"0.5"},"720":{"cpu":"0.0","mem":"0.0"},"721":{"cpu":"0.3","mem":"1.8"},"723":{"cpu":"0.0","mem":"0.5"},"726":{"cpu":"0.0","mem":"0.0"},"727":{"cpu":"0.0","mem":"0.0"},"728":{"cpu":"0.0","mem":"0.0"},"729":{"cpu":"0.0","mem":"0.0"},"730":{"cpu":"0.0","mem":"0.0"},"731":{"cpu":"2.7","mem":"0.5"},"734":{"cpu":"3.0","mem":"0.5"},"737":{"cpu":"4.7","mem":"0.4"},"738":{"cpu":"5.3","mem":"0.6"},"742":{"cpu":"4.3","mem":"0.4"},"743":{"cpu":"0.0","mem":"0.0"},"744":{"cpu":"9.0","mem":"0.5"},"748":{"cpu":"0.0","mem":"0.0"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"88":{"cpu":"0.0","mem":"0.1"},"97":{"cpu":"0.0","mem":"0.0"},"98":{"cpu":"0.0","mem":"1.0"},"MiB":{"cpu":"7698.1","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"3"},"top":{"cpu":"users,","mem":"load"}},"dahua":{"camera_fps":5.0,"capture_pid":738,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":744,"pid":729,"process_fps":0.0,"skipped_fps":0.0},"detection_fps":0.0,"detectors":{"tensorrt":{"detection_start":0.0,"inference_speed":30.82,"pid":721}},"gpu_usages":{"Quadro K620":{"gpu":"4 %","mem":"28.9 %"}},"service":{"last_updated":1697734197,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":227.1,"mount_type":"tmpfs","total":268.4,"used":41.4},"/media/frigate/clips":{"free":4836047.2,"mount_type":"fuse.shfs","total":11996500.4,"used":7160453.2},"/media/frigate/recordings":{"free":4836047.2,"mount_type":"fuse.shfs","total":11996500.4,"used":7160453.2},"/tmp/cache":{"free":41.6,"mount_type":"tmpfs","total":100.0,"used":58.4}},"temperatures":{},"uptime":362,"version":"0.12.1-367d724"},"yi_pasillo":{"camera_fps":5.0,"capture_pid":731,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":742,"pid":727,"process_fps":0.0,"skipped_fps":0.0},"yi_salon":{"camera_fps":5.0,"capture_pid":734,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":737,"pid":728,"process_fps":0.0,"skipped_fps":0.0}}

Operating system

UNRAID

Install method

Docker CLI

Coral version

CPU (no coral)

Network connection

Wired

Camera make and model

dahua, yi

Any other information that may be helpful


+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro K620                    Off | 00000000:08:00.0 Off |                  N/A |
| 50%   61C    P0               5W /  30W |    592MiB /  2048MiB |      5%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     18721      C   frigate.detector.tensorrt                   192MiB |
|    0   N/A  N/A     18747      C   ffmpeg                                       87MiB |
|    0   N/A  N/A     18752      C   ffmpeg                                      106MiB |
|    0   N/A  N/A     18754      C   ffmpeg                                      201MiB |
+---------------------------------------------------------------------------------------+```
NickM-27 commented 1 year ago

Doesn't look like you're setting the labelmap

qubex22 commented 1 year ago

Doesn't look like you're setting the labelmap

Same error setting the default labelmap.txt

2023-10-19 21:50:36.775702346  [2023-10-19 21:50:36] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +191, GPU +0, now: CPU 249, GPU 175 (MiB)
2023-10-19 21:50:36.834194250  [2023-10-19 21:50:36] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 51 MiB
2023-10-19 21:50:36.942934829  [2023-10-19 21:50:36] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +8, GPU +11, now: CPU 337, GPU 440 (MiB)
2023-10-19 21:50:36.944769341  [2023-10-19 21:50:36] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +51, now: CPU 0, GPU 51 (MiB)
2023-10-19 21:50:37.162729684  [2023-10-19 21:50:36] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 285, GPU 441 (MiB)
2023-10-19 21:50:37.167882604  [2023-10-19 21:50:37] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +35, now: CPU 0, GPU 86 (MiB)
2023-10-19 21:50:37.171323510  [2023-10-19 21:50:37] frigate.detectors.plugins.tensorrt WARNING : The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
2023-10-19 21:50:37.279631723  Process camera_processor:dahua:
2023-10-19 21:50:37.284457785  Traceback (most recent call last):
2023-10-19 21:50:37.284485185    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 21:50:37.284486445      self.run()
2023-10-19 21:50:37.284487745    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 21:50:37.284488795      self._target(*self._args, **self._kwargs)
2023-10-19 21:50:37.284496995    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 21:50:37.284497935      process_frames(
2023-10-19 21:50:37.284499015    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 21:50:37.284499955      detect(
2023-10-19 21:50:37.284500925    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 21:50:37.284501895      region_detections = object_detector.detect(tensor_input)
2023-10-19 21:50:37.284506175    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 21:50:37.284507745      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 21:50:37.284541815  KeyError: 636
2023-10-19 21:50:37.516822697  Process camera_processor:yi_pasillo:
2023-10-19 21:50:37.517654959  Traceback (most recent call last):
2023-10-19 21:50:37.517682519    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 21:50:37.517684528      self.run()
2023-10-19 21:50:37.517686188    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 21:50:37.517687688      self._target(*self._args, **self._kwargs)
2023-10-19 21:50:37.517689748    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 21:50:37.517691008      process_frames(
2023-10-19 21:50:37.517692368    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 21:50:37.517693618      detect(
2023-10-19 21:50:37.517694898    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 21:50:37.517696288      region_detections = object_detector.detect(tensor_input)
2023-10-19 21:50:37.517698598    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 21:50:37.517700018      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 21:50:37.517770758  KeyError: 636
2023-10-19 21:50:37.705598928  Process camera_processor:yi_salon:
2023-10-19 21:50:37.706428250  Traceback (most recent call last):
2023-10-19 21:50:37.706439350    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-19 21:50:37.706440590      self.run()
2023-10-19 21:50:37.706442690    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-19 21:50:37.706444110      self._target(*self._args, **self._kwargs)
2023-10-19 21:50:37.706445550    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-10-19 21:50:37.706446740      process_frames(
2023-10-19 21:50:37.706455339    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-10-19 21:50:37.706456269      detect(
2023-10-19 21:50:37.706457269    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-10-19 21:50:37.706458279      region_detections = object_detector.detect(tensor_input)
2023-10-19 21:50:37.706459329    File "/opt/frigate/frigate/object_detection.py", line 221, in detect
2023-10-19 21:50:37.706460299      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-19 21:50:37.706461129  KeyError: 636
NickM-27 commented 1 year ago

what labelmap was this model trained on though? it seems to be referencing an index of a label (636) that is not in the labelmap

qubex22 commented 1 year ago

what labelmap was this model trained on though? it seems to be referencing an index of a label (636) that is not in the labelmap

I just took the default yolov8 trained with coco dataset, I don't think there's a 636 class. So I think my problem is somewhere else, but I really don't know...

NickM-27 commented 1 year ago

The error is occurring on line 221, the only ting that could have a KeyError would be self.labels

qubex22 commented 1 year ago

I have noticed another error in the logs only showing when I change to yolov8 model. But I can't see the relationship between both.

This error is repeating every 5 seconds aprox and when I return to yolov7 it goes away.

2023-10-19 22:08:25.206001055  [2023-10-19 22:08:25] asyncio                        DEBUG   : Using selector: EpollSelector
2023-10-19 22:08:27.796673749  [2023-10-19 22:08:27] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."label", "t1"."sub_label", "t1"."camera", "t1"."start_time", "t1"."end_time", "t1"."top_score", "t1"."false_positive", "t1"."zones", "t1"."thumbnail", "t1"."has_clip", "t1"."has_snapshot", "t1"."region", "t1"."box", "t1"."area", "t1"."retain_indefinitely", "t1"."ratio", "t1"."plus_id" FROM "event" AS "t1" WHERE ((("t1"."camera" = ?) AND (("t1"."end_time" IS ?) OR ("t1"."end_time" >= ?))) AND "t1"."has_clip") ORDER BY "t1"."start_time"', ['dahua', None, 1697746093.0])
2023-10-19 22:08:27.797147994  [2023-10-19 22:08:27] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-10-19 22:08:27.797151064  [2023-10-19 22:08:27] frigate.record                 ERROR   : list index out of range
503Dev commented 1 year ago

I run a very similar setup and the labelmap is required or my custom models never work. You usually need the labelmap.txt and then you can reference it from the model config block:

labelmap_path: /trt-models/labelmap.txt

qubex22 commented 1 year ago

I run a very similar setup and the labelmap is required or my custom models never work. You usually need the labelmap.txt and then you can reference it from the model config block:

labelmap_path: /trt-models/labelmap.txt

Got the labelmap from here: https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/datasets/coco.yaml

but the same error. I'm thinking that maybe I made some mistake when building the model... I will redo everything from 0

qubex22 commented 1 year ago

Built it again, now size 416. Curiously the KeyError now is 411 and 412, before the size was 640 and the KeyError was 636... It makes me think that it has something to do with the tensor input.

Besides I noticed some differences debugging the start with yolov7 and yolov8:

yolov7:

072521  [2023-10-20 00:26:25] frigate.detectors.plugins.tensorrt DEBUG   : Input has Shape (1, 3, 416, 416)
2023-10-20 00:26:25.748155740  [2023-10-20 00:26:25] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding detections Memory 298116 Bytes (74529 * DataType.FLOAT)
2023-10-20 00:26:25.748243180  [2023-10-20 00:26:25] frigate.detectors.plugins.tensorrt DEBUG   : Output has Shape (1, 74529, 1, 1)

Allocated Tensor Binding detections Output has Shape (1, 74529, 1, 1)

yolov8:

256828  [2023-10-20 00:23:33] frigate.detectors.plugins.tensorrt DEBUG   : Input has Shape (1, 3, 416, 416)
2023-10-20 00:23:33.715343897  [2023-10-20 00:23:33] frigate.detectors.plugins.tensorrt WARNING : The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
2023-10-20 00:23:33.715416567  [2023-10-20 00:23:33] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding output0 Memory 1192464 Bytes (298116 * DataType.FLOAT)
2023-10-20 00:23:33.715483306  [2023-10-20 00:23:33] frigate.detectors.plugins.tensorrt DEBUG   : Output has Shape (1, 84, 3549)

Allocated Tensor Binding output0 Output has Shape (1, 84, 3549)

Cheerpipe commented 1 year ago

Managed to export a "working" yolov8s 320x320 model and get more or less same error

2023-10-20 00:31:09.563516982  Process camera_processor:cocina:
2023-10-20 00:31:09.564002861  Traceback (most recent call last):
2023-10-20 00:31:09.564024978    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-20 00:31:09.564026224      self.run()
2023-10-20 00:31:09.564027330    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-20 00:31:09.564028434      self._target(*self._args, **self._kwargs)
2023-10-20 00:31:09.564034015    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-20 00:31:09.564034743      process_frames(
2023-10-20 00:31:09.564037305    File "/opt/frigate/frigate/video.py", line 673, in process_frames
2023-10-20 00:31:09.564038035      detect(
2023-10-20 00:31:09.564039055    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-20 00:31:09.564040109      region_detections = object_detector.detect(tensor_input)
2023-10-20 00:31:09.564043452    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-20 00:31:09.564044753      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-20 00:31:09.564063273  KeyError: 323
2023-10-20 00:31:09.635967535  Process camera_processor:entrada:
2023-10-20 00:31:09.636616826  Traceback (most recent call last):
2023-10-20 00:31:09.636641096    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-20 00:31:09.636641781      self.run()
2023-10-20 00:31:09.636642544    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-20 00:31:09.636643244      self._target(*self._args, **self._kwargs)
2023-10-20 00:31:09.636643913    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-20 00:31:09.636644529      process_frames(
2023-10-20 00:31:09.636645221    File "/opt/frigate/frigate/video.py", line 673, in process_frames
2023-10-20 00:31:09.636645784      detect(
2023-10-20 00:31:09.636646412    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-20 00:31:09.636647111      region_detections = object_detector.detect(tensor_input)
2023-10-20 00:31:09.636648718    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-20 00:31:09.636649442      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-20 00:31:09.636673615  KeyError: 316
2023-10-20 00:31:09.749621360  Process camera_processor:salon:
2023-10-20 00:31:09.750146784  Traceback (most recent call last):
2023-10-20 00:31:09.750148239    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-20 00:31:09.750148962      self.run()
2023-10-20 00:31:09.750149644    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-20 00:31:09.750150284      self._target(*self._args, **self._kwargs)
2023-10-20 00:31:09.750150932    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-20 00:31:09.750151573      process_frames(
2023-10-20 00:31:09.750152273    File "/opt/frigate/frigate/video.py", line 673, in process_frames
2023-10-20 00:31:09.750152774      detect(
2023-10-20 00:31:09.750153467    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-20 00:31:09.750154124      region_detections = object_detector.detect(tensor_input)
2023-10-20 00:31:09.750154766    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-20 00:31:09.750155440      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-20 00:31:09.750155977  KeyError: 316
2023-10-20 00:31:10.091766485  Process camera_processor:dormitorio_pequeno:
2023-10-20 00:31:10.092264021  Traceback (most recent call last):
2023-10-20 00:31:10.092279667    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-20 00:31:10.092280409      self.run()
2023-10-20 00:31:10.092281193    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-20 00:31:10.092281863      self._target(*self._args, **self._kwargs)
2023-10-20 00:31:10.092285099    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-20 00:31:10.092285716      process_frames(
2023-10-20 00:31:10.092286450    File "/opt/frigate/frigate/video.py", line 673, in process_frames
2023-10-20 00:31:10.092287091      detect(
2023-10-20 00:31:10.092287748    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-20 00:31:10.092288483      region_detections = object_detector.detect(tensor_input)
2023-10-20 00:31:10.092289174    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-20 00:31:10.092289842      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-20 00:31:10.092292592  KeyError: 315
2023-10-20 00:31:10.398648547  Process camera_processor:pasillo:
2023-10-20 00:31:10.399180929  Traceback (most recent call last):
2023-10-20 00:31:10.399191755    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-20 00:31:10.399192707      self.run()
2023-10-20 00:31:10.399193820    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-20 00:31:10.399194940      self._target(*self._args, **self._kwargs)
2023-10-20 00:31:10.399196000    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-20 00:31:10.399196975      process_frames(
2023-10-20 00:31:10.399200942    File "/opt/frigate/frigate/video.py", line 673, in process_frames
2023-10-20 00:31:10.399201746      detect(
2023-10-20 00:31:10.399202813    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-20 00:31:10.399203877      region_detections = object_detector.detect(tensor_input)
2023-10-20 00:31:10.399204973    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-20 00:31:10.399205977      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-20 00:31:10.399206768  KeyError: 313
2023-10-20 00:31:10.696797233  Process camera_processor:salon_secundaria:
2023-10-20 00:31:10.697404962  Traceback (most recent call last):
2023-10-20 00:31:10.697414135    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-20 00:31:10.697415156      self.run()
2023-10-20 00:31:10.697416335    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-20 00:31:10.697417364      self._target(*self._args, **self._kwargs)
2023-10-20 00:31:10.697418437    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-20 00:31:10.697419316      process_frames(
2023-10-20 00:31:10.697423473    File "/opt/frigate/frigate/video.py", line 673, in process_frames
2023-10-20 00:31:10.697424307      detect(
2023-10-20 00:31:10.697425270    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-20 00:31:10.697426347      region_detections = object_detector.detect(tensor_input)
2023-10-20 00:31:10.697427551    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-20 00:31:10.697428656      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-20 00:31:10.697429489  KeyError: 317
2023-10-20 00:31:12.676415218  Process camera_processor:comida_luke:
2023-10-20 00:31:12.676987508  Traceback (most recent call last):
2023-10-20 00:31:12.677003806    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-20 00:31:12.677004579      self.run()
2023-10-20 00:31:12.677005419    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-20 00:31:12.677006109      self._target(*self._args, **self._kwargs)
2023-10-20 00:31:12.677006910    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-20 00:31:12.677007549      process_frames(
2023-10-20 00:31:12.677010482    File "/opt/frigate/frigate/video.py", line 673, in process_frames
2023-10-20 00:31:12.677011086      detect(
2023-10-20 00:31:12.677011802    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-20 00:31:12.677012489      region_detections = object_detector.detect(tensor_input)
2023-10-20 00:31:12.677013184    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-20 00:31:12.677013876      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-20 00:31:12.677018769  KeyError: 314
2023-10-20 00:31:12.687816034  Process camera_processor:dormitorio_principal:
2023-10-20 00:31:12.688602909  Traceback (most recent call last):
2023-10-20 00:31:12.688617582    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-20 00:31:12.688618310      self.run()
2023-10-20 00:31:12.688619017    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-20 00:31:12.688619687      self._target(*self._args, **self._kwargs)
2023-10-20 00:31:12.688620318    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-20 00:31:12.688620923      process_frames(
2023-10-20 00:31:12.688624741    File "/opt/frigate/frigate/video.py", line 673, in process_frames
2023-10-20 00:31:12.688625294      detect(
2023-10-20 00:31:12.688625970    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-20 00:31:12.688626633      region_detections = object_detector.detect(tensor_input)
2023-10-20 00:31:12.688627280    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-20 00:31:12.688627929      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-20 00:31:12.688631683  KeyError: 318
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01             Driver Version: 535.113.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA TITAN RTX               On  | 00000000:01:00.0 Off |                  N/A |
| 40%   34C    P8              14W / 280W |   1003MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   1663531      C   python3                                    1000MiB |
+---------------------------------------------------------------------------------------+

I am using frigate 0.13 beta 2 btw and used TensorRT 8.5.3 to export the engine file. So .ay be just yolov8 being incompatible with Frigate TensorRT detector?

qubex22 commented 12 months ago

Managed to export a "working" yolov8s 320x320 model and get more or less same error

I am using frigate 0.13 beta 2 btw and used TensorRT 8.5.3 to export the engine file. So .ay be just yolov8 being incompatible with Frigate TensorRT detector?

From what I've seen, the trt-models that come with frigate are built with the old tensorrt api that requires weights+cfg and they are darknet models, while yolov8 is from ultralytics and is built with trtexec. So I'm thinking that maybe you are right and frigate doesn't support tensorrt yolov8 yet.

Cheerpipe commented 12 months ago

output format seems to be different so it may need a different postprocessing. This requires someone with the adecuate knowledge (and hw) to dev and test yolov8 postprocessing to make tensorrt detector works with yolov8 .

========================== Tried with a different post processing for an output shape like [x,x,x] but didn't work. I toke some code from here

https://blog.kubwa.co.kr/yolov8-with-tensorrt-nvidia-triton-server-f6aa900a53b8

and here

https://github.com/ultralytics/ultralytics/blob/main/examples/YOLOv8-OpenCV-ONNX-Python/main.py

I am not a vision or python dev so maybe someone with better skills may make this works. At this point i don't know if it is a code or a model problem.

 def detect_raw(self, tensor_input):
        self.inputs[0].host = np.ascontiguousarray(
            tensor_input.astype(self.input_shape[-1])
        )
        trt_outputs = self._do_inference()
        ################################################################################
        df = pd.DataFrame(trt_outputs)

        outputs = np.array([cv2.transpose(trt_outputs[0].astype(np.float32))])
        #df.to_csv("/yolov8/detections.csv")
        rows = outputs.shape[1]

        boxes = []
        scores = []
        class_ids = []

        # Iterate through output to collect bounding boxes, confidence scores, and class IDs
        for i in range(rows):
            classes_scores = outputs[0][i][4:]
            (minScore, maxScore, minClassLoc, (x, maxClassIndex)) = cv2.minMaxLoc(classes_scores)
            if maxScore >= 0.25:
                box = [
                    outputs[0][i][0] - (0.5 * outputs[0][i][2]), outputs[0][i][1] - (0.5 * outputs[0][i][3]),
                    outputs[0][i][2], outputs[0][i][3]]
                boxes.append(box)
                scores.append(maxScore)
                class_ids.append(maxClassIndex)

        # Apply NMS (Non-maximum suppression)
        result_boxes = cv2.dnn.NMSBoxes(boxes, scores, 0.25, 0.45, 0.5)

        if len(result_boxes) == 0:
            return np.zeros((20, 6), np.float32)

        detections = []

        # Iterate through NMS results to draw bounding boxes and labels
        for i in range(len(result_boxes)):
            index = result_boxes[i]
            detection = self.process_yolo(class_ids[index], scores[index], boxes[index])
            detections.append(detection)

        return detections
    def process_yolo(self, class_id, conf, pos):
        return [
            class_id,  # class ID
            conf,  # confidence score
            (pos[1] - (pos[3])) / self.h,  # y_min
            (pos[0] - (pos[2])) / self.w,  # x_min
            (pos[1] + (pos[3])) / self.h,  # y_max
            (pos[0] + (pos[2])) / self.w,  # x_max
        ]

I still have KeyError but with values tha makes no sense like over 5000

An alternative may be build a detector uppon https://github.com/triple-Mu/YOLOv8-TensorRT/blob/main/infer-det.py

qubex22 commented 12 months ago

Tried 13 beta 2, with minimum config, just camera stream and detect, and letting it build the default model following the 13b2 docs and also the same KeyError... This is driving me crazy.

2023-10-22 22:54:10.154052435  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 147 MiB
2023-10-22 22:54:10.301427938  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublas.so.11
2023-10-22 22:54:10.301500887  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublas.so.11
2023-10-22 22:54:10.318515156  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as plugin tactic source
2023-10-22 22:54:10.318590775  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as core library tactic source
2023-10-22 22:54:10.318686164  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +8, now: CPU 265, GPU 290 (MiB)
2023-10-22 22:54:10.318786483  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcudnn.so.8
2023-10-22 22:54:10.318940212  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcudnn.so.8
2023-10-22 22:54:10.319031681  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as plugin tactic source
2023-10-22 22:54:10.324295317  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as core library tactic source
2023-10-22 22:54:10.324385587  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 267, GPU 300 (MiB)
2023-10-22 22:54:10.328527445  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Deserialization required 61674 microseconds.
2023-10-22 22:54:10.328598954  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +147, now: CPU 0, GPU 147 (MiB)
2023-10-22 22:54:10.351216406  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublas.so.11
2023-10-22 22:54:10.354553692  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublas.so.11
2023-10-22 22:54:10.354629562  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as plugin tactic source
2023-10-22 22:54:10.354702661  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as core library tactic source
2023-10-22 22:54:10.354762810  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 120, GPU 293 (MiB)
2023-10-22 22:54:10.354815900  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcudnn.so.8
2023-10-22 22:54:10.354882849  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcudnn.so.8
2023-10-22 22:54:10.354940418  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as plugin tactic source
2023-10-22 22:54:10.354990758  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as core library tactic source
2023-10-22 22:54:10.355040947  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 120, GPU 301 (MiB)
2023-10-22 22:54:10.355089977  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner device persistent memory is 1074176
2023-10-22 22:54:10.355139156  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner host persistent memory is 197408
2023-10-22 22:54:10.355187586  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Allocated activation device memory of size 29286912
2023-10-22 22:54:10.355248705  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +29, now: CPU 0, GPU 176 (MiB)
2023-10-22 22:54:10.355303295  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : CUDA lazy loading is enabled.
2023-10-22 22:54:10.355352484  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding input Memory 1228800 Bytes (307200 * DataType.FLOAT)
2023-10-22 22:54:10.355402244  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Input has Shape (1, 3, 320, 320)
2023-10-22 22:54:10.355451203  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding detections Memory 176400 Bytes (44100 * DataType.FLOAT)
2023-10-22 22:54:10.355498683  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : Output has Shape (1, 44100, 1, 1)
2023-10-22 22:54:10.355547392  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT loaded. Input shape is ((320, 320), <class 'numpy.float32'>)
2023-10-22 22:54:10.355597602  [2023-10-22 22:54:10] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT version is 8
2023-10-22 22:54:14.501297524  Process camera_processor:yi_pasillo:
2023-10-22 22:54:14.502085186  Traceback (most recent call last):
2023-10-22 22:54:14.502103776    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-22 22:54:14.502105416      self.run()
2023-10-22 22:54:14.502106996    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-22 22:54:14.502108526      self._target(*self._args, **self._kwargs)
2023-10-22 22:54:14.502112116    File "/opt/frigate/frigate/video.py", line 433, in track_camera
2023-10-22 22:54:14.502166955      process_frames(
2023-10-22 22:54:14.502181195    File "/opt/frigate/frigate/video.py", line 680, in process_frames
2023-10-22 22:54:14.502182465      detect(
2023-10-22 22:54:14.502187785    File "/opt/frigate/frigate/video.py", line 471, in detect
2023-10-22 22:54:14.502189045      region_detections = object_detector.detect(tensor_input)
2023-10-22 22:54:14.502190425    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-22 22:54:14.502191755      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-22 22:54:14.502193045  KeyError: -1
NickM-27 commented 12 months ago

What driver are you running? Another user saw this error and it was because they were using an outdated nvidia driver.

qubex22 commented 12 months ago

What driver are you running? Another user saw this error and it was because they were using an outdated nvidia driver.

It is a K620 with 535.104.05 driver. From the tensorrt docs I see maxwell is supported yet in 8.5.4

The most updated driver is v535.113.01 though. However I think there's a beta of v545, I'll try it

NickM-27 commented 12 months ago

The most updated driver is v535.113.01 though. However I think there's a beta of v545, I'll try it

that's what I would suggest

qubex22 commented 12 months ago

I've just updated and tried it with no success.

root@H3RMES:~# nvidia-smi
Sun Oct 22 23:53:32 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.06              Driver Version: 545.23.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro K620                    Off | 00000000:08:00.0 Off |                  N/A |
| 53%   60C    P0               3W /  30W |    328MiB /  2048MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      6448      C   frigate.detector.tensorrt                   219MiB |
|    0   N/A  N/A      6466      C   ffmpeg                                      106MiB |
+---------------------------------------------------------------------------------------+

This is part of the log:

Generating the following TRT Models: yolov7-320
Downloading yolo weights
2023-10-22 23:48:54.693998801  [INFO] Preparing go2rtc config...
2023-10-22 23:48:55.279999791  [INFO] Starting go2rtc...
2023-10-22 23:48:55.358361354  23:48:55.358 INF go2rtc version 1.7.1 linux/amd64
2023-10-22 23:48:55.358584913  23:48:55.358 INF [api] listen addr=0.0.0.0:1984
2023-10-22 23:48:55.358747912  23:48:55.358 INF [rtsp] listen addr=0.0.0.0:8554
2023-10-22 23:48:55.358892872  23:48:55.358 INF [webrtc] listen addr=0.0.0.0:8555/tcp

Creating yolov7-320.cfg and yolov7-320.weights

Done.
2023-10-22 23:49:04.576457981  [INFO] Starting go2rtc healthcheck service...

Generating yolov7-320.trt. This may take a few minutes.

/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:147: DeprecationWarning: Use network created with NetworkDefinitionCreationFlag::EXPLICIT_BATCH flag instead.
  builder.max_batch_size = MAX_BATCH_SIZE
/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:149: DeprecationWarning: Use set_memory_pool_limit instead.
  config.max_workspace_size = 1 << 30
/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:172: DeprecationWarning: Use build_serialized_network instead.
  engine = builder.build_engine(network, config)

2023-10-22 23:51:10.011689421  [2023-10-22 23:51:10] detector.tensorrt              INFO    : Starting detection process: 806
2023-10-22 23:51:10.270448331  [2023-10-22 23:51:10] frigate.app                    INFO    : Output process started: 808
2023-10-22 23:51:10.270586861  [2023-10-22 23:51:10] frigate.app                    INFO    : Camera processor started for yi_pasillo: 813
2023-10-22 23:51:10.270682860  [2023-10-22 23:51:10] frigate.app                    INFO    : Capture process started for yi_pasillo: 815

2023-10-22 23:51:10.310125605  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Num Available Devices: 1
2023-10-22 23:51:10.313907082  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
2023-10-22 23:51:10.314015131  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchedNMS_TRT version 1
2023-10-22 23:51:10.314120181  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::BatchTilePlugin_TRT version 1
2023-10-22 23:51:10.314205411  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Clip_TRT version 1
2023-10-22 23:51:10.314289530  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CoordConvAC version 1
2023-10-22 23:51:10.314372600  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResizeDynamic version 1
2023-10-22 23:51:10.314454470  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::CropAndResize version 1
2023-10-22 23:51:10.314541389  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DecodeBbox3DPlugin version 1
2023-10-22 23:51:10.314623549  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::DetectionLayer_TRT version 1
2023-10-22 23:51:10.314708839  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
2023-10-22 23:51:10.314789049  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
2023-10-22 23:51:10.314869118  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
2023-10-22 23:51:10.314951268  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::EfficientNMS_TRT version 1
2023-10-22 23:51:10.315035468  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::FlattenConcat_TRT version 1
2023-10-22 23:51:10.315127527  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::fMHA_V2 version 1
2023-10-22 23:51:10.315209387  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::fMHCA version 1
2023-10-22 23:51:10.315288777  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GenerateDetection_TRT version 1
2023-10-22 23:51:10.315371597  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchor_TRT version 1
2023-10-22 23:51:10.315455446  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GridAnchorRect_TRT version 1
2023-10-22 23:51:10.315536416  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::GroupNorm version 1
2023-10-22 23:51:10.315622836  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 1
2023-10-22 23:51:10.315702825  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::InstanceNormalization_TRT version 2
2023-10-22 23:51:10.315786415  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LayerNorm version 1
2023-10-22 23:51:10.315872565  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::LReLU_TRT version 1
2023-10-22 23:51:10.315955005  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
2023-10-22 23:51:10.316035184  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultilevelProposeROI_TRT version 1
2023-10-22 23:51:10.316122694  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
2023-10-22 23:51:10.316210084  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMSDynamic_TRT version 1
2023-10-22 23:51:10.316292433  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::NMS_TRT version 1
2023-10-22 23:51:10.316375663  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Normalize_TRT version 1
2023-10-22 23:51:10.316454713  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PillarScatterPlugin version 1
2023-10-22 23:51:10.316533333  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PriorBox_TRT version 1
2023-10-22 23:51:10.316614902  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalDynamic version 1
2023-10-22 23:51:10.316698302  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ProposalLayer_TRT version 1
2023-10-22 23:51:10.316788992  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Proposal version 1
2023-10-22 23:51:10.316878601  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::PyramidROIAlign_TRT version 1
2023-10-22 23:51:10.316955081  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Region_TRT version 1
2023-10-22 23:51:10.317032391  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Reorg_TRT version 1
2023-10-22 23:51:10.317124101  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ResizeNearest_TRT version 1
2023-10-22 23:51:10.317215630  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ROIAlign_TRT version 1
2023-10-22 23:51:10.317293980  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::RPROI_TRT version 1
2023-10-22 23:51:10.317375530  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::ScatterND version 1
2023-10-22 23:51:10.317455299  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SeqLen2Spatial version 1
2023-10-22 23:51:10.317534089  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SpecialSlice_TRT version 1
2023-10-22 23:51:10.317613139  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::SplitGeLU version 1
2023-10-22 23:51:10.317692879  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::Split version 1
2023-10-22 23:51:10.317774358  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Registered plugin creator - ::VoxelGeneratorPlugin version 1
2023-10-22 23:51:10.477917168  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 147 MiB
2023-10-22 23:51:10.608822897  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublas.so.11
2023-10-22 23:51:10.608894947  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublas.so.11
2023-10-22 23:51:10.625204291  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as plugin tactic source
2023-10-22 23:51:10.625287371  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as core library tactic source
2023-10-22 23:51:10.625414400  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +8, now: CPU 268, GPU 181 (MiB)
2023-10-22 23:51:10.625571520  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcudnn.so.8
2023-10-22 23:51:10.625744969  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcudnn.so.8
2023-10-22 23:51:10.625828209  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as plugin tactic source
2023-10-22 23:51:10.631090911  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as core library tactic source
2023-10-22 23:51:10.631202561  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 270, GPU 191 (MiB)
2023-10-22 23:51:10.640966717  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Deserialization required 63731 microseconds.
2023-10-22 23:51:10.641061507  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +148, now: CPU 0, GPU 148 (MiB)
2023-10-22 23:51:10.660826399  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcublas.so.11
2023-10-22 23:51:10.660896619  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcublas.so.11
2023-10-22 23:51:10.660985948  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as plugin tactic source
2023-10-22 23:51:10.661068018  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cublas as core library tactic source
2023-10-22 23:51:10.661165898  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 122, GPU 184 (MiB)
2023-10-22 23:51:10.661252707  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Trying to load shared library libcudnn.so.8
2023-10-22 23:51:10.661374237  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Loaded shared library libcudnn.so.8
2023-10-22 23:51:10.661454747  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as plugin tactic source
2023-10-22 23:51:10.661539506  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Using cuDNN as core library tactic source
2023-10-22 23:51:10.661626246  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 122, GPU 192 (MiB)
2023-10-22 23:51:10.661708566  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner device persistent memory is 1064448
2023-10-22 23:51:10.661792345  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Total per-runner host persistent memory is 199056
2023-10-22 23:51:10.661872475  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Allocated activation device memory of size 29286400
2023-10-22 23:51:10.661959755  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +29, now: CPU 0, GPU 177 (MiB)
2023-10-22 23:51:10.662045545  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : CUDA lazy loading is enabled.
2023-10-22 23:51:10.662467553  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding input Memory 1228800 Bytes (307200 * DataType.FLOAT)
2023-10-22 23:51:10.662552103  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Input has Shape (1, 3, 320, 320)
2023-10-22 23:51:10.662633233  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Allocated Tensor Binding detections Memory 176400 Bytes (44100 * DataType.FLOAT)
2023-10-22 23:51:10.662710112  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : Output has Shape (1, 44100, 1, 1)
2023-10-22 23:51:10.662793262  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT loaded. Input shape is ((320, 320), <class 'numpy.float32'>)
2023-10-22 23:51:10.662880662  [2023-10-22 23:51:10] frigate.detectors.plugins.tensorrt DEBUG   : TensorRT version is 8
2023-10-22 23:51:14.987266359  [2023-10-22 23:51:14] asyncio                        DEBUG   : Using selector: EpollSelector
2023-10-22 23:51:16.579766952  Process camera_processor:yi_pasillo:
2023-10-22 23:51:16.580948828  Traceback (most recent call last):
2023-10-22 23:51:16.580993938    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-22 23:51:16.580996188      self.run()
2023-10-22 23:51:16.580997848    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-22 23:51:16.580999438      self._target(*self._args, **self._kwargs)
2023-10-22 23:51:16.581009397    File "/opt/frigate/frigate/video.py", line 501, in track_camera
2023-10-22 23:51:16.581010787      process_frames(
2023-10-22 23:51:16.581012267    File "/opt/frigate/frigate/video.py", line 876, in process_frames
2023-10-22 23:51:16.581013437      detect(
2023-10-22 23:51:16.581014917    File "/opt/frigate/frigate/video.py", line 575, in detect
2023-10-22 23:51:16.581016347      region_detections = object_detector.detect(tensor_input)
2023-10-22 23:51:16.581018427    File "/opt/frigate/frigate/object_detection.py", line 225, in detect
2023-10-22 23:51:16.581019577      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))
2023-10-22 23:51:16.581037467  KeyError: -1
2023-10-22 23:51:19.987372569  [2023-10-22 23:51:19] asyncio                        DEBUG   : Using selector: EpollSelector
2023-10-22 23:51:24.987429541  [2023-10-22 23:51:24] asyncio                        DEBUG   : Using selector: EpollSelector
2023-10-22 23:51:25.051001568  [2023-10-22 23:51:25] peewee                         DEBUG   : ('SELECT "t1"."start_time", "t1"."end_time" FROM "event" AS "t1" WHERE ((("t1"."camera" = ?) AND (("t1"."end_time" IS ?) OR ("t1"."end_time" >= ?))) AND "t1"."has_clip") ORDER BY "t1"."start_time"', ['yi_pasillo', None, 1698011471.0])
2023-10-22 23:51:25.053278069  [2023-10-22 23:51:25] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
2023-10-22 23:51:25.053354819  [2023-10-22 23:51:25] frigate.record.maintainer      ERROR   : list index out of range
github-actions[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.