search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.88k stars 1.72k forks source link

[Support]: issues with frigate:dev-f6a4c2a-tensorrt #9750

Closed bughattiveyron closed 8 months ago

bughattiveyron commented 8 months ago

Describe the problem you are having

I know this is a dev branch and unsupported, but i would like to see if I can get this to work. Currently it keeps restarting the docker container

Version

0.14 unstable

Frigate config file

mqtt:
  host: xx.xx.xx.240
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: user
  password: pass

birdseye:
  enabled: True
  width: 1920
  height: 1080
  quality: 8
  mode: objects

detectors:
  tensorrt:
    type: tensorrt

model:
  path: /model_cache/tensorrt/yolov7-tiny-416.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 416
  height: 416

cameras:
  Back_Drive:
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -ar 44100 -c:a aac
      inputs:
        - path: rtsp://liquid:pass@xx.xx.xx.191:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - rtmp
      hwaccel_args: preset-nvidia-h264
    objects:
      track:
        - person
        - car
        - truck
        - bicycle
        - motorcycle
        - dog
        - cat
    snapshots:
      enabled: True
    record:
      enabled: True
      retain:
        days: 180
        mode: motion
      events:
        retain:
          default: 180
          mode: active_objects
    detect:
      fps: 20
  Front_Garage:
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -ar 44100 -c:a aac
      inputs:
        - path: rtsp://admin:pass@xx.xx.xx.97:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - rtmp
      hwaccel_args: preset-nvidia-h264
    objects:
      track:
        - person
        - car
        - truck
        - bicycle
        - motorcycle
        - dog
        - cat
    snapshots:
      enabled: True
    record:
      enabled: True
      retain:
        days: 180
        mode: motion
      events:
        retain:
          default: 180
          mode: active_objects
    detect:
      fps: 20
  Back_Yard:
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -ar 44100 -c:a aac
      inputs:
        - path: rtsp://admin:pass@xx.xx.xx.205:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - rtmp
      hwaccel_args: preset-nvidia-h264
    objects:
      track:
        - person
        - car
        - truck
        - bicycle
        - motorcycle
        - dog
        - cat
    snapshots:
      enabled: True
    record:
      enabled: True
      retain:
        days: 180
        mode: motion
      events:
        retain:
          default: 180
          mode: active_objects
    detect:
      fps: 20
  Front_Side:
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -ar 44100 -c:a aac
      inputs:
        - path: rtsp://admin:pass@xx.xx.xx.131:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - rtmp
      hwaccel_args: preset-nvidia-h264
    objects:
      track:
        - person
        - car
        - truck
        - bicycle
        - motorcycle
        - dog
        - cat
    snapshots:
      enabled: True
    record:
      enabled: True
      retain:
        days: 180
        mode: motion
      events:
        retain:
          default: 180
          mode: active_objects
    detect:
      fps: 20

Relevant log output

s6-rc: info: service trt-model-prepare: starting
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
No models to convert.
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service trt-model-prepare successfully started
s6-rc: info: service frigate: starting
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service go2rtc-healthcheck successfully started
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2024-02-08 10:54:51.709098073  [INFO] Preparing Frigate...
2024-02-08 10:54:51.709791843  [INFO] Starting NGINX...
s6-rc: info: service legacy-services successfully started
2024-02-08 10:54:51.713743386  [INFO] Preparing new go2rtc config...
2024-02-08 10:54:51.718888694  [INFO] Starting Frigate...
2024-02-08 10:54:52.060171491  [INFO] Starting go2rtc...
2024-02-08 10:54:52.127137910  10:54:52.127 INF go2rtc version 1.8.5 linux/amd64
2024-02-08 10:54:52.127500531  10:54:52.127 INF [api] listen addr=:1984
2024-02-08 10:54:52.127554653  10:54:52.127 INF [rtsp] listen addr=:8554
2024-02-08 10:54:52.127704821  10:54:52.127 INF [webrtc] listen addr=:8555
2024-02-08 10:54:53.408818495  [2024-02-08 10:54:53] frigate.app                    INFO    : Starting Frigate (0.14.0-f6a4c2a)
2024-02-08 10:54:53.784100487  [2024-02-08 10:54:53] frigate.util.services          INFO    : Automatically detected nvidia hwaccel for video decoding
     
08 10:54:54.360226002  2024/02/08 10:54:54 [error] 158#158: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.66.66.202, server: , request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:5002/", host: "10.66.66.48:5000"
2024-02-08 10:54:54.360278222  10.66.66.202 - - [08/Feb/2024:10:54:54 -0700] "GET /ws HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
     
08 10:55:00.359101597  2024/02/08 10:55:00 [error] 159#159: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.66.66.202, server: , request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:5002/", host: "10.66.66.48:5000"
2024-02-08 10:55:00.359168865  10.66.66.202 - - [08/Feb/2024:10:55:00 -0700] "GET /ws HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
2024-02-08 10:55:01.712567379  [INFO] Starting go2rtc healthcheck service...
     
08 10:55:06.361361754  2024/02/08 10:55:06 [error] 160#160: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 10.66.66.202, server: , request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:5002/", host: "10.66.66.48:5000"
2024-02-08 10:55:06.361393071  10.66.66.202 - - [08/Feb/2024:10:55:06 -0700] "GET /ws HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
2024-02-08 10:55:06.678023921  2024/02/08 10:55:06 [error] 161#161: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/version HTTP/1.1", upstream: "http://127.0.0.1:5001version", host: "127.0.0.1:5000"
2024-02-08 10:55:08.207805157  [2024-02-08 10:55:08] peewee_migrate.logs            INFO    : Starting migrations
2024-02-08 10:55:08.211599141  [2024-02-08 10:55:08] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-02-08 10:55:08.219827897  [2024-02-08 10:55:08] frigate.app                    INFO    : Recording process started: 540
2024-02-08 10:55:08.222889661  [2024-02-08 10:55:08] frigate.app                    INFO    : go2rtc process pid: 102
2024-02-08 10:55:08.254782025  [2024-02-08 10:55:08] detector.tensorrt              INFO    : Starting detection process: 549
2024-02-08 10:55:08.269834025  [2024-02-08 10:55:08] frigate.app                    INFO    : Output process started: 551
2024-02-08 10:55:08.291868730  [2024-02-08 10:55:08] frigate.app                    INFO    : Camera processor started for Back_Drive: 559
2024-02-08 10:55:08.298977814  [2024-02-08 10:55:08] frigate.app                    INFO    : Camera processor started for Front_Garage: 561
2024-02-08 10:55:08.308102499  [2024-02-08 10:55:08] frigate.app                    INFO    : Camera processor started for Back_Yard: 563
2024-02-08 10:55:08.317465656  [2024-02-08 10:55:08] frigate.app                    INFO    : Camera processor started for Front_Side: 567
2024-02-08 10:55:08.326915218  [2024-02-08 10:55:08] frigate.app                    INFO    : Capture process started for Back_Drive: 570
2024-02-08 10:55:08.335846726  [2024-02-08 10:55:08] frigate.app                    INFO    : Capture process started for Front_Garage: 571
2024-02-08 10:55:08.341112917  Process detector:tensorrt:
2024-02-08 10:55:08.342107605  Traceback (most recent call last):
2024-02-08 10:55:08.342324762    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-02-08 10:55:08.342326987      self.run()
2024-02-08 10:55:08.342468716    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-02-08 10:55:08.342470887      self._target(*self._args, **self._kwargs)
2024-02-08 10:55:08.342592309    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-02-08 10:55:08.342594740      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-02-08 10:55:08.342732219    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2024-02-08 10:55:08.342734531      self.detect_api = create_detector(detector_config)
2024-02-08 10:55:08.342855687    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-02-08 10:55:08.342857726      return api(detector_config)
2024-02-08 10:55:08.342991929    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 220, in __init__
2024-02-08 10:55:08.342994232      self.engine = self._load_engine(detector_config.model.path)
2024-02-08 10:55:08.343117950    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 88, in _load_engine
2024-02-08 10:55:08.343120436      with open(model_path, "rb") as f, trt.Runtime(self.trt_logger) as runtime:
2024-02-08 10:55:08.343272747  FileNotFoundError: [Errno 2] No such file or directory: '/trt-models/yolov7-tiny-416.trt'
2024-02-08 10:55:08.343466648  Exception ignored in: <function TensorRtDetector.__del__ at 0x7fa029aa71f0>
2024-02-08 10:55:08.343590440  Traceback (most recent call last):
2024-02-08 10:55:08.343694382    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 239, in __del__
2024-02-08 10:55:08.344057678      if self.outputs is not None:
2024-02-08 10:55:08.344225973  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'
2024-02-08 10:55:08.353761120  [2024-02-08 10:55:08] frigate.app                    INFO    : Capture process started for Back_Yard: 576
2024-02-08 10:55:08.368239094  [2024-02-08 10:55:08] frigate.app                    INFO    : Capture process started for Front_Side: 588
2024-02-08 10:55:28.657572882  [2024-02-08 10:55:28] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service go2rtc-healthcheck: stopping
2024-02-08 10:55:28.676406069  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
s6-rc: info: service go2rtc-healthcheck successfully stopped
2024-02-08 10:55:28.746967994  [INFO] Service NGINX exited with code 0 (by signal 0)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service nginx-log: stopping
s6-rc: info: service frigate: stopping
2024-02-08 10:55:28.749647270  [2024-02-08 10:55:28] frigate.app                    INFO    : Stopping...
s6-rc: info: service nginx-log successfully stopped
2024-02-08 10:55:28.751112178  [2024-02-08 10:55:28] frigate.storage                INFO    : Exiting storage maintainer...
2024-02-08 10:55:28.751271104  [2024-02-08 10:55:28] frigate.stats                  INFO    : Exiting stats emitter...
2024-02-08 10:55:28.751770298  [2024-02-08 10:55:28] frigate.record.cleanup         INFO    : Exiting recording cleanup...
2024-02-08 10:55:28.751964241  [2024-02-08 10:55:28] frigate.ptz.autotrack          INFO    : Exiting autotracker...
2024-02-08 10:55:28.752487168  [2024-02-08 10:55:28] frigate.events.cleanup         INFO    : Exiting event cleanup...
2024-02-08 10:55:28.752625151  [2024-02-08 10:55:28] frigate.watchdog               INFO    : Exiting watchdog...
2024-02-08 10:55:28.758382588  [2024-02-08 10:55:28] frigate.object_processing      INFO    : Exiting object processor...
2024-02-08 10:55:28.879021032  [2024-02-08 10:55:28] frigate.comms.ws               INFO    : Exiting websocket client...
2024-02-08 10:55:29.676941288  [2024-02-08 10:55:29] frigate.events.maintainer      INFO    : Exiting event processor...
2024-02-08 10:55:29.677300829  [2024-02-08 10:55:29] peewee.sqliteq                 INFO    : writer received shutdown request, exiting.
2024-02-08 10:55:29.681205996  [2024-02-08 10:55:29] frigate.record.maintainer      INFO    : Exiting recording maintenance...

FFprobe output from your camera

# ffprobe rtsp://admin:pass3@xx.xx.xx.205:554/cam/realmonitor?channel=1&subtype=0
# ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[rtsp @ 0x5568c88ac140] method DESCRIBE failed: 404 Not Found
rtsp://admin:pass@xx.xx.xx.205:554/cam/realmonitor?channel=1: Server returned 404 Not Found

Frigate stats

No response

Operating system

Debian

Install method

Docker CLI

Coral version

CPU (no coral)

Network connection

Wired

Camera make and model

Amcrest 5MP Turret POE Camera • UltraHD Outdoor IP Camera POE with Mic/Audio • 5-Megapixel Security Surveillance Cameras

Any other information that may be helpful

I changed nothing on the config file that worked fine on 0.12 dev, i did see the trt models location changed and that in version 0.14 stated it would generate on its on.

nvidia-smi

root@liquidnas:~# nvidia-smi
Thu Feb  8 11:18:43 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1080        Off | 00000000:01:00.0 Off |                  N/A |
| 42%   66C    P2              53W / 180W |   1698MiB /  8192MiB |     13%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2470      G   /usr/lib/xorg/Xorg                           12MiB |
|    0   N/A  N/A   1262947      C   ffmpeg                                      420MiB |
|    0   N/A  N/A   1262950      C   ffmpeg                                      420MiB |
|    0   N/A  N/A   1262958      C   ffmpeg                                      420MiB |
|    0   N/A  N/A   1262964      C   ffmpeg                                      420MiB |
+---------------------------------------------------------------------------------------+

docker cli

docker run -d \
  --name frigate14 \
  --restart=unless-stopped \
  --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \
  --shm-size=512m \
  --gpus=all \
  -v /zfsmount/Media/frigate/media:/media/frigate \
  -v /zfsmount/Media/frigate/config:/config \
  -v /etc/localtime:/etc/localtime:ro \
  -e FRIGATE_RTSP_PASSWORD='pass' \
  -e YOLO_MODELS=yolov7-tiny-416,yolov4-tiny-288,yolov4-tiny-416 \
  -e USE_FP16=False \
  -e "NVIDIA_DRIVER_CAPABILITIES=all" \
  -e "NVIDIA_VISIBLE_DEVICES=all" \
  -p 5000:5000 \
  -p 8554:8554 \
  -p 8555:8555/tcp \
  -p 8555:8555/udp \
  ghcr.io/blakeblackshear/frigate:dev-f6a4c2a-tensorrt
NickM-27 commented 8 months ago

your problem is No such file or directory: '/trt-models/yolov7-tiny-416.trt'

bughattiveyron commented 8 months ago

whelp, im an idiot lol. Apparently i need to read more in depth.

set to this in config, now it works

model:
  path: /config/model_cache/tensorrt/yolov7-tiny-416.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 416
  height: 416

So next question and I can start a new thread. In testing this, i left the old docker with media and config intact and created a new folder on my nas to hold this new one. Is there a best way to transfer all events and such from the old container to new one?

NickM-27 commented 8 months ago

if the db and all files are copied then it will work