search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.22k stars 1.76k forks source link

[Detector Support]: How to use Google Coral TPU and OpenVINO at the same time #5962

Closed to4ko closed 1 year ago

to4ko commented 1 year ago

Describe the problem you are having

Hello!

I'm running local server with i5-10600 CPU with Debian and HA Supervised. Frigate installed as HA Addon. Google Coral TPU M2 card.

Trying to use both detectors - TPU and integrated Intel GPU - gives error and Frigate crashing every time.

Please advise how these two can be used simultaneously.

Version

0.12.0

Frigate config file

detectors:
  coral:
    type: edgetpu
    device: pci:0
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

docker-compose file or Docker CLI command

NA

Relevant log output

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service go2rtc-healthcheck successfully started
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2023-04-10 08:00:54.002778906  [INFO] Preparing go2rtc config...
2023-04-10 08:00:54.003441077  [INFO] Starting Frigate...
2023-04-10 08:00:54.004016290  [INFO] Starting NGINX...
s6-rc: info: service legacy-services successfully started
2023-04-10 08:00:54.027136536  [INFO] Got IP address from supervisor: 192.168.1.14
2023-04-10 08:00:54.051162095  [INFO] Got WebRTC port from supervisor: 8556
2023-04-10 08:00:54.231828641  [INFO] Starting go2rtc...
2023-04-10 08:00:54.281177264  11:00:54.281 INF go2rtc version 1.2.0 linux/amd64
2023-04-10 08:00:54.281358318  11:00:54.281 INF [api] listen addr=:1984
2023-04-10 08:00:54.281513462  11:00:54.281 INF [rtsp] listen addr=:8554
2023-04-10 08:00:54.281645092  11:00:54.281 INF [srtp] listen addr=:8443
2023-04-10 08:00:54.281758986  11:00:54.281 INF [webrtc] listen addr=:8555
2023-04-10 08:00:54.756429202  [2023-04-10 11:00:54] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-04-10 08:00:54.775234213  [2023-04-10 11:00:54] frigate.config                 WARNING : Customizing more than a detector model path is unsupported.
2023-04-10 08:00:54.779078718  [2023-04-10 11:00:54] frigate.app                    INFO    : Creating directory: /tmp/cache
2023-04-10 08:00:54.780227900  [2023-04-10 11:00:54] peewee_migrate                 INFO    : Starting migrations
2023-04-10 08:00:54.782717120  [2023-04-10 11:00:54] peewee_migrate                 INFO    : There is nothing to migrate
2023-04-10 08:00:54.792562270  [2023-04-10 11:00:54] detector.coral                 INFO    : Starting detection process: 587
2023-04-10 08:00:54.828016831  [2023-04-10 11:00:54] detector.ov                    INFO    : Starting detection process: 589
2023-04-10 08:00:54.828118776  [2023-04-10 11:00:54] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:0
2023-04-10 08:00:54.828184086  [2023-04-10 11:00:54] frigate.app                    INFO    : Output process started: 591
2023-04-10 08:00:54.828244933  [2023-04-10 11:00:54] frigate.app                    INFO    : Camera processor started for entrance: 594
2023-04-10 08:00:54.828404554  [2023-04-10 11:00:54] frigate.app                    INFO    : Camera processor started for hall: 596
2023-04-10 08:00:54.828407162  [2023-04-10 11:00:54] frigate.app                    INFO    : Camera processor started for street: 597
2023-04-10 08:00:54.828411969  [2023-04-10 11:00:54] frigate.app                    INFO    : Camera processor started for dixy: 599
2023-04-10 08:00:54.828429965  [2023-04-10 11:00:54] frigate.app                    INFO    : Camera processor started for window: 600
2023-04-10 08:00:54.828487122  [2023-04-10 11:00:54] frigate.app                    INFO    : Capture process started for entrance: 602
2023-04-10 08:00:54.828583751  [2023-04-10 11:00:54] frigate.app                    INFO    : Capture process started for hall: 606
2023-04-10 08:00:54.828643021  [2023-04-10 11:00:54] frigate.app                    INFO    : Capture process started for street: 610
2023-04-10 08:00:54.828700134  [2023-04-10 11:00:54] frigate.app                    INFO    : Capture process started for dixy: 621
2023-04-10 08:00:54.831991128  [2023-04-10 11:00:54] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-04-10 08:00:54.833369546  [2023-04-10 11:00:54] frigate.app                    INFO    : Capture process started for window: 636
2023-04-10 08:00:54.977291973  [2023-04-10 11:00:54] frigate.detectors.plugins.openvino INFO    : Model Input Shape: {1, 300, 300, 3}
2023-04-10 08:00:54.977377457  [2023-04-10 11:00:54] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: {1, 1, 100, 7}
2023-04-10 08:00:54.977421807  [2023-04-10 11:00:54] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
2023-04-10 08:00:55.792538880  Process detector:coral:
2023-04-10 08:00:55.792664686  Traceback (most recent call last):
2023-04-10 08:00:55.792689962    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-10 08:00:55.792691382      self.run()
2023-04-10 08:00:55.792692829    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-10 08:00:55.792693947      self._target(*self._args, **self._kwargs)
2023-04-10 08:00:55.792695077    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2023-04-10 08:00:55.792696350      detections = object_detector.detect_raw(input_frame)
2023-04-10 08:00:55.792714079    File "/opt/frigate/frigate/object_detection.py", line 71, in detect_raw
2023-04-10 08:00:55.792715483      return self.detect_api.detect_raw(tensor_input=tensor_input)
2023-04-10 08:00:55.792716619    File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 55, in detect_raw
2023-04-10 08:00:55.792717908      self.interpreter.set_tensor(self.tensor_input_details[0]["index"], tensor_input)
2023-04-10 08:00:55.792719303    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 572, in set_tensor
2023-04-10 08:00:55.792720443      self._interpreter.SetTensor(tensor_index, value)
2023-04-10 08:00:55.792722339  ValueError: Cannot set tensor: Dimension mismatch. Got 300 but expected 320 for dimension 1 of input 7.
2023-04-10 08:00:58.784761023  172.30.32.1 - - [10/Apr/2023:11:00:58 +0300] "GET /api/stats HTTP/1.1" 200 3592 "-" "HomeAssistant/2023.4.2 aiohttp/3.8.4 Python/3.10" "-"
2023-04-10 08:01:04.004025859  [INFO] Starting go2rtc healthcheck service...
2023-04-10 08:01:06.788174498  172.30.32.1 - - [10/Apr/2023:11:01:06 +0300] "GET /api/stats HTTP/1.1" 200 3670 "-" "HomeAssistant/2023.4.2 aiohttp/3.8.4 Python/3.10" "-"
2023-04-10 08:01:14.784226962  172.30.32.1 - - [10/Apr/2023:11:01:14 +0300] "GET /api/stats HTTP/1.1" 200 3610 "-" "HomeAssistant/2023.4.2 aiohttp/3.8.4 Python/3.10" "-"
2023-04-10 08:01:15.049073035  [2023-04-10 11:01:15] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-04-10 08:01:15.054742086  [2023-04-10 11:01:15] detector.coral                 INFO    : Starting detection process: 1190
2023-04-10 08:01:15.059999673  [2023-04-10 11:01:15] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:0
2023-04-10 08:01:15.062471293  [2023-04-10 11:01:15] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-04-10 08:01:15.097194698  Process detector:coral:
2023-04-10 08:01:15.097799256  Traceback (most recent call last):
2023-04-10 08:01:15.097801408    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-10 08:01:15.097802460      self.run()
2023-04-10 08:01:15.097803652    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-10 08:01:15.097804695      self._target(*self._args, **self._kwargs)
2023-04-10 08:01:15.097805739    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2023-04-10 08:01:15.097806782      detections = object_detector.detect_raw(input_frame)
2023-04-10 08:01:15.097807841    File "/opt/frigate/frigate/object_detection.py", line 71, in detect_raw
2023-04-10 08:01:15.097808849      return self.detect_api.detect_raw(tensor_input=tensor_input)
2023-04-10 08:01:15.097809928    File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 55, in detect_raw
2023-04-10 08:01:15.097826819      self.interpreter.set_tensor(self.tensor_input_details[0]["index"], tensor_input)
2023-04-10 08:01:15.097828086    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 572, in set_tensor
2023-04-10 08:01:15.097829525      self._interpreter.SetTensor(tensor_index, value)
2023-04-10 08:01:15.097830677  ValueError: Cannot set tensor: Dimension mismatch. Got 300 but expected 320 for dimension 1 of input 7.
2023-04-10 08:01:22.807003912  172.30.32.1 - - [10/Apr/2023:11:01:22 +0300] "GET /api/stats HTTP/1.1" 200 3627 "-" "HomeAssistant/2023.4.2 aiohttp/3.8.4 Python/3.10" "-"
2023-04-10 08:01:25.053493013  [2023-04-10 11:01:25] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service go2rtc-healthcheck: stopping
2023-04-10 08:01:25.060524190  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
s6-rc: info: service go2rtc-healthcheck successfully stopped
2023-04-10 08:01:25.144335268  [INFO] Service NGINX exited with code 0 (by signal 0)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service nginx-log: stopping
s6-rc: info: service frigate: stopping
2023-04-10 08:01:25.147046846  [2023-04-10 11:01:25] frigate.app                    INFO    : Stopping...
s6-rc: info: service nginx-log successfully stopped
2023-04-10 08:01:25.148144991  [2023-04-10 11:01:25] root                           INFO    : Waiting for detection process to exit gracefully...
2023-04-10 08:01:25.148361952  [2023-04-10 11:01:25] frigate.storage                INFO    : Exiting storage maintainer...
2023-04-10 08:01:25.148538561  [2023-04-10 11:01:25] frigate.events                 INFO    : Exiting event cleanup...
2023-04-10 08:01:25.149053511  [2023-04-10 11:01:25] frigate.stats                  INFO    : Exiting stats emitter...
2023-04-10 08:01:25.149229384  [2023-04-10 11:01:25] frigate.record                 INFO    : Exiting recording cleanup...
2023-04-10 08:01:25.149371575  [2023-04-10 11:01:25] frigate.watchdog               INFO    : Exiting watchdog...
2023-04-10 08:01:25.149657048  [2023-04-10 11:01:25] frigate.record                 INFO    : Exiting recording maintenance...
2023-04-10 08:01:25.149659418  [2023-04-10 11:01:25] detector.ov                    INFO    : Signal to exit detection process...
2023-04-10 08:01:25.149661064  [2023-04-10 11:01:25] detector.ov                    INFO    : Exited detection process...
2023-04-10 08:01:25.152182258  [2023-04-10 11:01:25] frigate.object_processing      INFO    : Exiting object processor...
2023-04-10 08:01:25.178846919  [2023-04-10 11:01:25] root                           INFO    : Detection process has exited...
2023-04-10 08:01:25.318283875  [2023-04-10 11:01:25] frigate.comms.ws               INFO    : Exiting websocket client...
2023-04-10 08:01:25.896086328  [2023-04-10 11:01:25] frigate.events                 INFO    : Exiting event processor...
2023-04-10 08:01:25.896727495  [2023-04-10 11:01:25] peewee.sqliteq                 INFO    : writer received shutdown request, exiting.
2023-04-10 08:01:25.905022894  [2023-04-10 11:01:25] watchdog.entrance              INFO    : Terminating the existing ffmpeg process...
2023-04-10 08:01:25.905364258  [2023-04-10 11:01:25] watchdog.hall                  INFO    : Terminating the existing ffmpeg process...
2023-04-10 08:01:25.905675077  [2023-04-10 11:01:25] watchdog.street                INFO    : Terminating the existing ffmpeg process...
2023-04-10 08:01:25.905968073  [2023-04-10 11:01:25] watchdog.window                INFO    : Terminating the existing ffmpeg process...
2023-04-10 08:01:25.906200892  [2023-04-10 11:01:25] watchdog.street                INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-10 08:01:25.906451824  [2023-04-10 11:01:25] watchdog.entrance              INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-10 08:01:25.906674563  [2023-04-10 11:01:25] watchdog.dixy                  INFO    : Terminating the existing ffmpeg process...
2023-04-10 08:01:25.906861136  [2023-04-10 11:01:25] watchdog.hall                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-10 08:01:25.907044512  [2023-04-10 11:01:25] watchdog.window                INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-10 08:01:25.907235457  [2023-04-10 11:01:25] watchdog.dixy                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-04-10 08:01:26.372285690  [2023-04-10 11:01:26] frigate.output                 INFO    : exiting output process...
2023-04-10 08:01:26.773189498  [2023-04-10 11:01:26] frigate.video                  INFO    : hall: exiting subprocess
2023-04-10 08:01:26.800761872  [2023-04-10 11:01:26] frigate.video                  INFO    : entrance: exiting subprocess
2023-04-10 08:01:26.887688717  [2023-04-10 11:01:26] frigate.video                  INFO    : window: exiting subprocess
2023-04-10 08:01:30.443352202  /usr/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 26 leaked shared_memory objects to clean up at shutdown
2023-04-10 08:01:30.443355463    warnings.warn('resource_tracker: There appear to be %d '
2023-04-10 08:01:30.444941858  [INFO] Service Frigate exited with code 0 (by signal 0)
s6-rc: info: service frigate successfully stopped
s6-rc: info: service go2rtc: stopping
s6-rc: info: service frigate-log: stopping
2023-04-10 08:01:30.447415113  exit OK
s6-rc: info: service frigate-log successfully stopped
2023-04-10 08:01:30.449534798  [INFO] The go2rtc service exited with code 0 (by signal 0)
s6-rc: info: service go2rtc successfully stopped
s6-rc: info: service go2rtc-log: stopping
s6-rc: info: service go2rtc-log successfully stopped
s6-rc: info: service log-prepare: stopping
s6-rc: info: service s6rc-fdholder: stopping
s6-rc: info: service log-prepare successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service s6rc-fdholder successfully stopped
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Operating system

Debian

Install method

HassOS Addon

Coral version

M.2

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

Using different types of detectors is not currently supported

xekil commented 2 months ago

Using different types of detectors is not currently supported

Hello, Has there been any development on this subject? Indeed, I have a Coral USB and I wonder if it is possible to use OpenVino at the same time? Thanks

NickM-27 commented 2 months ago

you can only mix detectors when the models are identical

xekil commented 2 months ago

you can only mix detectors when the models are identical

Ok thank you for you reply