[Detector Support]: Dual TPU detected but crashes

[kcraig73]

Describe the problem you are having

Staring with minimal configuration, dual TPU detected yet immediately crashes saying no TPU detected. No errors when enabling only one TPU.

Version

0.12.0

Frigate config file

mqtt:
  enabled: False

detectors:
  coral1:
    type: edgetpu
    device: pci:0
  # coral2:
  #   type: edgetpu
  #   device: pci:1

cameras:
  familyroom:
    ffmpeg:
      inputs:
        - path: rtsps://192.168.1.1:7441
          roles:
            - detect
      hwaccel_args: preset-nvidia-h264      
    detect:
      enabled: True 
      width: 640 
      height: 360

docker-compose file or Docker CLI command

Installed via Community Apps. Unsure how to share config.

Repository:
ghcr.io/blakeblackshear/frigate:stable
Docker Hub URL:
https://hub.docker.com/r/blakeblackshear/frigate/
Icon URL:
https://raw.githubusercontent.com/yayitazale/unraid-templates/main/frigate.png
WebUI:
http://[IP]:[PORT:5000]
Extra Parameters:
--shm-size=256mb --mount type=tmpfs,target=/tmp/cache,tmpfs-size=100000000 --restart unless-stopped --runtime=nvidia
Post Arguments:
CPU Pinning:
CPU:
HT:
Network Type:

Bridge
Console shell command:

Shell
Privileged:
OFF
Config Path:
/mnt/user/appdata/frigate-1
EDITREMOVE
Container Path: /config

Media path:
/mnt/disks/CCTV/frigate/
EDITREMOVE
Container Path: /media/frigate

HTTP port:
5000
EDITREMOVE
Container Port: 5000

RTSP port:
8556
EDITREMOVE
Container Port: 8554

Frigate RTSP Password:
*redacted*
EDITREMOVE
Container Variable: FRIGATE_RTSP_PASSWORD

Coral TPU/NCS2 Mapping:
/dev/apex_0
EDITREMOVE
Container Device: /dev/bus/usb
Use /dev/bus/usb for USB devices and /dev/apex_0 for PCIe devices (you must install the drivers first for PCIe devices). Remove this if you are not using it.
Nvidia Visible Devices:
GPU-a25a7943-2e63-ef1c-bd67-31b442324b6b
EDITREMOVE
Container Variable: NVIDIA_VISIBLE_DEVICES
This info can be found in the NVidia driver plugin.. Only for Nvidia GPU decoding and/or detection. Remove this if you are not using it.
Nvidia Driver Capabilities:
compute,utility,video
EDITREMOVE
Container Variable: NVIDIA_DRIVER_CAPABILITIES
Only for Nvidia GPU decoding and/or detection. Remove this if you are not using it.
go2rtc TCP port:
8555
EDITREMOVE
Container Port: 8555
If you are not going to use MSE/WebRCT you can delete all the WebRTC entrys
go2rtc UDP port:
8555
EDITREMOVE
Container Port: 8555
If you are not going to use MSE/WebRCT you can delete all the WebRTC entrys
go2rtc server port:
1984
EDITREMOVE
Container Port: 1984
For Home Assistant WebRTC integration. Remove this if you are not using it.

 Hide more settings ...
Localtime:
/etc/localtime
EDITREMOVE
Container Path: /etc/localtime

Relevant log output

2023-08-12 22:59:56.107581280  [INFO] Starting Frigate...
2023-08-12 22:59:57.040687579  [2023-08-12 22:59:57] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-08-12 22:59:57.053777059  [2023-08-12 22:59:57] peewee_migrate                 INFO    : Starting migrations
2023-08-12 22:59:57.057717847  [2023-08-12 22:59:57] peewee_migrate                 INFO    : There is nothing to migrate
2023-08-12 22:59:57.072271062  [2023-08-12 22:59:57] detector.coral1                INFO    : Starting detection process: 576
2023-08-12 22:59:57.108744180  [2023-08-12 22:59:57] detector.coral2                INFO    : Starting detection process: 578
2023-08-12 22:59:57.108844279  [2023-08-12 22:59:57] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:0
2023-08-12 22:59:57.111736162  [2023-08-12 22:59:57] frigate.app                    INFO    : Output process started: 581
2023-08-12 22:59:57.111851770  [2023-08-12 22:59:57] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:1
2023-08-12 22:59:57.111906203  [2023-08-12 22:59:57] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-08-12 22:59:57.112010308  [2023-08-12 22:59:57] frigate.detectors.plugins.edgetpu_tfl ERROR   : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
2023-08-12 22:59:57.112013945  Process detector:coral2:
2023-08-12 22:59:57.112065853  [2023-08-12 22:59:57] frigate.app                    INFO    : Camera processor started for familyroom: 586
2023-08-12 22:59:57.112128321  [2023-08-12 22:59:57] frigate.app                    INFO    : Capture process started for familyroom: 589
2023-08-12 22:59:57.113606613  Traceback (most recent call last):
2023-08-12 22:59:57.113612484    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
2023-08-12 22:59:57.113615721      delegate = Delegate(library, options)
2023-08-12 22:59:57.113639465    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
2023-08-12 22:59:57.113647480      raise ValueError(capture.message)
2023-08-12 22:59:57.113650055  ValueError
2023-08-12 22:59:57.113665564  
2023-08-12 22:59:57.113668730  During handling of the above exception, another exception occurred:
2023-08-12 22:59:57.113670975  
2023-08-12 22:59:57.113673419  Traceback (most recent call last):
2023-08-12 22:59:57.113690952    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-08-12 22:59:57.113693878      self.run()
2023-08-12 22:59:57.113696863    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-08-12 22:59:57.113699458      self._target(*self._args, **self._kwargs)
2023-08-12 22:59:57.113720568    File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector
2023-08-12 22:59:57.113723674      object_detector = LocalObjectDetector(detector_config=detector_config)
2023-08-12 22:59:57.113726529    File "/opt/frigate/frigate/object_detection.py", line 52, in __init__
2023-08-12 22:59:57.113729164      self.detect_api = create_detector(detector_config)
2023-08-12 22:59:57.113732671    File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector
2023-08-12 22:59:57.113762797      return api(detector_config)
2023-08-12 22:59:57.113765302    File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 37, in __init__
2023-08-12 22:59:57.113768719      edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
2023-08-12 22:59:57.113771343    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
2023-08-12 22:59:57.113811950      raise ValueError('Failed to load delegate from {}\n{}'.format(
2023-08-12 22:59:57.113834913  ValueError: Failed to load delegate from libedgetpu.so.1.0
2023-08-12 22:59:57.113837358

Operating system

UNRAID

Install method

Docker CLI

Coral version

M.2

Any other information that may be helpful

Under version previous to 0.12.0, no issues identified with Dual TPU.

[NickM-27]

Most likely you haven't configured the second /dev/apex_1 to be passed to the frigate container.

[kcraig73]

Although this didn't appear to be an issue in previous releases, exposing device: /dev/apex_1 to the container addressed the issue in 0.12.0. Thank you for the quick fix!

[EvanSnyder]

Most likely you haven't configured the second /dev/apex_1 to be passed to the frigate container.

This needs to go into the documentation somewhere, unless I missed it. Thank you!

[blakeblackshear]

Right here: https://docs.frigate.video/configuration/object_detectors#multiple-pciem2-corals

[EvanSnyder]

Right here: https://docs.frigate.video/configuration/object_detectors#multiple-pciem2-corals

I had that part in the config and the docker container was in a boot loop when it tried connecting to the second Coral. What fixed it was adding the /dev/apex_1:/dev/apex_1 to devices in the docker compose file. The documentation for the config,yml is fine, it was the additional piece in the docker-compose.yml that was the missing piece.

    devices:
      - /dev/apex_0:/dev/apex_0
      - /dev/apex_1:/dev/apex_1

Edit: spelling is hard sometimes