search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
17.71k stars 1.62k forks source link

[Support]: Detection gets stuck on Coral TPU #2783

Closed Sloth-on-meth closed 2 years ago

Sloth-on-meth commented 2 years ago

Describe the problem you are having

my detection process gets stuck

Version

DEBUG 0.9.4-26AE608

Frigate config file

na

Relevant log output

[2022-02-10 12:22:14] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
[2022-02-10 12:22:14] root                           INFO    : Waiting for detection process to exit gracefully...
[2022-02-10 12:22:17] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58662]
[2022-02-10 12:22:44] root                           INFO    : Detection process didnt exit. Force killing...
[2022-02-10 12:22:44] detector.coral                 INFO    : Starting detection process: 566
[2022-02-10 12:22:44] frigate.edgetpu                INFO    : Attempting to load TPU as usb
[2022-02-10 12:22:47] frigate.edgetpu                INFO    : TPU found
[2022-02-10 12:23:44] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
[2022-02-10 12:23:44] root                           INFO    : Waiting for detection process to exit gracefully...

FFprobe output from your camera

n/a

Frigate stats

n/a

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

reolink 310/510

Any other information that may be helpful

the coral is plugged into an external USB hub with 12v power.

blakeblackshear commented 2 years ago

I need more information about your hardware

Sloth-on-meth commented 2 years ago

of course,

SABERTOOTH X79 I7 3930K 32GB DDR3 GTX 1050TI CORAL.AI accellerator https://coral.ai/products/accelerator

blakeblackshear commented 2 years ago

What OS and are you running frigate directly on the OS with compose?

Sloth-on-meth commented 2 years ago

Ubuntu 21.10 frigate directly on OS using compose

Sloth-on-meth commented 2 years ago

fyi, other tests do work with the TPU - just frigate is having issues

blakeblackshear commented 2 years ago

Do you have other things plugged into the same hub. Can you try without the hub? The Coral can only be used by a single process. If it is being used by something else, Frigate cannot use it.

Sloth-on-meth commented 2 years ago

Do you have other things plugged into the same hub. Can you try without the hub? The Coral can only be used by a single process. If it is being used by something else, Frigate cannot use it.

i actually purposely plugged it into the hub because of this issue, since I found another issue where it was power related. this happens with or without the HUB. no other process, as far as I know, is using the TPU, how can I make sure?

Sloth-on-meth commented 2 years ago

i can run this without issue: 


root@skynetbams:~/coral/pycoral# python3 examples/classify_image.py --model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --labels test_data/inat_bird_labels.txt --input test_data/parrot.jpg 
----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
17.1ms
4.9ms
4.9ms
4.9ms
4.9ms
-------RESULTS--------
Ara macao (Scarlet Macaw): 0.76953```
Sloth-on-meth commented 2 years ago

is there a way to get more verbose logging on why it gets stuck

Sloth-on-meth commented 2 years ago

FYI @blakeblackshear - This did NOT occur on my old motherboard, but DOES occur on my new motherboard.

my old motherboard: MSI Z75 E33 with I7 3770K my new motherboard: SABERTOOTH X79 I7 3990K

NickM-27 commented 2 years ago

If it had to do with switching motherboards, have you checked your BIOS for any USB sleep, USB mode settings, or anything else that might affect USB?

Sloth-on-meth commented 2 years ago

Good point. I shall find a display to plug into it

On Thu, Feb 10, 2022, 15:54 Nicolas Mowen @.***> wrote:

If it had to do with switching motherboards, have you checked your BIOS for any USB sleep, USB mode settings, or anything else that might affect USB?

— Reply to this email directly, view it on GitHub https://github.com/blakeblackshear/frigate/issues/2783#issuecomment-1035014457, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKJVL4DJWDMAVH7UD433A3DU2PGQRANCNFSM5OAMX4HQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

Sloth-on-meth commented 2 years ago

image these are the only options in the usb menu. would EHCI handoff do anything

blakeblackshear commented 2 years ago

Keep in mind that frigate uses the Coral at max speed often. You won't see some issues in a car if you are idling around the parking lot. It's quite a bit different to drive the same car on the highway with the pedal all the way to the floor for hours.

Sloth-on-meth commented 2 years ago

I see what you're saying, but it keeps restarting over and over without ever detecting

Sloth-on-meth commented 2 years ago

enabled EHCI handoff - this happened - image

Sloth-on-meth commented 2 years ago

checked with lsusb, the device does show up as USB 3.10 so it is connected properly. running a standalone test works too. I'm trying to find other tpu apps for docker to see if this is frigate specific

Sloth-on-meth commented 2 years ago 
root@skynetbams:~# lsusb -D /dev/bus/usb/008/003
Device: ID 18d1:9302 Google Inc. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.10
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         9
  idVendor           0x18d1 Google Inc.
  idProduct          0x9302 
  bcdDevice            1.00
  iManufacturer           0 
  iProduct                0 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0060
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              896mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           6
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol    255 Vendor Specific Protocol
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               1
        bMaxBurst               0
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x0016
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000002
      HIRD Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000c
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   2
      Lowest fully-functional device speed is High Speed (480Mbps)
    bU1DevExitLat           0 micro seconds
    bU2DevExitLat           0 micro seconds
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)
Sloth-on-meth commented 2 years ago

quick note - the TPU does work running the coral example - https://www.toptal.com/developers/hastebin/eyahoperun.yaml

Sloth-on-meth commented 2 years ago

im sadly still unable to use frigate - changing bios settings regarding xhci did not change anything

Sloth-on-meth commented 2 years ago

i uh... replaced the usb cable of the accelerator and uh... fixed it

NickM-27 commented 2 years ago

i uh... replaced the usb cable of the accelerator and uh... fixed it

Cables really are silent killers

weboide commented 9 months ago

Changing cables also fixed it for me!

It's baffling that they provide a cable that doesn't even work properly.