search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
17.7k stars 1.62k forks source link

[Detector Support]: USB Coral keeps resetting despite new PC hardware #6445

Closed LaurenceGough closed 1 year ago

LaurenceGough commented 1 year ago

Describe the problem you are having

Hi all,

I always had the odd reset of the USB Coral on my Pi 4, I suspected it was just due to the limited USB power it could provide. I was a bit shocked to see the same thing happening, despite a whole new PC. I've now got a mini desktop PC (Intel Celeron) which should have ample power, it has a 30W power supply and draws around 8W by itself.

I am not using any custom model file, just the standard one.

Output of dmesg errors, which is exact same as I used to get on my Pi 4.

[159263.023770] usb 2-2: reset SuperSpeed USB device number 3 using xhci_hcd
[159263.035674] usb 2-2: LPM exit latency is zeroed, disabling LPM.
[167524.287925] usb 2-2: reset SuperSpeed USB device number 3 using xhci_hcd
[167524.301045] usb 2-2: LPM exit latency is zeroed, disabling LPM.

Pi 4 dmesg errors:

Mar  8 20:24:40 Pi kernel: [2758710.649344] usb 2-2: reset SuperSpeed USB device number 4 using xhci_hcd
Mar  8 20:24:40 Pi kernel: [2758710.668836] usb 2-2: LPM exit latency is zeroed, disabling LPM.
Mar  8 23:51:35 Pi kernel: [2771126.257350] usb 2-2: reset SuperSpeed USB device number 4 using xhci_hcd
Mar  8 23:51:35 Pi kernel: [2771126.276745] usb 2-2: LPM exit latency is zeroed, disabling LPM.

I am currently on the latest dev build, but this was happening before on stable 12 release on my Pi 4.

Could my USB Coral be faulty? It's temps are not above 38c on the heatsink, and I'm using the cable that came with it out of the box from an official reseller direct. It's plugged into a USB 3 port.

I've disabled USB suspension: cat /sys/bus/usb/devices/*/power/control on on on on on

The OS before was Diet Pi (Debian based) and now I'm using Clear Linux server OS.

Thank you

Version

0.13.0-E3B9998

Frigate config file

I am just copying what I think is relevant to it:
detectors:
  coral:
    type: edgetpu
    device: usb

Cam1:
    detect:
        width: 1920
        height: 1080
        fps: 8

Cam2:
    detect:
      width: 640
      height: 360

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:dev-e3b9998
    shm_size: "128mb"
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128

Relevant log output

2023-05-08 13:07:41.792907035  [2023-05-08 13:07:41] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-05-08 13:07:41.793068629  [2023-05-08 13:07:41] root                           INFO    : Waiting for detection process to exit gracefully...
2023-05-08 13:08:11.799798860  [2023-05-08 13:08:11] root                           INFO    : Detection process didnt exit. Force killing...
2023-05-08 13:08:11.806219416  [2023-05-08 13:08:11] root                           INFO    : Detection process has exited...
2023-05-08 13:08:11.825268304  [2023-05-08 13:08:11] detector.coral                 INFO    : Starting detection process: 158812
2023-05-08 13:08:14.440345313  [2023-05-08 13:08:11] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-05-08 13:08:14.449308733  [2023-05-08 13:08:14] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-05-08 15:25:23.060842490  [2023-05-08 15:25:23] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-05-08 15:25:23.061033726  [2023-05-08 15:25:23] root                           INFO    : Waiting for detection process to exit gracefully...
2023-05-08 15:25:53.065874816  [2023-05-08 15:25:53] root                           INFO    : Detection process didnt exit. Force killing...
2023-05-08 15:25:53.099814981  [2023-05-08 15:25:53] root                           INFO    : Detection process has exited...
2023-05-08 15:25:53.105894453  [2023-05-08 15:25:53] detector.coral                 INFO    : Starting detection process: 167969
2023-05-08 15:25:55.930304018  [2023-05-08 15:25:53] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-05-08 15:25:55.934036238  [2023-05-08 15:25:55] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

Operating system

Other Linux

Install method

Docker Container (via Portainer)

Coral version

USB

Any other information that may be helpful

No response

rknobbe commented 1 year ago

I'm having the same problem.

May 8 21:49:04 elitedesk kernel: [ 661.298797] usb 2-10.4: reset SuperSpeed USB device number 6 using xhci_hcd May 8 21:49:04 elitedesk kernel: [ 661.318686] usb 2-10.4: LPM exit latency is zeroed, disabling LPM.

loops about every minute

blakeblackshear commented 1 year ago

Sometimes it's an issue with the cable.

LaurenceGough commented 1 year ago

Sometimes it's an issue with the cable.

I am using the cable that come out of the new usb coral box, and it's never been bent by myself but I guess there is a chance it was damaged before it went in the box or something else. I'll try another high quality usb 3 cable and I'll report back. I've also contacted the Google coral team to see what they say. It works fine 98%+ of the time.

Have there been other reports of the cable being faulty?

Thanks

NickM-27 commented 1 year ago

I found the included cable to be low quality and using a high quality cable led to better stability and lower inference speeds. Switching back to the old cable would immediately show the worse results.

rknobbe commented 1 year ago

Any recommendations on a "high quality" USB cable? I'm using the provided cable as well and have been seeing frequent instability. Switched to OpenVINO entirely and am seeing better results. I'd like to use the Coral, if I can find a good stable solution.

On Tue, May 9, 2023 at 5:09 AM Nicolas Mowen @.***> wrote:

I found the included cable to be low quality and using a high quality cable led to better stability and lower inference speeds. Switching back to the old cable would immediately show the worse results.

— Reply to this email directly, view it on GitHub https://github.com/blakeblackshear/frigate/issues/6445#issuecomment-1540019347, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGD5GKYRVADSVTE4Z4V5YDXFIXYRANCNFSM6AAAAAAX27MYQM . You are receiving this because you commented.Message ID: @.***>

LaurenceGough commented 1 year ago

I changed the cable from the one included in the box and now my USB Coral is now cold to the touch, whereas before it was always very warm (38c on heatsink hotspot measured by IR meter) 24/7. It seems highly unusual but it appears to be working. The only things I can think of is the standard cable had some kind of short, or the new cable isn't good enough (no complaints in the logs that I can see yet), or the new, longer cable is acting as some kind of heatsink, but it all seems far too cold for that.

Are your USB Coral's cold generally?

So strange! I'll report back with the progress.

LaurenceGough commented 1 year ago

Well this is insane, the coral is now completely cold still which was unheard of ever before for me at around 10 months of 24/7 use. So clearly the cable that came with it is broken. Now, how much has the life of my usb coral been shortened due to running on the edge of overheating all of this time, it doesn't have thermal throttling either like the pcie one. I think I'll ask for a replacement. Cheers guys.

P.s no error yet, I'm sure this is the solution.

NickM-27 commented 1 year ago

Thanks for the update. I'm go ahead and close this since the root issue seems to be solved.

blakeblackshear commented 1 year ago

What cable did you end up purchasing?

LaurenceGough commented 1 year ago

I just used a really cheap ugreen usb3 cable I had laying around as a test. I really didn't think it would do anything. I really would love to know if your usb corals are warm or not guys. The difference is insane from pretty warm to just cold all the time.only 2 cams currently at 20c ambient. Inference speed is now a few ms slower but nothing in it really, it's still ~20ms. Unless it's running at a lower clock or something, but I thought that could only be set in software (drivers installed during the frigate container install).

I don't understand why Google would include basically a faulty cable. Especially after all the work they did on usb 3 cable standards etc. Surely I must have got an exception to the norm...

This may have stopped my pi 4 crashing the once a month it did as I think it upset the usb SSD with a power draw spike, and it relied on the SSD a lot for swap etc, 2gb wasn't enough for Frigate from what I could see. HA was running on it too but that didn't use too much RAM. The usb coral draining so much all the time couldn't have helped things.

blakeblackshear commented 1 year ago

Mine definitely gets warm, but it is heavily dependent on utilization. The warmth directly correlates with the detect fps. I have several and have always used the included USB cable. I have one thats been plugged in for years inside a hot server cabinet without any issues at all.

weboide commented 9 months ago

My findings on this matter are: