blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
17.7k stars 1.62k forks source link

[Detector Support]: Detection appears to be stuck. Restarting detection process... #8552

Closed tomhusband closed 9 months ago

tomhusband commented 9 months ago

Describe the problem you are having

I have HA/Frigate running in a Proxmox VM on an Intel NUC. I recently installed a Coral Accelerator and have been getting "Detection appears to be stuck. Restarting detection process..." errors in the Frigate log. It starts again but a few hours later it happens again. I replaced the original USB cable hoping it might be the problem and it did run smoothly for a day but last night it restarted several times again.

The Frigate log is below but I also get the following in the Proxmox Syslog at the time the Coral shuts down and restarts:

Nov 08 22:18:25 pve kernel: usb 2-2: reset SuperSpeed USB device number 4 using xhci_hcd Nov 08 22:18:25 pve kernel: usb 2-2: LPM exit latency is zeroed, disabling LPM.

Version

0.12.1-367D724

Frigate config file

mqtt:
  host: 192.168.1.185
  user: Tom
  password: password

detectors:
  coral:
    type: edgetpu
    device: usb

logger:
  default: info
  logs:
    frigate.record: error

cameras:
  driveway_cam:
    ffmpeg:
      inputs:
        - path: rtsp://admin:AnnieMarie@192.168.1.15:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            - detect
            - record
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 10

    objects:
      track:
        - car
            mask:954,372,960,482,935,563,758,650,557,664,481,649,455,595,414,437,423,285,434,169,490,123,665,81,721,28,808,32
        - person
        - dog
        - cat
        - bicycle
      filters:
        person:
          min_score: 0.6
          threshold: 0.7
        dog:
          min_score: 0.6
          threshold: 0.75
    record:
      enabled: True
      retain:
        days: 0
        mode: active_objects
      events:
        retain:
          default: 3
          mode: active_objects

    motion:
      improve_contrast: True

docker-compose file or Docker CLI command

I have HA/Frigate in a VM not Docker.

Relevant log output

2023-11-08 15:08:11.910168363  [INFO] Starting Frigate...
2023-11-08 15:08:12.937387765  [2023-11-08 15:08:12] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-11-08 15:08:12.950306025  [2023-11-08 15:08:12] frigate.app                    INFO    : Creating directory: /tmp/cache
2023-11-08 15:08:12.950790710  [2023-11-08 15:08:12] peewee_migrate                 INFO    : Starting migrations
2023-11-08 15:08:12.955283928  [2023-11-08 15:08:12] peewee_migrate                 INFO    : There is nothing to migrate
2023-11-08 15:08:13.001244355  [2023-11-08 15:08:12] frigate.app                    INFO    : Output process started: 229
2023-11-08 15:08:13.008287365  [2023-11-08 15:08:13] frigate.app                    INFO    : Camera processor started for driveway_cam: 235
2023-11-08 15:08:13.028957915  [2023-11-08 15:08:13] frigate.app                    INFO    : Capture process started for driveway_cam: 238
2023-11-08 15:08:15.778177120  [2023-11-08 15:08:12] detector.coral                 INFO    : Starting detection process: 227
2023-11-08 15:08:15.778180580  [2023-11-08 15:08:12] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-11-08 15:08:15.782393577  [2023-11-08 15:08:15] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-11-08 22:17:54.303812454  [2023-11-08 22:17:54] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-11-08 22:17:54.304235144  [2023-11-08 22:17:54] root                           INFO    : Waiting for detection process to exit gracefully...
2023-11-08 22:18:24.325368533  [2023-11-08 22:18:24] root                           INFO    : Detection process didnt exit. Force killing...
2023-11-08 22:18:24.329569399  [2023-11-08 22:18:24] root                           INFO    : Detection process has exited...
2023-11-08 22:18:24.341291445  [2023-11-08 22:18:24] detector.coral                 INFO    : Starting detection process: 19714
2023-11-08 22:18:27.124089560  [2023-11-08 22:18:24] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-11-08 22:18:27.129554979  [2023-11-08 22:18:27] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

Operating system

Proxmox

Install method

HassOS Addon

Coral version

USB

Any other information that may be helpful

No response

tpjanssen commented 9 months ago

It seems that your Coral device is running at USB 2 speed. USB 2 has limited bandwidth, and probably it can't keep up. Make sure it is connected to a USB 3 port.

tomhusband commented 9 months ago

Hmmm, I wondered about that and went with the one that had a blue tab thinking that was USB 3. Why can't they just mark it with a 2 or 3? Now I wonder if I even have a USB 3 port available.

Thanks for your reply.

tomhusband commented 9 months ago

lsusb gives me this:

root@pve:~# lsusb Bus 002 Device 003: ID 1058:2620 Western Digital Technologies, Inc. Elements 2620 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 003: ID 8087:0a2b Intel Corp. Bluetooth wireless interface Bus 001 Device 002: ID 10c4:ea60 Silicon Labs CP210x UART Bridge Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Do I even have a USB 3 available besides the one connected to my hard drive?

tpjanssen commented 9 months ago

Based on the list I would say that bus 2 is USB 3, so that should be OK. When you ran lsusb, the Coral was not connected? In that case I would rerun the command with the Coral detected.

I see that you have a connected a external HDD as well? In that case it might be power related, the Coral devices are very power hungry, especially high peak currents.

tomhusband commented 9 months ago

Yes, here's lsusb with the Coral connected:

root@pve:~# lsusb Bus 002 Device 003: ID 1058:2620 Western Digital Technologies, Inc. Elements 2620 Bus 002 Device 006: ID 18d1:9302 Google Inc. Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 003: ID 8087:0a2b Intel Corp. Bluetooth wireless interface Bus 001 Device 002: ID 10c4:ea60 Silicon Labs CP210x UART Bridge Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

If I don't have a spare USB 3 what happens if I just continue to let the Coral restart?

tpjanssen commented 9 months ago

So the Coral and the HDD share the same USB bus. As long as you don't use the HDD intensively I don't expect too much problems on that. My first guess would be the USB power supply being insufficient.

If I don't have a spare USB 3 what happens if I just continue to let the Coral restart?

You mean to leave the situation as is now? I think you just miss detections in that case. Can you test without the HDD?

tomhusband commented 9 months ago

Can you test without the HDD? Yes, I guess so. The HDD is used just for backups. So just disconnect the HDD and see what happens?

Thanks, again, for your advice.