google-coral / edgetpu

Coral issue tracker (and legacy Edge TPU API source)
https://coral.ai
Apache License 2.0
426 stars 125 forks source link

Errorr when running for a long period of time #297

Open Co2Link opened 3 years ago

Co2Link commented 3 years ago

I am using one Coral Edge Tpu on Raspberry Pi 4. I run yolov5s created using this repo at about 5 ~ 10 fps. I have encountered the following error consistenly when running the code for a long period of time (couple of hours) but also occasionally in shorter amounts of time(several minutes).

F :859] transfer on tag 2 failed. Abort. Deadline exceeded: USB transfer error 2 [LibUsbDataOutCallback]

As mentioned in #11 , it may be a power issue. But I had used an USB Power Meter to confirm that the Tpu consistenly draw about 4.95V and 0.20A (even at the moment the above error showing up). The only thing i connected to the Pi 4 is the Tpu and a webcam, so i don't thing power is an issue.

Moreover, I encountered the the error after i switching from MobileNet SSD V2 (Downloaded from the coral official site) to yolov5s.

Here is the edgetpu compiler log for the yolov5s: compiler_log

Link for .tflite file: yolov5s-int8_edgetpu.tflite

I hope I gave you all the information needed, please do not hesitate to ask for more precisions.

Thanks in advance.

Namburger commented 3 years ago

Hello, it seems to be a usb3 issue, which I sometimes have with the RPI3. Could you try to see if it is reproducible on a us2 port or with a different cable?

Co2Link commented 3 years ago

@Namburger Thank you. I will try using a different cable first then usb2.

Co2Link commented 3 years ago

@Namburger Both options "using a different cable" and "use usb2 port" still encounter the same error.

This error could be related to yolov5s model, cuase my code work fine with MobileNet SSD V2

The code from .7z file i attached here should be able to reproduce the error if you want to have a try. yolov5s.7z

Inside here you will see three files

It may take up to 20 hours until the erorr show up.

Co2Link commented 3 years ago

As mentioned here Running on an ubuntu PC for 48 hours without error. so, it may be an OS issue?

Co2Link commented 3 years ago

I tested on a different set of hardware(RPI4 and TPU), the error still show up.

Co2Link commented 3 years ago

I found the problem, but don't know how to fix it.

import numpy as np
import time
from models.yolov5 import yolov5
from utils.camera import ImageGenerator
model = yolov5('weights/yolov5s-int8_edgetpu.tflite',
               'weights/yolov5s.yaml', 0.5)

gen = ImageGenerator()
# if comment out the following line, error won't show up
gen.start_read()
ret = True
while ret:
    img = np.random.randint(0, 256, (600, 800, 3)).astype('uint8')
    model.predict(img)

start_read() method will start a new thread to read frame from usb webcam constantly. predict() method do preprocessing and invoke() and postpreocessing using edge tpu. I found that error won't show up if i comment out gen.start_read() (89hours without error).

So can it be the limitation of the usb bandwidth?