search

google-coral / edgetpu

Coral issue tracker (and legacy Edge TPU API source)
https://coral.ai
Apache License 2.0
426 stars 125 forks source link

USB transfer error while running inference for Posenet model #99

Closed ghost closed 4 years ago

ghost commented 4 years ago

Platform: aarch64, IMX8 Edge TPU Release: 2.13 Edge TPU Device: USB stick Model used: posenet_mobilenet_v1_075_481_641_quant_decoder_edgetpu.tflite

Having a strange USB transfer error while trying to run a pose estimation inference. Had no problem running the USB stick with some simple object detection, this seems only to occur when trying to implement the pose detection stuff.

F :1133] HandleQueuedBulkIn transfer in failed. Not found: USB transfer error 5 [LibUsbDataInCallback]

Tested several ways to initialize the interpreter and prepare the input but with same result. Right now I've used the code available in the /src/cpp folder, just to see if it made any difference;

basic_engine.h/.cc basic_engine_native.h/.cc posenet_decoder.h/.cc posenet_decoder_op.h/.cc error_reporter.h/.cc edgetpu_resource_manager.h/.cc

So basically:

m_engine = new coral::BasicEngine(poseModelPath);

std::vector<uint8_t> input;
formatInput(inputImage, input);

const std::vector<std::vector<float>> &raw_outputs = m_engine->RunInference(input);

The crash happens when Invoke() is called in 

EdgeTpuApiStatus BasicEngineNative::RunInference(const uint8_t *const input,
    const size_t in_size,
    float const **const output,
    size_t *const out_size);

Thanks!

Namburger commented 4 years ago

@alfred-tang What's your model input size and are you using a USB2 or USB3 port?

ghost commented 4 years ago

@Namburger I'm using the posenet_mobilenet_v1_075_481_641_quant_decoder_edgetpu.tflite model available in the repo (edgetpu/test_data/posenet/). Currently using USB2 which worked fine for some object detection earlier.

Namburger commented 4 years ago

@alfred-tang Just realized: HandleQueuedBulkIn is usually due to multiple processing attempting to access the tpu at the same time. Are you trying to invoke() with multiple processes by any chance?

ghost commented 4 years ago

@Namburger Nope, invoke() is only called once and i've got a check so it doesn't run inference until the first is completed. Also ran it through the debugger and yeah it crashes when it reaches the invoke() in BasicEngineNative.

Namburger commented 4 years ago

@alfred-tang Apolologies. This is not expected, but I couldn't reproduce on my side. I'm basically running this test could you try to see if you are getting the same error running it? Instructions:

# from root of this project
$ make tests CPU=aarch64
...
# copy to you platform
$ scp -r edgetpu /your/board
# on your board cd into the project
$ LD_LIBRARY_PATH=libedgetpu/direct/aarch64 ./out/aarch64/tests/src/cpp/posenet/models_test
ghost commented 4 years ago

Seems to be a problem with _481_641 model;

./out/aarch64/tests/src/cpp/posenet/models_test 
[==========] Running 3 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 3 tests from PosenetModelCorrectnessTest
[ RUN      ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_353_481
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0421 13:28:22.775524 28695 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_353_481_quant_decoder.tflite
I0421 13:28:31.610836 28695 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_353_481_quant_decoder_edgetpu.tflite
[       OK ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_353_481 (15471 ms)
[ RUN      ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_481_641
I0421 13:28:38.246665 28695 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_481_641_quant_decoder.tflite
I0421 13:28:46.319895 28695 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_481_641_quant_decoder_edgetpu.tflite
F :1133] HandleQueuedBulkIn transfer in failed. Not found: USB transfer error 5 [LibUsbDataInCallback]
Aborted
Namburger commented 4 years ago

@alfred-tang I sense something isn't right, this test passes for both by rpi(arm32) and dev board:

{20-04-21 9:12}raspberrypi:~ pi% ./armv7a/tests/src/cpp/posenet/models_test 
[==========] Running 3 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 3 tests from PosenetModelCorrectnessTest
[ RUN      ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_353_481
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0421 09:12:22.555716  5893 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_353_481_quant_decoder.tflite
I0421 09:12:26.293758  5893 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_353_481_quant_decoder_edgetpu.tflite
[       OK ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_353_481 (7168 ms)
[ RUN      ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_481_641
I0421 09:12:29.723809  5893 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_481_641_quant_decoder.tflite
I0421 09:12:33.683826  5893 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_481_641_quant_decoder_edgetpu.tflite
[       OK ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_481_641 (7449 ms)
[ RUN      ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_721_1281
I0421 09:12:37.172334  5893 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_721_1281_quant_decoder.tflite
I0421 09:12:42.362380  5893 test_utils.cc:297] Testing model: test_data/posenet/posenet_mobilenet_v1_075_721_1281_quant_decoder_edgetpu.tflite
[       OK ] PosenetModelCorrectnessTest.TestPoseNetWithDecoder_721_1281 (8903 ms)
[----------] 3 tests from PosenetModelCorrectnessTest (23520 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test suite ran. (23520 ms total)
[  PASSED  ] 3 tests.

Can you share the out puts of this:

$ dpkg -l | grep edgetpu
ghost commented 4 years ago

@Namburger Just tested the 353_481 model with my implementation and it seems to work fine. Unfortunately I cannot run dpkg on the current unit I've got my stick hooked up to, it's not available in the running Linux.

Not sure what the real issue is, maybe too slow transfer as the unit is using USB 2.0? Or just too little system resources for that amount of data which is produced by the model maybe? As you see the first test took more than double the amount of time compared to your output.

Since you cannot re-produce this on several platforms, should I close it?

Namburger commented 4 years ago

@alfred-tang Actually, on USB2.0, I'm getting 8515 ms on the first test, your results seems very odd but that won't tell much, because the time measured included both CPU model and TPU model... On another note, we found that 2.0 is actually more reliable than 3.0, it's just a little slower. My suspicion is that you have an older version of libedgetpu, maybe? Since we can't show dpkg, maybe we can try md5sum?

mendel@purple-orange:~$ md5sum /usr/lib/aarch64-linux-gnu/libedgetpu.so.1.0
42eab946e268728a84f6548d0eee5d67  /usr/lib/aarch64-linux-gnu/libedgetpu.so.1.0

Another thing I suspect is not enough power is supplying to the tpu. That would also give a transfer error message, but I remember it's different than the HandleQueuedBulkIn so I'm not sure. Anyways, closing or not is up to you, although I don't see any faulty issues on our library. Thanks!

ghost commented 4 years ago

Alright I see, interesting. Yeah this is the output from md5sum;

md5sum /opt/aarch64/libedgetpu.so.1.0
cd8a37758d678513e59a9c0980817df2  /opt/aarch64/libedgetpu.so.1.0

Apparently, I'm running 1.14. Just checked and I'm running from the master branch.

Yeah I was also thinking something about power earlier.

Thanks again for your help.

Namburger commented 4 years ago

Hey, looks like you are running the latest version with the max frequencies instead of the the standard frequencies. I just switched all of mine to max and still didn't run into this issue :/ Apologies, right now I have no other suggestions besides power, check this one for some references

ghost commented 4 years ago

@Namburger Thanks for looking into this, I also starting to think that its power related. Apparently the HW I'm using is shutting off its usb hub if it exceeds 600mA. So yeah probably power related, next up is trying to connect an external usb hub with extra power.

I'm closing this now!

Thanks again!

BernardinD commented 4 years ago

@Namburger did you confirm that this is a power issue?

Update: After switching the accelerator to USB 2.0 I was able to get it to work. But I need it to work on 3.0. Any suggestions?

cliffjao commented 4 years ago

@Namburger did you confirm that this is a power issue?

Update: After switching the accelerator to USB 2.0 I was able to get it to work. But I need it to work on 3.0. Any suggestions?

I came across this issue as well. For me, it turned out that the included USB cable is somehow bad. I switched it with another one that I had lying around, and it seems to be a lot more stable now. Haven't ran into this problem so far.

brunovollmer commented 3 years ago

Hey @Namburger, I have a similar problem and similar to @BernardinD I was able to fix it by running the tpu stick over USB 2.0. Any suggestions on how to fix the Issue with USB 3.0?

Luke1962 commented 3 years ago

Hi, everybody I want to report similar problem. However it happen after some loops. Could it be generated by some power saving agents? My Edgetpu is on USB3.0 HUB , shared with USB camera an arduino platform connected to Jetson Nano board (Ubuntu 16.04LTS) and running two inference models (obj detection + face detection)

... logging to /home/pi/.ros/log/33c9b67c-118a-11ec-868a-0242abf3735e/roslaunch-nano-14279.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://192.168.195.11:40595/

SUMMARY
========

PARAMETERS
 * /edgetpu/camera_info_url: /home/pi/ros/src/...
 * /edgetpu/camera_link: usbcam_bottom_link
 * /edgetpu/detection_label_path: /home/pi/ros/src/...
 * /edgetpu/detection_treshold: 0.5
 * /edgetpu/device_id: 2
 * /edgetpu/facedetection_model_path: /home/pi/ros/src/...
 * /edgetpu/fps: 30
 * /edgetpu/img_from_cv2: True
 * /edgetpu/json_facelabel_path: /home/pi/ros/src/...
 * /edgetpu/known_faces_path: /home/pi/ros/src/...
 * /edgetpu/mouse_click_topic: /mouse_left_click
 * /edgetpu/noderate_hz: 100
 * /edgetpu/persondetection_model_path: /home/pi/ros/src/...
 * /edgetpu/publish_img: True
 * /edgetpu/show_detections: True
 * /edgetpu/topic_img_in: /robot/usbcam/ima...
 * /edgetpu/topic_imgprocessed_out: /edgetpu_imgcompr...
 * /edgetpu/tracker_center_half_width: 30
 * /edgetpu/use_face_recognition: 0
 * /rosdistro: melodic
 * /rosversion: 1.14.11

NODES
  /
    edgetpu (rm_edgetpu_detect/edgetpu_node.py)

ROS_MASTER_URI=http://192.168.195.11:11311

process[edgetpu-1]: started with pid [14646]
OpenCV version :  4.5.3-dev
======== rm_edgetpu_detect_and_recognize.py node  =====
Project path: /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect
------------------
Python v.  3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0]
------------------
Package path:  /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect
 Person Detection Model path: /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/models/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite 
detection label path /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/models/coco_labels_it.txt
detection_treshold 0.5
 Face Detection Model path: /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/models/facedetection_320_240_edgetpu.tflite 
 File with face names: /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/pretrained_model/label.json 

Camera Info URL: /home/pi/ros/src/myrobot/src/vision/rm_usbcam/camera_info/ps3eye.yaml

Reading camera info YAML from:  /home/pi/ros/src/myrobot/src/vision/rm_usbcam/camera_info/ps3eye.yaml
 Used video device id : 2 
Show locally detection captures
IMMAGINI ACQUISITE DIRETTAMENTE DA OPENCV SU DEVICE  2
[ WARN:0] global /home/pi/opencv/modules/videoio/src/cap_gstreamer.cpp (1081) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
 Publishing original Image compressed to: /robot/usbcam/image_raw/compressed 
Publishing Image with recognition to  topic:  /edgetpu_imgcompressed
 Publishing elaborated Image compressed with detections to: /edgetpu_imgcompressed 
[INFO] [1631211003.291965]: Loading Neural network interpreter ...
loading labels from:  /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/models/coco_labels_it.txt
{0: 'persona', 1: 'bicicletta', 2: 'automobile', 3: 'moto', 4: 'aeropplano', 5: 'autobus', 6: 'treno', 7: 'camion', 8: 'barca', 9: 'semaforo', 10: 'idrante', 12: 'segno di stop', 13: 'parchimetro', 14: 'panchina', 15: 'uccello', 16: 'gatto', 17: 'cane', 18: 'cavallo', 19: 'pecora', 20: 'mucca', 21: 'elefante', 22: 'orso', 23: 'zebra', 24: 'giraffa', 26: 'zaino', 27: 'ombrello', 30: 'borsa', 31: 'cravatta', 32: 'valigia', 33: 'frisbi', 34: 'sci', 35: 'snoboard', 36: 'palla sportiva', 37: 'kite', 38: 'mazza da baseball', 39: 'guanto da besboal', 40: 'skateboard', 41: 'surfboard', 42: 'racchetta da tennis', 43: 'bottiglia', 45: 'bicchiere di vino', 46: 'tazza', 47: 'forchetta', 48: 'coltello', 49: 'cucchiaio', 50: 'ciotola', 51: 'banana', 52: 'mela', 53: 'panino', 54: 'arancia', 55: 'broccoli', 56: 'carota', 57: 'hot dog', 58: 'pizza', 59: 'ciambella', 60: 'torta', 61: 'sedia', 62: 'divano', 63: 'pianta in vaso', 64: 'letto', 66: 'tavolo da pranzo', 69: 'vater', 71: 'televisione', 72: 'laptop', 73: 'mouse', 74: 'telecomando', 75: 'tastiera', 76: 'telefono', 77: 'microonde', 78: 'forno', 79: 'tostapane', 80: 'lavello', 81: 'frigorifero', 83: 'libro', 84: 'orologio', 85: 'vaso', 86: 'forbici', 87: 'orsacchiotto di peluche', 88: 'asciuga capelli', 89: 'spazzolino da denti'}
 FACE RECOGNITION DISABLED BY PARAMETER 
 Thread 'thPubServo' avviato 

__init__ completed 

Main Loop started....

Traceback (most recent call last):
  File "/home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/scripts/edgetpu_node.py", line 2022, in <module>
    main( sys.argv)
  File "/home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/scripts/edgetpu_node.py", line 2014, in main
    detector.main_loop(rate)
NameError: name 'rate' is not defined
^C[edgetpu-1] killing on exit
[INFO] [1631211012.607219]: detect_and_recognise.py: Shutting down
Exception ignored in: <module 'threading' from '/usr/lib/python3.6/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 1294, in _shutdown
    t.join()
  File "/usr/lib/python3.6/threading.py", line 1056, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
  File "/home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/scripts/edgetpu_node.py", line 309, in _signal_handler
    self.exit()
  File "/home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/scripts/edgetpu_node.py", line 306, in exit
    exit()
  File "/usr/lib/python3.6/_sitebuiltins.py", line 26, in __call__
    raise SystemExit(code)
SystemExit: None
 ServoThread END 
shutting down processing monitor...
... shutting down processing monitor complete
done
pi@nano:~$ edgetpu
... logging to /home/pi/.ros/log/33c9b67c-118a-11ec-868a-0242abf3735e/roslaunch-nano-25334.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://192.168.195.11:39235/

SUMMARY
========

PARAMETERS
 * /edgetpu/camera_info_url: /home/pi/ros/src/...
 * /edgetpu/camera_link: usbcam_bottom_link
 * /edgetpu/detection_label_path: /home/pi/ros/src/...
 * /edgetpu/detection_treshold: 0.5
 * /edgetpu/device_id: 2
 * /edgetpu/facedetection_model_path: /home/pi/ros/src/...
 * /edgetpu/fps: 30
 * /edgetpu/img_from_cv2: True
 * /edgetpu/json_facelabel_path: /home/pi/ros/src/...
 * /edgetpu/known_faces_path: /home/pi/ros/src/...
 * /edgetpu/mouse_click_topic: /mouse_left_click
 * /edgetpu/noderate_hz: 100
 * /edgetpu/persondetection_model_path: /home/pi/ros/src/...
 * /edgetpu/publish_img: True
 * /edgetpu/show_detections: True
 * /edgetpu/topic_img_in: /robot/usbcam/ima...
 * /edgetpu/topic_imgprocessed_out: /edgetpu_imgcompr...
 * /edgetpu/tracker_center_half_width: 30
 * /edgetpu/use_face_recognition: 0
 * /rosdistro: melodic
 * /rosversion: 1.14.11

NODES
  /
    edgetpu (rm_edgetpu_detect/edgetpu_node.py)

ROS_MASTER_URI=http://192.168.195.11:11311

process[edgetpu-1]: started with pid [25701]
OpenCV version :  4.5.3-dev
======== rm_edgetpu_detect_and_recognize.py node  =====
Project path: /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect
------------------
Python v.  3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0]
------------------
Package path:  /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect
 Person Detection Model path: /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/models/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite 
detection label path /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/models/coco_labels_it.txt
detection_treshold 0.5
 Face Detection Model path: /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/models/facedetection_320_240_edgetpu.tflite 
 File with face names: /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/pretrained_model/label.json 

Camera Info URL: /home/pi/ros/src/myrobot/src/vision/rm_usbcam/camera_info/ps3eye.yaml

Reading camera info YAML from:  /home/pi/ros/src/myrobot/src/vision/rm_usbcam/camera_info/ps3eye.yaml
 Used video device id : 2 
Show locally detection captures
IMMAGINI ACQUISITE DIRETTAMENTE DA OPENCV SU DEVICE  2
[ WARN:0] global /home/pi/opencv/modules/videoio/src/cap_gstreamer.cpp (1081) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
 Publishing original Image compressed to: /robot/usbcam/image_raw/compressed 
Publishing Image with recognition to  topic:  /edgetpu_imgcompressed
 Publishing elaborated Image compressed with detections to: /edgetpu_imgcompressed 
[INFO] [1631211523.135779]: Loading Neural network interpreter ...
loading labels from:  /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/models/coco_labels_it.txt
{0: 'persona', 1: 'bicicletta', 2: 'automobile', 3: 'moto', 4: 'aeropplano', 5: 'autobus', 6: 'treno', 7: 'camion', 8: 'barca', 9: 'semaforo', 10: 'idrante', 12: 'segno di stop', 13: 'parchimetro', 14: 'panchina', 15: 'uccello', 16: 'gatto', 17: 'cane', 18: 'cavallo', 19: 'pecora', 20: 'mucca', 21: 'elefante', 22: 'orso', 23: 'zebra', 24: 'giraffa', 26: 'zaino', 27: 'ombrello', 30: 'borsa', 31: 'cravatta', 32: 'valigia', 33: 'frisbi', 34: 'sci', 35: 'snoboard', 36: 'palla sportiva', 37: 'kite', 38: 'mazza da baseball', 39: 'guanto da besboal', 40: 'skateboard', 41: 'surfboard', 42: 'racchetta da tennis', 43: 'bottiglia', 45: 'bicchiere di vino', 46: 'tazza', 47: 'forchetta', 48: 'coltello', 49: 'cucchiaio', 50: 'ciotola', 51: 'banana', 52: 'mela', 53: 'panino', 54: 'arancia', 55: 'broccoli', 56: 'carota', 57: 'hot dog', 58: 'pizza', 59: 'ciambella', 60: 'torta', 61: 'sedia', 62: 'divano', 63: 'pianta in vaso', 64: 'letto', 66: 'tavolo da pranzo', 69: 'vater', 71: 'televisione', 72: 'laptop', 73: 'mouse', 74: 'telecomando', 75: 'tastiera', 76: 'telefono', 77: 'microonde', 78: 'forno', 79: 'tostapane', 80: 'lavello', 81: 'frigorifero', 83: 'libro', 84: 'orologio', 85: 'vaso', 86: 'forbici', 87: 'orsacchiotto di peluche', 88: 'asciuga capelli', 89: 'spazzolino da denti'}
 FACE RECOGNITION DISABLED BY PARAMETER 
 Thread 'thPubServo' avviato 

__init__ completed 

Main Loop started....

 CAPTURE OK 
[INFO] [1631211527.472422]: fps: 3
[INFO] [1631211532.623626]: fps: 5
[INFO] [1631211537.771487]: fps: 3
[INFO] [1631211542.947235]: fps: 4
[INFO] [1631211548.163975]: fps: 4
[INFO] [1631211553.452893]: fps: 1
[INFO] [1631211558.699951]: fps: 2
[INFO] [1631211563.851722]: fps: 2
[INFO] [1631211569.127918]: fps: 3
[INFO] [1631211574.150576]: fps: 3
 Detection [torta] at 157.0,161.0   id: 60 , score: 0.7890625  
 Detection [torta] at 183.5,161.5   id: 60 , score: 0.5  
[INFO] [1631211579.607873]: fps: 1
 Detection [banana] at 186.0,155.5   id: 51 , score: 0.58203125  
 Detection [banana] at 161.5,166.5   id: 51 , score: 0.5  
[INFO] [1631211584.833714]: fps: 3
 Detection [idrante] at 166.5,181.0   id: 10 , score: 0.58203125  
 Detection [idrante] at 164.0,181.5   id: 10 , score: 0.66015625  
 Detection [banana] at 158.5,158.5   id: 51 , score: 0.66015625  
 Detection [banana] at 158.5,159.0   id: 51 , score: 0.5  
 Detection [banana] at 158.5,158.5   id: 51 , score: 0.58203125  
[INFO] [1631211589.998382]: fps: 3
 Detection [banana] at 158.5,158.5   id: 51 , score: 0.5  
 Detection [mela] at 158.5,158.5   id: 52 , score: 0.66015625  
 Detection [mela] at 158.5,158.5   id: 52 , score: 0.58203125  
 Detection [mela] at 158.5,158.5   id: 52 , score: 0.66015625  
 Detection [mela] at 158.5,158.5   id: 52 , score: 0.66015625  
 Detection [banana] at 158.5,157.0   id: 51 , score: 0.73046875  
[INFO] [1631211595.239139]: fps: 2
[INFO] [1631211600.886437]: fps: 1
[INFO] [1631211605.925757]: fps: 5
[INFO] [1631211611.037876]: fps: 3
 Detection [vaso] at 233.0,222.5   id: 85 , score: 0.5  
[INFO] [1631211616.197554]: fps: 4
 Detection [borsa] at 235.5,129.5   id: 30 , score: 0.5  
 Detection [borsa] at 219.0,129.0   id: 30 , score: 0.5  
 Detection [borsa] at 227.5,138.5   id: 30 , score: 0.58203125  
 Detection [borsa] at 230.5,139.0   id: 30 , score: 0.5  
 Detection [borsa] at 234.0,129.0   id: 30 , score: 0.5  
 Detection [borsa] at 231.0,129.0   id: 30 , score: 0.5  
[INFO] [1631211621.242037]: fps: 4
[INFO] [1631211626.386743]: fps: 6
 Detection [semaforo] at 148.5,179.0   id: 9 , score: 0.73046875  
 Detection [semaforo] at 151.5,218.0   id: 9 , score: 0.73046875  
 Detection [semaforo] at 118.5,217.0   id: 9 , score: 0.7890625  
 Detection [semaforo] at 111.0,214.0   id: 9 , score: 0.73046875  
 Detection [lavello] at 99.0,193.5   id: 80 , score: 0.5  
 Detection [vater] at 99.0,181.0   id: 69 , score: 0.7890625  
[INFO] [1631211631.611478]: fps: 3
 Detection [vater] at 80.0,178.0   id: 69 , score: 0.5  
 Detection [persona] at 204.0,158.5   id: 0 , score: 0.5  
 TRACKING PERSON 
 delta_x  44     delta_deg_x  -9 
 delta_y  -41     delta_deg_y  8 
 Detection [persona] at 178.5,156.0   id: 0 , score: 0.58203125  
 TRACKING PERSON 
 delta_x  18     delta_deg_x  -3 
 delta_y  -42     delta_deg_y  8 
[INFO] [1631211636.744531]: fps: 3
 Detection [persona] at 172.5,158.5   id: 0 , score: 0.58203125  
 TRACKING PERSON 
 delta_x  12     delta_deg_x  -2 
 delta_y  -40     delta_deg_y  8 
 Detection [vater] at 118.0,196.0   id: 69 , score: 0.83984375  
[INFO] [1631211641.877873]: fps: 4
 Detection [borsa] at 232.5,136.0   id: 30 , score: 0.5  
 Detection [persona] at 195.5,161.0   id: 0 , score: 0.5  
 TRACKING PERSON 
 delta_x  35     delta_deg_x  -7 
 delta_y  -36     delta_deg_y  7 
[INFO] [1631211646.953316]: fps: 3
[INFO] [1631211651.969112]: fps: 3
[INFO] [1631211657.219135]: fps: 3
[INFO] [1631211662.871499]: fps: 1
[INFO] [1631211667.997949]: fps: 6
[INFO] [1631211673.301733]: fps: 3
[INFO] [1631211678.509430]: fps: 4
[INFO] [1631211683.627036]: fps: 3
[INFO] [1631211688.734689]: fps: 3
[INFO] [1631211693.807655]: fps: 4
[INFO] [1631211698.967588]: fps: 3
[INFO] [1631211704.367031]: fps: 2
[INFO] [1631211709.606932]: fps: 2
[INFO] [1631211714.822089]: fps: 3
[INFO] [1631211720.018266]: fps: 3
[INFO] [1631211725.104279]: fps: 4
[INFO] [1631211730.320860]: fps: 4
[INFO] [1631211735.542035]: fps: 2
[INFO] [1631211740.635724]: fps: 7
[INFO] [1631211745.715543]: fps: 3
[INFO] [1631211750.838119]: fps: 2
[INFO] [1631211755.856170]: fps: 3
[INFO] [1631211761.111532]: fps: 2
[INFO] [1631211766.160178]: fps: 4
[INFO] [1631211771.258991]: fps: 2
[INFO] [1631211776.568861]: fps: 2
[INFO] [1631211781.671919]: fps: 3
[INFO] [1631211786.824948]: fps: 5
[INFO] [1631211792.004130]: fps: 4
[INFO] [1631211797.236492]: fps: 0
[INFO] [1631211802.347454]: fps: 4
[INFO] [1631211807.647876]: fps: 2
[INFO] [1631211812.656887]: fps: 4
[INFO] [1631211817.965451]: fps: 1
[INFO] [1631211823.166618]: fps: 2
[INFO] [1631211828.301155]: fps: 4
[INFO] [1631211833.408357]: fps: 4
[INFO] [1631211838.646809]: fps: 2
[INFO] [1631211843.776500]: fps: 4
[INFO] [1631211848.928910]: fps: 4
[INFO] [1631211854.139534]: fps: 3
[INFO] [1631211859.210857]: fps: 1
[INFO] [1631211864.405087]: fps: 2
[INFO] [1631211869.432307]: fps: 5
[INFO] [1631211874.488996]: fps: 3
[INFO] [1631211879.693287]: fps: 2
[INFO] [1631211885.208428]: fps: 1
[INFO] [1631211890.402648]: fps: 5
[INFO] [1631211895.466701]: fps: 2
[INFO] [1631211900.703559]: fps: 2
[INFO] [1631211905.795683]: fps: 5
[INFO] [1631211910.958337]: fps: 4
[INFO] [1631211916.066036]: fps: 3
[INFO] [1631211921.165003]: fps: 3
[INFO] [1631211926.414975]: fps: 3
[INFO] [1631211931.661435]: fps: 3
[INFO] [1631211936.820073]: fps: 2
[INFO] [1631211941.945880]: fps: 3
[INFO] [1631211946.961304]: fps: 5
[INFO] [1631211952.093739]: fps: 3
[INFO] [1631211957.107336]: fps: 5
[INFO] [1631211962.221087]: fps: 5
[INFO] [1631211967.271934]: fps: 2
[INFO] [1631211972.399119]: fps: 4
[INFO] [1631211977.511641]: fps: 3
[INFO] [1631211982.544523]: fps: 3
[INFO] [1631211987.828058]: fps: 2
[INFO] [1631211992.989187]: fps: 5
[INFO] [1631211998.138411]: fps: 2
[INFO] [1631212003.339292]: fps: 3
[INFO] [1631212008.469089]: fps: 4
[INFO] [1631212013.648347]: fps: 4
[INFO] [1631212018.864417]: fps: 2
[INFO] [1631212024.073397]: fps: 2
[INFO] [1631212029.244026]: fps: 3
[INFO] [1631212034.349195]: fps: 5
[INFO] [1631212039.494167]: fps: 5
[INFO] [1631212045.282241]: fps: 1
[INFO] [1631212050.581936]: fps: 1
[INFO] [1631212055.589744]: fps: 1
[INFO] [1631212060.705941]: fps: 2
[INFO] [1631212065.772302]: fps: 5
[INFO] [1631212071.185102]: fps: 2
[INFO] [1631212076.428905]: fps: 3
[INFO] [1631212081.690045]: fps: 2
[INFO] [1631212086.852314]: fps: 3
[INFO] [1631212092.203147]: fps: 1
 Detection [kite] at 256.0,172.5   id: 37 , score: 0.5  
[INFO] [1631212097.211319]: fps: 6
[INFO] [1631212102.440438]: fps: 4
[INFO] [1631212107.611812]: fps: 5
[INFO] [1631212112.618757]: fps: 4
[INFO] [1631212117.824487]: fps: 2
[INFO] [1631212122.924126]: fps: 4
[INFO] [1631212127.945889]: fps: 6
[INFO] [1631212133.105087]: fps: 5
[INFO] [1631212138.132056]: fps: 6
[INFO] [1631212143.573118]: fps: 1
[INFO] [1631212148.671319]: fps: 4
[INFO] [1631212153.888478]: fps: 3
[INFO] [1631212159.033012]: fps: 6
[INFO] [1631212164.145214]: fps: 5
[INFO] [1631212169.150338]: fps: 6
[INFO] [1631212174.438710]: fps: 1
[INFO] [1631212179.737458]: fps: 1
[INFO] [1631212184.751251]: fps: 3
[INFO] [1631212189.963563]: fps: 1
[INFO] [1631212195.131298]: fps: 6
[INFO] [1631212200.207240]: fps: 4
[INFO] [1631212205.400014]: fps: 5
[INFO] [1631212210.456179]: fps: 1
[INFO] [1631212215.537128]: fps: 2
[INFO] [1631212220.759526]: fps: 2
[INFO] [1631212225.926209]: fps: 5
[INFO] [1631212231.416849]: fps: 1
[INFO] [1631212236.431305]: fps: 3
[INFO] [1631212241.700757]: fps: 3
[INFO] [1631212246.961635]: fps: 2
[INFO] [1631212252.052159]: fps: 3
[INFO] [1631212257.140488]: fps: 5
[INFO] [1631212262.381285]: fps: 4
[INFO] [1631212267.456088]: fps: 5
[INFO] [1631212272.760458]: fps: 3
[INFO] [1631212278.043958]: fps: 1
[INFO] [1631212283.182254]: fps: 2
[INFO] [1631212288.302504]: fps: 2
[INFO] [1631212293.929147]: fps: 1
[INFO] [1631212299.172267]: fps: 3
[INFO] [1631212304.188389]: fps: 6
[INFO] [1631212309.275358]: fps: 6
[INFO] [1631212314.442809]: fps: 3
[INFO] [1631212320.692030]: fps: 0
[INFO] [1631212325.750988]: fps: 4
[INFO] [1631212330.759288]: fps: 3
[INFO] [1631212335.862189]: fps: 5
[INFO] [1631212341.041385]: fps: 3
[INFO] [1631212346.184529]: fps: 4
[INFO] [1631212351.373257]: fps: 5
[INFO] [1631212356.630514]: fps: 2
[INFO] [1631212362.112197]: fps: 2
[INFO] [1631212367.113543]: fps: 3
[INFO] [1631212372.253342]: fps: 3
[INFO] [1631212377.363927]: fps: 3
[INFO] [1631212382.525723]: fps: 5
[INFO] [1631212387.612936]: fps: 4
[INFO] [1631212392.690478]: fps: 4
[INFO] [1631212397.725153]: fps: 4
[INFO] [1631212402.762683]: fps: 4
[INFO] [1631212408.151552]: fps: 1
[INFO] [1631212413.241533]: fps: 5
[INFO] [1631212418.493451]: fps: 2
[INFO] [1631212423.642059]: fps: 6
[INFO] [1631212428.693465]: fps: 4
[INFO] [1631212433.770837]: fps: 3
[INFO] [1631212439.104464]: fps: 1
[INFO] [1631212444.410951]: fps: 1
[INFO] [1631212449.913692]: fps: 0
[INFO] [1631212455.016466]: fps: 6
[INFO] [1631212460.151907]: fps: 6
[INFO] [1631212465.529386]: fps: 2
[INFO] [1631212470.728579]: fps: 4
[INFO] [1631212475.878384]: fps: 5
[INFO] [1631212481.325658]: fps: 1
[INFO] [1631212486.366763]: fps: 3
[INFO] [1631212491.371143]: fps: 5
[INFO] [1631212496.409792]: fps: 2
[INFO] [1631212502.233085]: fps: 0
[INFO] [1631212507.248760]: fps: 3
[INFO] [1631212512.389155]: fps: 4
[INFO] [1631212517.532846]: fps: 5
[INFO] [1631212522.704541]: fps: 5
[INFO] [1631212527.988784]: fps: 2
[INFO] [1631212533.149953]: fps: 5
[INFO] [1631212538.334840]: fps: 4
[INFO] [1631212543.503671]: fps: 4
[INFO] [1631212548.745190]: fps: 4
[INFO] [1631212553.770662]: fps: 4
[INFO] [1631212559.031308]: fps: 3
[INFO] [1631212564.086475]: fps: 4
[INFO] [1631212569.255930]: fps: 4
[INFO] [1631212574.451984]: fps: 3
[INFO] [1631212579.635544]: fps: 2
[INFO] [1631212584.718123]: fps: 2
[INFO] [1631212589.903687]: fps: 4
[INFO] [1631212594.945110]: fps: 6
[INFO] [1631212599.955248]: fps: 5
[INFO] [1631212605.102016]: fps: 5
[INFO] [1631212610.176747]: fps: 5
[INFO] [1631212615.191335]: fps: 4
[INFO] [1631212620.312369]: fps: 1
[INFO] [1631212625.447764]: fps: 5
[INFO] [1631212630.562707]: fps: 5
[INFO] [1631212635.639164]: fps: 5
[INFO] [1631212640.752218]: fps: 6
[INFO] [1631212645.834112]: fps: 3
F driver/usb/usb_driver.cc:1148] HandleQueuedBulkIn transfer in failed. Not found: USB transfer error 5 [LibUsbDataInCallback]
[edgetpu-1] process has died [pid 25701, exit code -6, cmd /home/pi/ros/src/myrobot/src/vision/rm_edgetpu_detect/scripts/edgetpu_node.py __name:=edgetpu __log:=/home/pi/.ros/log/33c9b67c-118a-11ec-868a-0242abf3735e/edgetpu-1.log].
log file: /home/pi/.ros/log/33c9b67c-118a-11ec-868a-0242abf3735e/edgetpu-1*.log
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
done
BernardinD commented 3 years ago

@Luke1962 for me it was a power issue. Using a USB camera along with the USB 3.0 accelerators required too much power. Once switching to a camera with a different connector everything worked fine

Luke1962 commented 3 years ago

@BernardinD Thank you. I cannot exclude this possibility, even if I'm powering USB3 HUB with a dedicated dc-dc power supply, different from Jetson power supply (which has its power jack , 20W dedicated dc-dc converter and running on 3cpu @1GHz instead of 4cpu). The strange thing is that this problem appears on a "static" scenario without detections

aslanpour commented 2 years ago

I face this error when a container loads an object detection model while a previous container has done so already. The older container fails and the new one gets the TPU, but I have no idea how to catch the error so the container does not fail.