search

google-coral / pycoral

Python API for ML inferencing and transfer-learning on Coral devices
https://coral.ai
Apache License 2.0
360 stars 149 forks source link

Using multi-threading for unsupported operations #112

Closed JHeldens closed 1 year ago

JHeldens commented 1 year ago

Description

Hi,

I have trained an object detection model using the Tensorflow Lite Model maker. The last three unsupported operations need to be run by the CPU. I have manually splitted the tflite file into two tflite files, a head segment and tail segment. By invoking the head segment using the PyCoral API (contains only supported operations) and the tail segment (contains the unsupported operations) by tflite-runtime, which supports multi-threading. An increase of 2 FPS is achieved.

An edge tpu compiled tflite model in which the last three operations of the AI-model are not supported by the Coral, and therefore needs to be run by the CPU are not performed using multi-threading. I believe the PyCoral API does not provide the option to select the amount of threads that needs to be used for unsupported operations. Is it possible to run the unsupported operations using multi-threading with the PyCoral API?

Frame rate PyCoral for full AI-model: 5.2 FPS Frame rate PyCoral + tflite-runtime: 7.3 FPS

Click to expand! ### Issue Type Performance ### Operating System Linux ### Coral Device USB Accelerator ### Other Devices Rapsberry Pi 4 ### Programming Language Python 3.9 ### Relevant Log Output _No response_
hjonnala commented 1 year ago

Is it possible to run the unsupported operations using multi-threading with the PyCoral API?

No, its not possible.

But, Pycoral is just a wrapper around tflite-runtime. So, it should be fine if you were able to work with PyCoral + tflite-runtime. Thanks!!!

JHeldens commented 1 year ago

Alright! I just thought it would be interesting if this option was implemented into the PyCoral. The process of splitting a tflite file was rather complex.

hjonnala commented 1 year ago

Frame rate PyCoral for full AI-model: 5.2 FPS

Perhaps, you may try loading the model using tflite interpreter with num_threads instead of using pycoraledgetpu.make_interpreter.

JHeldens commented 1 year ago

Thanks for the suggestion. I have tried it, the frame rate is the same, i.e. around 5.2 - 5.4 FPS. Setting the amount of threads does not seem to influence the performance during inference.

JHeldens commented 1 year ago

Perhaps I should post this issue at https://github.com/iCorv/tflite-runtime/issues ?

google-coral-bot[bot] commented 1 year ago

Are you satisfied with the resolution of your issue? Yes No