search

google-coral / edgetpu

Coral issue tracker (and legacy Edge TPU API source)
https://coral.ai
Apache License 2.0
422 stars 124 forks source link

Model Does Not Contain Only Custom Operations #587

Closed vyang2968 closed 2 years ago

vyang2968 commented 2 years ago

Description

Hello, I am new to Google Coral's Edge TPU, but am familiar with Tensorflow. To get started, I ran this Colab notebook. I compiled the model with the edgetpu_compiler, I got an output which said that 3 operations would still run on the CPU. I still downloaded the model and ran inferencing on my Dev Board with edgetpu_detect_server --source /dev/video1:YUY2:800x600:20/1 --model exported_model.tflite. I got about 17 fps. I did the same thing, but with a pretrained model from this link with edgetpu_detect_server --source /dev/video1:YUY2:800x600:20/1 --model tf2_ssd_mobilenet_v2_coco17_ptq_edgetpu.tflite. With this model, however, I got around 76 fps. Since Google Coral says that "If part of your model executes on the CPU, you should expect a significantly degraded inference speed compared to a model that executes entirely on the Edge TPU. We cannot predict how much slower your model will perform in this situation", I assume that the 3 operations that run on the CPU are the cause of this difference. It's weird that an official Google Coral Colab Notebook would not have a model that has all the operations happening on the TPU. How can I make sure all of those 3 operations run on the TPU? I will be happy to provide any additional information. Thanks in advance.

Click to expand! ### Issue Type Performance ### Operating System Mendel Linux ### Coral Device Dev Board ### Other Devices _No response_ ### Programming Language _No response_ ### Relevant Log Output ```shell Edge TPU Compiler version 16.0.384591198 Started a compilation timeout timer of 180 seconds. Model compiled successfully in 3299 ms. Input model: exported.tflite Input size: 4.24MiB Output model: exported_edgetpu.tflite Output size: 5.57MiB On-chip memory used for caching model parameters: 4.21MiB On-chip memory remaining for caching model parameters: 3.29MiB Off-chip memory used for streaming uncached model parameters: 0.00B Number of Edge TPU subgraphs: 1 Total number of operations: 267 Operation log: exported_edgetpu.log Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs. Number of operations that will run on Edge TPU: 264 Number of operations that will run on CPU: 3 See the operation log file for individual operation details. Compilation child process completed within timeout period. Compilation succeeded! ```
hjonnala commented 2 years ago

Hi, It is not possible to map those 3 operations to TPU. As EdgeTPU does not support dequantize and TFLite_Detection_PostProcess custom operation.

Please go through this page for model requirements and supported operations: https://coral.ai/docs/edgetpu/models-intro/.

vyang2968 commented 2 years ago

That's unfortunate. How can I improve my model's FPS then? Does it depend on the architecture? Does SSD MobileNet V2 perform better than the EfficientLite-D0?

hjonnala commented 2 years ago

Yes in terms of latency SSD Mobilenet V2 perform better than the EfficientLite-D0, but accuracy would be higher in case of EfficientLite-D0.

vyang2968 commented 2 years ago

Alright thank you. I will try it with SSD Mobilenet V2.

google-coral-bot[bot] commented 2 years ago

Are you satisfied with the resolution of your issue? Yes No