search

sithu31296 / PyTorch-ONNX-TFLite

Conversion of PyTorch Models into TFLite
MIT License
356 stars 48 forks source link

channel problem #2

Open rose-jinyang opened 3 years ago

rose-jinyang commented 3 years ago

Hello How are you? Thanks for contributing to this project. I converted my PyTorch model to the TFLite model as your method. But this TFLite model is different from the original TFLite model from the Keras model.

This figure is the structure of the base TFLite model converted from the Keras model.

image

The below figure is the structure of the TFLite model converted from the Pytorch model.

image

The below figure is the structure of the ONNX model converted from the Pytorch model.

image

The main problem is that the TFLite model converted from the Pytorch model runs more slowly than the base TFLite model converted from the Keras model.

As u know, the PyTorch model uses channel-first as tensor ordering and the Tensorflow uses channel-last. I looked at the structure of the TFLite model converted from the PyTorch model. Comparing with the base TFLite model converted from the Keras model, there are many Transpose layers in the TFLite model converted from the PyTorch model. I think that this affects the model inference speed. Can we get the TFLite model of using channel-last ordering from the PyTorch model?

sithu31296 commented 3 years ago

Hi Thanks for your interest in this repo.

PyTorch and ONNX are written in channel first format. PyTorch has beta support for NCHW format. Please check here. But this will not convert the model to NCHW format. Although ONNX has some support for NHWC format, but ONNX-tf has no support for NHWC. So, as far as I know, you cannot do that.

But if your main concern is speed, run onnx model with onnxruntime. I see a performance gain from 100ms to 1ms with just running onnx model with onnxruntime rather than using pytorch model.

If you have time, please also check out this guide for optimizing pytorch model.

PINTO0309 commented 3 years ago

I don't mean to interrupt this thread, but there is a way. You can use the tool I created to convert it to NHWC. Since it is still a WIP, there are occasional bugs. Eliminate all Transpose layers.

PyTorch (NCHW) -> ONNX (NCHW) -> OpenVINO (NCHW) -> optimized -> saved_model (NHWC) -> tflite (NHWC) https://github.com/PINTO0309/openvino2tensorflow

sample image https://github.com/PINTO0309/openvino2tensorflow#8-model-structure

sithu31296 commented 3 years ago

That is a great tool! But I have a question. That channel conversion you implemented is done in OpenVINO to tf model step. Right? Even official onnx_tf doesn't have any support for NHWC. Is there any speed comparison you tested?

PINTO0309 commented 3 years ago

That channel conversion you implemented is done in OpenVINO to tf model step. Right?

Yes. That's right.

Even official onnx_tf doesn't have any support for NHWC. Is there any speed comparison you tested?

Unfortunately, it generates a model that is far more optimized and faster than the onnx-tf developed by Microsoft. Sorry. I didn't record the benchmark because it is very troublesome. It's fast enough to feel, since it doesn't contain any transpose. However, only GroupConvolution is known to be very slow because tflite does not support it. The tool I created reads all the weights and structures during the conversion process and automatically transposes NCHW to NHWC.

By the way, if you want to reverse-translate tfite (NHWC) files, the following tool is also available.

"Generate saved_model, tfjs, tf-trt, EdgeTPU, CoreML, quantized tflite, ONNX, OpenVINO, Myriad Inference Engine blob and .pb from .tflite. Support for building environments with Docker. It is possible to directly access the host PC GUI and the camera to verify the operation. NVIDIA GPU (dGPU) support." https://github.com/PINTO0309/tflite2tensorflow

Here are 122 models that I have committed to transformations generated from PyTorch and other sources. You can try them out. https://github.com/PINTO0309/PINTO_model_zoo

Specially optimized TFLite runtime installer https://github.com/PINTO0309/TensorflowLite-bin

sithu31296 commented 3 years ago

Thanks for your answer! I will check it out your works!

PINTO0309 commented 3 years ago

I am looking forward to your contribution. I am a lone committer.

lee-man commented 3 years ago

I also face this problem when I try to deploy the model on Android phones using Google's official demo. When I try to replace the TF models, the channel ordering is different so I cannot directly reuse other codes.

PINTO0309 commented 3 years ago

@lee-man You can freely change the channel order and the shape of Transpose and Reshape. https://github.com/PINTO0309/openvino2tensorflow#6-7-replace-weights-or-constant-values-in-const-op-and-add-transpose-or-reshape-or-cast-just-beforeafter-the-operation-specified-by-layer_id

lee-man commented 3 years ago

@lee-man You can freely change the channel order and the shape of Transpose and Reshape. https://github.com/PINTO0309/openvino2tensorflow#6-7-replace-weights-or-constant-values-in-const-op-and-add-transpose-or-reshape-just-before-the-operation-specified-by-layer_id

Got it. I will check your tool later. Thanks for your help!

MPolaris commented 2 years ago

You can use this tools, it will auto transform NCHW to NHWC: https://github.com/MPolaris/onnx2tflite