search

PaddlePaddle / Paddle2ONNX

ONNX Model Exporter for PaddlePaddle
Apache License 2.0
703 stars 162 forks source link

int8量化后的模型导出onnx时，有两个op不支持 #402

Open zhangxianjie1314 opened 2 years ago

zhangxianjie1314 commented 2 years ago

fake_quantize_dequantize_moving_average_abs_max fake_channel_wise_quantize_dequantize_abs_max

yeliang2258 commented 2 years ago

您好，请拉这个PR测试：https://github.com/PaddlePaddle/Paddle2ONNX/pull/399

yeliang2258 commented 2 years ago

另外，用户您好，请问你转换为ONNX模型的目的是？用于部署的话，请问部署引擎是？

jiangjiajun commented 2 years ago

Hi, @zhangxianjie1314

我们准备了一份简单的问题调研（半分钟可完成），如有时间可帮忙完成调研，帮助Paddle2ONNX做得更好 https://iwenjuan.baidu.com/?code=r8hu2s

zhangxianjie1314 commented 2 years ago

另外，用户您好，请问你转换为ONNX模型的目的是？用于部署的话，请问部署引擎是？

想用onnx来进行部署，但是用tensorrt对量化后的onnx进行部署时，发现onnx2trt似乎不支持int8的onnx，所以想请教一下，如果对int8模型进行部署，有什么推荐的方法吗？谢谢。

yeliang2258 commented 2 years ago

可以直接使用paddleinference进行部署，paddleinference可以直接部署。 另外使用TensorRT是可以直接加载我们转换出来的ONNX量化模型的，具体可以参考TensorRT的例子，直接加载ONNX模型。

zhangxianjie1314 commented 2 years ago

好的，谢谢了，我尝试一下

zhangxianjie1314 commented 2 years ago

可以直接使用paddleinference进行部署，paddleinference可以直接部署。 另外使用TensorRT是可以直接加载我们转换出来的ONNX量化模型的，具体可以参考TensorRT的例子，直接加载ONNX模型。

image 为什么TensorRT官网上说不支持int8的onnx进行部署？

yeliang2258 commented 2 years ago

请安装TensorRT8+，具体链接：https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#working-with-int8 image

yeliang2258 commented 2 years ago

你也可以使用如下类似的命令进行转换： trtexec --onnx=quant_model.onnx --workspace=2048 --explicitBatch --int8

zhangxianjie1314 commented 2 years ago

你也可以使用如下类似的命令进行转换： trtexec --onnx=quant_model.onnx --workspace=2048 --explicitBatch --int8

好的好的，谢谢

edwardnguyen1705 commented 2 years ago

Dear @yeliang2258 , I have been trying your work (#399), but it failed. Do you have any suggestion? Thanks for your time. Here is the log:

2022-01-13 08:41:22 [WARNING]
======================

 Not Find Name : hardswish_38.tmp_0

======================

2022-01-13 08:41:22 [WARNING]
======================

 Not Find Name : hardswish_42.tmp_0

======================

2022-01-13 08:41:22 [WARNING]
======================

 Not Find Name : hardswish_46.tmp_0

======================

2022-01-13 08:41:22 [WARNING]
======================

 Not Find Name : hardswish_50.tmp_0

======================

2022-01-13 08:41:23 [INFO]      ONNX model generated is valid.
Traceback (most recent call last):
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/utils.py", line 43, in check_model
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/onnx/checker.py", line 104, in check_model
    C.check_model(protobuf_string)
onnx.onnx_cpp2py_export.checker.ValidationError: Unrecognized attribute: split for operator Split

==> Context: Bad node spec for node. Name: Split_0 OpType: Split

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/python3.7.0/bin/paddle2onnx", line 33, in <module>
    sys.exit(load_entry_point('paddle2onnx==0.9.0', 'console_scripts', 'paddle2onnx')())
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/command.py", line 195, in main
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/command.py", line 159, in program2onnx
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/convert.py", line 85, in program2onnx
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/convert.py", line 38, in export_onnx
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/graph/onnx_graph.py", line 367, in export_proto
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/utils.py", line 45, in check_model
Exception: ONNX model is not valid.
yeliang2258 commented 2 years ago

@edwardnguyen1705 hello, please pull this PR to test: https://github.com/PaddlePaddle/Paddle2ONNX/pull/399

edwardnguyen1705 commented 2 years ago

@edwardnguyen1705 hello, please pull this PR to test: #399

Hi @yeliang2258 , Thanks for your time. I checked that you have already fixed the split issue. However, I have just tried to pull PR #399, but the result is the same.

$ git clone https://github.com/yeliang2258/Paddle2ONNX
$ cd Paddle2ONNX
$ git checkout quantize_dev
$ python setup.py install
onnx: 1.9.0
onnxruntime : 1.10.0
paddle2onnx: 0.9.0
2022-01-14 02:15:17 [WARNING]
======================

 Not Find Name : hardswish_38.tmp_0

======================

2022-01-14 02:15:17 [WARNING]
======================

 Not Find Name : hardswish_42.tmp_0

======================

2022-01-14 02:15:18 [WARNING]
======================

 Not Find Name : hardswish_46.tmp_0

======================

2022-01-14 02:15:18 [WARNING]
======================

 Not Find Name : hardswish_50.tmp_0

======================

2022-01-14 02:15:18 [INFO]      ONNX model generated is valid.
Traceback (most recent call last):
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/utils.py", line 43, in check_model
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/onnx/checker.py", line 104, in check_model
    C.check_model(protobuf_string)
onnx.onnx_cpp2py_export.checker.ValidationError: Unrecognized attribute: split for operator Split

==> Context: Bad node spec for node. Name: Split_0 OpType: Split

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/python3.7.0/bin/paddle2onnx", line 33, in <module>
    sys.exit(load_entry_point('paddle2onnx==0.9.0', 'console_scripts', 'paddle2onnx')())
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/command.py", line 195, in main
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/command.py", line 159, in program2onnx
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/convert.py", line 85, in program2onnx
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/convert.py", line 38, in export_onnx
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/graph/onnx_graph.py", line 367, in export_proto
  File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle2onnx-0.9.0-py3.7.egg/paddle2onnx/utils.py", line 45, in check_model
Exception: ONNX model is not valid.
edwardnguyen1705 commented 2 years ago

Hi @yeliang2258 , Regarding the hardwish activation layer, could you take a look: https://github.com/ultralytics/yolov5/issues/825

nongwoluanlai666 commented 2 years ago

@yeliang2258 请问，这两个op支持了么，试过了好多分支了，仍然解决不了，目前的需求是：将paddle模型量化后转为onnx =========== fake_channel_wise_quantize_dequantize_abs_max =========== =========== fake_quantize_dequantize_moving_average_abs_max ===========

yeliang2258 commented 2 years ago

@nongwoluanlai666 用这个PR来转吧：https://github.com/PaddlePaddle/Paddle2ONNX/pull/651 另外，请问你导出为ONNX用什么推理引擎部署呢？

nongwoluanlai666 commented 2 years ago

@yeliang2258 您好，我是打算用在MNN部署的。目前MNN可以用protobuffer 格式作为量化参数的输入，因为不熟悉protobuffer，我用#651导出的txt转换仍有困难; MNN官网的量化参数定义如下： MNN_compression.proto