transpose layer outputs value is nan

[Lennon-cheng]

paddle ernie model convert to openvino。 There is transpose layer, after this transpose layer, why some positions of the output is become nan。 netron：

transpose inputs：

transpose outputs：

[xczhai]

@Lennon-cheng Thanks for your detailed log and info. That's really helpful to OpenVINO. But I need to confirm something to accelerate the debug progress.

1. The model is from https://www.paddlepaddle.org.cn/lite/develop/demo_guides/intel_openvino.html, right?
2. Could you provide the reproduced script if possible?
3. Does this issue occur in the master branch?

Thanks

[Lennon-cheng]

I finetune paddlenlp "ernie-gram-zh" pretrain model used paddlepaddle framework. this is load model code pretrained_model = paddlenlp.transformers.ErnieGramModel.from_pretrained( 'ernie-gram-zh') tokenizer = ppnlp.transformers.ErnieGramTokenizer.from_pretrained( 'ernie-gram-zh')

then use paddle2onnx convert paddle to onnx. model.onnx in zip is onnx file. But CumSum not support. Then I modify some node and result is model1.onnx. This code convert.py is in zip. Then I convert onnx to openvino used script.sh which is in zip. Then I infer openvino used infer.py which is in zip.
Run: python infer.py 你 好. Can find the problem transpose node which name p2o.Transpose.65

zip download from baidu drive 链接: https://pan.baidu.com/s/1h5osf5ad5HIu7vx17RMPfw?pwd=dv25 提取码: dv25

[yuxu42]

@Lennon-cheng as OV has already supported input of Paddle format model that OV handles the paddle to ngraph conversion internally, so that it can save additional Paddle to ONNX to IR conversion. Could you please have a try? https://docs.openvino.ai/nightly/openvino_docs_OV_UG_Integrate_OV_with_your_application.html#step-2-compile-the-model

[xczhai]

@Lennon-cheng Hi,

1. I downloaded the zip file but failed to load the model.onnx in netron and openvino. It seems to be broken. Could you take a check?
2. For CumSum not support problem. I try to use paddle2onnx to convert the ernie_1.0/model.pdmodel in https://www.paddlepaddle.org.cn/lite/develop/demo_guides/intel_openvino.html instead. Yes, running the generated onnx produces "CumSum layer with name 'p2o.CumSum.0' doesn't support 'axis' input tensor with non scalar rank". I think the root cause is the paddle2onnx may generate an unsupported corner case. I will look into such the problem. Don't be discouraged, you can load and run ernie_1.0/model.pdmodel directly. That works well.
3. The script is to do the inference of ernie-gram-zh model. The recommended step is pretrained_model ==> paddle finetune ==> paddle save inference model ==> *.pdmodel & *.pdiparams format ==> OpenVINO.
4. OpenVINO can load and run the native paddle inference natively. I have checked that cumsum has been supported in OpenVINO. Could you take a try as the above comments? https://github.com/openvinotoolkit/openvino/blob/master/docs/MO_DG/prepare_model/Supported_Frameworks_Layers.md
5. BTW, I find some lagacy api is still in your script. It is recommended to replace them with API2.0. That supports dynamic shape well. https://docs.openvino.ai/latest/openvino_2_0_inference_pipeline.html

Thanks.

[xczhai]

@Lennon-cheng Hi,

• I try https://paddlelite-demo.bj.bcebos.com/NNAdapter/models/PaddleNLP/ernie_1.0.tgz and convert pdmodel to onnx using paddle2onnx. The model architecture is the same as your provided.
• Could you please try to run pdmodel using openvino because we want to exclude the paddle2onnx effect?
• It is appreciated if you help provide pdmodel and pdiparams or the command line to convert paddle to onnx.

Thanks.

[Lennon-cheng]

@xczhai Thank you very much for your help！ First, I succeeded in running pdmodel using openvino！

I reorganized the whole process code. Step is paddle save inference model ==> .pdmodel & .pdiparams format ==> onnx ==> OpenVINO

The explanation is in the readme of the zip. Specifically pdmodel、pdiparams and command line is in zip.

链接: https://pan.baidu.com/s/1oiW5SGqUPurg_IGZfDZt0A?pwd=65qu 提取码: 65qu

[xczhai]

@Lennon-cheng Thanks for your attachment.

• I think the legacy API causes this bug. Please refer to the uploaded python scritpt. ov_infer.zip
• If you still encounter the cumsum bug, please refer to the fixed code in my temp branch https://github.com/xczhai/openvino/compare/fix_onnx_cumsum...openvinotoolkit:openvino:fix_onnx_cumsum
• We will make an internal review of the cumsum op before merged to master.
• BTW, if you want to hit the best latency or throughput performance, please refer to https://docs.openvino.ai/latest/openvino_docs_deployment_optimization_guide_dldt_optimization_guide.html#
• If it works, please tell us and close it.

Thanks.

[Lennon-cheng]

@xczhai The problem is solved. Thank you very much for your help!!!