Tencent / TNN

TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.
Other
4.38k stars 766 forks source link

模型导出为fp16时失败(工具输出结果为succeed但模型文件仍为fp32) #1911

Open polarispw opened 1 year ago

polarispw commented 1 year ago

1. 使用环境(environment)

---------- convert model, please wait a moment ----------

Converter ONNX to TNN Model...

Converter ONNX to TNN check_onnx_dim...

Converter ONNX to TNN check_onnx_dim...

Model will be converted to FP16!!! // 这行是为了看half是否被启用自己加的 The onnx2tnn command:python3 onnx2tnn.py /workspace/matmul_linear1.onnx -version=202303210906 -optimize=0 -half=1 -o /workspace

E/tnn: onnx2tnn_convert [File /opt/TNN/tools/onnx2tnn/onnx-converter/onnx2tnn_convert.cc][Line 153] The input_info

0.----onnx version:1.6.0

algo_optimize 0

onnx_net_opt_path /workspace/matmul_linear1.onnx

2.----onnx2tnn: /workspace/matmul_linear1.onnx

3.----onnx2tnn status: 0

D/tnn: OnnxExtractBlobWeights [File /opt/TNN/tools/onnx2tnn/src/core/onnx2tnn.cc][Line 388] weight = linear.bias

D/tnn: OnnxExtractBlobWeights [File /opt/TNN/tools/onnx2tnn/src/core/onnx2tnn.cc][Line 388] weight = onnx::MatMul_6

D/tnn: TNNWriteProto [File /opt/TNN/tools/onnx2tnn/src/core/onnx2tnn.cc][Line 247] input_blob_shape dim_size: 3

D/tnn: TNNWriteProto [File /opt/TNN/tools/onnx2tnn/src/core/onnx2tnn.cc][Line 330] node:/linear/MatMul_output_0 onnx:MatMul -> tnn:MatMul

D/tnn: TNNWriteProto [File /opt/TNN/tools/onnx2tnn/src/core/onnx2tnn.cc][Line 330] node:output_1 onnx:Add -> tnn:Add

WriteTensorData [Line 160] tersor (onnx::MatMul_6) data type: 1 item_size: 1179648

WriteTensorData [Line 160] tersor (linear.bias) data type: 1 item_size: 1536

WriteTensorData [Line 160] tersor (linear.bias) data type: 1 item_size: 1536

D/tnn: Interpret [File /opt/TNN/source/tnn/interpreter/tnn/model_interpreter.cc][Line 103] model params md5: dc0cb66c1ba4de01cd48e0eea9107e00

D/tnn: Interpret [File /opt/TNN/source/tnn/interpreter/tnn/model_interpreter.cc][Line 103] model params md5: e0266ddf9b4f40ea35f25d0b443827e7

D/tnn: CpuSupportFp16 [File /opt/TNN/source/tnn/utils/cpu_utils.cc][Line 356] CpuUtils::CpuSupportFp16, TNN_ARM82 is off, fp16arith = 0.

D/tnn: Init [File /opt/TNN/source/tnn/core/default_network.cc][Line 86] support fp 16: 0

D/tnn: CpuSupportFp16 [File /opt/TNN/source/tnn/utils/cpu_utils.cc][Line 356] CpuUtils::CpuSupportFp16, TNN_ARM82 is off, fp16arith = 0.

D/tnn: ReshapeLayers [File /opt/TNN/source/tnn/core/default_network.cc][Line 711] ReshapeLayers Output Shape: [name: /linear/MatMul_output_0 data type: 0 shape: [ 16 128 1536 ]]

D/tnn: ReshapeLayers [File /opt/TNN/source/tnn/core/default_network.cc][Line 711] ReshapeLayers Output Shape: [name: output_1 data type: 0 shape: [ 16 128 1536 ]]

D/tnn: Forward [File /opt/TNN/source/tnn/core/default_network.cc][Line 600] layer name: /linear/MatMul, forward result: 0

D/tnn: Forward [File /opt/TNN/source/tnn/core/default_network.cc][Line 601] Output Shape: [name: /linear/MatMul_output_0 data type: 0 shape: [ 16 128 1536 ]]

D/tnn: Forward [File /opt/TNN/source/tnn/core/default_network.cc][Line 600] layer name: /linear/Add, forward result: 0

D/tnn: Forward [File /opt/TNN/source/tnn/core/default_network.cc][Line 601] Output Shape: [name: output_1 data type: 0 shape: [ 16 128 1536 ]]

Converter ONNX to TNN model succeed!



  **5. 截图(Screenshots)**
 If applicable, add screenshots to help explain your problem.
polarispw commented 1 year ago

上述问题可以通过拉取支持fp16模型转换的最新版本解决,使用半精度的onnx模型直接转换为tnn模型。 但是转换出的tnn模型(Bert-base-uncased from huggingface)在TNNTest中无法运行:

/TNNTest -mp Bert_simp_half.tnnproto -dt=ARM -ic=5 -wc=1 -pr=NORMAL                                         <
2023-03-27 14:59:42 773: E source/tnn/optimizer/graph_matcher/ir.cc:584 the graph is not connected.
E/tnn: virtual tnn::Status tnn::optimizer::NetOptimizerConvertMatMulToConv::Optimize(tnn::NetStructure *, tnn::NetResource *) [File source/tnn/optimizer/net_optimizer_convert_matmul_to_conv.cc][Line 77] code: 0x6000 msg: source/tnn/optimizer/graph_matcher/ir.cc:584 the graph is not connected.E/tnn: static tnn::DimsVector tnn::DimsFunctionUtils::StrideSlice(const tnn::DimsVector, tnn::DimsVector &, tnn::DimsVector &, const tnn::DimsVector, const tnn::DimsVector, tnn::Status *) [File source/tnn/utils/dims_function_utils.cc][Line 164] StrideSliceV2Layer param of axes, ends, strides size is invalid
E/tnn: virtual tnn::Status tnn::CpuSubLayerAcc::Calculate(const std::vector<Blob *> &, const std::vector<void *> &, const std::vector<DimsVector> &, tnn::Blob *) [File source/tnn/device/cpu/acc/cpu_sub_layer_acc.cc][Line 48] Error: CpuSubLayerAcc don't support data type: 1
E/tnn: virtual tnn::Status tnn::DefaultNetwork::Forward() [File source/tnn/core/default_network.cc][Line 603] Forward error code: 0x4000 msg: Error: CpuSubLayerAcc don't support data type, exit
E/tnn: bool tnn::test::CheckResult(std::string, tnn::Status) [File test/test.cc][Line 348] create instance failed: code: 0x4000 msg: Error: CpuSubLayerAcc don't support data type 

@gttiankai