search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
13.59k stars 2.77k forks source link

ONNX with FloatTensorType when inferred from C++ returns different label everytime #15665

Open VPU-tech opened 1 year ago

VPU-tech commented 1 year ago

My ONNX Model has been built on float pointing numbers, where number accuracy is of high importance upto 10 decimal precision . ONNX model input is array of float numbers and output is label.

When I am inferring this ONNX model in C++ using 'Microsoft.ML.OnnxRuntime\1.13.1\runtimes', for same set of input, output label keeps differing. Also sometimes it is not even matching with results, when inferred same ONNX model from python, However while executing it from python, label output is not only consistent, but also same as pickle file output.

`C++ Code Snippet for ONNX Inference

// Set up options for session Ort::SessionOptions sessionOptions; sessionOptions.SetIntraOpNumThreads(1);

// Sets graph optimization level (Here, enable all possible optimizations) sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);

std::string instanceName{ "Test Inference" }; mEnv = std::make_shared(OrtLoggingLevel::ORT_LOGGING_LEVEL_WARNING, instanceName.c_str());

// use it.... mSession = std::make_shared(*mEnv, pwcsName, sessionOptions);

// Create input tensors of ORT::Value, which is a tensor format used by ONNX Runtime Ort::MemoryInfo memoryInfo = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtDeviceAllocator, OrtMemType::OrtMemTypeCPU);

auto input_tensor = Ort::Value::CreateTensor(memoryInfo, inputTensorValues.data(), inputTensorSize, mInputDims.data(), mInputDims.size());

auto output_tensors = mSession->Run(Ort::RunOptions{ nullptr }, inputNames.data(), &input_tensor, 1, outputNames.data(), 1);

// Get the inference result float* outArr = output_tensors.front().GetTensorMutableData() assert(abs(outArr[0] - 0.000045) < 1e-6);

// score the model, and print scores for label classes for (int i = 0; i < count; i++) { std::cout << "Score for class [" << i << "] = " << outArr[i] << '\n';

} std::cout << std::flush;

int64_t cls_idx = std::max_element(outArr, outArr + count) - outArr; m_label = labels[cls_idx]; `

std::vectorstd::string labels = { "setosa", "versicolor", "virginica" }; Prediction Label differs only when inferred from C++

Please help me understand why results from these 2 Inferences are not matching.

Expected : Output labels from Python and C++ inferences shall be same

xadupre commented 1 year ago

What about the probabilities? It seems you trained a model for a multiclass classification. What is the model (tree, linear classifier) and how did you convert it? There are known issues when the model is trained with double and the conversion uses float. The label may be different after the probabilites are rounded if they were close to ties. The discrepancies may be even bigger for trees. You will find more at Issues when switching to float.

VPU-tech commented 1 year ago

We have used RandomForest Classifier for predicting label for Iris Data `Code used to convert RF model into ONNX

from skl2onnx import convert_sklearn

from skl2onnx.common.data_types import FloatTensorType

initial_type = [('float_input', FloatTensorType([None, 4]))] clr = RandomForestClassifier() onx = convert_sklearn(clr, initial_types=initial_type)

with open("rf_iris.onnx", "wb") as f:

    f.write(onx.SerializeToString()) `

<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

python_onnx_inference_results | python_onnx_inference_probab | c_onnx_inference_results | C_onnx_inference_probab -- | -- | -- | -- 2 | {0: 0.0, 1: 0.17000000178813934, 2: 0.8299995064735413} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.829999506] 2 | {0: 0.0, 1: 0.0, 2: 0.9999993443489075} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.999999344] 2 | {0: 0.0, 1: 0.06999999284744263, 2: 0.9299994111061096} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.929999411] 1 | {0: 0.0, 1: 0.9999993443489075, 2: 0.0} | 0 | [0: 0.0,  1: 0.000000000,  2: 0.000000000] 0 | {0: 0.9999993443489075, 1: 0.0, 2: 0.0} | 0 | [0: 0.0,  1: 0.000000000,  2: 0.000000000] 2 | {0: 0.0, 1: 0.0, 2: 0.9999993443489075} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.999999344] 2 | {0: 0.0, 1: 0.12999998033046722, 2: 0.8699994683265686} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.869999468] 0 | {0: 0.9999993443489075, 1: 0.0, 2: 0.0} | 0 | [0: 0.0,  1: 0.000000000,  2: 0.000000000] 2 | {0: 0.0, 1: 0.0, 2: 0.9999993443489075} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.999999344] 2 | {0: 0.0, 1: 0.0, 2: 0.9999993443489075} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.999999344] 0 | {0: 0.9999993443489075, 1: 0.0, 2: 0.0} | 0 | [0: 0.0,  1: 0.000000000,  2: 0.000000000] 1 | {0: 0.0, 1: 0.9799993634223938, 2: 0.019999999552965164} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.020000000] 2 | {0: 0.0, 1: 0.17000000178813934, 2: 0.8299995064735413} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.829999506] 0 | {0: 0.9999993443489075, 1: 0.0, 2: 0.0} | 0 | [0: 0.0,  1: 0.000000000,  2: 0.000000000] 1 | {0: 0.0, 1: 0.9899993538856506, 2: 0.009999999776482582} | 2 | [0: 0.0,  1: 0.000000000,  2: 0.010000000] 0 | {0: 0.9999993443489075, 1: 0.0, 2: 0.0} | 0 | [0: 0.0,  1: 0.000000000,  2: 0.000000000] 0 | {0: 0.9999993443489075, 1: 0.0, 2: 0.0} | 0 | [0: 0.0,  1: 0.000000000,  2: 0.000000000] 0 | {0: 0.9999993443489075, 1: 0.0, 2: 0.0} | 0 | [0: 0.0,  1: 0.000000000,  2: 0.000000000]

VPU-tech commented 1 year ago

Iris Data and Reference Code available here: https://github.com/microsoft/onnxruntime/files/11275159/Iris_ONNX_Inference_Sample.zip

It contains:

  1. Code for ONNX inference in C++
  2. Code for ONNX inference in Python
  3. ONNX created for Iris data using Random Forest classifier.
  4. Pickle file for Random Forest classifier.
  5. Iris test data
xadupre commented 1 year ago

I tried your script and it produces the following. It seems to work for me. Which version of skl2onnx, onnx, onnxruntime do you use?

Actual_val,Model_predict_results,python_onnx_inference_results,python_pkl_results
2,2,2,2
0,0,0,0
0,0,0,0
1,1,1,1
2,2,2,2
1,1,1,1
0,0,0,0
2,2,2,2
...
VPU-tech commented 1 year ago

I tried your script and it produces the following. It seems to work for me. Which version of skl2onnx, onnx, onnxruntime do you use?

Actual_val,Model_predict_results,python_onnx_inference_results,python_pkl_results
2,2,2,2
0,0,0,0
0,0,0,0
1,1,1,1
2,2,2,2
1,1,1,1
0,0,0,0
2,2,2,2
...

While inferring in C++, we are using Microsoft.ML.OnnxRuntime.1.13.1 and in python skl2onnx 1.14.0 version is being used. We are also getting same predictions for above categories, the problem is, it's not matching with 'C_onnx_inference_results'

VPU-tech commented 1 year ago

Even after using same 1.14.0 version at both sides (skl2onnx and Microsoft.ML.OnnxRuntime), there is difference in prediction results and Prediction probabilities.

VPU-tech commented 1 year ago

@yf711, I have provided required information for the issue reproduction. Can we please change the issue state to something else? @xadupre, can you please help us to identify the root cause of the issue? As per the shared probability data, Probabilities are always zero for first 2 categories / labels. Is any correction required in C++ code?

Neuroosapiens commented 4 months ago

I know this is very late. I have been working recently on this.

When you created model you need to see what is the data type for output labels. For IRIS it is usually "int".

// Get the inference result float* outArr = output_tensors.front().GetTensorMutableData() assert(abs(outArr[0] - 0.000045) < 1e-6);

change it to

// Get the inference result int* outArr = output_tensors.front().GetTensorMutableData();

The above output will give you label index. That should solve issue.

cheers