Different outputs in Python and C++

[yoyococo700]

Describe the issue

I have trained a model (1 inputs node (1x12), 3 outputs but i'm only interessed by the first one) and it behave well when I'm using a python inference. When I switched to C++, for the same input I got another output, very far from the expected python output. Do you have any ideas of what can cause such behavior?

To reproduce

Here is the python code used for testing:

onnx_path = "6DDL.onnx"
onnx_model = onnx.load(onnx_path)
onnx.checker.check_model(onnx_model)
ort_sess = ort.InferenceSession(onnx_path)
print("Input shape:", ort_sess.get_inputs()[0].shape)
#Input shape: [1, 12]
print("Input shape:", ort_sess.get_inputs()[0].type)
#Input shape: tensor(float)
input = np.array([[0.3, 0, -0.5, 0.707107, 0.707107, 0, 0, -0.0375, 0, 0, -0.2, -0.124991]],dtype=np.float32)
action = ort_sess.run(None, {"input": input})
print(action)
#[array([[-0.0197425]], dtype=float32), array([[21.29765]], dtype=float32), array([4.232593], dtype=float32)]

Here is the C++ equivalent, it come from the "model-explorer" sample from the sample repository

#include <algorithm>  // std::generate
#include <cassert>
#include <cstddef>
#include <cstdint>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>
#include "onnxruntime_cxx_api.h"

// pretty prints a shape dimension vector
std::string print_shape(const std::vector<std::int64_t>& v) {
  std::stringstream ss("");
  for (std::size_t i = 0; i < v.size() - 1; i++) ss << v[i] << "x";
  ss << v[v.size() - 1];
  return ss.str();
}

int calculate_product(const std::vector<std::int64_t>& v) {
  int total = 1;
  for (auto& i : v) total *= i;
  return total;
}

template <typename T>
Ort::Value vec_to_tensor(std::vector<T>& data, const std::vector<std::int64_t>& shape) {
  Ort::MemoryInfo mem_info =
      Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemType::OrtMemTypeDefault);
  auto tensor = Ort::Value::CreateTensor<T>(mem_info, data.data(), data.size(), shape.data(), shape.size());
  return tensor;
}

#ifdef _WIN32
int wmain(int argc, ORTCHAR_T* argv[]) {
#else
int main(int argc, ORTCHAR_T* argv[]) {
#endif
  if (argc != 2) {
    std::cout << "Usage: ./onnx-api-example <onnx_model.onnx>" << std::endl;
    return -1;
  }

  std::basic_string<ORTCHAR_T> model_file = argv[1];

  // onnxruntime setup
  Ort::Env env(ORT_LOGGING_LEVEL_VERBOSE, "example-model-explorer");
  Ort::SessionOptions session_options;
  Ort::Session session = Ort::Session(env, model_file.c_str(), session_options);

  // print name/shape of inputs
  Ort::AllocatorWithDefaultOptions allocator;
  std::vector<std::string> input_names;
  std::vector<std::int64_t> input_shapes;
  std::cout << "Input Node Name/Shape (" << input_names.size() << "):" << std::endl;
  for (std::size_t i = 0; i < session.GetInputCount(); i++) {
    input_names.emplace_back(session.GetInputNameAllocated(i, allocator).get());
    input_shapes = session.GetInputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();
    std::cout << "\t" << input_names.at(i) << " : " << print_shape(input_shapes) << std::endl;
  }
  // some models might have negative shape values to indicate dynamic shape, e.g., for variable batch size.
  for (auto& s : input_shapes) {
    if (s < 0) {
      s = 1;
    }
  }

  // print name/shape of outputs
  std::vector<std::string> output_names;
  std::cout << "Output Node Name/Shape (" << output_names.size() << "):" << std::endl;
  for (std::size_t i = 0; i < session.GetOutputCount(); i++) {
    output_names.emplace_back(session.GetOutputNameAllocated(i, allocator).get());
    auto output_shapes = session.GetOutputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();
    std::cout << "\t" << output_names.at(i) << " : " << print_shape(output_shapes) << std::endl;
  }

  // Create a single Ort tensor of random numbers
  auto input_shape = input_shapes;
  auto total_number_elements = calculate_product(input_shape);

  std::vector<float> input_tensor_values(total_number_elements);

  input_tensor_values[0] = (float)-0.3;
  input_tensor_values[1] = (float)0.;
  input_tensor_values[2] = (float)-0.5;
  input_tensor_values[3] = (float)0.707107;
  input_tensor_values[4] = (float)0.707107;
  input_tensor_values[5] = (float)0.;
  input_tensor_values[6] = (float)0.;
  input_tensor_values[7] = (float)-0.0375;
  input_tensor_values[8] = (float)0.;
  input_tensor_values[9] = (float)0.;
  input_tensor_values[10] = (float)-0.2;
  input_tensor_values[11] = (float)-0.124991;

  std::vector<Ort::Value> input_tensors;
  input_tensors.emplace_back(vec_to_tensor<float>(input_tensor_values, input_shape));

  for (int i = 0;i<12;i++)
      std::cout << "vec[" << i << "] = " << input_tensors.front().GetTensorMutableData<float>()[i] << std::endl;

  // double-check the dimensions of the input tensor
  assert(input_tensors[0].IsTensor() && input_tensors[0].GetTensorTypeAndShapeInfo().GetShape() == input_shape);
  std::cout << "\ninput_tensor shape: " << print_shape(input_tensors[0].GetTensorTypeAndShapeInfo().GetShape()) << std::endl;

  // pass data through model
  std::vector<const char*> input_names_char(input_names.size(), nullptr);
  std::transform(std::begin(input_names), std::end(input_names), std::begin(input_names_char),
                 [&](const std::string& str) { return str.c_str(); });

  std::vector<const char*> output_names_char(output_names.size(), nullptr);
  std::transform(std::begin(output_names), std::end(output_names), std::begin(output_names_char),
                 [&](const std::string& str) { return str.c_str(); });

  std::cout << "Running model..." << std::endl;
  try {
    auto output_tensors = session.Run(Ort::RunOptions{nullptr}, input_names_char.data(), input_tensors.data(),
                                      input_names_char.size(), output_names_char.data(), output_names_char.size());
    std::cout << "Done!" << std::endl;
    std::cout << "DLC = " << (float)output_tensors.front().GetTensorMutableData<float>()[0] << std::endl;
    // double-check the dimensions of the output tensors
    // NOTE: the number of output tensors is equal to the number of output nodes specifed in the Run() call
    assert(output_tensors.size() == output_names.size() && output_tensors[0].IsTensor());
  } catch (const Ort::Exception& exception) {
      std::cout << "ERROR running model inference: " << exception.what() << std::endl;
    exit(-1);
  }
}

And the output of the C++ code

Input Node Name/Shape (0):
input : 1x12
Output Node Name/Shape (0):
53 : 1x1
24 : 1x1
51 : 1
Total number of elements = 12
vec[0] = -0.3
vec[1] = 0
vec[2] = -0.5
vec[3] = 0.707107
vec[4] = 0.707107
vec[5] = 0
vec[6] = 0
vec[7] = -0.0375
vec[8] = 0
vec[9] = 0
vec[10] = -0.2
vec[11] = -0.124991

input_tensor shape: 1x12
Running model...
Done!
Sortie 0: -0.501966
Sortie 1: -20.8855
Sortie 2: 2.46406

Urgency

No response

Platform

Linux

OS Version

24

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.19.2

ONNX Runtime API

Python

Architecture

X86

Execution Provider

Default CPU

Execution Provider Library Version

No response

[yuslepukhin]

the output would tell you if your shape matches the element count that you are initializing (12). Your input values code is not driven by the shape product.

[yoyococo700]

I may have uncorrectly explained the issue, sorry about that. I have only one input node that takes an 1x12 tensor shape. The C++ code is working without error, every assert is passed. The value of the output node is just very wrong (-0.5 instead of -0.019 on a {-1,1} scale ). I have modified the initial question to include the output of the C++ code.

I have checked that the total_number_elements of the input is equal to 12. I didn't understand what do you mean by

the output would tell you if your shape matches the element count that you are initializing (12)

Thank you in advance for your answer