Confusions when using Ort::Value::CreateMap with value are multidimensional

[codyuan]

Describe the issue

I encountered a question when I use RNN model in my project, it requires a dict input to get the hidden state containing history information. After I export the model in python with the codes provided below, I need to use it to infer in C++.

example =torch.randn(1,22).to(params.device)
example_dict = {
    'h': torch.randn(1,3,128).to(params.device),  
    'c': torch.randn(1,3,128).to(params.device),
}
torch.onnx.export(model, (example,example_dict,{}), "my_model.onnx", verbose=True, input_names=['obs','hidden'],output_names=['logits','hidden'])

It is quite clear to create the first input "example" in C++ using Ort::Value::CreateTensor. However, the situation of the second input is complex, for it is a map. I cannot find any case that works for my situation that the value of the map is multidemensional. Specifically, I always get a segmentation fault either when I use CreateMap or when I create tensor for the multidimensional value.

I wonder are there any case for me to refer because I cannot find any util now. Plus, I wonder if my code is right when a input of model is dict. If you could show me the case that the input of the model is not single and be with multiple type, I will be sincerely appreciate.

The code that could reproduce my problem is provided in "To reproduce".

To reproduce

Ort::MemoryInfo info("Cpu", OrtDeviceAllocator, 0, OrtMemTypeDefault);

const int NUM_KV_PAIRS = 2;
std::vector<Ort::Value> in;
const char* keys_arr[NUM_KV_PAIRS] = {"h", "c"};
std::vector<std::string> keys{keys_arr, keys_arr + NUM_KV_PAIRS};
std::vector<int64_t> keys_dim = {2};
std::vector<float> values(2*3*128,0.f);
std::vector<int64_t> values_dim = {2,3,128};

// create key tensor
Ort::Value keys_tensor = Ort::Value::CreateTensor(info, keys.data(), keys.size() * sizeof(int64_t),keys_dim.data(), keys_dim.size(),ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64);

// create value tensor
Ort::Value values_tensor = Ort::Value::CreateTensor(info, values.data(), values.size() * sizeof(float), values_dim.data(), values_dim.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT);

// create map ort value
Ort::Value ort_map=Ort::Value::CreateMap(keys_tensor, values_tensor);

Urgency

No response

Platform

Linux

OS Version

Ubuntu 22.04.3

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.15.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[yuslepukhin]

Map tensors that represent values or keys are always 1-D. Their dimensions must match. That's for primitive value types. We will address the segmentation fault.

[skottmckay]

Given the key is a string, can you instead have an input for each item in the dictionary so the model would have 3 tensor inputs ('obs', 'h', 'c')?

[codyuan]

Map tensors that represent values or keys are always 1-D. Their dimensions must match. That's for primitive value types. We will address the segmentation fault.

Thanks for your reply. I made it creating map for the multi dimensional data by simply changing values_dim to 1-D. Codes are now:

const int NUM_KV_PAIRS = 2;
std::vector<Ort::Value> in;
const char* keys_arr[NUM_KV_PAIRS] = {"h","c"};
std::vector<std::string> keys{keys_arr, keys_arr + NUM_KV_PAIRS};
std::vector<int64_t> keys_dim = {2};
std::vector<float> value(128,0.f);
std::vector<std::vector<std::vector<float>>> values={{value,value,value},{value,value,value}};
std::vector<int64_t> values_dim = {2};

// create key tensor
Ort::Value keys_tensor = Ort::Value::CreateTensor(memory_info, keys.data(), keys.size() * sizeof(std::string),
                                                  keys_dim.data(), keys_dim.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING);
// create value tensor
Ort::Value values_tensor = Ort::Value::CreateTensor(memory_info, values.data(), values.size() * sizeof(float),
                                                    values_dim.data(), values_dim.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT);
// create map ort value
Ort::Value input_map=Ort::Value::CreateMap(keys_tensor, values_tensor);

However, I met another question for I now set the keys_tensor to be string. As shown in my last case, the type of keys_tensor is set to be int64 due to my careless while it should be string. Therefore I want to correct it. The error message I got is still segmentation faults. It's weird that this case works when keys_tensor is set to be int64 but failed for string.

[skottmckay]

Can you share the onnx model so we can see the inputs it's expecting? AFAIK there are very two ONNX operators that support map input - DictVectorizer and CastMap and I don't believe either of those expect multidimensional values in the map.

[codyuan]

Can you share the onnx model so we can see the inputs it's expecting? AFAIK there are very two ONNX operators that support map input - DictVectorizer and CastMap and I don't believe either of those expect multidimensional values in the map.

Thanks for your reply. We have uploaded the onnx model image below. btw I have made it createing maps using Ort::Value::CreateMap through adjusting the values_dim I set (refer to my last comment), and the new errors occur when the keys_tensor is set to be string. I will learn the two approaches you provided, if there are any operating mistakes of mine to save or load this model in C++, please let me know, thank you.

For the corresponding codes to save the model, please kindly refer to my first submission.

[skottmckay]

The input data needs to be contiguous and a vector of a vector of a vector is not so you're going to get a seg fault by claiming it is. That is all irrelevant though.

The model does not take a map as input. There are 3 inputs:

• 'obs' has shape {1, 22}
• 'hidden.1' has shape {1, 3, 128}
• '2' has shape {1, 3, 128}

I would guess the 'h' in the map equates to 'hidden.1' and the 'c' equates to '2'.

Each input is a single tensor.

And I would guess the output with name 'hidden' is passed back through as 'hidden.1' on the next execution, and '116' is passed back in as '2'.

[codyuan]

The input data needs to be contiguous and a vector of a vector of a vector is not so you're going to get a seg fault by claiming it is. That is all irrelevant though.

The model does not take a map as input. There are 3 inputs:

• 'obs' has shape {1, 22}
• 'hidden.1' has shape {1, 3, 128}
• '2' has shape {1, 3, 128}

I would guess the 'h' in the map equates to 'hidden.1' and the 'c' equates to '2'.

Each input is a single tensor.

And I would guess the output with name 'hidden' is passed back through as 'hidden.1' on the next execution, and '116' is passed back in as '2'.

Thanks, your guess are correct, I now understand my faults. But I still wonder why I saved the onnx model with the dict input in python(check my first submission), but I have to load it with all tensor input in C++.

[skottmckay]

I assume the model exporter would have understood that you can't have that sort of dictionary input to an ONNX model and split things out into separate inputs.