Freeing tensor data created via CreateTensor

[vymao]

Describe the issue

I am wondering whether or not we need to manually free the data created via CreateTensor. This API takes in a pointer to our underlying tensor data, but I'm not sure if the pointer is managed by Ort::Value itself (and hence will be destructed/deallocated once the Ort::Value goes out of scope), or if we need to still free the data ourselves.

To reproduce

For example, if we have the following:

float* new_arr = new float[1];
*new_arr[0] = 1;
Ort::MemoryInfo mem_info =
      Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemType::OrtMemTypeDefault);
  auto tensor = Ort::Value::CreateTensor<T>(mem_info, new_arr, ...);

Ignoring the possibility of a dangling pointer via new_arr, do we have to manually free new_arr, or will auto_tensor free the data once it goes out of scope?

Urgency

No response

Platform

Mac

OS Version

13.5

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.16.2

ONNX Runtime API

C++

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[laxnpander]

According to C documentation:

Create a tensor backed by a user supplied buffer
Create a tensor with user's buffer. You can fill the buffer either before calling this function or after. p_data is owned by caller. ReleaseValue won't release p_data.

ORT_API2_STATUS(CreateTensorWithDataAsOrtValue, _In_ const OrtMemoryInfo* info, _Inout_ void* p_data,
                size_t p_data_len, _In_ const int64_t* shape, size_t shape_len, ONNXTensorElementDataType type,
                _Outptr_ OrtValue** out);

Correct me if I am wrong, but this sounds to me like it will NOT be released and user has to take care of it. There could be others function signatures which may take ownership though. Not 100% sure.

[HectorSVC]

That's correct @laxnpander @vymao, user owns the raw buffers they created. Here's some example code for your reference. https://github.com/microsoft/onnxruntime-inference-examples/blob/main/c_cxx/OpenVINO_EP/Windows/squeezenet_classification/squeezenet_cpp_app.cpp

[yuslepukhin]

Ort::Value class acts as a smart pointer for the underyling C structures. Exception safety is the purpose of the C++ API, so it will automatically release the C API structure it OWNS.

https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_cxx_api.h#L10

[manickavela29]

Hi @HectorSVC @yuslepukhin

void m1() {
    std::vector<float> new_arr = {}// initialize
    Ort::MemoryInfo mem_info =
        Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemType::OrtMemTypeDefault);
    auto tensor = Ort::Value::CreateTensor<T>(mem_info, new_arr, ...);
    Ort::value output =  RunModel(std::move(tensor);
}

we have a std::vector new_arr, and ort::value object buffer is mapped to this new_arr, if there is a call to another method RunModel with std::move() over the

std::move generally moves the reference out of its scope so std::move is only moving the ort::value (and its info about tensor) but not the actual data in the buffer(created by std::vector), it only holds the pointer to memory.

but when m1 completes , vector goes out of scope and memory buffer will also be deallocated, am i right?