ONNXRuntime. Access violation error if you run detection twice in a row.

[AlienRouge]

Hello! I'm trying to run image detection twice (for one Yolov8Onnx object), but the second time I get an error: Exception thrown at 0x00007FFFE4CD1CC6 (onnxruntime.dll) in yolov8-opencv-onnxruntime-cpp.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF. What could be the problem?

[UNeedCryDear]

Can you confirm if the names of your input/output nodenames has changed during your second run?

Or, what method did you use to run the second time? Please show me code you modified yourself.

[AlienRouge]

Sorry, this is my mistake. I just started working with ORT and sometimes I get confused. Above, I tried to run the FP16 model with FP32 tensors, which eventually led to an error. I am currently trying to adapt the postprocessing stage for the FP16 tensor, but after converting Ort::Float16_t to float, I encountered a lack of detections... Can you help with that?

Creating an input tensor:

blob = cv::dnn::blobFromImage(borderImg, 1 / 255.0, input_size, cv::Scalar(0, 0, 0), true, false);

// FP16
int64_t input_tensor_length = VectorProduct(_inputTensorShape);
std::vector<Ort::Value> input_tensors;
input_tensors.push_back(Ort::Value::CreateTensor(ortMemoryInfo, blob.data, input_tensor_length * 2,
                                                     _inputTensorShape.data(), _inputTensorShape.size(),
                                                     ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16));

Transformation of the output tensor:

Ort::Float16_t* all_data = outputTensors[0].GetTensorMutableData<Ort::Float16_t>();
_outputTensorShape = outputTensors[0].GetTensorTypeAndShapeInfo().GetShape();

auto type_info = outputTensors[0].GetTypeInfo();
auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
const auto element_count = tensor_info.GetElementCount();

std::vector<float> output_values;
output_values.reserve(element_count);
std::span fp32_data(all_data, element_count);
std::ranges::transform(fp32_data, std::back_inserter(output_values), [](const Ort::Float16_t& fp16)
{
    return static_cast<float>(fp16);
});

[UNeedCryDear]

In CPP, only float type (FP32) is available.

You should choose "keep_io_types=True" when exporting onnx, keeping inputs/outputs datatype is FP32 and FP16 for other tensors. Otherwise, you will need to deal with the datatype with fp16 bit by bit yourself.

[AlienRouge]

Yes, I have already launched FP16 inference, but I did not see any acceleration. It would be better if the inputs and outputs remained in FP32, but the total size of the model on disk remained as FP16. If I export the Yolov8s model from ".pt" to ".onnx" via the native ultralyrics exporter, how can I use the "keep_io_types" parameter?

[UNeedCryDear]

try code:(maybe be you should pip install onnxconverter_common )

from onnxconverter_common import float16
import onnx
f="./best.onnx"   #FP32
fout="./best_oi_fp32_fp16.onnx"
model_onnx = onnx.load(f)  # load onnx model
onnx.checker.check_model(model_onnx)  # check onnx model
# Metadata
d = {'stride': int(max(model.stride)), 'names': model.names}
for k, v in d.items():
    meta = model_onnx.metadata_props.add()
    meta.key, meta.value = k, str(v)

model_fp16=float16.convert_float_to_float16(model_onnx,keep_io_types=True, disable_shape_infer=True)
onnx.save(model_fp16, fout)

[AlienRouge]

I commented out the loop on d.items() because d is undefined and the model was exported with some warnings like that: UserWarning: the float32 number -1.8842254145834403e-11 will be truncated to -1e-07 warnings.warn("the float32 number {} will be truncated to {}".format(neg_max, -min_positive_val))

But an error occurs at the ONNX session creation: Load model from C:/Users/rudov/PycharmProjects/yolov8/best_oi_fp32_fp16.onnx failed:This is an invalid model. Type Error: Type 'tensor(float16)' of input parameter (/model.10/Constant_output_0) of operator (Resize) in node (/model.10/Resize) is invalid.

[UNeedCryDear]

try again:

from ultralytics import YOLO
import cv2
from onnxconverter_common import float16
import onnx
model_path="yolov8n-seg.pt"
dynamic_shape=True
model = YOLO(model_path)
results = model.export(format='onnx',dynamic=dynamic_shape,opset=16)

f="yolov8n-seg.onnx"   #FP32
fout="./best_oi_fp32_fp16.onnx"
model_onnx = onnx.load(f)  # load onnx model
onnx.checker.check_model(model_onnx)  # check onnx model
# Metadata
d = { 'names': model.names}
for k, v in d.items():
    meta = model_onnx.metadata_props.add()
    meta.key, meta.value = k, str(v)

model_fp16=float16.convert_float_to_float16(model_onnx,keep_io_types=True, disable_shape_infer=dynamic_shape)
onnx.save(model_fp16, fout)

[AlienRouge]

Thanks, this code caused the same loading error, but setting disable_shape_infer=False solved it.

[UNeedCryDear]

I forgot that this parameter is suitable for dynamic export. Because I have set "dynamic=True" in my default.yaml. I have made changes to the code. Thank you for your reminder.