[Bug]: PrePostProcessor.postprocess().convert_color()

openvinotoolkit / openvino

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

Apache License 2.0

6.83k stars 2.18k forks source link

OpenVINO Version

2024.3.0.16041-1e3b88e4e3f-releases/2024/3

Operating System

Windows System

Device used for inference

GPU

Framework

None

Model used

https://drive.google.com/drive/folders/1Na9z_qT3WC0gBUERhUK7d3EmDLGet-XB?usp=drive_link

Issue description

My model is a video interpolation model. The input of the model is two images. My code is below:

ov::preprocess::PrePostProcessor ppp = ov::preprocess::PrePostProcessor::PrePostProcessor(model); //hzk

ppp.input(0).tensor().set_spatial_static_shape(1080, 1920);

ppp.input(0).tensor().set_element_type(ov::element::u8);
ppp.input(0).tensor().set_color_format(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE);

ppp.input(1).tensor().set_spatial_static_shape(1080, 1920);

ppp.input(1).tensor().set_element_type(ov::element::u8);

ppp.input(1).tensor().set_color_format(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE);

ppp.input(0).preprocess() //hzk
    .convert_color(ov::preprocess::ColorFormat::RGB)
    .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
    .convert_element_type(ov::element::f32)
    .scale({ 1.0f / 255.0f,1.0f / 255.0f,1.0f / 255.0f });

ppp.input(1).preprocess() //hzk
    .convert_color(ov::preprocess::ColorFormat::RGB)
    .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
    .convert_element_type(ov::element::f32)
    .scale({ 1.0f / 255.0f,1.0f / 255.0f,1.0f / 255.0f });
ppp.input(0).tensor().set_layout("NHWC");

ppp.input(1).tensor().set_layout("NHWC");

ppp.input(0).model().set_layout("NCHW");

ppp.input(1).model().set_layout("NCHW");

std::cout << "Preprocessor: " << ppp << std::endl; setmodel = ppp.build();

It is OK now. But there is a error when I continue using the following code

ov::preprocess::PrePostProcessor pppimage = ov::preprocess::PrePostProcessor::PrePostProcessor(setmodel); //hzk

// pppimage.output(0).tensor().set_element_type(ov::element::f32); // pppimage.output(0).tensor().set_layout("NCHW"); // ov::preprocess::OutputModelInfo& OutputModel = pppimage.output(0).model(); // OutputModel.set_layout("NHWC"); // OutputModel.set_color_format(ov::preprocess::ColorFormat::RGB);

pppimage.output().tensor().set_layout("NCHW");
pppimage.output().model().set_layout("NCHW");
pppimage.output().postprocess()
   .convert_layout("NHWC")
    .convert_element_type(ov::element::u8)
    .convert_color(ColorFormat::NV12_SINGLE_PLANE); // wrong !
std::cout << "Postprocessor: " << pppimage << std::endl;
model = pppimage.build();

If I change NV12_SINGLE_PLANE to BGR, I still get a error.

Thank you !

Step-by-step reproduction

No response

Relevant log output

No response

Issue submission checklist

[X] I'm reporting an issue. It's not a question.
[X] I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
[X] There is reproducer code and related data files such as images, videos, models, etc.

[ INFO ] Loading model files: video_interprolation.xml Model name :main_graph Number of inputs :2 Number of outputs :1 input_tensor_0 name : input input_tensor_0 shape : [1,3,1080,1920] input_tensor_1 name : onnx::Concat_1 input_tensor_1 shape : [1,3,1080,1920] input_tensor_1 name : output input_tensor_1 shape : [1,3,1080,1920] Preprocessor: Input "input" (color NV12 (single plane)): User's input tensor: [1,1620,1920,1], [N,H,W,C], u8 Model's expected tensor: [1,3,1080,1920], [N,C,H,W], f32 Pre-processing steps (4): convert color (RGB): ([1,1620,1920,1], [N,H,W,C], u8, NV12 (single plane)) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB) resize to model width/height: ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB) convert type (f32): ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB) scale (0.00392157,0.00392157,0.00392157): ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB) Implicit pre-processing steps (1): convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB) Input "onnx::Concat_1" (color NV12 (single plane)): User's input tensor: [1,1620,1920,1], [N,H,W,C], u8 Model's expected tensor: [1,3,1080,1920], [N,C,H,W], f32 Pre-processing steps (4): convert color (RGB): ([1,1620,1920,1], [N,H,W,C], u8, NV12 (single plane)) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB) resize to model width/height: ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB) convert type (f32): ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB) scale (0.00392157,0.00392157,0.00392157): ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB) Implicit pre-processing steps (1): convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB)

Postprocessor: Output "output": Model's data tensor: [1,3,1080,1920], [N,C,H,W], f32 Post-processing steps (3): convert layout [N,H,W,C]: ([1,3,1080,1920], [N,C,H,W], f32) -> ([1,1080,1920,3], [N,H,W,C], f32) convert type (u8): ([1,1080,1920,3], [N,H,W,C], f32) -> ([1,1080,1920,3], [N,H,W,C], u8) convert color (BGR): ([1,1080,1920,3], [N,H,W,C], u8

After I add ".convert_color(ColorFormat::NV12_SINGLE_PLANE) or .convert_color(ColorFormat:BGR). I get the following error. Unhandled exception at 0x00007FF88B16FABC Microsoft C++ exception: ov::Exception at memory location 0x000000AA5AD4C788. 0x00007FF88B16FABC and 0x000000AA5AD4C788 ? what's wrong?

By the way, after I delete the convert_color function, it is OK ,but I do not want an image using RGB format, while I want an image using NV12 format. I noticed that in " convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB)" ,there is a RGB string. But "Post-processing steps (3): convert layout [N,H,W,C]: ([1,3,1080,1920], [N,C,H,W], f32) -> ([1,1080,1920,3], [N,H,W,C], f32)" I do not see any RGB or NV12 (single plane) string I add try catch in this code. I get Source color format 'Unknown' is not convertible to 'NV12 (single plane)

Thank you very much !

openvinotoolkit / openvino