[Bug]: PrePostProcessor.postprocess().convert_color()

hzk7287 commented 2 months ago

OpenVINO Version

2024.3.0.16041-1e3b88e4e3f-releases/2024/3

Operating System

Windows System

Device used for inference

GPU

Framework

None

Model used

https://drive.google.com/drive/folders/1Na9z_qT3WC0gBUERhUK7d3EmDLGet-XB?usp=drive_link

Issue description

My model is a video interpolation model. The input of the model is two images. My code is below:

ov::preprocess::PrePostProcessor ppp = ov::preprocess::PrePostProcessor::PrePostProcessor(model); //hzk

ppp.input(0).tensor().set_spatial_static_shape(1080, 1920);

ppp.input(0).tensor().set_element_type(ov::element::u8);
ppp.input(0).tensor().set_color_format(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE);

ppp.input(1).tensor().set_spatial_static_shape(1080, 1920);

ppp.input(1).tensor().set_element_type(ov::element::u8);

ppp.input(1).tensor().set_color_format(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE);

ppp.input(0).preprocess() //hzk
    .convert_color(ov::preprocess::ColorFormat::RGB)
    .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
    .convert_element_type(ov::element::f32)
    .scale({ 1.0f / 255.0f,1.0f / 255.0f,1.0f / 255.0f });

ppp.input(1).preprocess() //hzk
    .convert_color(ov::preprocess::ColorFormat::RGB)
    .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
    .convert_element_type(ov::element::f32)
    .scale({ 1.0f / 255.0f,1.0f / 255.0f,1.0f / 255.0f });
ppp.input(0).tensor().set_layout("NHWC");

ppp.input(1).tensor().set_layout("NHWC");

ppp.input(0).model().set_layout("NCHW");

ppp.input(1).model().set_layout("NCHW");

std::cout << "Preprocessor: " << ppp << std::endl; setmodel = ppp.build();

It is OK now. But there is a error when I continue using the following code

ov::preprocess::PrePostProcessor pppimage = ov::preprocess::PrePostProcessor::PrePostProcessor(setmodel); //hzk

// pppimage.output(0).tensor().set_element_type(ov::element::f32); // pppimage.output(0).tensor().set_layout("NCHW"); // ov::preprocess::OutputModelInfo& OutputModel = pppimage.output(0).model(); // OutputModel.set_layout("NHWC"); // OutputModel.set_color_format(ov::preprocess::ColorFormat::RGB);

pppimage.output().tensor().set_layout("NCHW");
pppimage.output().model().set_layout("NCHW");
pppimage.output().postprocess()
   .convert_layout("NHWC")
    .convert_element_type(ov::element::u8)
    .convert_color(ColorFormat::NV12_SINGLE_PLANE); // wrong !
std::cout << "Postprocessor: " << pppimage << std::endl;
model = pppimage.build();

If I change NV12_SINGLE_PLANE to BGR, I still get a error.

Thank you !

Step-by-step reproduction

No response

Relevant log output

No response

Issue submission checklist

[X] I'm reporting an issue. It's not a question.
[X] I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
[X] There is reproducer code and related data files such as images, videos, models, etc.

ilya-lavrenov commented 2 months ago

Hi @hzk7287 What error do you observe?

hzk7287 commented 2 months ago

[ INFO ] Loading model files: video_interprolation.xml Model name :main_graph Number of inputs :2 Number of outputs :1 input_tensor_0 name : input input_tensor_0 shape : [1,3,1080,1920] input_tensor_1 name : onnx::Concat_1 input_tensor_1 shape : [1,3,1080,1920] input_tensor_1 name : output input_tensor_1 shape : [1,3,1080,1920] Preprocessor: Input "input" (color NV12 (single plane)): User's input tensor: [1,1620,1920,1], [N,H,W,C], u8 Model's expected tensor: [1,3,1080,1920], [N,C,H,W], f32 Pre-processing steps (4): convert color (RGB): ([1,1620,1920,1], [N,H,W,C], u8, NV12 (single plane)) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB) resize to model width/height: ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB) convert type (f32): ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB) scale (0.00392157,0.00392157,0.00392157): ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB) Implicit pre-processing steps (1): convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB) Input "onnx::Concat_1" (color NV12 (single plane)): User's input tensor: [1,1620,1920,1], [N,H,W,C], u8 Model's expected tensor: [1,3,1080,1920], [N,C,H,W], f32 Pre-processing steps (4): convert color (RGB): ([1,1620,1920,1], [N,H,W,C], u8, NV12 (single plane)) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB) resize to model width/height: ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], u8, RGB) convert type (f32): ([1,1080,1920,3], [N,H,W,C], u8, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB) scale (0.00392157,0.00392157,0.00392157): ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,1080,1920,3], [N,H,W,C], f32, RGB) Implicit pre-processing steps (1): convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB)

Postprocessor: Output "output": Model's data tensor: [1,3,1080,1920], [N,C,H,W], f32 Post-processing steps (3): convert layout [N,H,W,C]: ([1,3,1080,1920], [N,C,H,W], f32) -> ([1,1080,1920,3], [N,H,W,C], f32) convert type (u8): ([1,1080,1920,3], [N,H,W,C], f32) -> ([1,1080,1920,3], [N,H,W,C], u8) convert color (BGR): ([1,1080,1920,3], [N,H,W,C], u8

After I add ".convert_color(ColorFormat::NV12_SINGLE_PLANE) or .convert_color(ColorFormat:BGR). I get the following error. Unhandled exception at 0x00007FF88B16FABC Microsoft C++ exception: ov::Exception at memory location 0x000000AA5AD4C788. 0x00007FF88B16FABC and 0x000000AA5AD4C788 ? what's wrong?

By the way, after I delete the convert_color function, it is OK ,but I do not want an image using RGB format, while I want an image using NV12 format. I noticed that in " convert layout [N,C,H,W]: ([1,1080,1920,3], [N,H,W,C], f32, RGB) -> ([1,3,1080,1920], [N,C,H,W], f32, RGB)" ,there is a RGB string. But "Post-processing steps (3): convert layout [N,H,W,C]: ([1,3,1080,1920], [N,C,H,W], f32) -> ([1,1080,1920,3], [N,H,W,C], f32)" I do not see any RGB or NV12 (single plane) string I add try catch in this code. I get Source color format 'Unknown' is not convertible to 'NV12 (single plane)

Thank you very much !

praasz commented 3 hours ago

@hzk7287 I've tried to reproduce your issue on 2024.3 version and recent master branch, but I cannot reproduce the issue. Here is my code sample (base on your input), maybe I missed something.

#include "openvino/openvino.hpp"

#include <stdexcept>
#include <iostream>
#include <string>

int main() {
    ov::Core core;
    auto model = core.read_model("video.xml");

    auto ppp = ov::preprocess::PrePostProcessor::PrePostProcessor(model); //hzk
    ppp.input(0).tensor().set_spatial_static_shape(1080, 1920);

    ppp.input(0).tensor().set_element_type(ov::element::u8);
    ppp.input(0).tensor().set_color_format(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE);
    ppp.input(1).tensor().set_spatial_static_shape(1080, 1920);
    ppp.input(1).tensor().set_element_type(ov::element::u8);
    ppp.input(1).tensor().set_color_format(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE);

    ppp.input(0).preprocess()
        .convert_color(ov::preprocess::ColorFormat::RGB)
        .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
        .convert_element_type(ov::element::f32)
        .scale({ 1.0f / 255.0f,1.0f / 255.0f,1.0f / 255.0f });

    ppp.input(1).preprocess()
        .convert_color(ov::preprocess::ColorFormat::RGB)
        .resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR)
        .convert_element_type(ov::element::f32)
        .scale({ 1.0f / 255.0f,1.0f / 255.0f,1.0f / 255.0f });
    ppp.input(0).tensor().set_layout("NHWC");
    ppp.input(1).tensor().set_layout("NHWC");
    ppp.input(0).model().set_layout("NCHW");
    ppp.input(1).model().set_layout("NCHW");

    std::cout << "Preprocessor: " << ppp << std::endl;
    auto set_model = ppp.build();
    auto ppp_image = ov::preprocess::PrePostProcessor::PrePostProcessor(set_model); //hzk

    ppp_image.output(0).tensor().set_element_type(ov::element::f32);
    ppp_image.output(0).tensor().set_layout("NCHW");
    auto& OutputModel = ppp_image.output(0).model();
    OutputModel.set_layout("NHWC");
    OutputModel.set_color_format(ov::preprocess::ColorFormat::RGB);

    ppp_image.output().tensor().set_layout("NCHW");
    ppp_image.output().model().set_layout("NCHW");
    ppp_image.output().postprocess()
      .convert_layout("NHWC")
      .convert_element_type(ov::element::u8)
      .convert_color(ov::preprocess::ColorFormat::NV12_SINGLE_PLANE); // wrong !

    std::cout << "Postprocessor: " << ppp_image << std::endl;
    model = ppp_image.build();

    auto compiled_model = core.compile_model(model, "GPU");

    return 0;
}

openvinotoolkit / openvino