search

openvinotoolkit / openvino

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
https://docs.openvino.ai
Apache License 2.0
6.34k stars 2.09k forks source link

[Good First Issue]: [OP CONFORMANCE][TEMPLATE] Fix Reshape Inference test in op conformance #23546

Open iefode opened 3 months ago

iefode commented 3 months ago

Context

OP conformance suite is validation tool checks a plugin conditions from operation implementation status perspective. OP conformance is based on extracted from OMZ model scope operations and graphs by ov_subgraphs_dumper tool. Extracted graphs are saved as IR (OpenVINO Intermediate representation) and is stored in the public share. OP conformance suite contains the following test types:

Template is a simple plugin run inference using reference implementation. It means if we will run conformance over template, we should compare TEMPLATE vs TEMPLATE results. Sometimes we have bugs inside the plugin, reference implementation or test infrastructure leads to negative test result.

What needs to be done?

Description: We have several Reshape tests failed in operation conformance over TEMPLATE plugin. Do contributor should fix them:

How to reproduce:

  1. Build the OV using -DENABLE_TESTS=ON -DENABLE_FUNCTIONAL_TESTS=ON
  2. Build ov_op_conformance_tests target
  3. Run OP conformance to download conformance IRs and run executable file with args: 
    python3 /openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/run_conformance.py -d=TEMPLATE --gtest_filter="conformance_Gather/ReadIRTest.Inference/Op=Gather.8_Type=f32_Shape=dynamic_IR=1c727cc96123227a9fe6c3079a497fd64a04f273bff45b5ea56a3c0d577eca8e_Device=TEMPLATE_Config=()"

    NOTE: run_conformance log contains all commands to run ov_op_conformance_tests! You can take them to debug!

  4. Check the logs inside working directory and get the results as an example: 
    
[ RUN      ] conformance_Reshape/ReadIRTest.Inference/Op=Reshape.1_Type=i64_Shape=dynamic_IR=c5ff38504273a230addadadf4fef517ef73154c5f9f10ef2ace961b1dc3cb794_Device=TEMPLATE_Config=()

MEM_USAGE=71872KB [ CONFORMANCE ] Influence coefficient: 0.000946854 [ PLUGIN ] SubgraphBaseTest::compile_model() is started [ PLUGIN ] SubgraphBaseTest::compile_model() is finished successfully. Duration is 0.0796472s [ PLUGIN ] SubgraphBaseTest::get_plugin_outputs() is started /home/efode/repo/openvino/src/tests/functional/shared_test_classes/src/base/ov_subgraph.cpp:94: Failure Exception from src/inference/src/cpp/infer_request.cpp:223: Check 'backward_compatible_check || in_out_elements_equal' failed at src/core/shape_inference/include/reshape_shape_inference.hpp:354: While validating node 'opset1::Reshape Reshape-1 (opset1::Parameter Reshape-1_0[0]:i64[1,1], opset1::Parameter Reshape-1_1[0]:i64[2]) -> (i64[?,?])' with friendly_name 'Reshape-1': Shape inference input shapes {[1,1],[2]} Requested output shape [5,4] is incompatible with input shape

[ FAILED ] conformance_Reshape/ReadIRTest.Inference/Op=Reshape.1_Type=i64_Shape=dynamic_IR=c5ff38504273a230addadadf4fef517ef73154c5f9f10ef2ace961b1dc3cb794_Device=TEMPLATE_Config=(), where GetParam() = ("/home/efode/repo/openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/2023.3.0-13657-d5b0f4d2d73/operation/dynamic/Reshape-1/i64/c5ff38504273a230addadadf4fef517ef73154c5f9f10ef2ace961b1dc3cb794.xml", "") (340 ms)

5. To debug the test in C++ Use the following command (just an example, refer to note in third item):

openvino/bin/intel64/Release/ov_op_conformance_tests --device=TEMPLATE --input_folders=openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/conformance_ir_files.lst, --report_unique_name --output_folder="openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/report/parallel" --gtest_filter="conformance_Reshape/ReadIRTest.Inference/Op=Reshape.1_Type=i64_Shape=dynamic_IR=c5ff38504273a230addadadf4fef517ef73154c5f9f10ef2ace961b1dc3cb794_Device=TEMPLATE_Config=()" --config_path="" --shape_mode=



>Important NOTE: The issue can be connected with [input generation for Gather Operation](src/tests/functional/shared_test_classes/src/base/utils/generate_inputs.cpp). We should generate input tensor according OV specification!

### Example Pull Requests

As an example of extending input generation per specific layer: https://github.com/openvinotoolkit/openvino/pull/20897/

### Resources

- [Contribution guide - start here!](https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING.md)
- [Intel DevHub Discord channel](https://discord.gg/7pVRxUwdWG) - engage in discussions, ask questions and talk to OpenVINO developers
- [How to link your Pull Request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#manually-linking-a-pull-request-to-an-issue-using-the-pull-request-sidebar)
- [Conformance readme](https://github.com/openvinotoolkit/openvino/tree/master/src/tests/functional/plugin/conformance/test_runner)
- [TEMPLATE plugin](https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/template/README.md)
- [OV CONTRIBUTION GUIDE](https://github.com/openvinotoolkit/openvino/tree/master/?tab=readme-ov-file#how-to-contribute)

### Contact points

@iefode

### Ticket

_No response_
22ema commented 3 months ago

.take

github-actions[bot] commented 3 months ago

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

22ema commented 3 months ago

Current progress

MEM_USAGE=44600KB [ CONFORMANCE ] Influence coefficient: 0.000946854 [ PLUGIN ] SubgraphBaseTest::compile_model() is started [ PLUGIN ] SubgraphBaseTest::compile_model() is finished successfully. Duration is 0.00809249s [ PLUGIN ] SubgraphBaseTest::get_plugin_outputs() is started [ REFERENCE ] SubgraphBaseTest::calculate_refs() is started [ REFERENCE ] Calculate reference in runtime [ REFERENCE ] SubgraphBaseTest::calculate_refs() is started src/tests/functional/shared_test_classes/src/base/ov_subgraph.cpp:94: Failure Exception from src/inference/src/cpp/infer_request.cpp:223: Check 'backward_compatible_check || in_out_elements_equal' failed at src/core/shape_inference/include/reshape_shape_inference.hpp:354: While validating node 'opset1::Reshape Reshape-1 (opset1::Parameter Reshape-1_0[0]:i64[1,1], opset1::Parameter Reshape-1_1[0]:i64[2]) -> (i64[?,?])' with friendly_name 'Reshape-1': Shape inference input shapes {[1,1],[2]} Requested output shape [5,4] is incompatible with input shape

[ FAILED ] conformance_Reshape/ReadIRTest.Inference/Op=Reshape.1_Type=i64_Shape=dynamic_IR=c5ff38504273a230addadadf4fef517ef73154c5f9f10ef2ace961b1dc3cb794_Device=TEMPLATE_Config=(), where GetParam() = ("/home/Projects/yunsunghan/openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/2023.3.0-13657-d5b0f4d2d73/operation/dynamic/Reshape-1/i64/c5ff38504273a230addadadf4fef517ef73154c5f9f10ef2ace961b1dc3cb794.xml", "") (63 ms)

22ema commented 3 months ago

@iefode

It appears that a problem occurs when -1 is written in dimension.

In the above problem, if the dimension is written as -1, how can it be reshaped? Do I need to enter the appropriate number?

ex.) input { [-1, -1], [4] } output {[-1, -1, -1, -1]} With the current implementation, the output of the example above is as follows: input shape {[1, 1], [4]) output {[5, 4, 1, 4]} The problem appears to occur because the above input shape and output shape are different. So it seems that the resolve_minus_one_dim() function in the reshape_shape_inference.hpp file needs to be modified.

Please review the direction of progress.

iefode commented 3 months ago

@22ema

Thanks for taking the issue!

Please, check the Openvino Reshape operator specification to get more details for the operator. https://docs.openvino.ai/2022.3/openvino_docs_ops_shape_Reshape_1.html

Moreover, -1 in the input dims can be interpreted as dynamic.

iefode commented 2 months ago

Hello @22ema!

What is the status of the issue? Please, let us know in case of any issues

We will unassign the ticket from you in case of inaction