search

openvinotoolkit / openvino

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
https://docs.openvino.ai
Apache License 2.0
7.28k stars 2.27k forks source link

[Good First Issue]: [OP CONFORMANCE][TEMPLATE] Add test in op conformance fails #23556

Open iefode opened 8 months ago

iefode commented 8 months ago

Context

OP conformance suite is validation tool checks a plugin conditions from operation implementation status perspective. OP conformance is based on extracted from OMZ model scope operations and graphs by ov_subgraphs_dumper tool. Extracted graphs are saved as IR (OpenVINO Intermediate representation) and is stored in the public share. OP conformance suite contains the following test types:

Template is a simple plugin run inference using reference implementation. It means if we will run conformance over template, we should compare TEMPLATE vs TEMPLATE results. Sometimes we have bugs inside the plugin, reference implementation or test infrastructure leads to negative test result.

What needs to be done?

Description: We have several Reshape tests failed in operation conformance over TEMPLATE plugin. Do contributor should fix them:

How to reproduce:

  1. Build the OV using -DENABLE_TESTS=ON -DENABLE_FUNCTIONAL_TESTS=ON
  2. Build ov_op_conformance_tests target
  3. Run OP conformance to download conformance IRs and run executable file with args: 
    python3 /openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/run_conformance.py -d=TEMPLATE --gtest_filter="conformance_Gather/ReadIRTest.Inference/Op=Gather.8_Type=f32_Shape=dynamic_IR=1c727cc96123227a9fe6c3079a497fd64a04f273bff45b5ea56a3c0d577eca8e_Device=TEMPLATE_Config=()"

    NOTE: run_conformance log contains all commands to run ov_op_conformance_tests! You can take them to debug!

  4. Check the logs inside working directory and get the results as an example: 
    
[ RUN      ] conformance_Add/ReadIRTest.Inference/Op=Add.1_Type=f32_Shape=dynamic_IR=f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815_Device=TEMPLATE_Config=()

MEM_USAGE=71868KB [ CONFORMANCE ] Influence coefficient: 2.89071e-06 [ PLUGIN ] SubgraphBaseTest::compile_model() is started [ PLUGIN ] SubgraphBaseTest::compile_model() is finished successfully. Duration is 0.106113s [ PLUGIN ] SubgraphBaseTest::get_plugin_outputs() is started [ PLUGIN ] SubgraphBaseTest::get_plugin_outputs() is finished successfully. Duration is 0.00207637s [ REFERENCE ] SubgraphBaseTest::calculate_refs() is started [ REFERENCE ] Calculate reference in runtime [ REFERENCE ] SubgraphBaseTest::calculate_refs() is started [ REFERENCE ] SubgraphBaseTest::calculate_refs() is finished successfully. Duration is 0.00844726s [ REFERENCE ] SubgraphBaseTest::calculate_refs() is finished successfully. Duration is 0.0104847s [ COMPARATION ] ov_tensor_utils.hpp::compare() is started [ COMPARATION ] rel_threshold: 1.79769e+308 [ COMPARATION ] abs_threshold: 0.396962 [ COMPARATION ] ov_tensor_utils.hpp::compare() is finished successfully. Duration is 0.00496672s [ PLUGIN ] SubgraphBaseTest::get_plugin_outputs() is started [ PLUGIN ] SubgraphBaseTest::get_plugin_outputs() is finished successfully. Duration is 10.222s [ REFERENCE ] SubgraphBaseTest::calculate_refs() is started [ REFERENCE ] Calculate reference in runtime [ REFERENCE ] SubgraphBaseTest::calculate_refs() is started [ REFERENCE ] SubgraphBaseTest::calculate_refs() is finished successfully. Duration is 12.747s [ REFERENCE ] SubgraphBaseTest::calculate_refs() is finished successfully. Duration is 12.748s [ COMPARATION ] ov_tensor_utils.hpp::compare() is started [ COMPARATION ] rel_threshold: 1.79769e+308 [ COMPARATION ] abs_threshold: 0.396962 [ COMPARATION ] ov_tensor_utils.hpp::compare() is finished successfully. Duration is 82.1662s [ PLUGIN ] SubgraphBaseTest::get_plugin_outputs() is started /home/efode/repo/openvino/src/tests/functional/shared_test_classes/src/base/ov_subgraph.cpp:94: Failure Exception from src/inference/src/cpp/infer_request.cpp:223: Exception from src/core/src/runtime/allocator.cpp:69: std::bad_alloc

[ FAILED ] conformance_Add/ReadIRTest.Inference/Op=Add.1_Type=f32_Shape=dynamic_IR=f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815_Device=TEMPLATE_Config=(), where GetParam() = ("/home/efode/repo/openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/2023.3.0-13657-d5b0f4d2d73/operation/dynamic/Add-1/f32/f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815.xml", "") (115025 ms)

5. To debug the test in C++ Use the following command (just an example, refer to note in third item):

openvino/bin/intel64/Release/ov_op_conformance_tests --device=TEMPLATE --input_folders=openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/conformance_ir_files.lst, --report_unique_name --output_folder="openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/report/parallel" --gtest_filter="conformance_Add/ReadIRTest.Inference/Op=Add.1_Type=f32_Shape=dynamic_IR=f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815_Device=TEMPLATE_Config=()" --config_path="" --shape_mode=



### Resources

- [Contribution guide - start here!](https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING.md)
- [Intel DevHub Discord channel](https://discord.gg/7pVRxUwdWG) - engage in discussions, ask questions and talk to OpenVINO developers
- [How to link your Pull Request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#manually-linking-a-pull-request-to-an-issue-using-the-pull-request-sidebar)
- [Conformance readme](https://github.com/openvinotoolkit/openvino/tree/master/src/tests/functional/plugin/conformance/test_runner)
- [TEMPLATE plugin](https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/template/README.md)
- [OV CONTRIBUTION GUIDE](https://github.com/openvinotoolkit/openvino/tree/master/?tab=readme-ov-file#how-to-contribute)

### Contact points

@iefode

### Ticket

_No response_
Aryan8912 commented 8 months ago

assign to me

Qasim-Sajjad commented 7 months ago

Hi @iefode I would like to work on this task. I am new to open source. Can you please guide me how to quickly setup openvino on windows and some guidelines what should i do so that i can complete this task.

Qasim-Sajjad commented 7 months ago

.take

github-actions[bot] commented 7 months ago

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

iefode commented 7 months ago

@Qasim-Sajjad

Thanks for taking the issue!

OpenVINO team provided instruction to build OV on Windows https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_windows.md

Also please check the issue description to get details!

Thanks!

himanshugupta11002 commented 7 months ago

Is this issue is open to take?

himanshugupta11002 commented 7 months ago

.take

github-actions[bot] commented 7 months ago

Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.

iefode commented 6 months ago

Hello @himanshugupta11002 !

What is the status of the issue? Please, let us know in case of any issues

We will unassign the ticket in case of inaction. Thanks!

himanshugupta11002 commented 6 months ago

@iefode , But this issue was not assigned to me as you can see because it was already assigned to @Qasim-Sajjad . I was not working on this issue, Also if you can assign me I will start working on this.

Qasim-Sajjad commented 6 months ago

Hello @himanshugupta11002 . You can work on this issue as i am currently busy with my exams. I will unassign it so you can work on it.

p-wysocki commented 6 months ago

Hello @himanshugupta11002, are you still working on that issue? Do you need any help?

ajo2705 commented 3 months ago

.take

github-actions[bot] commented 3 months ago

Thank you for looking into this issue! Please let us know if you have any questions or require any help.