Conformance test denylist reduce

Abdurrahheem commented 2 weeks ago

Comment out all passing tests in test_onnx_conformance_layer_filter_opencv_all_denylist.inl.hpp file.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

[x] I agree to contribute to the project under Apache 2 License.
[x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
[x] The PR is proposed to the proper branch
[x] There is a reference to the original bug report and related work
[x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name.
[x] The feature is well documented and sample code can be built with the project CMake

Abdurrahheem commented 2 weeks ago

@asmorkalov should I completely remove test names (instead of commenting)?

asmorkalov commented 2 weeks ago

@Abdurrahheem I propose to remove and replace "FAIL" with some failure reason, e.g. exception message or some investigation result. Example: "dynamic shape support is required", "atribute qwerty is not supported for now", etc.

asmorkalov commented 2 weeks ago

Also, please add reference to dependent PRs.

asmorkalov commented 2 weeks ago

cc @alexlyulkov OpenVINO fails list:

[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_default_axis_example_NGRAPH_CPU, where GetParam() = (test_argmax_default_axis_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_default_axis_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_default_axis_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_default_axis_random_NGRAPH_CPU, where GetParam() = (test_argmax_default_axis_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_default_axis_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_default_axis_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmax_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmax_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_negative_axis_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmax_negative_axis_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_negative_axis_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_negative_axis_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_negative_axis_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmax_negative_axis_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_negative_axis_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_negative_axis_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_no_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmax_no_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_no_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_no_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_no_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmax_no_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmax_no_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmax_no_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_default_axis_example_NGRAPH_CPU, where GetParam() = (test_argmin_default_axis_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_default_axis_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_default_axis_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_default_axis_random_NGRAPH_CPU, where GetParam() = (test_argmin_default_axis_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_default_axis_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_default_axis_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmin_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmin_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_negative_axis_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmin_negative_axis_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_negative_axis_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_negative_axis_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_negative_axis_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmin_negative_axis_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_negative_axis_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_negative_axis_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_no_keepdims_example_NGRAPH_CPU, where GetParam() = (test_argmin_no_keepdims_example, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_no_keepdims_example_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_no_keepdims_example_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_no_keepdims_random_NGRAPH_CPU, where GetParam() = (test_argmin_no_keepdims_random, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_argmin_no_keepdims_random_select_last_index_NGRAPH_CPU, where GetParam() = (test_argmin_no_keepdims_random_select_last_index, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_0_NGRAPH_CPU, where GetParam() = (test_gather_0, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_1_NGRAPH_CPU, where GetParam() = (test_gather_1, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_2d_indices_NGRAPH_CPU, where GetParam() = (test_gather_2d_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_elements_0_NGRAPH_CPU, where GetParam() = (test_gather_elements_0, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_elements_1_NGRAPH_CPU, where GetParam() = (test_gather_elements_1, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_elements_negative_indices_NGRAPH_CPU, where GetParam() = (test_gather_elements_negative_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_gather_negative_indices_NGRAPH_CPU, where GetParam() = (test_gather_negative_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_maxunpool_export_without_output_shape_NGRAPH_CPU, where GetParam() = (test_maxunpool_export_without_output_shape, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_axis_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_axis, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_duplicate_indices_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_duplicate_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_negative_indices_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_negative_indices, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_reduction_max_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_reduction_max, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_reduction_min_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_reduction_min, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_without_axis_NGRAPH_CPU, where GetParam() = (test_scatter_elements_without_axis, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_with_axis_NGRAPH_CPU, where GetParam() = (test_scatter_with_axis, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_without_axis_NGRAPH_CPU, where GetParam() = (test_scatter_without_axis, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_NGRAPH_CPU, where GetParam() = (test_scatternd, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_add_NGRAPH_CPU, where GetParam() = (test_scatternd_add, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_max_NGRAPH_CPU, where GetParam() = (test_scatternd_max, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_min_NGRAPH_CPU, where GetParam() = (test_scatternd_min, NGRAPH/CPU)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatternd_multiply_NGRAPH_CPU, where GetParam() = (test_scatternd_multiply, NGRAPH/CPU)

opencv / opencv

Conformance test denylist reduce #25442

Pull Request Readiness Checklist