Closed ScottTodd closed 2 months ago
Config files (mostly used for the ONNX test suite) use this schema: https://github.com/nod-ai/SHARK-TestSuite/blob/16812962da23c85302c03ef505f4c654cb54b46c/iree_tests/conftest.py#L25-L52
(aside: that schema could be encoded in a file for validation/reference, rather than just included in a comment)
Right now test cases are included in one of these lists or not mentioned at all
skip_compile_tests
skip_run_tests
expected_compile_failures
expected_run_failures
While this lets us add new tests without needing to update existing files, it doesn't make it clear how many tests are included and which are passing.
Now that test results can be automatically reflected back into config files using https://github.com/nod-ai/SHARK-TestSuite/blob/main/iree_tests/update_config_xfails.py, we could for example
A) also list passing tests:
"passing_tests": [ "test_abs", ]
B) list test statuses directly:
"tests": { "test_abs": "pass", "test_add": "skip_compile", "test_div": "skip_run", "test_mul": "fail_compile", "test_sub": "fail_run", }
ONNX op tests moved to iree-test-suites. Moving this issue to https://github.com/iree-org/iree-test-suites/issues/25.
Config files (mostly used for the ONNX test suite) use this schema: https://github.com/nod-ai/SHARK-TestSuite/blob/16812962da23c85302c03ef505f4c654cb54b46c/iree_tests/conftest.py#L25-L52
(aside: that schema could be encoded in a file for validation/reference, rather than just included in a comment)
Right now test cases are included in one of these lists or not mentioned at all
skip_compile_tests
skip_run_tests
expected_compile_failures
expected_run_failures
While this lets us add new tests without needing to update existing files, it doesn't make it clear how many tests are included and which are passing.
Now that test results can be automatically reflected back into config files using https://github.com/nod-ai/SHARK-TestSuite/blob/main/iree_tests/update_config_xfails.py, we could for example
A) also list passing tests:
B) list test statuses directly: