Refactor matmul test suite.

ScottTodd commented 2 months ago

Progress on https://github.com/iree-org/iree-test-suites/issues/2. See also the long Discord thread here.

Summaries of changes

Further decoupled test suites from the core CMake project

Forked iree_native_test.cmake to iree_test_suites_native_test.cmake
- Dropped support (temporarily?) for testing on Android, RISC-V, and ARM with SME
Forked iree_e2e_generated_runner_test.cmake to iree_test_suites_runner_test.cmake
- Dropped support (temporarily?) for filtering within the build system which tests are defined and compile .vmfb files
Now we can set -DIREE_BUILD_TESTS=OFF and avoid pulling in IREE's other tests
Added a new hand-authored linalg_ops/matmul/CMakeLists.txt that runs tests on each backend using default flags

Simplified the test generator

Dropped unused functions
Folded GPU-specific shapes into generic "small" and "large" shape test suites

Ran the `generate_e2e_matmul_tests.py` script offline and checked in the generated files

Currently 56 files totaling 1.90MB on disk (~27000 lines of code according to GitHub)
Now we can inspect the test cases without needing to run the generator locally, and I fixed a few formatting issues
I think this makes test suite management easier, and having the generated files in this test suites repository doesn't cost the main repository much (just extra git checkout time), but I could see a case for more tightly coupling the generator with the test runner

What is left to do?

I want to iterate some more on the linalg_ops/matmul/CMakeLists.txt file or move to a different test runner somehow. I mainly want to support XFAIL in some way for both compiling and running.
We should add back tests using CPU features like AVX512, GPU features like Vulkan float16 extensions, and other non-default flags somehow. Either infer what the compiler can from the host / target, or add test suites explicitly.

ScottTodd commented 2 months ago

Requested reviews from a few people that might have suggestions. Sorry, the code is kind of gross before and after... such is the nature of test suites x_x

ScottTodd commented 2 months ago

Ping @erman-gurses / others that might have opinions?

ScottTodd commented 1 month ago

I've discussed this on and off with @erman-gurses . We'd like to merge and then iterate.

Probably worth diffing the changes in this repo with https://github.com/iree-org/iree/pull/18725 to make sure we pick up all the improvements.

ScottTodd commented 1 month ago

Whoops, I should have known better than to merge without syncing and re-running the tests ;P

https://github.com/iree-org/iree-test-suites/actions/runs/11353574142/job/31578892613#step:8:445 <unknown>:0: error: HIP target not set; did you forget to pass '--iree-hip-target'?

This repo is low traffic and this PR touched a good chunk of large files so I'll fix-forward instead of revert.

iree-org / iree-test-suites