search

iree-org / iree

A retargetable MLIR-based machine learning compiler and runtime toolkit.
http://iree.dev/
Apache License 2.0
2.58k stars 579 forks source link

Rework check tests to join built+supported backends vs. hardcoding to specific backends. #4644

Open benvanik opened 3 years ago

benvanik commented 3 years ago

The way check tests are defined in cmake/bazel needs to change to better enable out-of-tree backends - but also it's pretty unwieldy today even for in-tree backends. Which backends a check test runs on is defined by what backends are built joined with an optional (and hopefully small) exclude list that is managed per backend (vs per test).

This allows in-tree backend tests to be more uniform and have less boilerplate and out-of-tree backend tests to be possible (where we otherwise would have to reference them in the in-tree build files and configuration).

We can also use this opportunity to clean up the names - the current style is very annoying in IDEs/tooling as the test target name is not easily inferable from the test file.

Concretely, this may mean something of the form of #4643 where the in-tree tests are just filegroups/exported test files and then the actual test rules are instantiated per target (iree/test/e2e/tosa_ops/vulkan/...) - meaning that out-of-tree/experimental backends can also instantiate them (experimental/cuda/test/e2e/tosa_ops/cuda/... or experimental/cuda/test/... in a single file, etc). Just as with #4643 we should be able to cmake install the test files and stamp out a new instantiation of them out-of-tree.

ScottTodd commented 10 months ago

I drafted what a test suite could look like in Bazel here: https://gist.github.com/ScottTodd/8379663fd2cbd1ea8a973b88c2a32a92 (before I noticed/remembered this issue). The next thing I was thinking was how to map that to CMake, and having a "registration" function like iree_compiler_register_plugin could be a good fit (declare each test suite then attach configurations to them).

ScottTodd commented 10 months ago

Could we generate check tests from other test suites, like JAX / StableHLO and PyTorch? I'd like to improve coverage (frameworks and platforms) as well as ergonomics here, but without that it feels like any work would be in service of one or the other.

As more targets like ROCm and AMD-AIE come online, this feels like it would be especially useful.

ScottTodd commented 8 months ago

For out of tree projects that define compiler plugins providing new target backends, it would be great to leverage the existing check test suites in some way. I spent a bit of time looking at how that could look for https://github.com/nod-ai/iree-amd-aie, and these checks are overly restrictive: https://github.com/openxla/iree/blob/eaefd3ff964f14e700b1c468aa2b4f20a4401028/build_tools/cmake/iree_check_test.cmake#L97-L103

I had two ideas for this:

  1. Expand on the filtering by letting iree_compiler_register_plugin() specify a list of compiler target backends the registered plugin includes (and similarly which input types it can convert)
  2. Simplify the filtering in the rule and instead filter at the directory/caller level. This would require reworking files like https://github.com/openxla/iree/blob/main/tests/e2e/stablehlo_ops/CMakeLists.txt. The files in the project already use iree_check_single_backend_test_suite extensively (and iree_check_test_suite is never used... maybe I should drop it from https://github.com/openxla/iree/pull/16225). This could also be complimentary to some of the other test suite refactoring ideas I've had.

I think 2 is the better option.

As the CMake logic changes, it may also help to just drop Bazel support for these test suites.