search

nod-ai / iree-amd-aie

IREE plugin repository for the AMD AIE accelerator
Apache License 2.0
69 stars 30 forks source link

[Testing] Make it possible to run individual tests #876

Closed newling closed 2 weeks ago

newling commented 2 weeks ago

After this PR it is possible to run individual tests. 

./run.py --help

Prints

...
  --tests TESTS         A comma-separated list of test names or sets to run. Available test sets: BatchMatmul, Convolution, ConvolutionNHWCQ, Matmul, MatmulFullBias, MatmulThinBias, MatmulTruncf, MultipleDispatches, UKernel, VanillaMatmul, All. Available individual tests: batch_matmul_1_128_128_256_bf16_f32, batch_matmul_1_128_128_256_i32_i32,
                        batch_matmul_2_64_64_64_bf16_f32, batch_matmul_2_64_64_64_i32_i32, conv_2d_nhwc_hwcf_2_14_bf16_f32, conv_2d_nhwc_hwcf_2_14_i32_i32, conv_2d_nhwc_hwcf_2_14_i8_i32, convolution_nhwc_q, depthwise_conv_2d_nhwc_hwc_1_14_i32_i32, matmul_f32_8_4_8, matmul_f32_8_8_4, matmul_full_bias_128_128_256_i32_i32,
                        matmul_thin_bias_1024_1024_512_bf16_f32_ukernel0, matmul_thin_bias_1024_1024_512_bf16_f32_ukernel1, matmul_truncf_128_256_bf16_f32, matmul_truncf_8_8_bf16_f32, two_matmul_switching, vanilla_matmul_32_32_64_bf16_f32.
...

So you can do ./run.py --tests=batch_matmul_1_128_128_256_bf16_f32 ... for example, to run just the one test. But you can also do ./run.py --tests=Matmul ... to run all tests with matmuls. I think this is a better design for debugging.

jtuyls commented 2 weeks ago

Also, for my understanding : Why do we still need to maintain run_matmul_test.sh ? I understand it has "repeat runs", but that shouldn't be the only motivating delta to keep it, right?

'repeat runs' is a very important 'delta'. When we're removing run_matmul_test.sh, we should keep at least the same coverage.

Abhishek-Varma commented 2 weeks ago

Also, for my understanding : Why do we still need to maintain run_matmul_test.sh ? I understand it has "repeat runs", but that shouldn't be the only motivating delta to keep it, right?

'repeat runs' is a very important 'delta'. When we're removing run_matmul_test.sh, we should keep at least the same coverage.

Absolutely! My main intention was to understand why maintain a separate file just for the repeat runs and not include that delta within run.py itself.

newling commented 2 weeks ago

run.py has a repeat_runs option too. I intend to deprecate run_matmul_test.sh soon (move the tests to run.py)