Open ScottTodd opened 4 months ago
https://github.com/iree-org/iree/pull/17846 added a few more CTS test binaries, so this is getting more important.
CI run before: https://github.com/iree-org/iree/actions/runs/9884624163/job/27301313444 (22 minutes) CI run after: https://github.com/iree-org/iree/actions/runs/9885502550/job/27303651412 (25 minutes)
Yeah I thought about that when doing it ;( I think we can change the embedded driver name string into a flag - the test is compiled into a library without the hardcoded driver name and then we could produce one binary per driver that just has a main() that sets the flag and calls into the library (to keep easy "run this binary without flags") or a single binary with the flag and find a way to make cmake create synthetic targets that pass the flag that are still debuggable.
Another data point: https://github.com/iree-org/iree/actions/runs/9914405645/job/27394002144#step:4:10
gcloud storage cp gs://iree-github-actions-postsubmit-artifacts/9914405645/1/build-android-arm_64.tar
2m10s to download a 1.6GB archive onto an android test runner, then another 4m45s to extract it. Of the 1.6GB, 960MB is under runtime/src/iree/hal/
, with 11MB per CTS executable 15 CTS tests 5 configs (local_sync vmvx, local_sync llvm-cpu, local_task vmvx, local_task llvm-cpu, vulkan) = 825MB
I noticed multiple minutes of build time and large chunks of build directory disk space being spent on HAL CTS test object files and binary linking at https://github.com/iree-org/iree/pull/17145#discussion_r1576873396 and https://github.com/iree-org/iree/pull/16273#issuecomment-1918174524. Each driver that instantiates the CTS (e.g.
hal/drivers/vulkan/cts/CMakeLists.txt
) creates one test binary per test case file (e.g.allocator_test
,event_test
,semaphore_test
). We should be able to share part of the build time and binary size between similar tests (across drivers) and/or drivers (across tests).Some other ideas from Ben:
Code:
Progress/sketches at https://github.com/ScottTodd/iree/tree/cts-improve-link-time