yzhang93
commented
1 week ago LGTM. Thanks for adding this feature.
If we're brave we can use the execution time as a threshold above which the test fails (thoughts on this?)
This is a good idea. Main IREE also has some performance check in ci. We'd just need to be careful about the threshold.
We could also create an artifact and upload these numbers to table and make nice plots of our progress (although I'd prefer > someone more experienced with this stuff to take over from me)
Yeah, I think this feature can be added later. There are some folks in the group have this experience.
I'll remove some spacing, but above is where you can see the kernel execution time in the CI logs.
Alternatives considered:
1) Suggestion here: https://github.com/nod-ai/iree-amd-aie/pull/892#discussion_r1838636704 very nearly worked, but it resulted in large amounts of trace information being written to terminal, which I was worried would swamp the CI logs. Played around with
IREE_TRACING_CONSOLE_FILEbut not really designed for this.2) Tried tracy, but there is no simple text file dumped with tracy (as far as I can tell) so this didn't work
3) Tried making a standalone test based on https://github.com/nod-ai/iree-amd-aie/blob/main/runtime/src/iree-amd-aie/driver/xrt/cts/matmul_dispatch_test.cc but this got more complicated than I could handle.
FWIW running the m=n=512 k=4096 matmul in a loop (ie having warm up runs) does not effect the time to run the kernel, at least that is what I've observed.
If we're brave we can use the execution time as a threshold above which the test fails (thoughts on this?)
We could also create an artifact and upload these numbers to table and make nice plots of our progress (although I'd prefer someone more experienced with this stuff to take over from me)