search

nod-ai / SHARK-TestSuite

Temporary home of a test suite we are evaluating
Apache License 2.0
2 stars 23 forks source link

Update benchmarking to check benchmarking results and generate artifacts #252

Closed saienduri closed 3 weeks ago

saienduri commented 1 month ago

This commit adds support, so that we actually make use of what we are benchmarking. We check against a golden time to make sure there are no regression for e2e and the individual submodels as well (unet, clip, vae). I also made the golden time a command line argument so that we can easily update it in the workflow file in either SHARK-TestSuite or iree whenever a patch has a performance boost, and there is no annoying repo dependence there.

There is also a job summary that is posted to the summary in CI, where there is an overview of the whole SDXL benchmarking that happened: https://github.com/nod-ai/SHARK-TestSuite/actions/runs/9455228567

saienduri commented 4 weeks ago

The w7900 runner (this is a different one from the iree repo's 2 runners) has gone down. Will get it back up, but this PR doesn't change any of the cpu testing code, so should be good. Update: got the runner back up :)