nod-ai / SHARK-TestSuite

Temporary home of a test suite we are evaluating
Apache License 2.0
2 stars 23 forks source link

Add support for SD3 testing along with a refactor of the suite. #266

Open saienduri opened 3 weeks ago

saienduri commented 3 weeks ago

This commit adds in the necessary support for integrating tests for all the sd3 sub models. Also refactored the test suite and made changes along the way to get everything working smoothly. We no longer have input, output, and module flags coming from the json configs. Enabled expansion of the real weight file, so we can use environment variables while running the module. With this, I switched to using an environment variable, so we can just use one real_weights file for sdxl-scheduled-unet (dealing with cpu and rocm pipeline vmfbs). These changes allow us to group sd3 and sdxl testing into the same json configs for a clean integration :) All the azure artifacts for each submodel (inputs, outputs, weights) can be found in sharkpublic/sai/sd3-mmdit, sharkpublic/sai/sd3-vae, and sharkpublic/sai/sd3-prompt-encoder. The mlirs and flags are all set up here.

dan-garvey commented 3 weeks ago

Looks good to me for now, the only thing is we should move the artifacts to a "stable" folder of some sort, and then update that regularly with artifacts generated on newer versions of turbine as soon as they pass