Closed pgmpablo157321 closed 1 month ago
@pgmpablo157321 yes, that's correct. We need to use two different deltas for CLIP
and FID
scores.
WG comments: Only change FID in TEST01 - that increases the range All existing results passing current checker are valid
When checking for the allowed delta percentage between test01 and baseline accuracy, the submission checker sets it to 1 or 0.1 based on the name of the benchmark
https://github.com/mlcommons/inference/blob/28144996310f5303a09355d55f238786737dc346/tools/submission/submission_checker.py#L2474-L2478
I think this behavior is incorrect for
stable-diffusion-xl
, where theCLIP_SCORE
metric has a tolerance of0.2%
and theFID_SCORE
of2%
based on the allowed ranges.The allowed delta percentage for those cases should be
0.2%
and2%
respectively @arjunsuresh Can you confirm this is the correct behavior?