We get a reasonable number of spurious QoR failures in CI on the small benchmarks in things like arithmetic/figure8.
We should loosen the relevant QoR metrics by making new small pass requirements and pointing at them. As circuits fail, we can point the relevant tests at this set of looser criteria instead of constantly updating golden.
This is better practice than having spurious failures that people learn to update -- that takes longer, and teaches a bad habit of expecting some failures.
We get a reasonable number of spurious QoR failures in CI on the small benchmarks in things like arithmetic/figure8. We should loosen the relevant QoR metrics by making new small pass requirements and pointing at them. As circuits fail, we can point the relevant tests at this set of looser criteria instead of constantly updating golden.
This is better practice than having spurious failures that people learn to update -- that takes longer, and teaches a bad habit of expecting some failures.