should fuzzy() tests account for pixel color distance?

[leeoniya]

Hi,

I see you guys constantly tweaking the fuzziness of various ref-tests. Perhaps a better fuzziness capability would be max color distance (eg Euclidean) of any pixel. This way inconsequential changes don't cause reftests to fail. An absolute pixel difference count should still be available as a warning for verification but not cause failures. Stuff like [1] seems rather tedious.

[1] https://github.com/servo/webrender/pull/3229/files#diff-1b6a7a062bdd2d11b504fa0976dc76ccL61

[kvark]

@leeoniya how would this look with the color distance approach?

[leeoniya]

so basically for all differing pixels, also compute something like [1]. the reftests will have some maximum allowable distance for any one differing pixel before failing. this way if 1,000 new pixels change by some imperceptible %, the reftest would still pass but with warnings and not require adjustments (just eyeballs). also maybe accumulate into buckets of small diff, medium diff, big diff and do something with that?

i don't know how generalized this approach is, especially with stuff like blur quality changes, compositing/alpha variations and anti-aliasing or pixel snapping adjustments.

[1] https://github.com/leeoniya/RgbQuant.js/blob/master/src/rgbquant.js#L703-L729

[Gankra]

cc @staktrace

note that our current design matches gecko's reftest infra, which is useful for integrating with their tooling