Allow overriding failure threshold per test

[mihkeleidast]

Same as #15, but that was closed a few years ago, so filing a new issue seems reasonable.

Again, I'm trying to migrate from jest-image-snapshot. That lib allows specifying threshold and other options through the matcher, like so:

expect(screenshot).toMatchImageSnapshot({
    failureThreshold: 0.0004,
    failureThresholdType: 'percent',
});

Wondering if you would support something similar? The reasoning is quite obvious - some screenshot tests are quite flaky when run through Chrome/Puppeteer, so in some rare cases we raise the failure threshold so tests wouldn't fail all the time.

[chrismansfield]

I second this, as apparent by my PR attempting to implement it.

I would argue that it is more logical to be able to adjust these setting per tests, as some tests might require some leeway in the pixel comparison, while some need to be super strict.

The scenario I was in which prompted me to add this functionality, was setting up visual testing for a UI library we're building. Because of font rendering difference between platforms (snapshots generated on Windows and verified on an Ubuntu server), our components have small differences. These differences vary between components, and within variants of components. For instance;

<Button size="large">Label</Button> // about 1% difference due to fonts
<Button size="small">Label</Button> // about 3% difference due to fonts

Given the above scenario, I would like to set the relative threshold per test. so that the large button does not accept more than 1% difference, while the small button accepts 3%.

[mihkeleidast]

We run our tests in a Docker container, so cross platform issues do not come into play that much, but have had some issues with SVG rendering diff even when using the same container.

[chrismansfield]

Docker is for sure one solution to my specific problem, but for various reasons that's off the table for now. Thanks for the suggestion though! Also interesting to hear that even ensuring the same environment might not be enough.

Still, I believe that in the generic sense, configuring this on the test level makes a lot of sense.