search

meinaart / cypress-plugin-snapshots

Plugin for snapshot tests in Cypress.io
MIT License
489 stars 117 forks source link

(feat) pixelmatch threshold control #236

Open jehoshua02 opened 10 months ago

jehoshua02 commented 10 months ago

Is your feature request related to a problem? Please describe.

I want to be able to control pixelmatch threshold so I can control how different a pixel must be before pixelmatch considers it different.

Describe the solution you'd like

I'd like to be able to control pixelmatch threshold directly, instead of cypress-plugin-snapshots passing a hard coded 0.01.

Describe alternatives you've considered

Forking this repo. Using a different package. Adjusting threshold in cypress-plugin-snapshots config. Living with flaky tests and accepting new snapshots every day, which is annoying and hinders adoption of visual testing in my team.

Additional context

A snapshot that has imperceptible changes failed a test. I used pixelmatch directly to dial in a threshold that would tolerate the difference:

I like keeping the threshold for cypress-plugin-snapshots at 0 pixels, because I don't want to allow any room for visual regressions.

Here's the code where threshold is utilized to determine if tests pass or not. Just above this, you can see where the hard coded pixelmatch threshold is passed. https://github.com/meinaart/cypress-plugin-snapshots/blob/6b95c7a930364b99f2496922159a6d693ad684a2/src/utils/tasks/imageSnapshots.js#L130C1-L138C6

ibrahimelement commented 5 months ago

Thanks for spotting this, also running into the same issue - due to color distance between two different renders. Adjusting the threshold seems to solve this problem.

Edit: Upon looking closer, it seems that you're trying to control the difference threshold passed to pixelmatch, rather than the hardcoded 0.01 threshold difference. From my tests, the difference in pixels reported by changing the threshold from 0.01 to 0.00 is 4.7x - which shows that it's significantly more sensitive in what it reports. This ratio is also highly variable. So what you're looking for by setting it to 0.00 is a pixel perfection, then adjusting the difference ratio.

Edit: While using this, I also ran into issues with threshold control and weird bugs for just checking images. I solved this by implementing my own image snapshot logic using pixelmatch - essentially the same thing this package is doing, but gives us full control over image comparison.