search

jaredpalmer / cypress-image-snapshot

Catch visual regressions in Cypress
MIT License
888 stars 160 forks source link

How do you make snapshot test useful if they can fail when nothing changed? #150

Open travis5491811 opened 4 years ago

travis5491811 commented 4 years ago

How do you make snapshot test useful if they can fail when nothing changed that we can really see? I'm having this problem and have seen multiple other people report a similar problem in this project and other projects like it.

To get around the issue they use failureThreshold and just keep raising the value. The problem with that is sometimes that 0.03% change in detail is something we want to know about so we can fix it. An example is one case with an extra character in a word. That is just a quick example. I realize i can increase threshold and make a test to manually check every piece of text but the hopes was that snapshot testing would eliminate having to manually check pages to see if something changed and also not be forced to write test cases for every single thing on the page when i could be checked against a screenshot.

If I reduce the failure threshold enough to catch it, it's also throwing errors for things that did not change like a image. So increasing failureThreshold just increases the number of passing test that should have failed while also allowing some test to pass that should pass.

There is no way to block areas (only with https://docs.cypress.io/api/cypress-api/screenshot-api.html#Arguments)

Am i missing something?

ivoiv commented 3 years ago

Image comparison is it's own science and not a problem easily solved. This plugin provides with you the tools to solve the problem in a specific way, via treshold. This is not a perfect way, and indeed as you report it has a lot of caveats and false-positives that you need to sort out on your own, based on your specific issues.

If you want a more advanced solution, that "just works", there's plenty of projects like https://applitools.com/ which use more advanced methods to recognise differences.

Tiagoperes commented 3 years ago

I'm facing the same problem and I tried multiple libs. The issue is that these libs make very simple image comparison. If your entire image shifts 1px to the right, for instance, the test will fail saying as much as 30% of the image is different. These shifts will happen as frequently as you change devices, hence the need to execute the tests always in the exact same environment.

To notice that a small shift in the pixels is a very minor difference, we need more complex tools, like AI, and as pointed out by @ivoiv, there are some paid tools that can do this. I couldn't find anything free that suits my need to be able to execute snapshot tests reliably across multiple devices. At this moment my solution is to ignore the snapshot tests when running them in the developer machine, but enable them in the CI, which will always be the same environment.