Snapshot comparison never fails

[cjea]

I have been following the README step by step. Here is the state of my dummy project: Repo

I wrote this test: File

cy.visit("localhost:3000")
      .document()
      .toMatchImageSnapshot();

Running that test causes a screenshot to be saved. However, when I change what is rendered on the page, the tests still pass!

Expected result:

---

Actual result:

---

I do not understand why the tests pass, if what is rendered in cypress' browser looks different than the screenshot that is saved.

I do not understand how toMatchImageSnapshot works. What is the base image that it uses? What is the current image?

Could someone please advise me how to get this test failing as expected?

[mullens]

I'm also experiencing this. I'm on 1.2.9, and the toMatchImageSnapshot tests will pass, even if the image is completely different, or deleted from disk.

[uillis]

I see this behavior as well. Image Snapshot tests used to fail as expected but recently that has changed and now they always pass. 1.2.9

• Seems to be intermittent and works again if the tests are renamed , cypress restarted, and rerun - will continue to investigate

[viniciuskneves]

Having the same issue. Just installed the plugin and made the setup. The first screenshot is nice but the following ones never fail, doesn't matter what I change in the page.

@meinaart something we can do to support here? I could help debugging if you give me some context on the project itself. I tried versions up to 1.2.6 and none of them seems to work.

[netes]

I can confirm the bug is still out there. If there's some way for me to help - just let me know.

[wolfi3b]

Still there. As I wanted to use this as awesome plugin for common approach to snapshots not just images .. well this is kinda deal breaker.

If there are needed any data from us or something I believe all would be willing to help.

[viniciuskneves]

I went for Percy as I don't have a lot of screenshots/day and I can still use the free tier.

I definitely recommend others to give it a try as well.

[wolfi3b]

Hi, @meinaart please is there any chance this is going to be addressed sometime soon? Just yes/no would be very appreciated answer. I would prefer much more to use your solution (as its local and does not limit me what ever I do while doing just the core I need) against the percy/aplitools etc... cloud ones. But ofc if you don't have time and don't see it would change sometime soon I understand and I will move on to something else.

[wolfi3b]

[NOTE: My first message (I deleted to remove confusion) was incorrect due to forgotten testing code]

So guys (FYI @meinaart) ...It bothered me so I digged deep down to the code and found the issue :-)

The problem is simple.... we are using it wrong.

In documentation is writen simply

You can pass the following options to toMatchImageSnapshot to override default behavior.

followed up by example

{
  "createDiffImage": true,       // Should a "diff image" be created, can be disabled for performance
  "threshold": 0.01,             // Amount in pixels or percentage before snapshot image is invalid
  "name": "custom image name",   // Naming resulting image file with a custom name rather than concatenating test titles
  "thresholdType": "percent",    // Can be either "pixel" or "percent"
}

so it evokes we should put it directly to "toMatchImageSnapshot" function. But that is not correct.

in config.js is code which expects

return Object.keys(DEFAULT_IMAGE_CONFIG)
    .filter((key) => options.imageConfig && options.imageConfig[key] !== undefined)
    .reduce(

so actually correct call to the method is:

cy.get('.class').toMatchImageSnapshot({imageConfig: {"threshold": 0})

and voala....it finally fails at not matching screenshots!

[StormPooper]

Nice find @wolfi3b, it also seems to fix it if there's a value in your cypress.json file, like so:

{
  "env": {
    "cypress-plugin-snapshots": {
      "imageConfig": {
        "threshold": 0.01
      }
    }
  },
  "ignoreTestFiles": [
    "**/__snapshots__/*",
    "**/__image_snapshots__/*"
  ]
}

Side note, my instance would blow up unless I included cypress-plugin-snapshots in my config. I'm guessing the code for checking defaults isn't quite right based on both of these issues.

[linukkis]

@wolfi3b Does it still work though? Whenever I try to wrap my configs with "imageConfig" like this cy.document().toMatchImageSnapshot({imageConfig: { capture: 'viewport'}}), I always get an error saying 'imageConfig' does not exist in type '(Partial<{ createDiffImage: boolean; threshold: number; thresholdType: "percent" | "pixels"; name...'. And of course because of this it always ignores all my overridden configs.

[skivol]

Note also that for mostly white screen + just some text changing the value (or position) the threshold (for pixels which differ) should be really low (I settled on "0", because even "0.01" didn't fail the test when the text location was changing in my case).