cy.matchImageSnapshot() failing in CI #43

Closed awnclaim closed 6 months ago

Hi there,

All my specs fail when run in GitLab CI with the following error message:

Here is a spec example:

describe("Visual regression tests for Idea Tasks", () => {
  const sizes = [
    [1920, 1080],
    [768, 1024],
    [390, 844],
  ];

  const scrubbedElements = [
    ".one",
    ".two",
    ".three"
  ];

  sizes.forEach((size) => {
    it(`Idea Tasks when '${size}' resolution`, () => {
      // custom command
      cy.setResolution(size);

      cy.visit("");

      cy.wait(1000);

      cy.matchImageSnapshot({ blackout: scrubbedElements });
    });
  });
});

It's okay locally, and the specs not using the cypress-image-snapshot plugin work fine in CI as well.

Thank you in advance!

That would likely indicate that the snapshot hasn't matched, if it's failing in the then callback. I'd recommend this plugin to give you better feedback in CI

What does the console output say currently?

Hi Simon, thanks for your swift reply. With the plugin you recommended, the output isn't really clearer:

      cy:command ✔  wrap    
      cy:command ✔  task    Matching image snapshot, Object{13}
      cy:command ✔  screenshot  Visual regression tests for Idea Tasks -- Idea Tasks when '1920,1080' resolution
        cy:fetch ➟  GET ...
                    Status: 200
        cy:fetch ➟  GET ...
                    Status: 200
        cy:fetch ➟  GET ...
                    Status: 200
      cy:command ✔  task    Recording snapshot result
      cy:command ✘  then    function(){}

Also, normally, when the snapshot hasn't matched, I get screenshots with the word "diff" in the title, and in this case there're no diffs. To me, it just looks like an empty function is passed to .then() callback, as if the snapshots hadn't been recorded at all...

Does it work correctly locally both in run and open mode? Do you run it locally in the same environment as CI (i.e Docker)?

Nope, locally it only runs well in Open mode, so the issue comes from the Run mode, it seems!

Same issue here. Additionnally, I realized that the snapshots that raise the same issue @awnclaim describes also throws the following error in the console:

Error: Image size (464x559) different than saved snapshot size (464x560).

In fact, it generates the diff image as well as a failure screenshot. But in my specific case, that issue seems to stop the test suite prematurely.

Other failing snapshots that are not throwing the error above had their diff image generated just fine.

Cypress 13.2.0 Vite 5.0.10 Node 20.8 Windows 10 Enterprise

It would be great if one of you could create a minimal reproducible example that I can debug. I'm using this plugin across four different projects with dozens of tests and I've not encountered. Perhaps there is something unique to your setup that I haven't seen, perhaps around filenames/paths

Thanks

simonsmith / cypress-image-snapshot

cy.matchImageSnapshot() failing in CI #43