snapshots recorded during `cypress run` mismatch `cypress open`

[goldylucks]

To reproduce:

1. write in a test cy.matchImageSnapshot()
2. run $ node_modules/.bin/cypress run
3. run again $ node_modules/.bin/cypress run - test passes
4. run $ node_modules/.bin/cypress open
5. click on the aforementioned test - test fails

Starting with cypress open and then following with cypress run also gives an error.

This might be a bug/inconsistency on cypress's end, so let me know if I should open it there as well.

Many thanks for this awesome tool!

[goldylucks]

Seems like cypress run only takes screenshot of the visible screen, and cypress open scrolls and screenshots the entire page

I recorded now using cypress open and then did cypress run and this is the error:

Error: Image size (360x640) different than saved snapshot size (360x3566).

[goldylucks]

hmmm ... recording with cypress run didn't wait for content to load, while cypress open did. Maybe that's why there's such a difference in image height.

I will try to make cypress wait for content to load before taking the snapshot, and run the test again

[goldylucks]

I changed the capture to viewport: addMatchImageSnapshotCommand({ capture: 'viewport' })

so both tests are same dimensions, but they are still failing, tho I don't see any difference in the diff image:

In the following image, I think everything in the left picture is one pixel higher than the right image:

Any ideas why this is happening?

[goldylucks]

I ran it in gitlab ci, and there's a mismatch between the images taken by the ci and taken locally, with the same cypress command cypress run

[sebinsua]

You might have a different issue from me, but I ran into an issue due to headed mode using a different device scale factor that what is configured for headless mode, and I was able to fix it like so. Obviously, by default, browsers are started headless with cypress run and headed with cypress open.

However, if that doesn't fix your problem, I would be tempted to just modify the failureThreshold until it's OK, since your images look very similar to me.

[goldylucks]

@sebinsua the failure percentage varies from 5-28(!) percent :/

your solution disables the image matching in a headed electron browser? This means it won't compare the snapshots when I do cypress open ?

[sebinsua]

your solution disables the image matching in a headed electron browser? This means it won't compare the snapshots when I do cypress open?

Yes, but that is just because I can't control the device scale factor in headed mode, and without control of this it's meaningless to compare the images (they would always have a different size).

As I said, in your instance, I think you should increase the failureThreshold. Btw, I've also ran into similar issues with broken behaviours when screenshotting elements that leave the viewport; I have no idea how to fix these, because I think they occur at quite a low level (within chromium or Xvbf).

[ivoiv]

I get the same issue.

cypress.json

{
  "viewportWidth": 1400,
  "viewportHeight": 660
}

commands.js

addMatchImageSnapshotCommand({
  failureThreshold: 0.003, // threshold for entire image
  failureThresholdType: "percent", // percent of image or number of pixels
  customDiffConfig: { threshold: 0.1 }, // threshold for each pixel
  capture: 'viewport', // capture viewport in screenshot
)}

When I run tests in cypress open I get a snapshot resolution of 1400x600. When I run in cypress run I get a snapshot of 1280x660

One strange behaviour is that if i remove the viewport config from cypress.json i get a snapshot of 1000x660, which is the default viewport.

If i change cypress.json to

{
  "viewportWidth": 1390,
  "viewportHeight": 650
}

Then recorded snapshot in cypress run mode changes to 1280x650, so i reacts to viewportHeight but not to viewportWidth.

If you have a custom viewport setting, viewportWidth defaults to 1280.

[ivoiv]

Apparently this is a problem with Cypress and not with this plugin

https://github.com/cypress-io/cypress/issues/2102

[RichieRunner]

what can we do to get around this? what is the optimum viewport for both open and run to match?

[jsphstls]

My workaround is to only capture images in headless mode.

 Cypress.browser.isHeadless
    ? cy.matchImageSnapshot()
    : cy.log('Snapshot skipped in headed mode.')

[Karnikel]

Because I'm using the headed mode just for test developing...I used this workaround in my code:

    if (Cypress.browser.isHeadless) {
      cy.matchImageSnapshot('AnySnapshot');
    } else {
      cy.matchImageSnapshot('AnySnapshot', { failureThreshold: 0.1 });
      cy.log('Compared image with 10% failure threshold, because test is running in headed mode!');
      cy.log('PLEASE ADD NEW REFERENCE-SNAPSHOTS JUST IN HEADLESS MODE!');
    }

[jennifer-shehane]

I would refer to this issue in Cypress for this one: https://github.com/cypress-io/cypress/issues/3324

Cypress's current recommendation is to only take screenshots for comparison during cypress run. There's further explanation here: https://github.com/cypress-io/cypress/issues/3324#issuecomment-542414532

You could write logic to disable screenshotting during cypress open by using the isInteractive config option or browser.isHeadless option. https://docs.cypress.io/api/cypress-api/browser.html#Screenshot-only-in-headless-browser

if (Cypress.config('isInteractive')) {
  // interactive "cypress open" mode
} else {
  // "cypress run" mode
}

Cypress.Commands.overwrite('screenshot', (originalFn, subject, name, options) => {
  // only take screenshots in headless browser
  if (Cypress.browser.isHeadless) {
    // return the original screenshot function
    return originalFn(subject, name, options)
  }

  return cy.log('No screenshot taken when headed')
})

// only takes in headless browser
cy.screenshot()

[edouard-lopez]

Trying to overwrite matchImageSnapshot, I got the error:

Cannot overwite command for: matchImageSnapshot. An existing command does not exist by that name.

Overwriting screenshot I got the error:

Image was NaN% different from saved snapshot with undefined different pixels.
See diff for details: undefined

Asserted the headed vs headless mode seem to work but looks clumsy:

  Cypress.browser.isHeadless
    ? cy.matchImageSnapshot({
        customSnapshotIdentifier: 'in-iframe/conversation-is-open',
        failureThreshold: 0.03,
        failureThresholdType: 'percent', // percent of image or number of pixels
      })
    : cy.log('No screenshot taken when headed');

[harrync]

@edouard-lopez thanks for that, that's been the only way I've gotten it to be useable, what I did to clean it up a little was add that code to a custom Command, e.g.

In commands.js:

Cypress.Commands.add('matchImageHeadless', () => {
  Cypress.browser.isHeadless
    ? cy.matchImageSnapshot({
        customSnapshotIdentifier: 'in-iframe/conversation-is-open',
        failureThreshold: 0.03,
        failureThresholdType: 'percent',
      })
    : cy.log('No screenshot taken when headed');
});

In *.spec.js:

cy.matchImageHeadless();

Would love to be able to run the plugin in normal mode too though, but for now this works with our Husky setup