search

meinaart / cypress-plugin-snapshots

Plugin for snapshot tests in Cypress.io
MIT License
498 stars 117 forks source link

(bug) Screenshots do not match with different monitor setups. #117

Open zvaehn opened 4 years ago

zvaehn commented 4 years ago

Describe the bug Screenshots do not match if the monitor setup has changed from the initial screenshot monitor setup.

To Reproduce Steps to reproduce the behavior:

  1. Clear your screenshot folder
  2. Let your test run and let the runner create the reference screenshots
  3. Disconnect / Connect a Monitor (change the setup)
  4. Start the tests again

2nd way to reproduce: 1. Start Cypress in head mode (npx cypress open) GUI

  1. Clear your screenshot folder
  2. Start the Test on e.g. Monitor 1 (which is in my case a retina display and different resolition)
  3. Start the Test on e.g. Monitor 2

The screenshots will fail.

Expected behavior The Screenshots should match, no matter the resolution or amount of monitors. The Screenshots look almost the same, except some font rendering pixels, but even a threshold of 50% doesn't work.

Desktop (please complete the following information):

Additional context Thats basically the code i use. The Screenshot captures an iframe within a lightbox and some text. Due to the client's data protection reasons i cannot add a screenshot.

// testfile.js
cy.get('.lightbox').within(($overlay) => {
  cy.testPreviews($overlay, numberOfExpectedPreviews);
});

// support/command.js
Cypress.Commands.add('testPreviews', (scope) => {
    cy.wrap(scope).contains('teststring').should('be.visible');
    cy.wrap(scope).toMatchImageSnapshot({});`
}
sybrendotinga commented 4 years ago

I have a similar problem. Screenshots made while running in CI look slightly different than on a monitor. It looks like a color setting difference. The difference is ever so slightly that I only noticed it when I opened both screenshots in Photoshop on top of each other.

Expected image

Diff image

Actual image

NB: The screenshots are cropped by Github. Open the two images in two browser tabs (or on top of each other in Photoshop) to see the difference. Maybe also turn your brightness up.

And I can confirm that the above also is the case with my two monitors, the tests also fail when not on the same monitor.

Using: Mac OS 10.15.5 (19F101) Cypress 4.9.0 cypress-plugin-snapshots: 1.4.3 Node v12.16.2 Browser: Electron 80

revanth0212 commented 3 years ago

Exactly, I was able to reproduce this bug. For some reason, the plugin is not ignoring color changes even if I use a tolerance level as high as 1.

For instance, here are 2 snapshots, 1 taken on a mac with a retina display and the other on a 4K Samsung monitor. They both have different saturation levels and the plugin doesn't like it.

https://user-images.githubusercontent.com/35203638/117330889-c4168800-ae5b-11eb-900c-d8c070f93040.mov