Taking screenshots during `cypress open` versus `cypress run` result in different screenshot sizes

caaldrid commented 5 years ago

Current behavior:

Screenshots are not taken at the height and width set prior for the viewport even when using the setting the capture to be the viewport. This interestingly only happens if you trigger the test from the runner and not if you trigger the test from terminal.

Desired behavior:

For the screenshot height and width to match the height and width that the viewport is set to both when the test is triggered by the runner and when the test is triggered in the terminal.

Steps to reproduce: (app code and test code)

set code in test to take screenshot like so:

const visaulDiffWidth = 1280;
const visualDiffHight = 720;

options.capture = "viewport";

cy.viewport(visaulDiffWidth, visualDiffHight);

cy.wrap(subject, { log: false })
  .screenshot(screenshotTitle, options)
  .then(() => {
    cy.viewport(
      Cypress.config("viewportWidth"),
      Cypress.config("viewportHeight")
  );

  cy.task("imageMatch", taskData, { log: false });
});

Launch cypress
Start a test that takes a screenshot with the code above manually in the runner. (Screenshot will match whole display width and a height)
Exit runner
Start test from command line with yarn cypress run --spec '<path to test file>' (Screenshot will match the viewport)

Versions

Cypress 3.0.0 macOS Mojave 10.14.2 node v8.11.4 yarn 1.5.1

jennifer-shehane commented 5 years ago

Hey @xSky93, are you including this code as a custom command? Just wondering and needing where this subject is defined from the code above. Can you post the full code example and how it is called? Thanks!

caaldrid commented 5 years ago

@jennifer-shehane yes this is via a custom command full file is here:

function getTitle(test: any): string {
  return (
    (test.parent && test.parent.title ? `${getTitle(test.parent)}-` : "") +
    test.title
  )
    .split(" ")
    .join("_")
    .toLowerCase();
}

function afterScreenshot(data: Cypress.ImageMatchTaskData) {
  return ($el: JQuery, props: Cypress.Props) => {
    data.devicePixelRatio = $el.get(
      0
    ).ownerDocument.defaultView.devicePixelRatio;
    data.path = props.path;
  };
}

export const visualDiff = (
  subject: Cypress.Chainable<any>,
  options?: Partial<
    Cypress.Loggable & Cypress.Timeoutable & Cypress.ScreenshotOptions
  >
): Cypress.Chainable<null> => {
  const screenshotTitle = getTitle(Cypress.mocha.getRunner().test);

  const visaulDiffWidth = 1280;
  const visualDiffHight = 720;

  const taskData: Cypress.ImageMatchTaskData = {
    fileName: screenshotTitle,
    path: "",
    devicePixelRatio: -1
  };

  options.capture = "viewport";
  options.onAfterScreenshot = afterScreenshot(taskData);

  cy.viewport(visaulDiffWidth, visualDiffHight);
  return cy
    .wrap(subject, { log: false })
    .screenshot(screenshotTitle, options)
    .then(() => {
      cy.viewport(
        Cypress.config("viewportWidth"),
        Cypress.config("viewportHeight")
      );

      cy.task("imageMatch", taskData, { log: false });
    });
};

How I am calling it:

    it("Screenshots Full Doc", () => {
      // Wait for the composer to render then take screenshot
      cy.get(".cke_wysiwyg_div > p").then(() => {
        // Need to wait for all buttons to render into place
        cy.get(
          ":nth-child(2) > .ui-layout > .ui-layout__main > .ms-FocusZone"
        ).then(() => {
          // black out the chat content for consistentcy
          const options: Partial<Cypress.ScreenshotOptions> = {
            blackout: [".ui-chat"]
          };
          cy.document().visualDiff(options);
        });
      });
    });

caaldrid commented 5 years ago

I'm also doing some declaration merging with the cypress types to remove typing errors/warnings.

declare namespace Cypress {
  /**
   * Custom Chainable interface for non-array Subjects the merges with the official cypress declartion.
   * This is done to include the signatures of custom commands to Cypress.
   */
  interface Chainable<Subject = any> {
    visualDiff(
      options?: Partial<Loggable & Timeoutable & ScreenshotOptions>
    ): Chainable<null>;
  }

  interface ImageMatchTaskData {
    devicePixelRatio: number;
    fileName: string;
    path: string;
  }

  interface Props {
    path: string;
    size: string;
    dimensions: Dimensions;
    scaled: boolean;
    blackout: [];
    duration: number;
  }

  interface ScreenshotOptions {
    blackout: string[];
    capture: "runner" | "viewport" | "fullPage";
    clip: Dimensions;
    disableTimersAndAnimations: boolean;
    scale: boolean;
    onAfterScreenshot: ($el: JQuery, props) => void;
  }
}

caaldrid commented 5 years ago

@jennifer-shehane any insight on this? Is this being caused by something I am doing?

jennifer-shehane commented 5 years ago

I'm not seeing where you are calling Cypress.Commands.add for the custom command. Am I missing something?

caaldrid commented 5 years ago

Sorry @jennifer-shehane I call Cypress.Commands.add in command.js like this:

import { visualDiff } from "../src/visualDiff";

Cypress.Commands.add("visualDiff", { prevSubject: "true" }, visualDiff);

and in support/index.js I import command.js

import "./commands.js";

jennifer-shehane commented 5 years ago

Ok, so the base code in question is something like this:

 const visaulDiffWidth = 1280;
 const visualDiffHight = 720;

 options.capture = "viewport";

 cy.viewport(visaulDiffWidth, visualDiffHight);

 cy.document().then((doc) => {
   cy.wrap(subject, { log: false })
     .screenshot(screenshotTitle, options)
     .then(() => {
       cy.viewport(
         Cypress.config("viewportWidth"),
         Cypress.config("viewportHeight")
     );
  })

When you chain .wrap() to .screenshot(), it's essentially telling the screenshot command to restrict the screenshot to the document you pass in. Is there a reason you are doing this as opposed to just using cy.screenshot() by itself?

jennifer-shehane commented 5 years ago

@xSky93 Any update on this issue?

Saulis commented 5 years ago

@jennifer-shehane I can verify the behaviour regardless of any custom commands being used:

it('test', () => {
  cy.visit('https://www.google.com')

  cy.viewport(1280, 768)

  cy.screenshot('google')
})

Running cypress run with headless Electron produces screenshots in 1280x768 while as running cypress open both Electron (59) and Chrome (74) produce screenshots in 2560x1536.

Setting viewport(1600, 900) produces screenshots in 3200x1800 when running headed. So the screenshots are taken with 2x the viewport size for some reason.

I'm running: Cypress v3.2.0 MacOS 10.14.4 Yarn 1.12.3 Node v10.15.0

Lakitna commented 5 years ago

Could be display scaling. I don't know any macs that don't do display scaling due to their high pixel density screens.

giltayar commented 5 years ago

Yup. Seeing the same here. Very weird and inconsistent, BTW. Not sure I can see any logic, but sometimes the screenshot is the size of the viewport, and sometimes just the size the window is opened on. Interestingly enough, this sometimes happens in the runner too.

edelgado commented 5 years ago

I'm having this same issue. Running the specs with cypress run generates screenshots that are doubled in size than those generated with cypress open:

Error: Image size (1872x1256) different than saved snapshot size (936x628).

Any ideas? 😕

edelgado commented 5 years ago

@jennifer-shehane do we have any updates on this issue?

Other folks have reported this issue in the cypress-image-snapshot plugin (https://github.com/palmerhq/cypress-image-snapshot/issues/67), though it seems this is an issue with Cypress itself.

stefanoTron commented 5 years ago

same here @edelgado maybe @chrisbreiding or @brian-mann could give us a hand here?

jennifer-shehane commented 4 years ago

This actually appears to be a duplicate of https://github.com/cypress-io/cypress/issues/2102 So, closing in favor of that issue which outlines some workarounds at least.

edelgado commented 4 years ago

@jennifer-shehane this doesn't seem to be a duplicate of #2102. Similar, but different issues. Here, we are experiencing images twice as large/small. #2102 is something else. This issue should remain open.

jennifer-shehane commented 4 years ago

Hey @edelgado you mentioned that your issue is that during cypress run the screenshots are double the size than cypress open, but the original issue is that cypress open screenshots were double the size of cypress run. So you are experiencing a different issue than is described.

This issue of the screenshots being smaller in cypress run than in cypress open is what this issue was tracking.

I can recreate this behavior from this test:

it('test', () => {
  cy.visit('https://www.google.com')
  cy.viewport(1280, 768)
  cy.screenshot('google')
})

In cypress open - this is the picture that was taken on my Mac Mojave version 10.14.6 which is 2560 × 1536 in width.

google

If I do cypress run, then the picture taken is set to the max size of the display which is using Xvfb and is set to 1280x720 as explained in this issue - https://github.com/cypress-io/cypress/issues/2102

The issue appears to be in the scaling as described in this comment: https://github.com/cypress-io/cypress/issues/2102#issuecomment-521299946

I suppose this is technically a separate problem from https://github.com/cypress-io/cypress/issues/2102, but all of this is a larger problem with how we do scaling and display size calculations overall.

edelgado commented 4 years ago

Hi @jennifer-shehane, thanks so much for looking into this. I believe I misspoke earlier.

Indeed, my issue is that when I capture a screenshot via cypress run, the image generated is half the size of the same screenshot captured via cypress open. In this example, the top image was taken via the UI with a size of 1796x496, and the bottom image via the CLI with a size of 898x248. The middle image is the diff:

cypress-screenshot

Hope that clarifies my situation. Thanks again!

jennifer-shehane commented 4 years ago

Hey, @edelgado, while we do believe this is unexpected behavior - one may expect screenshots to be consistent across machines, headless and headed, this is how Cypress was programmed to work and we do not regard this as a bug in Cypress.

Taking screenshots in cypress open versus cypress run are not going to necessarily be the same size.

cypress open will run the tests and take screenshots within your current environment. So, your computer's screen size and your browser's size.
cypress run will run the tests and take screenshots within a headless environment (by . default), in Xvfb, which will be limited to the 1280x720 screen size and the max width of the browser.

We do not recommend comparing screenshots across environments (aka comparing screenshots taken in cypress open versus cypress run)

We do recommend comparing screenshots taken during cypress run on your local computer versus screenshots taken during cypress run in a CI environment.

We realize that our documentation is lacking in this area.

We need to clearly communicate best practices around performing screenshot comparison.
We need to communicate why the screenshot size is different from the viewport size.
We need to communicate how screenshots are different when in cypress open versus cypress run
We need to communicate how scaling is handled and not handled.

There's more than what I just listed and we are planning to clear all of this up and write some clearer documentation around this in our Visual Testing doc.

I created a new issue in our docs to document Visual Testing better here: https://github.com/cypress-io/cypress-documentation/issues/2160.

mikila85 commented 4 years ago

Fix this already!!!

Lakitna commented 4 years ago

Fix this already!!!

@mikila85 if you really need this fixed, consider opening a pull request and fixing it yourself. I'm sure that if you ask, the Cypress team will point you in the right direction.

0xIslamTaha commented 4 years ago

Here is a workaround for this bug, I have been doing visual testing and after a while I made it work by

adding this config to the plugin/index.js

on('before:browser:launch', (browser = {}, args) => {
if (browser.name === 'chrome') {
    args.push('--window-size=1280,1024');

    // whatever you return here becomes the new args
    return args
}

if (browser.name === 'electron') {
    args['width'] = 1280
    args['height'] = 1024
    args['resizable'] = false;

    // whatever you return here becomes the new args
    return args
}
})

Instead of taking the base screenshots from my machine, I managed to take the base screenshots from the cypress docker container based on the cypress/browsers:node12.13.0-chrome78-ff70 image by executing npx cypress run -b chrome --headless command. This is the same image I'm using for the CI, then the base screenshots and the regression screenshots are related to the same environment.
With the CI, run your test cases by executing npx cypress run -b chrome --headless

Hint: It would be better to build a docker-compose file for all your services and run all of them in the same network.

philcunliffe commented 4 years ago

Also experiencing this issue, it seems that when the screenshot command runs it removes the runner and lets the application in question fill the viewport which results in scaling based on screen size. To me this seems to run counter to the "scale: false" option (which is the default).

When the screenshot is taken in the headless "run" mode this doesn't happen because the headless browser "window" itself is not affected by the actual screen resolution it's being run on.

valentindotxyz commented 4 years ago

Hi @jennifer-shehane

We do not recommend comparing screenshots across environments (aka comparing screenshots taken in cypress open versus cypress run)

We do recommend comparing screenshots taken during cypress run on your local computer versus screenshots taken during cypress run in a CI environment.

While I totally understand the issue, I find it weird that screenshots took during run differ from the ones took with open. How shall we implement screenshot diff in that case? Prevent screenshots from being taken while in open mode and "force" other developers to use the run command for the "good" screenshots to be taken before pushing their changes?

It's a bit against your fast and easy from your tagline "Fast, easy and reliable testing for anything that runs in a browser." 😔

ajw015 commented 4 years ago

I will also state I am hitting this issue now where open and run snapshot differently, and would like to see this resolved.

PawelWesolowski commented 4 years ago

In my case the problem was with the Retina display on Macbook. I was getting exactly twice as big screenshots when I used open (2048 width instead of 1024). Then I moved the runner's Chrome instance to my second screen and voilà... open and run give the same results.

jennifer-shehane commented 4 years ago

@valentindotxyz Yes, this is exactly what we're advocating. During cypress open the browser opens headed by default - it has a literal screen and this screen is whatever the developer's machine is - that is the size the screenshots/videos will be taken.

So I could run my application tests on my Mac at resolution 2880x1800 taking screenshots to save to diff against, then Gleb opens up cypress open tomorrow and runs the screenshots to 'diff' against on his Macbook Air - resolution of 2560×1600. Now we're trying to diff the 2560×1600 sized screenshots against the 2880x1800 size screenshots. That's a problem.

cypress run defaults to running headless - there is no screen, it is imaginary, so you can set the size to whatever you want or use the imaginary size that Cypress has set as the default - this is the size the screenshots will be taken. You could diff screenshots using cypress open if you pass the --headless flag, because really the difference is headed versus headless.

To only take screenshots during headless runs - here is a recipe for this: https://on.cypress.io/browser#Screenshot-only-in-headless-browser

Cypress.Commands.overwrite('screenshot', (originalFn, subject, name, options) => {
  // only take screenshots in headless browser
  if (Cypress.browser.isHeadless) {
    // return the original screenshot function
    return originalFn(subject, name, options)
  }

  return cy.log('No screenshot taken when headed')
})

// only takes in headless browser
cy.screenshot()

bahmutov commented 4 years ago

I would never do something like this to you @jennifer-shehane

Saulis commented 4 years ago

you are clearly not helping with these comments and not going to help fix this so keep your comments to your self and maybe a developer will help solve this problem instead of telling everyone that we are the problem.

If you really want this issue to be solved, consider taking part in finding the solution. If you can’t, then leave it to someone else who can. Your hostile comments won’t help. I really hope this is not the way you talk to your co-workers (if there are any).

Phenomite commented 4 years ago

I resolved this by slightly modifying @0xIslamTaha code. In accordance with the supported args:

In cypress\plugins\index.js

on('before:browser:launch', (browser = {}, launchOptions) => {
    if (browser.name === 'chrome') {
      launchOptions.args.push('--window-size=1440,900');
      return launchOptions
    }
    if (browser.name === 'electron') {
      launchOptions.preferences['width'] = 1440;
      launchOptions.preferences['height'] = 900;
      return launchOptions
    }
  })

Mainly the electron part we are interested in because when operating as a runner it wouldn't respect anything but these preferences.

Perhaps make these a potential cypress environment config.

timarney commented 4 years ago

To only take screenshots during headless runs

I found @cypress/skip-test

Usage:

import { onlyOn } from "@cypress/skip-test";

...

sizes.forEach((size) => {
    pages.forEach((page) => {
      onlyOn("headless", () => {
        it(`it should render "${page}" on "${size}" screen'`, () => {

...

Output

Tara-E commented 3 years ago

You could diff screenshots using cypress open if you pass the --headless flag, because really the difference is headed versus headless.

I don't see any way to pass the --headless flag to cypress open. I think it would be pretty awesome if that option were added though. The test runner could open as normal and show all of the pretty results and logs etc, but that live preview section on the right isn't needed.

yanfengliu commented 3 years ago

I think this is the issue related to the retina screen on Macs that scale every screenshot to 2x resolution. Cypress has an official solution: https://docs.cypress.io/api/plugins/browser-launch-api.html#Set-screen-size-when-running-headless

In general, the physical device where you take the screenshot on will have an impact on the screenshot. For example, the color profile of the monitor will affect the actual color rendered.

zuzusik commented 2 years ago

cypress run defaults to running headless - there is no screen, it is imaginary, so you can set the size to whatever you want or use the imaginary size that Cypress has set as the default - this is the size the screenshots will be taken.

@jennifer-shehane I don't think this statement is actually correct in a part where it says so you can set the size to whatever you want - we have explicitly set cy.viewport("macbook-16"); in our code and while it works just great in headed mode, it has zero effect on headless mode - it still runs with 1280 screen width

Orgenus commented 1 year ago

cypress run defaults to running headless - there is no screen, it is imaginary, so you can set the size to whatever you want or use the imaginary size that Cypress has set as the default - this is the size the screenshots will be taken.

@jennifer-shehane I don't think this statement is actually correct in a part where it says so you can set the size to whatever you want - we have explicitly set cy.viewport("macbook-16"); in our code and while it works just great in headed mode, it has zero effect on headless mode - it still runs with 1280 screen width

I had the same problem as you though.

launchOptions.args.push("--window-size=1536,960");

I recommend you to try the command solved it for me.

BillSchumacher commented 1 year ago

The fix that was implemented to not need the launch options did not actually fix the problem and the work-around is still needed but no longer works.

When I set the viewport and then call the screenshot command with viewport specified, it should take a screenshot with the exact viewport size I specified and not limit it to my monitor's resolution/scaling on axis that exceeds that value.

Even if the image is scaled a few pixels in either direction, the size must match.

The behavior seems different across platforms (linux/OSX) as well. Headed and headless mode do not work as intended.

AmirL commented 6 months ago

The --force-device-scale-factor=1 fix works for Chrome (though I have to look at the non-Retina browser when I open it in UI mode).

And there is no such option for electron, which results in huge screenshot sizes on the Mac. (even in headless mode).

Don't you think it would be good to have an option for desirable screenshot sizes?

cypress-io / cypress