microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
66.64k stars 3.65k forks source link

[Bug]: expect(locator).toHaveScreenshot() generates different picture width when running test headed vs headless #33375

Open marcusNumminen opened 1 day ago

marcusNumminen commented 1 day ago

Version

1.48.2

Steps to reproduce

  1. Run the following test in headless mode ("Show browser" unset in VS code):

    test('screenshot test', async ({ page }) => {
    await page.goto('https://playwright.dev/');
    await expect(page.locator('.hero')).toBeVisible();
    await expect(page.locator('.hero')).toHaveScreenshot('heroBanner.png', {maxDiffPixelRatio: 0.5});
    });

    to generate the "baseline snapshot".

  2. Now set the "Show browser" option in VS code and run the test again => Test fails with the following error msg: Expected an image 1280px by 410px, received 1264px by 410px.

  3. Now delete the "baseline snapshot" and run the test again to generated the "baseline snapshot" in headed mode

  4. Now unset the "Show browser" option in vs code and run the test again => Test fail with the following error msg: Expected an image 1264px by 410px, received 1280px by 410px.

It seems like the maxDiffPixelRatio makes no difference here.

Expected behavior

It should not make any difference if I run the browser in headed or headless mode, the screenshot should be the same size

Actual behavior

I get different picture width when taking screenshots depending if I run the tests in headed or headless mode

Additional context

No response

Environment

System: OS: Windows 11 10.0.22631 CPU: (8) x64 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz Memory: 19.55 GB / 31.85 GB Binaries: Node: 18.18.0 - C:\Program Files\nodejs\node.EXE npm: 9.8.1 - C:\Program Files\nodejs\npm.CMD pnpm: 8.15.4 - C:\Program Files\nodejs\pnpm.CMD IDEs: VSCode: 1.95.0 - C:\Users\marcus.numminen\AppData\Local\Programs\Microsoft VS Code\bin\code.CMD Languages: Bash: 5.2.26 - C:\Program Files\Git\usr\bin\bash.EXE npmPackages: @playwright/test: ^1.48.2 => 1.48.2

Skn0tt commented 20 hours ago

Hi Marcus! Headed and headful will always behave slightly different, that's in the nature of rendering sadly. I'm a little surprised to see two different pixel sizes. I'm having trouble reproducing though - following the steps you're giving, the test is green both headed and headless. I'm using Chromium. What browser are you observing this on?

marcusNumminen commented 20 hours ago

I use Chromium, the one that comes when you do npx playwright install On my side this is 100% reproducible

marcusNumminen commented 20 hours ago

I created a gif of the problem Image