[BUG] Same test, different runs causes different heights with a weird compression effect

[filipesmedeiros]

Context:

• Playwright Version: 1.27.1
• Operating System: MacOS 13.0
• Node.js version: 16.15.0
• Browser: All
• Extra: Also CI using MacOS 12 (Github Actions)

Describe the bug

Subsequent screenshots (on different test runs) of the same page are coming out with different heights. This is mainly due to a weird "compression" (like smashing) effect on one of the screenshots.

These two example come from the same code, same server (NextJS locally) running, just yarn test two times in a row:

I'm sorry for the long image, but that may be part of the problem (although short pages sometimes have this problem too).

If you compare the images using something like https://www.diffchecker.com/image-diff/ you can see the weird effect.

Is this a known bug? The differences are even greater once we run the tests on CI (more screenshots are different), but it's flaky, so sometimes it happens and sometimes everything is good.

EDIT: Just to clarify, by compression I mean that the pixels themselves are smashed, so everything is shorter, which in turn causes the heights of the screenshots to differ, like on of them was "smashed", which is wrong, e.g. causes the square logo to turn into a rectangle, etc

Thanks in advance!

[pavelfeldman]

This looks like a bug to me. My only idea would be that the actual pixel data and the size of the page are captured in a racy manner and then the image is scaled based on the wrong size. Could you try waiting for layout to settle before you capture an image a little? Also, await expect.toHaveScreenshot should wait for the image to stabilize, are you using it?

[filipesmedeiros]

@pavelfeldman I used shouldMatchSnapshot on all runs, including the first. Is that bad? I know the docs say to use toHaveScreenshot but since shouldMatch also writes if not present, I assumed it was the same.

I can test with that, though! Thanks for the reply!

[filipesmedeiros]

Also, it's supposed to be expect(await page.screenshot).toMatchScreenshot right? We don't have to await expect like you said, I think (just checking)

[pavelfeldman]

We do, that's the point of resilient screenshotting, should be await expect(page).toHaveScreenshot().

[filipesmedeiros]

@pavelfeldman But expect doesn't return a Promise 😕

[filipesmedeiros]

Wait my bad, I'm not awaiting expect, but toHaveScreenshot. But toMatchSnapshot does not return a Promise

[filipesmedeiros]

Ok I see what's happening: I'm doing expect(await page.screenshot).toMatchSnapshot (like documented here https://playwright.dev/docs/test-assertions#screenshot-assertions-to-match-snapshot-1) but maybe that's wrong and I should be doing await expect(page).toHaveScreenshot. Let me try and I'll get back to you!

[filipesmedeiros]

Same thing happens 😞 e.g. in the above image you can see a 1 pixel height different between screenshots

[pavelfeldman]

That is something we might be able to fix!

[filipesmedeiros]

Ok I don't know if it helps, but running it inside Docker Linux (and CI on Linux) fixed it (or at least all the tests passed).

So the issue seems to be specific for Mac!

EDIT: nevermind, I think it was a flaky test that passed :(

[thraizz]

Wanted to report in that this happens on my team's MacBooks as well. We run both M1 and Intel chips. Our CI runs the official image. Locally generated, fullPage screenshots differ in 1px height from the CI screenshots.

We can mitigate this by generating screenshots in a docker image, but a fix for this would be awesome. If I can help anyhow, let me know :)

Playwright Version: 1.29.2 Playwright config: https://gist.github.com/thraizz/0b2ab81f31925265a467e75fec58935f macOS 13.1

[Dwlad90]

Any update on this issue?

[kr-anastasiia]

We are also experiencing the same problem. When we run the test using 'run test button' and save screenshots we don't get any problems. But when we try to run test using command line we get the error. This only applies to Google Chrome. Mac OS 13.2.1, M1 Pro

[maartenbreddels]

We had this issue as well (in https://github.com/widgetti/solara/pull/67), we had a screenshot that had a different width sometimes. Based on https://github.com/microsoft/playwright/issues/20015#issuecomment-1427285592, we had the idea of printing out the bounding box.

• On failure we got: {'x': 416.765625, 'y': 611, 'width': 134.5, 'height': 36}, and our resulting image was 136x36
• On success we got: {'x': 398, 'y': 611, 'width': 134.5, 'height': 36}, and our resulting image was 135x36

So I think it is a combination of something else on the page pushing an element to a non-integer pixel position (but not stable every time) that can cause an image to change by 1-pixel width or height.

Hopefully, this is useful for others and also the cause of this issue.

[dabbottQA]

Unfortunately this issue is limiting our ability to write and run concise tests. Our Bamboo test runs fail with this 1px issue. We have been able to work around the issue by specifying locators that capture a broader area which is not always ideal.

Error: Screenshot comparison failed:

  Expected an image 816px by 96px, received 817px by 96px. 

[maartenbreddels]

Did you check if your bounding box has a non integer x coordinate?

[adelara]

So I think it is a combination of something else on the page pushing an element to a non-integer pixel position (but not stable every time) that can cause an image to change by 1-pixel width or height.

You know, this is exactly the thought I have... and it's the same system we use over and over to run the tests. We run the stuff on a CI/CD pipeline, inside a Linux container and develop on Windows. We don't save the Windows generated images. When the images are captured on the container, it is then sent to a separate server where we keep the benchmarked images, and the other server does the comparison. When it works, great but more than 70% of time, we get a major pain in the arse as the images are different, sometimes by a couple pixels here and there, or by generating images with different sizes, which invalidates the image comparison. Another annoyance is that text renders in slightly different positions or the spacing between lines is different (this also causes the image to have different size).

So, yes, anything to help, is appreciated :-)

[mumair-pc]

Any update on this?

[pascalknupper]

Is there any progress on this issue or do you know some workaround for this issue:

Expected an image 768px by 1465px, received 768px by 1463px.

[afonte15]

Is there any updates on this issue or a workaround?

• Expected an image 201px by 35px, received 201px by 36px.

[maartenbreddels]

I'm not sure it's a playwright issue, I think it's a combination of something unstable left or above an elements which causes the bounding box to be at floating point coordinates (see https://github.com/microsoft/playwright/issues/18827#issuecomment-1513345600) which when rounded to integer pixels positions will sometimes jump by one pixel.

[unikitty37]

I think I'm seeing something similar, except it's the width, not the height.

If I generate the screenshot using a headless Chromium (either from pnpm run playwright test or using the VSCode extension with "Reuse Browser" set to false) it generates a different width screenshot from the one generated by using the VSCode extension with "Reuse Browser" set to true:

Expected an image 1280px by 1108px, received 1265px by 1108px.

Additionally, if I use the mask option, the mask is placed in the wrong place if generating the screenshot with "Reuse Browser" set to true. In these examples, the second year (2023) should have been masked.

Reuse Browser is false (correct positioning)

Reuse Browser is true (offset positioning)

(I was going to raise a separate issue for this, but it feels like it might be related to this one. Let me know if you want it raised separately!)

This is with Playwright 1.37.1, Playwright Test for VSCode version v1.0.15, macOS macOS 13.5 (22G74) on Apple M1 Max, and whatever Playwright browsers were current at 09:00 (Europe/London) on 2023-08-30 :)

[nikolay-yavorovskiy]

We have this issue too, the fix would be very helpful, this issue is causing a lot of flaky tests...

[JBill85]

This is happening for us as well... It's driving me crazy. Would be nice to be able to specify some tolerance for this

Expected an image 161px by 161px, received 161px by 160px

[PeterHewat]

Hello, we are also having this issue. Tests are always running inside the same environement (GitHub Action Linux Chrome headless)

Expected an image 344px by 335px, received 344px by 333px.
Expected an image 704px by 580px, received 702px by 580px.
Expected an image 1058px by 330px, received 1058px by 329px.
...

Using the latest playwright 1.38.1

[JBill85]

Hello, I'm curious if there are any plans to look at this issue. I posted last month and never heard from anyone.

[maartenbreddels]

I really think the issue is not with playwright: https://github.com/microsoft/playwright/issues/18827#issuecomment-1513345600 but having a bounding box with non-integer values (e.g 3.5) in combination with a non-stable page (e.g. fonts still loading).

[JBill85]

I really think the issue is not with playwright: #18827 (comment) but having a bounding box with non-integer values (e.g 3.5) in combination with a non-stable page (e.g. fonts still loading).

I mean they could give us the option to turn this check off or something. I'm using the image checks for egregious failures like the picture is missing. Not really interested in the way the failure works right now.

[maartenbreddels]

If the failure is due to what I describe, I don't think there is simple solution. It's something else in the page that is not stable (playwright cannot be blamed for that) that causes the rounded off width and height to differ by 1 or 2 pixels. Maybe playwright could warn/error when taking screenshots of non-integer bounding boxes.

[paulsmithkc]

@maartenbreddels PR submitted to a add configurable tolerance on the size of the image. https://github.com/microsoft/playwright/pull/27513

Due to floating point error in both CSS and JS, it isn't practical/possible to ensure exact pixel sizing in the vast ecosystem of tools and frameworks.

[crazyk2]

The same issue. Flaky tests. We always run tests on Linux CI with docker mcr.microsoft.com/playwright:v1.38.1-focal Expected an image 1680px by 4482px, received 1680px by 4281px. 997854 pixels (ratio 0.14 of all image pixels) are different. Sometimes tests pass, sometimes fail

[woodyADT]

The same: Expected an image 1263px by 1305px, received 1263px by 1296px. 184121 pixels (ratio 0.12 of all image pixels) are different.

[JBill85]

There has been a PR submitted above that allows you to provide a tolerance on this check. I'm hoping someone will graciously review it.

Link to PR posted for this: https://github.com/microsoft/playwright/pull/27513

[JBill85]

@dgozman I noticed you tagged this, I'm pretty sure that means someone is aware of it, and potentially considering adding it. Can someone give us an update when you have one.

Much appreciate your consideration on this.

Link to PR posted for this: https://github.com/microsoft/playwright/pull/27513

[paulsmithkc]

Fresh PR started to address this. https://github.com/microsoft/playwright/pull/28453

@dgozman @yury-s

[MartinPopovski1]

I found a possible solution that makes my visual tests pass. Basically the idea is to scroll the page to the beginning (width and height) before making a screenshot. The trick is that you have to wait for the scroll function to finish.

await page.evaluate(() => {
      window.scrollTo(0, 0);
    });
    await page.waitForFunction(() => window.scrollY === 0);
    await expect.soft(page).toHaveScreenshot(...

[paulsmithkc]

@yury-s @mxschmitt Can you review this proposed fix? https://github.com/microsoft/playwright/pull/28453

[jmaschle]

I have another idea. Since i dont care what is in my tables on a compare i use a mask. why cant playwright ignore the size difference of the masked regions (now 'ghost' regions) and compare everything else without them? Better yet disable visibility of the table itself before the snapshot. In my case, the grid column widths are dynamic as are the number of rows. And i have scrollbars. Also, should the results differ if i add --headed at the terminal command line if viewport is set to 1920x1200 and launchOptions: { args: ["--start-maximized"] }? Also - why doesnt stylePath work??? i get red squigglies
![Uploading vscver.png…]()

[pbarhyl]

To make easier to vote on this issue (because what count are 👍 for first comment) @filipesmedeiros pls edit your comment and add folding for this huge image

<details>
  <summary>Hello</summary>
  World!
</details>

Demo:

Hello

World!

[piotrtar]

For me this workaround is working at the moment:

  await page.evaluate(() => window.scrollTo(0, 0));
  await page.waitForFunction('window.scrollY === 0');
  await page.screenshot({ path: 'path-to-screenshot.png' });
  await expect(page).toHaveScreenshot('path-to-screenshot.png');

[nbaldzhiev]

I am struggling with this as well.

Playwright version is 1.37.1, running on MacOS 13.4.1 (22F82). I use toHaveScreenshot using a Locator and the element is, similar to the author's, one with big height. The error message I get (primarily on Chromium) is

Error: Screenshot comparison failed:

  Timeout 15000ms exceeded.

Call log:
  - expect.toHaveScreenshot(first-screenshot-owner.png) with timeout 15000ms
  -   verifying given screenshot expectation
  - waiting for locator('#pdf-preview-content')
  -   locator resolved to <div class="css-1m2vc5j" id="pdf-preview-content">…</div>
  - taking element screenshot
  -   disabled all CSS animations
  -   waiting for element to be visible and stable
  -   element is visible and stable
  - Expected an image 600px by 4865px, received 600px by 5105px. 
  - waiting 100ms before taking screenshot
  - waiting for locator('#pdf-preview-content')
  -   locator resolved to <div class="css-1m2vc5j" id="pdf-preview-content">…</div>
  - taking element screenshot
  -   disabled all CSS animations
  -   waiting for element to be visible and stable
  -   element is visible and stable
  - Expected an image 600px by 5105px, received 600px by 4865px. 
  - waiting 250ms before taking screenshot
  - waiting for locator('#pdf-preview-content')
  - Timeout 15000ms exceeded.

By the way, I've noticed that adding a sleep (just for experiment's sake), the assertion passes each time. When I remove the sleep, it starts failing again. So I feel like it's not really a problem with the application rendering the element's bounding box with different resolution; if it was that, a sleep would not have helped as the resolution would have still been different:

With a minimal waitForTimeout, I avoid the issue, but obviously I don't want to merge code with waitForTimeout:

    await preview.locator('header').scrollIntoViewIfNeeded()
    await page.waitForTimeout(1000)
    const firstScreenshotName = isShared ? 'first-screenshot.png' : 'first-screenshot-owner.png'
    await expect(page).toHaveScreenshot(firstScreenshotName, { maxDiffPixelRatio: 0.01 })

If I remove the waitForTimeout line, it immediately starts failing again, specifically in Chromium.

[eugenekravchenko]

Any update on this issue? Is it in plans to fix it? Thanks

[StefanSchoof]

I had also a problem with random position on the y-axis. I make my test with a ugly hack stable. Get the bounding box and move the clipped area based on the current position:

const panel = page.getByRole("dialog");
  const headingBox = await page
    .getByRole("heading")
    .boundingBox();
  const panelBox = await panel.boundingBox();
  if (panelBox == null) {
    throw new Error("PanelBox was null");
  }
  panelBox.height -= 1;
  if (headingBox?.y == 26) {
    panelBox.y += 1;
  }
  await expect(page).toHaveScreenshot({ clip: panelBox });