Minor pixel difference noise while capturing screenshot using page.screenshot api for chromium browser only

[sunilsurana]

Context:

• Playwright Version: 1.27.1 & 1.11.1
• Operating System: Linux
• Node.js version: 16.*
• Browser: Chromium only, work fine on firefox

Problem Description

TLDR When we capture screenshots on multiple runs on same machine using playwright api for Chromium browser we see noise in some pixels when we compare the screenshots. Attaching the diff image as an example

Details We capture screenshot for our tests using playwright for chromium browser. Sometimes few image show noise in some pixels even if there is no code change and even when run on same machine. With firefox it works fine and does not show any noise. I even tried replacing playwright with pupeteer and with pupeteer on chromium also it does not show any noise.

I tried making the chromium startup arguments here same as pupeteer startup arguments here but still I see the noise.

I'll be looking into implementation details of page.screenshot api and how it differs between firefox and chromium for playwright and chrome vs chrome for pupeteer. But meanwhile filing this issue in case if anyone already know about this and step to mitigate

[mxschmitt]

Can you provide us a reproduction so we can run it locally? If it works with Puppeteer, would be nice if you could provide us two scripts, one for Playwright and one for Puppeteer so we can investigate better.

Which exact host OS and Playwright version are you using? Since Linux can mean a lot and I see two different Playwright versions.

[sunilsurana]

Hi, Here are the repo steps. Ive checked in the html for which we capture screenshot inside the repo itself in storybook folder. Please let know if this works.

Basically it opens storybook url, wait for 4 sec, take screenshot, wait for 2 sec and again take screenshot.

The reason of taking 2 screenshot is to check if issue is in loading page or .screenshot api

1. Clone sunilsurana/repoissue
2. Run npm install
3. Run npm run build
4. Run npm run chrome multiple times for playwright chrome screenshots
5. Run npm run firefox multiple times for playwright firefox screenshots
6. Run npm run pupp-chrome multiple times for pupeteer chrome screenshots
7. Do ls or dir and see the differences in bytes

i tried in windows and ubuntu and on both we see same results. Playwright version 1.27.1

Also something interesting is if we open pupeteer in headful mode then sometimes it generates noise.

[sunilsurana]

And here are some dir outputs mentioning size of screenshots

Playwright with firefox

Pupeteer with chromium

Playwright with chromium

[mxschmitt]

Thank you! My colleague will take a look.

[aslushnikov]

@sunilsurana I ran multiple times the following without any luck:

npm run chrome
# check for differences in images
magick compare -verbose -metric mae  playwright-chrome-run_1_A.png playwright-chrome-run_1_B.png diff.png

Generated screenshots were always identical for me.

Can you please attach the two images you get here separately?

[sunilsurana]

Hi @aslushnikov You might need to run npm run chrome 2 times and see differences when chrome is launced 2 times. The _A and _B are when screenshots is captured on same browser. I just added that to check if the difference occur due to the screenshot method or when new instance of chrome is started

[aslushnikov]

@sunilsurana ah, still no luck.

for i in {1..15}; do npm run chrome; done

@sunilsurana so if you can help by attaching screenshots with differences, i'll appreciate this!

[sunilsurana]

Ohh. For you its consistent everytimg. I'll rerun and attach the screenshots by morning our time tomorrow. Meanwhile can you please let know what OS are you running?

[aslushnikov]

@sunilsurana yep, awesome, I was able to reproduce this on Linux!

[sunilsurana]

BTW just curious on which platform did you get consistent screenshots? Also just to let you know we have about 1400 screenshot out of which only these 2-3 screenshot show difference rest all are good. The screenshot is attached in first post of this thread. If you see its always on the border of the persons picture. I guess chrome might be doing some border smoothing or something. I tried to match playwright default start options with that in pupeeter but that didn't solve the issue.

[aslushnikov]

just curious on which platform did you get consistent screenshots?

@sunilsurana Results are stable on macOS for me

[sunilsurana]

@aslushnikov Just checking, any luck with finding cause?

[aslushnikov]

@sunilsurana well, long story short: browser rendering is not guaranteed to be deterministic, so the same input might yield different pixels from time to time. This looks like this case to me.

[sunilsurana]

Yeah. And the strange thing is with firefox the screenshot are always deterministic and also with pupeeter and chromium. Not sure if they are disabling or using certain different flags to render or capture screenshot

[sunilsurana]

FYI, I tried with webkit too and with webkit also the screenshots match everytime. Ive checked in the webkit related changes also in same repo command is "npm run webkit"

[aslushnikov]

@sunilsurana we're not aware of webkit or firefox rendering non-determinism, so I'd expect screenshots to match there.

[salos1982]

I have a similar issue on Windows. I try to make a screenshot and very often one jpeg image very slightly differs from one screenshot to another.