[REGRESSION] Change docker from 1.15.x to 1.16.x that all screenshots need to be updated

[xuanzhaopeng]

Context:

we have some screenshot comparing, once we upgrade docker image from v1.15.1 to v1.16.3 and use 1.16.1 playwright library, the html screenshots are different, so we have to update all screenshots. Pls notice we are using same version of Chromium for both v1.15.1 & 1.16.3 docker image:

 browserVersion:  mozilla/5.0 (x11; linux x86_64) applewebkit/537.36 (khtml, like gecko) 
headlesschrome/97.0.4666.0 safari/537.36

is it expected?

Some investigation:

I use v1.15.1 docker image + playwright 1.16.3 + headless chromium 97.0.4666.0 => No changes of screenshot, test pass I use v1.16.3 docker image + playwright 1.16.3 + headless chromium 97.0.4666.0 => Screenshot has been changed, tests are failed

Screenshot comparing:

[xuanzhaopeng]

I think the "issue" is coming from here: https://github.com/microsoft/playwright/pull/9599/files

previously in 1.15.x , the test are executed in docker linux directly from 1.16.x, the test are executed within x11vnc/xvfb display on the top of docker linux

can we have a switch to disable xvfb and vnc? xvfb and vnc will increase the memory and cpu usage, however for functional testing(based on our experience with Selenoid with Selenium) it doesn't bring big values in real practice especially we have a lot of E2E tests and we have a lot of paralleled executors.

Or we should build different docker images:

• docker with vnc and xvfb
• docker without VNC and XVFB

what do you think @aslushnikov ?

Cheers

[aslushnikov]

@xuanzhaopeng in docker 1.16.3, we no longer run XVFB / VNC. They are installed but are not used.

In the screenshots you submitted, it looks like fonts did change. Could it be the case? Can you share a simple repro so that I can play with it?

[pavelfeldman]

We need more information to act on this report. Please file a new one and link to this issue when you get back to it!