Open manuman94 opened 2 years ago
Do you see the screenshots that don't include <canvas/>
element?
Looks like another issue caused by different anti-aliasing behavior is different between headed and headless modes. Unfortunately there is no good way to make the browsers produce identical images in both modes. We are adding additional settings to toMatchSnapshot
matcher in the next release (see #12167), namely pixelCount
and pixelRatio
. They will enable you to increase the tolerance level of the diff algorithm to allow for small discrepancies like this. You can already give it a try with npm i @playwright/test@next
Hi @yury-s,
Yes, we have a test that uses SVGs and the screenshots doesn't match between headful and headless. In fact, the screenshots above are of an SVG, it's not a canvas object.
Short HTML code behind the screenshots:
Okay! So the only way to avoid this issue is to play with tolerance, as I understand it is a browser issue.
Thank you so much for your support.
Greetings.
@manuman94 the maxDiffPixels
and maxDiffPixelRatio
were released yesterday in 1.20 release.
Summary: browsers do render differently in headless and headed modes, so comparing headless screenshots with headful screenshots becomes complicated.
Rendering issues include:
Our options:
screenshotsDir
path?
@aslushnikov @dgozman Seems that maxDiffPixels doesn't have any effect on the test
the 15 pixels are the difference between headed and headless mode
the 15 pixels are the difference between headed and headless mode
maxDiffPixels
only comes into play when image dimensions are equal.
@aslushnikov The problem with our case is that headed vs headless renders the image with one pixel difference of dimensions. I'm not sure I understand how maxDiffPixelRatio
will work. Does it also require maxDiffPixels
to be set? Or does it replace this?
I am working with a defect where a component shouldn't display a scrollbar, when I run with headed mode on Chromium or Firefox it shows it and I can throw the error with toMatchSnapshot()
, but it doesn't show up when it's running in headless mode (local or in CI).
Having a option to enable scrollbars on headless mode would make total sense as we're testing Visual here.
@aslushnikov it seems you already have an prepared option for that in browser_patches/firefox/juggler/content/main.js#L61
is it possible to make it available as option?
what is the status on this? Noticing these issues when creating baseline images from headless / headed and vice versa. Normally I'm able to get away with an insane .0001 maxpixeldifference!!
When running cross headed / headless not even .01 get's them to pass
@aslushnikov
I'm noticing maxDiffPixels: 5, is not working.... On ci there is a 1 pixel difference of the snapshot say 1x1 vs 1x2 and it fails. Tried 1000 pixels still fails
Expected an image 60px by 125px, received 60px by 124px......
@aslushnikov Hello! I have exactly the same problem, the size of the element differs by 1 - 2 pixels. Screenshot comparison failed: Expected an image 640px by 694px, received 640px by 696px.
This appears for both page.screenshot and element.screenshot.
For example, in jest-image-snapshot package https://www.npmjs.com/package/jest-image-snapshot there is an allowSizeMismatch parameter that allows you to solve this problem.
@playwright/test 1.27.1
Please find a solution to fix this problem as soon as possible. Due to this problem, I can not completely switch to playwright without using jest
Please find a solution to fix this problem as soon as possible.
I think this is an inappropriate comment considering that the software is provided to you free of charge and without warranty
Please find a solution to fix this problem as soon as possible.
I think this is an inappropriate comment considering that the software is provided to you free of charge and without warranty
I agree with you, I don't mean to offend the playwright team. My mistake. I really love this product and enjoy using it. I just found a few requests about this problem, so I asked to try to solve it.
Would the new Chrome headless mode (not yet officially supported in Playwright) solve this issue? https://developer.chrome.com/articles/new-headless/#whats-new-in-headless
As I understand it, the performance drops quite a bit, but for me, if the rendering becomes identical to the headful browser, and if it increases the usability of visual testing (developing in headful mode and running in CI in headless mode), then I would prefer the new headless mode of Chrome.
aslushnikov +1 to "shall we introduce an option to enable scrollbars in headless mode?"
But perhaps more importantly, the new headless mode that @eveltman-twinfield mentioned above sounds even better. A relevant quote from that article: "This created a confusing situation where any automated browser test might pass in headful mode but fail in Headless mode, or vice versa — a major pain point for automation engineers."
What's needed to try out the new chrome --headless=new
flag?
@eveltman-twinfield mentioned that this isn't officially supported in Playwright yet, but should I be able to try it by using the latest Playwright version, setting browser launch option Headless
to false, and then pass --headless=new
in the Args
browser launch option? :)
EDIT: Above worked out for me. Screenshot comparison now acts as expected for us no matter if we run headless (=new) or with head.. 👏
Since https://github.com/microsoft/playwright/commit/b21c81d3448dd47277997683d042c21aab095978 there is a new env var option PLAYWRIGHT_CHROMIUM_USE_HEADLESS_NEW
but it seems to be undocumented (?)
Hi, are there any updates on this bug. I am currently running into this issue between headless and headed modes. I am currently experiencing this issue.
Hello, im running in this issue with the width difference in headless creation & headless test. So taking snapshots with a headless environment on the same machine and testing against those snapshots on the same machine produces :
Expected an image 1125px by 229px, received 1133px by 229px.
I tested to define the viewport without success. I tested chrome without success.
I guess I will crop the resulting snapshots, so the test might work.
System:
Binaries:
Languages:
Playwright env.:
The bug
Hi there! We are testing a kind of "analytics" web where some charts and diagrams are shown using amCharts library. The first approach is being made with snapshot testing.
The issue comes with head/headless option, as we develop with headless option set to
false
and the CI environment uses headless executions.When the snapshots are made with headless option enabled and then we launch the tests in headful mode the tests in Chromium and Webkit are failing, the images are not equal. This doesn't happen with Firefox, that makes identically component screenshots.
Evidences
We made a simple project that enters an amChart example and makes a snapshot of a pie chart component. The test consist on comparing a snapshot made with headless option set to
true
and then repeat the test in headful mode (and vice versa).https://github.com/manuman94/playwright-head-vs-headless-rendering
Further, I tried to compare the images using pixelmatch library (which Playwright is using too I believe) and get the image difference. Here you can see the both images and the difference image: Headless:![headless](https://user-images.githubusercontent.com/6721904/157857322-aae6224e-54bf-4eb3-96f9-a244d9edf7c3.jpeg)
HeadFul:![noHeadless](https://user-images.githubusercontent.com/6721904/157857310-945baa23-ec08-4ed9-b1da-3659957bdcec.jpeg)
Difference:![difference](https://user-images.githubusercontent.com/6721904/157857333-dcb4d673-3856-43c7-8b00-900a9f0609b7.jpeg)
Previous issues
There was an issue opened on Jan 4th 2021 (https://github.com/microsoft/playwright/issues/11177) regarding this issue, but as for today is closed and a pull request was created to solve the font family problems.
There is another closed issue (https://github.com/microsoft/playwright/issues/5371) but it seems that it was reporting a completely different problem.
Side notes
Thank you so much, greetings.