[Question/Bug suspition] Java - ScreenshotOptions().setAnimations(ScreenshotAnimations.DISABLED) seem to alter unanimated graphics

[mumaguma]

Hi, my test: 1) opens https://qualityminds.com/de/portfolio/ 2) makes screenshot of not-animated locator "ul.et_pb_social_media_follow" - which should look like this: 3) compares that against image of that element saved earlier

Finally I've got test working correctly, but earlier struggle with failures made me discover, that ScreenshotOptions for disabling animations alters the screenshot even if element is not animated.

byte[] elementScreenshot1 = myLocator.screenshot();
byte[] elementScreenshot2 = myLocator.screenshot(new Locator.ScreenshotOptions().setAnimations(ScreenshotAnimations.DISABLED));

Result:

Is it a defect of feature?

[pavelfeldman]

Good question, over to @aslushnikov

[aslushnikov]

@mumaguma comparing screenshots byte-by-byte is a wrong idea; these are some encoded images, after all, and these might yield different results.

For example, just taking two exact screenshots might yield different results. For example, the following prints "false" for me every other time:

import { chromium } from '@playwright/test';

const browser = await chromium.launch({ });

const page = await browser.newPage();
await page.goto('https://qualityminds.com/de/portfolio/');
const buffer1 = await page.locator('ul.et_pb_social_media_follow').screenshot();
const buffer2 = await page.locator('ul.et_pb_social_media_follow').screenshot();
console.log(buffer1.equals(buffer2));
await browser.close();

Instead, you should use some image comparison algorithm to assert that images are the same.

For example, the built-in image comparison in Playwright reliably reports screenshots as identical for me:

I know you're in Java and we don't provide image comparison in Java yet, so you'll have to fallback to third-party solutions for now.

Hope it helps!

[mumaguma]

Hi,

Instead, you should use some image comparison algorithm to assert that images are the same.

I'd love to, but you've provided a typescript solution. Java users are left with "compare images on your own hand" as far as I remember. So I'm using image-comparison library ... byte-by-byte comparison was to show here in the ticket that option alters image. Comparison kept on failing, when for one screenshot I've did not use disable animations option, and for another I used it. When both screenshots have same setting (both animations disabled, or both animaitons enabled), comparison is successful.

Also, in this particular case I need to compare screenshot against the saved image.

Anyway, your example in typescript that fails during comparison proves my point - even in typescript "animations: 'disabled'," seems to corrupt the screenshot.

[aslushnikov]

Anyway, your example in typescript that fails during comparison proves my point - even in typescript "animations: 'disabled'," seems to corrupt the screenshot.

@mumaguma vice-versa: the example shows that the same screenshot is byte-different. (the code block was a wrong snippet; i edited my comment to align codeblock with the screenshot.)

Comparison kept on failing, when for one screenshot I've did not use disable animations option, and for another I used it.

Would you mind providing with the code snippet and resulting diff?

[mumaguma]

Hi @aslushnikov, sure. The code that stores element screenshot to file was

 locator.scrollIntoViewIfNeeded();
        assertThat(locator).isVisible();
        byte[] elementScreenshot = locator.screenshot(new Locator.ScreenshotOptions().setAnimations(ScreenshotAnimations.DISABLED));   // HERE
        File outputfile = new File(elementScreenshotsPath+scrshotFileName);
        try {
            FileUtils.writeByteArrayToFile(outputfile, elementScreenshot);
(...)
        }

And this "generator" class would be run only, when manual operator is sure that at the moment page works ok. Screenshot files are stored in test repo and test running at preset times takes a fresh screenshot and compares with the screenshot from file.

Now the test is successfull only in place marked "HERE" in generator above and in test below:

• both use screenshot options to disable animations
• both have no screenshot options

        byte[] actualElement = locator.screenshot(new Locator.ScreenshotOptions().setAnimations(ScreenshotAnimations.DISABLED));      // HERE
        byte[] expectedElement = Files.readAllBytes(Paths.get(elementScreenshotsPath + resourcePath));

        float diffPerc = imageDifferenceMarkup(expectedElement, actualElement, elementExpectedThreshold, 0, 0);

where imageDifferenceMarkup core is using https://github.com/romankh3/image-comparison, with core code more or less:

   public static float imageDifferenceMarkup(byte[] expected, byte[] actual) {

        BufferedImage expectedBI = convertByteArrayToBufferedImage(expected);
        BufferedImage actualBI = convertByteArrayToBufferedImage(actual);
        // here goes dimension check and if needed - they are expanded to have same dimensions, as ImageComparison method requires same dimesions.

      ImageComparisonResult imageComparisonResult = new ImageComparison(expectedBI, actualBI).setThreshold(0)
                .setDifferenceRectangleFilling(true, 5)
                .setExcludedRectangleFilling(true, 5)
                .setRectangleLineWidth(4).setDifferenceRectangleColor(magenta)   
                .compareImages();
        float comparisonDifferencePercent = imageComparisonResult.getDifferencePercent();

        if (comparisonDifferencePercent > 0.0F)  {
        attachImageDiffToReport(expected, actual, convertBufferedImageToByteArray(imageComparisonResult.getResult(), "png"));
        }
}

[aslushnikov]

@mumaguma so what exactly does the diff look like? Can you share the diff image?

Regarding the script: any chance you can come up with something that I can run locally to reproduce your behavior?

[mumaguma]

Hi @aslushnikov - the one with animations disabled has some color artifacts. And images were saved from byte[] to png file with the same method.

- diff

I'll try to extract a working code example.

[mumaguma]

Hi @aslushnikov - I've extracted what's needed to reproduce/analyse. https://github.com/mumaguma/playwright-20354/tree/master after running main, result files (no-options, with-options, diff) will be stored in /results/

If I run main "as is", image difference is ~0.34%. If in row 24 I remove options for elementScreenshotWOAnimations, image difference is 0% If in row 24 I have options, and also copy them to row 23, difference is 0%.

So the difference is only between screenshot w/o options and screenshot with options. It is also visible if you zoom / resize pictures to see particular pixels. Using options for both, or not using for both creates consistent results, even if compared screenshots are from different hosts (dev / staging / production sites)

[yury-s]

@mumaguna, I tried running the test as is and without the screenshot option and the result is Percentage difference is 0.0 in both cases. Which playwright version and OS are you using?

Here is the output I got: results.zip

[mumaguma]

@yury-s - as in pom.xml, I used 1.29.0. Is it possible, that playwright uses some other libraries that may be different on our machines? Like it would be dependent on java version or something else?

[aslushnikov]

@mumaguma Looking closely at your diff, it looks like subpixel antialiasing kicked in for you. Are you using LCD monitor? Are you on Windows? Could you please try launching browser with the following set of flags and let me know if it helps?

--font-render-hinting=none
--disable-skia-runtime-opts
--disable-font-subpixel-positioning
--disable-lcd-text

[mumaguma]

This happened to me on:

• windows local machine with lcd, both headless and normal
• jenkins running ubuntu docker container (https://hub.docker.com/_/microsoft-playwright-java) in headless mode

Now the tricky part: something else must have changed, and now I get Percentage difference is 0.0 every time; even without adding browser args. I guess that as for today, this ticket could be closed. Thank you!