[Bug]: toHaveScreenshot method does not take maxDiffPixels or maxDiffPixelRatio inputs into consideration when there are visual differences and these thresholds are higher than the differences.

PramodKumarYadav commented 3 months ago

Version

1.45.0

Steps to reproduce

NOTE: I have the setup working on a customer project so I dont have a generically accessible code to reproduce this at the moment but am sure the same issue can be verified if we run tests in a headed (VSCode plugin -> Settings -> Show Browser) or headless mode on any UI application. If this is not the case, do let me know and I will try to create a reproducible example and share.

For now, here are the steps and issue.

Run tests only for a particular input field without giving any options in headless mode.
Command I use to run: npx playwright test --grep=user-mgmt-search --project=ux-tests. With this the below test passes:

test("user-mgmt-search", async ({ page, userManagementPage, }) => {
  await userManagementPage.goTo();
  const searchUserInput = page.getByPlaceholder("Start typing to search");
  await expect(searchUserInput).toHaveScreenshot({});
});

2 passed (9.2s)
::notice title=🎭 Playwright Run Summary::  2 passed (9.2s)

Now run the same test in headed mode by going to VSCode plugin -> Settings -> Show Browser option.
This time, run test from clicking the button from UI and verify results.
Test throws an error - Expected an image 537px by 47px, received 529px by 47px. 98 pixels (ratio 0.01 of all image pixels) are different.
Now go to test and change pixel thresholds above the ones shown above. i.e. more than 98 pixels or more than 0.01. I will set both for this particular test.

test("user-mgmt-search", async ({ page, userManagementPage, }) => {
  await userManagementPage.goTo();
  const searchUserInput = page.getByPlaceholder("Start typing to search");
  await expect(searchUserInput).toHaveScreenshot({
    maxDiffPixels: 100, // maximum number of different pixels
    maxDiffPixelRatio: 0.1, // maximum ratio of different pixels
    threshold: 0.5 // color comparison threshold (0-1)
  });
});

Run test again from UI.

Expected behavior

Since the ratios are higher than the differences, I expect the tests to pass.

Actual behavior

Even if I make these ratios much higher - even to their max limit, the test still fails with below error. - Expected an image 537px by 47px, received 529px by 47px.

Additional context

Due to this reason I am not able to use the visual comparison tests for isolated fields in our tests. Kindly let me know if I am doing something wrong here.

Environment

System:
    OS: macOS 14.0
    CPU: (8) arm64 Apple M2
    Memory: 60.63 MB / 16.00 GB
  Binaries:
    Node: 21.7.1 - /opt/homebrew/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.8.1 - /opt/homebrew/bin/npm
  Languages:
    Bash: 3.2.57 - /bin/bash
  npmPackages:
    @playwright/test: ^1.45.0 => 1.45.0

mxschmitt commented 3 months ago

The code you provided is unfortunately not complete - would it be possible to share a minimal reproduction which we can clone and run locally? Ideally with expected and actual outcome. Thanks for your understanding!

PramodKumarYadav commented 3 months ago

Will setup a working example today and share. Thanks for your prompt response @mxschmitt . Appreciate it.

PramodKumarYadav commented 3 months ago

@mxschmitt: Here are requested details:

Test to replicate the bug: https://github.com/PramodKumarYadav/playwright-sandbox/blob/main/tests/ux-tests/sample-field-visual.test.ts
Video showing the issue in headed mode (and not in headless mode): https://www.loom.com/share/7e54e172c8db4512a9bc79ff572b2190?sid=d1120c23-683f-4cb1-bba0-52668f500e59
Screenshot location is set in playwright.config.ts file here: snapshotPathTemplate:.screenshots/{testFilePath}/{arg}{ext},
Screenshot can be found here: https://github.com/PramodKumarYadav/playwright-sandbox/blob/main/.screenshots/ux-tests/sample-field-visual.test.ts/Validate-get-started-field-visual-test-1.png
Command to run test in headless mode is: npx playwright test --grep=visual-test --project=chromium

mxschmitt commented 2 months ago

Thank you for your loom, that was helpful! I tried to run your sample-field test locally and for me it passes headless and headed via the VSCode extension with the lowest maxDiffPixels: of 74. I tried doing it via VSCode and via the command line. When looking at your code, I see that you specify the ratio and the pixels. In this case, we take the minimum when both is specified, that might end up in some confusion on your end: https://github.com/microsoft/playwright/blob/f374f8db38c8ab1fe9674a1839bd9fdbc5023842/packages/playwright-core/src/utils/comparators.ts#L85-L86