microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
66.42k stars 3.63k forks source link

[QUESTION] How to set screen resolution (and not only viewport)? #27310

Closed hbaron12 closed 1 year ago

hbaron12 commented 1 year ago

System info

Config file

// playwright.config.ts
import {defineConfig, devices} from '@playwright/test';

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
// require('dotenv').config();

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({
    testDir: './tests',
    /* Run tests in files in parallel */
    fullyParallel: true,
    /* Fail the build on CI if you accidentally left test.only in the source code. */
    forbidOnly: !!process.env.CI,
    /* Retry on CI only */
    retries: process.env.CI ? 2 : 0,
    /* Opt out of parallel tests on CI. */
    workers: process.env.CI ? 1 : undefined,
    /* Reporter to use. See https://playwright.dev/docs/test-reporters */
    reporter: 'html',
    expect: {
        timeout: 10000,
        toHaveScreenshot: {
            threshold: 0.2,
            maxDiffPixels: 80,
            maxDiffPixelRatio: 0.2
        },
        toMatchSnapshot: {
            threshold: 0.2,
            maxDiffPixels: 80,
            maxDiffPixelRatio: 0.2
        }
    },
    timeout: 60000,
    /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
    use: {
        /* Base URL to use in actions like `await page.goto('/')`. */
        // baseURL: 'http://127.0.0.1:3000',

        /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
        trace: 'on-first-retry',
    },

    /* Configure projects for major browsers */
    projects: [
        {
            name: 'chromium',
            use:  {
                ...devices['Desktop Chrome'],
                contextOptions: {
                    screen: {
                        width: 1280,
                        height: 720
                    }
                },
                viewport: {
                    width: 1280,
                    height: 720
                }
            },
        }
    ],

    /* Run your local dev server before starting the tests */
    webServer: {
        command:             'npm run lib:storybook:development',
        url:                 'http://localhost:9000/',
        reuseExistingServer: true,
    }
});

Test file (self-contained)

export async function loadStory(page: Page, storyId: string) {
    const url = `http://localhost:9000/iframe.html?viewMode=story&id=components-${storyId}`;

    await page.goto(url, {waitUntil: 'load'});
}

test('More Than Five Levels Overflow Breadcrumbs', async ({page}) => {
    await loadStory(page, 'breadcrumbs--more-than-five-levels-overflow-breadcrumbs-story');

    await expect(page).toHaveScreenshot();

    const ellipsisButton = await page.getByText('...');

    await ellipsisButton.click();

    await expect(page).toHaveScreenshot();
});

Steps

Expected

When running the test on different screens, the first time on 24-inch (1920 870 resolution) and the second time on 13-inch (1440 758 resolution), the screenshot comparison succeeds.

Actual

When running the test on different screens, the first time on 24-inch (1920 870 resolution) and the second time on 13-inch (1440 758 resolution), the screenshot comparison failed due to "773 pixels are different" Screenshot 2023-09-26 at 12 30 49

Question

How can I ensure that screenshots remain the same regardless of the screen they run on (as long as nothing changes in the source code)?

yury-s commented 1 year ago

How can I ensure that screenshots remain the same regardless of the screen they run on (as long as nothing changes in the source code)?

You need to ensure that the tests are running in a standardized environment, to ensure consistency. This way, you can rule out variables introduced by different monitors or system settings.

The problem is related to the system's rendering differences rather than just the browser itself, it depends on the device pixel configuration of the connected monitor. It's outside the control of Playwright and occurs at the OS or hardware (GPU) level.

Closing as there is no planned worked in this direction.

yury-s commented 1 year ago

See also related issue: https://github.com/microsoft/playwright/issues/14460

TheophilusRuben commented 5 months ago

Hey there, try removing the Viewport options and it should work fine!