microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
66.91k stars 3.67k forks source link

How to disable Playwright's auto-renaming of the snapshot filename #22168

Closed brk9394 closed 1 year ago

brk9394 commented 1 year ago

Discussed in https://github.com/microsoft/playwright/discussions/17404

Originally posted by **brk9394** September 16, 2022 Is there a way to disable Playwright's auto-renaming of the snapshot filename? Specifically, I'd like it to not include the browser name in the filename.
For example: If this assertion is in the test named '@myTest': `expect(await myLocator.screenshot()).toMatchSnapshot('snapshot.png');`
Then run the test to generate the snapshot file (the file generated if it doesn't exist): `npx playwright test --grep '@myTest' --browser=firefox`
The new generated filename contains `firefox` in the name: `snapshot-firefox-win32.png`
However, if the browser is set in `playwright.config.ts` instead of from the command-line e.g. `use: { browserName: 'firefox' }`)
Then run the test with: `npx playwright test --grep '@myTest'`
The resulting generated filename does not contain `firefox`: e.g `snapshot-win32.png`
It would be ideal if the different methods of using a specific browser generated the same snapshot filename. But since it doesn't, I'd like to manage the filename myself to include the browser name, headed/headless mode, etc.
UPDATE: After further testing, I've discovered that the snapshot filename is generated using the project name, and not the browser name (although they could be the same). When the command-line argument `--browser=firefox` is passed, then the `testInfo.project.name` is populated with `firefox`, which is then used in the generated filename.
Also, if a project is added to `playwright.config.ts`, then the `projects.name` from the config is used in the generated filename. E.g. After adding `myProject` to the config: ``` projects: [ { name: 'myProject', use: { ...devices['Desktop Firefox'], }, }, ] ``` Then generate a new snapshot file again using `--project` e.g. `npx playwright test --grep '@myTest' --project=myProject`
The resulting generated filename is: `snapshot-myProject-win32.png`
To me this seems like a bigger problem than what I first thought, since that if/when I implement projects in the config, then I'd also have to generate/maintain snapshots for each project as well, in addition to each browser, headed mode, and OS (due to differences in the base snapshots for rendering variations).
pavelfeldman commented 1 year ago

https://playwright.dev/docs/next/api/class-testproject#test-project-snapshot-path-template - are you looking for this?

brk9394 commented 1 year ago

Thanks @pavelfeldman, yes indeed that looks like it should solve my problem. I'll close this issue and create a new discussion/issue if I run into any new problems with the snapshotPathTemplate option.

However, I think there may still be a bug with the inconsistent behaviour in the filename generation when running the CLI --browsers param vs. using projects in the config (although I haven't tested this using snapshotPathTemplate since PW v1.28).

To me it seems logical that if using either method it should create the same default snapshot filename. But of course that's up to you guys to determine if this is by design or not.

My apologies for not reading the latest Release Notes, I haven't kept up with all the recent Playwright changes. My plan is to binge-watch all the release notes YT's this weekend :). Thank-you and all the PW team for such a fantastic testing tool and great support!