search

microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
65.51k stars 3.57k forks source link

[Question] Does `testConfig.updateSnapshots` have the ability to remove stale snapshots? #16582

Open virtuoushub opened 2 years ago

virtuoushub commented 2 years ago

When I first ran playwright test -u I expected it to remove any snapshots that were no longer related to any of my tests.

Does playwright have the similar ability to jest where it can remove "stale"/outdated snapshots?

I have tried out using updateSnapshots types "none" | "all" | "missing" in my config, and nothing seemed to result in the desired behavior.

Resources look at:

community resources

similar question #19597

aslushnikov commented 2 years ago

There's no such ability as of now.

Nhasa commented 2 years ago

I'd love to see this as build in feature in playwright as well.

leotg130 commented 1 year ago

This would be a nice feature.. Seems like there was another issue requesting this: https://github.com/microsoft/playwright/issues/19597

One obvious difficulty is when running the tests on multiple CI machines (sharding).

Jest snapshot feature supports removing obsolete snapshots, might give some clues as to approach this, though I don't know if they support sharding. Also I think Jest puts all the snapshots for a test in a single file, and thus has full ownership of said file. PW on the other hand usually puts snapshots into a directory, so it would have to be a dedicated directory for PW to have authority to just delete stuff.

Simply deleting all snapshots and running tests (to regenerate snapshots) isn't really viable option for CI, and I'd like for CI to enforce no obsolete snapshots.

Having the option of PW deleting obsolete snapshots would be sufficient, since CI can detect the changes to branch, though would be nice also to have a flag which causes failure if there are excess snapshots. On CI it's nice to enforce strictness: all snapshots match and there's nothing extra.

edumserrano commented 1 year ago

If it helps others, the way I've dealt with this was to create a custom script to provide this functionality. The script works like this:

  1. Run the tests using the --update-snapshots and setting the snapshot directory to be different than the default snapshots dir. Let's call the new snapshot directory the expected snapshots dir and the default snapshots dir the actual snapshots dir.
  2. Compare the snapshot files on the actual snapshots dir with the expected snapshots dir
  3. Delete the diff. Delete the snapshots that are found in the actual snapshots dir but don't exist on the expected snapshots dir.

Here is a gist that shows the script. Do read the first comment on the script that explains a bit more on how I'm using it.

Hope this helps others until we get this supported out of the box 🤞

edumserrano commented 8 months ago

I've decided to improve the initial version of the script provided on the gist from my previous comment.

I created the edumserrano/playwright-adventures repo to share some of the experiences I've had whilst using Playwright and one of them is a solution to deleting stale snapshots.

Check out the stale-screenshots-cleanup demo.

t3chguy commented 2 months ago

As a stopgap for this I added a custom reporter to fail CI when stale screenshots are present

https://github.com/matrix-org/matrix-react-sdk/pull/12743