microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
66.98k stars 3.68k forks source link

[Bug]: playwright.config.ts - updateSnapshots: "missing" will always make the test fail, when there is no existing snapshot #31503

Open radekcols opened 4 months ago

radekcols commented 4 months ago

Version

1.44.1

Steps to reproduce

  1. have some test with toHaveScreenshot() method
  2. delete the current snapshot (if there is any) 3 . in playwright.config.ts file have updateSnapshots set to "missing"
  3. the test from point 1 will fail. the new snapshot will be created, though and next run will be successful

Expected behavior

new snapshot should be created and the test should pass

Actual behavior

the test will always fail.

Additional context

pls note, that if we set updateSnapshots to 'all', the new snapshot will be also created, but the test will pass.

Environment

System:
    OS: Windows 11 10.0.22631
    CPU: (12) x64 Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz
    Memory: 6.79 GB / 15.79 GB
  Binaries:
    Node: 22.2.0 - C:\Program Files\nodejs\node.EXE
    npm: 10.7.0 - C:\Program Files\nodejs\npm.CMD
  IDEs:
    VSCode: 1.90.2 - C:\Users\borsky\AppData\Local\Programs\Microsoft VS Code\bin\code.CMD
  npmPackages:
    @playwright/test: ^1.44.1 => 1.44.1
    playwright: ^1.39.0 => 1.44.1
mxschmitt commented 4 months ago

updateSnapshot: missing is the default behaviour, and we need to fail for it on CI. So unfortunately we cannot change this behaviour. We could introduce a new mode for that, marking it as a feature request.

radekcols commented 4 months ago

We could introduce a new mode for that, marking it as a feature request.

that would be awesome :)