[Bug]: attachments are wrong/confusing when `expectScreenshot` fails on time limit

urugator commented 2 weeks ago

Version

1.42.1

Steps to reproduce

Create test, where expectScreenshot fails, because it's unable to capture stable snapshot within a time limit. Attachments will contain 3 screenshots: actual (last screenshot taken), expected (golden snapshot) and diff. Normally, if expectScreenshot fails on comparison, it creates diff between actual and expected. However, if expectScreenshot fails because it's unable to capture same consecutive images (stable snapshot) within a time limit, it produces a diff between the actual and the previous (attempt): https://github.com/microsoft/playwright/blob/3f085d568935b9a4422f689185f720c1206bac84/packages/playwright-core/src/server/page.ts#L644 The problem is that previous isn't provided as an attachment and therefore you see diff that is completely unrelated to the other two screenshots. Here is the caller, which ignores previous and assumes the returned diff is always between actual and expected https://github.com/microsoft/playwright/blob/3f085d568935b9a4422f689185f720c1206bac84/packages/playwright/src/matchers/toMatchSnapshot.ts#L426-L439

Sorry for not providing an actual test.

Expected behavior

expected attachment should be the previous screenshot or attachments should include an extra previous screenshot.

Actual behavior

Diff image is not a diff between actual and expected images when expectScreenshot is unable to capture stable snapshot within a time limit.

Additional context

No response

Environment

System:
    OS: Windows 10 10.0.19045
    CPU: (4) x64 Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
    Memory: 1.18 GB / 7.98 GB
  Binaries:
    Node: 20.13.1 - C:\Program Files\nodejs\node.EXE     
    npm: 10.5.2 - C:\Program Files\nodejs\npm.CMD        
    pnpm: 9.1.3 - C:\Program Files\nodejs\pnpm.CMD
  IDEs:
    VSCode: 1.92.2 - C:\Users\urug\AppData\Local\Programs\Microsoft VS Code\bin\code.CMD
  Languages:
    Bash: 5.0.17 - C:\WINDOWS\system32\bash.EXE
  npmPackages:
    @playwright/test: 1.42.1 => 1.42.1

caineblood commented 2 weeks ago

The above comment asking you to download a file is malware to steal your account; do not under any circumstances download or run it. The post needs to be removed. If you have attempted to run it please have your system cleaned and your account secured immediately.

Skn0tt commented 2 weeks ago

This sounds more like a feature request than a bug report to me. Please open a feature request and include both your usecase and an actual reproduction repository.

microsoft / playwright