Snapshot testing framework reports xfail tests as failing.

Textualize / textual

The lean application framework for Python. Build sophisticated user interfaces with a simple Python API. Run your apps in the terminal and a web browser.

https://textual.textualize.io/

MIT License

25.04k stars 764 forks source link

Snapshot testing framework reports xfail tests as failing. #2282

Open rodrigogiraoserrao opened 1 year ago

rodrigogiraoserrao commented 1 year ago

When a snapshot test is marked as xfail, pytest understands the test was expected to fail but the framework still reports a failing test.

What I expected:

A yellow notice saying that 1 snapshot x-failed instead of a red notice saying 1 failed.
As for the snapshot report, it being generated is probably still useful.

This makes it look like the tests failed:

Screenshot 2023-04-13 at 14 59 29

But pytest is perfectly happy with an xfail:

Screenshot 2023-04-13 at 14 59 45

rodrigogiraoserrao commented 1 year ago

After talking to @darrenburns this might be a bug/design decision of syrupy.