search

joseph-roitman / pytest-snapshot

A plugin for snapshot testing with pytest.
MIT License
114 stars 12 forks source link

Bug: snapshot contents are ignored #49

Closed konstmsu closed 2 years ago

konstmsu commented 2 years ago

Snapshot contents are ignored and the tests pass even when snapshot doesn't match baseline.

https://github.com/joseph-roitman/pytest-snapshot/blob/2d5fb6c4086648ce366dbda7019820b9caf1aaca/pytest_snapshot/plugin.py#L40

https://github.com/joseph-roitman/pytest-snapshot/blob/2d5fb6c4086648ce366dbda7019820b9caf1aaca/pytest_snapshot/plugin.py#L187

Current tests don't reveal it because of pytest assert rewriting https://docs.pytest.org/en/latest/how-to/writing_plugins.html?highlight=pytest.register_assert_rewrite#assertion-rewriting

but can be reproduced if assertion rewriting is disabled:

❯ pytest --assert=plain 
========================================================================= test session starts =========================================================================
platform darwin -- Python 3.10.0, pytest-7.0.0, pluggy-1.0.0
rootdir: /Users/kaspir/projects/pytest-snapshot
plugins: snapshot-0.8.0
collected 105 items                                                                                                                                                   

tests/test_assert_match.py ...FF............................xF...                                                                                               [ 36%]
tests/test_assert_match_dir.py .F..........................                                                                                                     [ 62%]
tests/test_misc.py .......................................                                                                                                      [100%]
joseph-roitman commented 2 years ago

Hello, Thanks for the bug report. It was a bit confusing to understand what caused the failures. It turns out that the outer pytest and the inner pytest's assert rewriting behavior affects one another since they run in the same python process by default. This makes it tricky to find a good approach to testing assert=plain.

Tomorrow I will merge #50 which fixes the bug and tests it by mocking "assert=plain" behavior. Running the test suite with --assert=plain will still cause failures since the previous tests assume rewrite behavior.

Another approach that might be better would be if the tests ran in a subprocess with testdir.runpytest_subprocess('-v', '--assert=plain') though that might cause other problems: i.e. test speed, multi-process bugs, and runpytest_subprocess is more low-level than runpytest.

If you have any other ideas I'd be happy to listen.

joseph-roitman commented 2 years ago

This is fixed now in v0.8.1.