tophat / syrupy

:pancakes: The sweeter pytest snapshot plugin
https://tophat.github.io/syrupy
Apache License 2.0
501 stars 33 forks source link

Incorrectly delete unused snapshot in pytest bdd when target run using -k filter #770

Open mawaliya opened 1 year ago

mawaliya commented 1 year ago

Describe the bug

In the pytest using pytest_bdd. When we have several test using scenario outlines. When different scenario outlines have the same examples, then we run targeted test into only one example of a scenario outline using -k filter. Syrupy with JSONSnapshotExtension extension, will think other snapshots with same example from different scenario outlines are unused.

To reproduce

  1. Have tests in bdd format like this: my_feature.feature

    @test1
    Scenario Outline: Test1
        Then The result is <file_type>
    
        Examples:
            | file_type |
            | JSON      |
            | CSV       |
    
    @test2
    Scenario Outline: Test2
        Then The result is <file_type>
    
        Examples:
            | file_type |
            | JSON      |
            | CSV       |
  2. Implement the test and snapshot using JSONSnapshotExtension extension. test_feature.py
    
    import pytest
    from pytest_bdd import parsers, scenarios, then
    from syrupy.assertion import SnapshotAssertion
    from syrupy.extensions.json import JSONSnapshotExtension

scenarios("my_feature.feature")

@pytest.fixture def custom_snapshot(snapshot: SnapshotAssertion): return snapshot.use_extension(JSONSnapshotExtension)

@then(parsers.cfparse("The result is {file_type}")) def check_the_file_type(file_type: str, custom_snapshot: SnapshotAssertion): assert file_type == custom_snapshot


3. Run the snapshot generation
```pytest --snapshot-update```
4. Modify Test1  JSON snapshot to fail the test
5. Run the targeted test only for test 1 JSON example
`pytest -m 'test1' -k 'JSON'`
6. Observe that the run will suggest unused snapshots for Test2 JSON.

**Expected behavior**

<!-- A clear and concise description of what you expected to happen. -->
Snapshot from other test with the same examples are not considered unused.

**Screenshots**

<!-- If applicable, add screenshots to help explain your problem. -->
![image](https://github.com/tophat/syrupy/assets/30797285/b4e97fe2-2b35-46ab-b274-fd60194fa534)

**Environment (please complete the following information):**
wsl Ubuntu, Windows 11
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

platform linux -- Python 3.10.6, pytest-7.3.2, pluggy-1.2.0
configfile: pytest.ini
plugins: metadata-3.0.0, bdd-6.1.1, allure-pytest-bdd-2.13.2, bdd-html-0.1.14a0, html-3.2.0, syrupy-4.0.6, check-2.1.5

**Additional context**

<!-- Add any other context about the problem here. -->
BarrensZeppelin commented 5 months ago

I don't think this is a tool compatibility issue.

Consider a pytest file like:

import pytest

@pytest.mark.parametrize("param", ["one", "two", "three"])
def test_parameterized(param, snapshot):
    assert param == snapshot

If you run pytest -k one you get the correct output:

collected 3 items / 2 deselected / 1 selected                                                   

test_syrupy.py .                                                                          [100%]

------------------------------------ snapshot report summary ------------------------------------
1 snapshot passed.
================================ 1 passed, 2 deselected in 0.02s ================================

But if you run pytest -k 'not one' you get an error for unused snapshots:

collected 3 items / 1 deselected / 2 selected                                                   

test_syrupy.py ..                                                                         [100%]

------------------------------------ snapshot report summary ------------------------------------
2 snapshots passed. 1 snapshot unused.

Re-run pytest with --snapshot-update to delete unused snapshots.
================================ 2 passed, 1 deselected in 0.02s ================================

EDIT: This is a completely separate issue related to syrupy's re-implementation of test selection with -k.