vberlier / pytest-insta

A practical snapshot testing plugin for pytest.
MIT License
68 stars 1 forks source link

feature request: snapshot name suffix #225

Open tamird opened 8 months ago

tamird commented 8 months ago

Thanks for making this! Insta is great.

Suppose I am testing the old and new versions of an API - I may want two flavors of every snapshot. Naively I'd write:

assert snapshot("v2.json") == (await client.get_new_api()).json()
assert snapshot("v1.json") == (await client.get_old_api()).json()

That would work fine as long as I called this only once. But what happens when my test wants to call each API multiple times? If I run with --insta update, I end up with the last response saved. If I run without --insta update, the test obviously fails.

There are two problems here:

I'd be happy to send a pull request! Let me know how you feel about this.

tamird commented 8 months ago

There's another issue with not being able to specify a suffix:

vberlier commented 8 months ago

There's no concept of a "suffix". Either you specify the full name, and the extension is extracted to select the correct snapshot format, or you just specify the extension, and the name is generated for you.

So here snapshot("v2.json") uniquely identifies a specific snapshot for the current test. You can only match it multiple times if all the assertions are supposed to produce the same output. So for example matching the same snapshot in a loop with different inputs won't work.

The closest thing to a suffix would be making pytest-insta recognize a custom extension for your different API versions by defining a custom snapshot format in your conftest.py:

# conftest.py
from pytest_insta import FmtJson

class FmtApiV1(FmtJson):
    extension = ".v1.json"

class FmtApiV2(FmtJson):
    extension = ".v2.json"

Then the code you showed earlier will generate a unique name with the custom extension.

Otherwise I'd suggest writing separate tests for v1 and v2, or using a parametrized test with a single snapshot("json") assertion that will run for each API version:

# test_api.py
@pytest.mark.parametrize("api_version", ["v2", "v1"])
async def test_foo(api_version, client):
    if api_version == "v2":
        result = await client.get_new_api()
    else:
        result = await client.get_old_api()
    assert snapshot("json") == result.json()

The generated snapshot name will automatically take into account the parameterized test (e.g.: api__foo__v2__0.json).

The snapshot counter resets for each parameter so the index won't "slide" and invalidate existing snapshots if you add or remove api versions this way.

Depending on your use case you could even bundle this into a fixture and have a hybrid client for tests:

# conftest.py
@pytest.fixture(scope="module", params=["v2", "v1"])
def hybrid_client(request):
    if request.param == "v2":
        return ClientV2()
    else:
        return ClientV1()
# test_api.py
async def test_foo(hybrid_client):
    assert snapshot("json") == (await hybrid_client.get_common_api()).json()
    if isinstance(hybrid_client, ClientV2):
        assert snapshot("json") == (await hybrid_client.get_new_api()).json()
    if isinstance(hybrid_client, ClientV1):
        assert snapshot("json") == (await hybrid_client.get_old_api()).json()

Hope this helps :)

tamird commented 8 months ago

There's no concept of a "suffix". Either you specify the full name, and the extension is extracted to select the correct snapshot format, or you just specify the extension, and the name is generated for you.

So here snapshot("v2.json") uniquely identifies a specific snapshot for the current test. You can only match it multiple times if all the assertions are supposed to produce the same output. So for example matching the same snapshot in a loop with different inputs won't work.

Yep, this is what I wrote in the issue description.

The closest thing to a suffix would be making pytest-insta recognize a custom extension for your different API versions by defining a custom snapshot format in your conftest.py:

# conftest.py
from pytest_insta import FmtJson

class FmtApiV1(FmtJson):
    extension = ".v1.json"

class FmtApiV2(FmtJson):
    extension = ".v2.json"

Then the code you showed earlier will generate a unique name with the custom extension.

This would work, but it requires a separate class for every version, which is quite tedious.

Otherwise I'd suggest writing separate tests for v1 and v2, or using a parametrized test with a single snapshot("json") assertion that will run for each API version:

# test_api.py
@pytest.mark.parametrize("api_version", ["v2", "v1"])
async def test_foo(api_version, client):
    if api_version == "v2":
        result = await client.get_new_api()
    else:
        result = await client.get_old_api()
    assert snapshot("json") == result.json()

The generated snapshot name will automatically take into account the parameterized test (e.g.: api__foo__v2__0.json).

This works of course, but imagine that I have a test with with very expensive setup - I don't want to repeat that setup for each test case. I could do something complicated with fixture scopes, but it doesn't work in the most general case (e.g. where something about the test case wants to change the setup, but I still want to test all API versions).

The snapshot counter resets for each parameter so the index won't "slide" and invalidate existing snapshots if you add or remove api versions this way.

Depending on your use case you could even bundle this into a fixture and have a hybrid client for tests:

# conftest.py
@pytest.fixture(scope="module", params=["v2", "v1"])
def hybrid_client(request):
    if request.param == "v2":
        return ClientV2()
    else:
        return ClientV1()
# test_api.py
async def test_foo(hybrid_client):
    assert snapshot("json") == (await hybrid_client.get_common_api()).json()
    if isinstance(hybrid_client, ClientV2):
        assert snapshot("json") == (await hybrid_client.get_new_api()).json()
    if isinstance(hybrid_client, ClientV1):
        assert snapshot("json") == (await hybrid_client.get_old_api()).json()

Hope this helps :)

Again, this doesn't solve for the scenario I describe above :(

vberlier commented 8 months ago

Then I'd simply recommend making your own wrapper for implementing a naming scheme that suits your specific use-case:

# conftest.py
@pytest.fixture
def api_snapshot(snapshot: SnapshotFixture):
    d: dict[str, int] = {}

    def wrapper(version: str):
        i = d.setdefault(version, 0)
        d[version] = i + 1
        return snapshot(f"{i}_{version}.json")

    return wrapper
assert api_snapshot("v2") == (await client.get_new_api()).json()
assert api_snapshot("v1") == (await client.get_old_api()).json()

This way you can have independent counters for all API versions without relying on a parametrized test. This will prevent snapshot indices from clashing when you add or remove versions, and you're free to expand it further to meet any additional requirement you might have.