syrusakbary / snapshottest

Snapshot Testing utils for Python 📸
MIT License
537 stars 102 forks source link

Should verify snapshots came from supported snapshottest version #143

Open medmunds opened 4 years ago

medmunds commented 4 years ago

Currently, the generated snapshot file includes "v1" in a header comment, indicating v1 of the snapshot file format. (I believe this approach was borrowed from jest-snapshot.)

Current snapshot file: (ignoring Python 2 compatibility lines)

# snapshottest: v1 - https://goo.gl/zC4yUc
from snapshottest import Snapshot

snapshots = Snapshot()

snapshots['TestDemo::test_api_me 1'] = {
    'url': '/me'
}

However, nothing actually checks that "v1" anywhere. If we change the snapshot file format in the future, that could lead to confusing errors or incorrect results if developers on a team somehow have different versions of snapshottest installed. (Which, you know, development teams should really try to avoid, but we don't control that.)

In particular, if developer A has snapshottest v0.6.0 installed, and developer B updates snapshots using some future, incompatible version, developer A's snapshottest won't be able to detect that case. (So we should really add version checking soon, not wait until we need to change the file format.)

Rather than trying to parse version numbers out of source comments, I suggest we use a module level variable, which could be easily checked in SnapshotModule.load_snapshots.

Proposed snapshot file:

# generated by snapshottest: https://pypi.org/project/snapshottest/
from snapshottest import Snapshot

_snapshottest_format = 1
snapshots = Snapshot()

snapshots['TestDemo::test_api_me 1'] = {
    'url': '/me'
}

(I've also changed the project link from the opaque goo.gl link shortener to PyPI. The PyPI project link should be stable for the lifetime of this package, allows moving the project home if ever desired, and is less mysterious.)

Some variations:

Also, for reference, Jest's version checking tests include their error message text, which we might want to borrow.

paulmelnikow commented 4 years ago

Great idea. I'd also consider backporting this change and releasing it as 0.6.1 for people who can't or don't make the leap to 1.x.