Way to fail in CI when running tests and the test writes a new snapshot

[jakebolam]

Is there a way to fail when running tests and there will be a new snapshot written?

[yourbuddyconner]

I think this should probably be the default behavior, as CI should fail if your tests are not fully implemented.

[ygormutti]

Changing the default behavior is backward-incompatible, so IMHO this should be done in a major release. What do you think of providing a flag for that in a minor release and later flip its default value once the project accumulate breaking changes that justify a major release?

[paulmelnikow]

Or, go ahead and ship the major release with this change.

This is the default behavior I'd expect for a snapshot library.

[davidshepherd7]

I implemented this here: https://github.com/davidshepherd7/snappytest/commit/4450fda2db97a62890889505e67666dd9f5ae3dc

There's no command line flag to disable it, but I think it would be straightforward to add one.

Let me know if you want a pull request.

[medmunds]

So, just for comparison, Jest does write new snapshot files by default, unless you are running in CI. (See conditions on when to write in the jest-snapshot code. Whether or not you're "in CI" is based on the is-ci node package.) Not sure how relevant Jest's behavior is for Python snapshottest, but it is cited as this package's inspiration.

One other wrinkle is that with unittest, there's currently no way to update snapshot files except deleting them and re-running the tests. (Raw unittest doesn't support command line args—see #51. This isn't an issue for the Nose and Django unittest derivatives.)

[davidshepherd7]

So, just for comparison, Jest does write new snapshot files by default, unless you are running in CI.

Good to know, but I think this corresponds to a difference between the philosophies of python vs js. Python says "explicit is better than implicit" but in JS it's common for things to guess what you want if you aren't explicit.

[kam193]

I think it's kind of behavior we will never choose best for everyone (I agree it should fail by default, but e.g. in my use with IDE it would be irritating to change the command on every test update). Maybe it's time to design some managing settings feature? I think there will be the best when everyone could adjust the behavior to their needs and environment.

I propose something like settings with following apply order:

1. Default settings (or maybe allow to change them by code).
2. Settings from file - e.g. section in project.toml
3. Env variable - I propose one env with syntax as command line arguments
4. Arguments from command line.

With this, settings could be applied in every environment, with every test runner (even e.g. CI with unittest, since we could set env variable). We eventually could also move all existing configuration to the one place (e.g. some singleton object?) to easier manage it and document (including my propose from #149). What do you think?

[samylaumonier]

Hey guys, here is my workaround:

File conftest.py:

from snapshottest.module import SnapshotModule

def pytest_sessionfinish(session):
    """Make the CI fail when there are some created or deprecated snapshots"""
    count_new_snapshots, _ = SnapshotModule.stats_new_snapshots()
    count_unvisited_snapshots, _ = SnapshotModule.stats_unvisited_snapshots()

    if count_new_snapshots or count_unvisited_snapshots:
        session.exitstatus = 1