Custom comparison for files

BenjaminHabert commented 5 years ago

Feature request: provide a custom comparison function for files.

Current method

At the moment the method filecmp.cmp() is used to compare two files

https://github.com/syrusakbary/snapshottest/blob/master/snapshottest/file.py#L52

    def assert_value_matches_snapshot(self, test, test_value, snapshot_value, formatter):
        snapshot_path = os.path.join(test.module.snapshot_dir, snapshot_value.path)
        files_identical = filecmp.cmp(test_value.path, snapshot_path, shallow=False)
        assert files_identical, "Stored file differs from test file"

Proposed API change

It would be convenient to provide a comparison function when creating the FileSnapshot object:

def my_comparison(path1, path2):
    # my custom code here
    return True

snap = FileSnapshot('path/to/file', comparison=my_comparison)
snapshot.assert_match(snap)

This custom function could be called by FileSnapshotFormatter

My use case

I am creating screenshots of a website (with selenium and chromedriver). Although the screenshots are very similar, the files are slightly different depending on the environment (my local machine, a docker container in CI). I would like to implement a comparison function that checks that the image difference is no higher than x%.

I think this could be useful in other instances:

comparing audio files that may be slightly different
created file contains metadata that can change but we only want to compare the actual content

paulmelnikow commented 3 years ago

See also #38.

paulmelnikow commented 3 years ago

Could this be solved by adding the option to pass a custom comparator / assertion function to assert_match()?

BenjaminHabert commented 3 years ago

The API change you are proposing is indeed clearer and more explicit than what I suggested.

I haven't used snapshottest for a while ; I don't know what would be the impact on the current implementation. I am willing to look into it before December 1st if you are interested.

syrusakbary / snapshottest