Output failing tests in 'ImageDiff' folder

[hollanderbart]

Hi @stephencelis,

Thanks for creating this snapshot testing framework! I was comparing this framework with the other snapshot framework (https://github.com/uber/ios-snapshot-test-case)(FBSnapshotTestCase) to know which features are supported.

What I like about this framework is the API, in which you can easily specify the device (iPhoneSE or iPad), while for with the FBSnapshotTestCase you have to run the test again on a different device (e.g. it will take the screen size from the simulator instead of via the test itself).

What I do like about the FBSnapshotTestCase framework is that it will output a folder called 'ImageDiff', which contains all the failing snapshots including the reference image and an image highlighting the difference. With this framework, you don't do that but output a git diff. I believe that having an output folder containing the failing test results in better developer experience and would be very helpful when integrating in a CI system.

Is it possible to also create such a feature where you can have an 'ImageDiff' folder containing the failing tests?

Thanks ✌️

[stephencelis]

Hi @hollanderbart! We output the failing snapshot to a temporary directory which can be reconfigured using an environment variable: SNAPSHOT_ARTIFACTS. I believe this is a partial solution to your problem: partial because we only output the new, different snapshot, we don't save the diff.

The "git diff" output is actually a configurable graphical diff tool output. You can even set SnapshotTesting.diffTool = "ksdiff" to get a copy-pastable output from a failing test.

And if you're using Xcode, we produce that image-based diff you like as an XCTAttachment. You can view these attachments from the "Report navigator".

All of this could use some better documentation, and we should definitely save the diff image to the temporary directory, too, for inspection, so I'll keep this issue open for now. We'll try to add this feature in the future, but if you'd like to contribute, let us know and we'll try to give you a push in the right direction!

[SpacyRicochet]

Something that would be interesting is if, at the end, you output a single copy-paste concatenated ksdiff command that you can put into a shell and immediately open all failure differences that way.

Not sure how feasible that is without manual script additions though.

(Listening to @hollanderbart speaking about this right now 😋)

[hollanderbart]

Agreed with @SpacyRicochet. In the ideal situation, I would like to see an 'ImageDiff' folder containing the 3 images (reference / test / diff) and a small script that would launch the ksdiff command. I will investigate a bit more, thanks to the comment of @stephencelis.

[hollanderbart]

Oh yeah. Yesterday, I gave a talk about implementing snapshot testing and I also spoke about this framework! It was recorded so I'll share you the link, once the video is uploaded 📹

[stephencelis]

@hollanderbart Nice!

And yeah the infrastructure is already in there for where to save the artifacts using a SNAPSHOT_ARTIFACTS environment variable (which defaults to a temporary directory). Just need to write the reference and diff image, which are currently only passed to Xcode as attachments to the test report.

[hollanderbart]

After the presentation @skyylex came to me to share that there's a github page with a script to create an image diff. Basically, it's the same as what Kaleidoscope does. I think it's worth to check it out!

[stephencelis]

@hollanderbart Have you checked out the Xcode report navigator and looked at the test report? We already generate an image diff there. We just don’t currently write it to disk, though we really should and will in the future!

[hollanderbart]

Yes, I rerun the test and in the report there's 'reference', 'failure' and 'difference'.

When I change the SNAPSHOT_ARTIFACTS in enviorment variables and rerun the tests I get a failed test with the following message;

failed - You don’t have permission to save the file “ListItemTest” in the folder “INGStyleKitSnapshotTests"”.

Is there something I'm missing to have it save the 'failure' snapshots in a different folder besides temp?

[lvnkmn]

@SpacyRicochet considering SNAPSHOT_ARTIFACTS is set to a folder named __FailedSnapshots__ in the same path as __Snapshots__ and Kaleidoscope and ksdiff are installed

The command cd pathToSnapshots ; ksdiff __FailedSnapshots__ __Snapshots__ has been giving me pretty much the desired result, because Kaleidoscope seems to work quite well with folders:

[adamwolf03]

@lvnkmn - Thanks for the suggestion! I just recently started trying this out as a replacement to FBSnapshot and I was really missing the ksdiff command from the test logs.

@stephencelis - Thanks for creating this, it is amazing!

I'd really like access to the failure diff that isn't written to disk. Is this something that still seems useful? I'd be happy to submit a PR but wanted to confirm this is something that seems useful.

My preferred workflow is to use TowerApp for source control and typically just looking at the diff file is sufficient to understand the problem. If not, then I use Kaleidoscope (ksdiff)

On a related note, is there documentation besides this issue on how to setup ksdiff? I'd be happy to write that up too. It is a good workaround that is easy to add.

[hlung]

considering SNAPSHOT_ARTIFACTS is set to a folder named FailedSnapshots in the same path as Snapshots

How did you do this? I'm wondering because each Snapshots folder can be in a different root folder, how did you specify the path?

[lvnkmn]

@hlung I think I only did this one a per folder basis at the time

[rkolevDevex]

@hollanderbart Have you checked out the Xcode report navigator and looked at the test report? We already generate an image diff there. We just don’t currently write it to disk, though we really should and will in the future!

Hey do you still plan on writing the image diff and reference image to disk? I am switching from https://github.com/uber/ios-snapshot-test-case)(FBSnapshotTestCase as well and for better readability on CI from non developers this 2 were really useful.