Support snapshot testing on iOS devices

[darknoon]

Snapshot testing is great and saves an immense amount of time vs manually checking data structures or images / tests. However, some tests need to be run on actual devices for correctness.

Why

One use case is apps that make heavy use of Metal API

• metal rendering on actual devices can vary, we want to test the result is correct on a range of them
• GPU features vary across devices, want to make sure rendering is equivalent
• Metal Performance Shaders is not supported on iOS simulator
• macOS doesn't support the texture formats as iOS, so can't run as a Mac test
• we want to test performance, but have to make sure it's still correct!

The way that swift-snapshot-testing tracks snapshot data currently does not work when the test app does not run where the source root is accessible (iOS, macOS sandbox). This has been mentioned before, eg https://github.com/pointfreeco/swift-snapshot-testing/issues/193 , perhaps https://github.com/pointfreeco/swift-snapshot-testing/issues/176#issuecomment-467328614

Proposed approach

• __Snapshot__ directories are copied into test bundle before testing begins
• When test runs, snapshots are verified against snapshots on device
• If a snapshot should be updated, it is written to an XCTAttachment
• After tests execute, a post-test script copies any recorded results from DerivedData/Project-XXXXX/Logs/Test-Project-YYYY.xcresult back to the source root

Related

• I may need to add info about the target device to paths as proposed in https://github.com/pointfreeco/swift-snapshot-testing/issues/182#issuecomment-468774418

Going to see if I can get this working, interested to hear if anyone else is trying something similar

[PauliusLiekis]

Any luck with this? I can not run the tests on a simulator, thus I need to run them on a real device too...

[PauliusLiekis]

You can actually make it work without modifying the library. Just use verifySnapshot which takes snapshotDirectory as an argument. Then all you have to do is place your snapshot images in a folder that gets packed into the bundle and then specify correct path for snapshotDirectory. That usually means that you can not keep your reference images in the __Snapshots__ folders next to the source, but I was ok with moving them all into one folder with some subfolders.

[winstondu]

@darknoon , any update?

[Fl0p]

Bump