pointfreeco / swift-snapshot-testing

📸 Delightful Swift snapshot testing.
https://www.pointfree.co/episodes/ep41-a-tour-of-snapshot-testing
MIT License
3.79k stars 581 forks source link

Snapshot counter breaks test repetitions (inc. retry on failure) #693

Closed lukeredpath closed 1 year ago

lukeredpath commented 1 year ago

Describe the bug There are some scenarios in which a test case can be run multiple times within the same XCTest process - once is when you have retry on failure configured in your test plan, the other is when using the "Run Repeatedly" function in Xcode.

By default, snapshot names take the form testName.<counter> where counter is an incrementing number starting from 1 - this is what allows multiple snapshots in a single test without explicit names. Unfortunately, because this counter is not reset between iterations, subsequent iterations will fail as it will look for the wrong snapshot name.

To Reproduce Create any test case with a single snapshot and no explicit name - use Xcode's "Run Repeatedly" to run the test more than once without relaunching between tests and the test will fail.

Expected behavior The test should not fail.

Additional context Ideally this counter would be reset between test iterations - I don't know if there is an API to check which iteration a test is on. If there is no automatic solution to this, an API to reset the counter that could be called from setup() would at least allow the counter to be reset explicitly between each test iteration.

It might be possible to use the XCTestObservationCenter to install an observer that resets this counter between test iterations.

Snapshots with explicit names (using the named: parameter) do not have this issue.

lukeredpath commented 1 year ago

Oh, looks like this was fixed in https://github.com/pointfreeco/swift-snapshot-testing/pull/585 - I tried searching but couldn't find this.