doordash-oss / swiftui-preview-snapshots

Apache License 2.0
167 stars 7 forks source link

Snapshot Fails if different simulator chosen #10

Open rfodge opened 7 months ago

rfodge commented 7 months ago

When we run the tests for the first time and it records the snapshots, they will pass as long as we have that same simulator selected in xcode.
Here:

Screenshot 2024-04-08 at 4 24 02 PM

But why does this device that is selected matter if we are passing in the device configuration when we record and run the snapshot tests. So our tests have something like this Testing_Previews.compactSnapshot.assertSnapshots(as: .image(layout: .device(config: .iPhone15ProMax(.portrait))))

If we were to run this test and have SE 3rd generation simulator selected for the device in xcode then this test will always run/pass if SE 3rd gen is picked. But if we run this test again another simulator like iPad mini then it will fail. Why does the device selected in Xcode menu matter if it allows us to pass in the device configuration we want used? 🤔

teameh commented 4 months ago

Known issue. See readme https://github.com/pointfreeco/swift-snapshot-testing?tab=readme-ov-file#snapshot-anything or https://github.com/pointfreeco/swift-snapshot-testing/issues/614

Warning Snapshots must be compared using the exact same simulator that originally took the reference to avoid discrepancies between images.

jflan-dd commented 1 month ago

@rfodge The device configuration passed to assertSnapshots is just a size and safe area insets to capture the images using.

Using a different simulator in Xcode means that the tests are actually run on a different device (simulator) so lots of things can vary such as text spacing, default element sizing, etc.