Open mschuetz72 opened 1 year ago
+1 for this. I wish I'd seen this sooner as I did a parallel investigation in https://github.com/pointfreeco/swift-snapshot-testing/issues/710 on Bitrise.
Hi,
We are facing the same issues in our company, seeing that this fix is already in place and working in this branch, are you planning on pushing it soon to the main branch then release it ? this would truly save a lot of people some headaches!
Thanks!
This would be very useful. I just ran into this on CircleCI, I had no idea our snapshot tests weren't doing anything on CI even though recording was working correctly 🤦🏻
What
Metal is currently unavailable on Github Actions. The
perceptuallyCompare
diffing algorithm relies on Metal render to obtain the level of correspondence to compare it with the given threshold. Snapshot assertions will falsely succeed, even if a given image does not match the stored reference image.Why
The snapshot assertion should fail if it hasn't got the requirements to execute the tests. Currently, the only indication that something's wrong is hidden inside the logs, stating
I'm aware that there are other initiatives, such as https://github.com/pointfreeco/swift-snapshot-testing/pull/666, which would be perfect, but until this is solved I think it's important to make people aware that the assertions might be wrong when relying on virtual machines without Metal support.
any real issue, any real inconsistency will be undetected when using
perceptuallyCompare
and running on a virtual machine without Metal support.How
Added a pre-condition check to
perceptuallyCompare
which will fail when Metal is unavailable.Testing
Any image assertion using
perceptuallyCompare
running on an environment without Metal support. See also https://github.com/mschuetz-viz/DemoSnapshotTest