Failing tests on CircleCI (color issues?)

[lukeredpath]

I was wondering if anyone else runs their snapshot tests on CircleCI and if so, you've had problems getting them working.

We have some snapshot tests for our iOS views that pass locally but not on CircleCI. We are using the same Xcode version and specifying the same OS and simulator version that we used to generate the tests.

For each view we take two snapshots:

assertSnapshot(matching: ui, as: .deviceImage(on: .iPhoneX))
assertSnapshot(matching: ui, as: .deviceImage(on: .iPhoneSe))

Unfortunately there doesn't seem to be an easy way to get access to the generated screenshots on Circle CI (I'm still looking into this) and worst-case scenario I can skip these tests on CI, but I'd prefer not to.

[pikaboo]

I ran in to the same problem. However, not on circleCI, but in the following scenario.

I changed computers.

And now all my ui tests are failing because the images don't match. Since it is easy to find the images on the local computer, i compared them visually, and saw no difference. Except in byte size.

One was a few bytes bigger than the other.

The right image is the original image taken when the test ran for the first time. The left image is the image taken when i ran the tests today.

There is 698 byte difference, and it's definitely not the image itself, something else was added from the image on the left. And what it is, I do not know.

[lukeredpath]

Hmm, this is very odd. I wonder if this is another symptom of an issue we've already seen where some team members see snapshot tests failing on their machines but they work fine on mine. Something very odd is going on.

[pikaboo]

They work fine on the machine they were recorded on. But since i changed computers, i no longer have the machine it was recorded on. Since circle CI is definitely not the machine it was recorded on, this is the result.

[lukeredpath]

So I managed to find a way to override the snapshot path so I could have them be saved to a known location on CircleCI and zip them up as saved artefacts and I can now see that the images to indeed not match.

I can't share the whole image because it contains sensitive IP but I can show you a partial screen grab of both images from Preview - the one on the left is the expected image taken from my machine, the one on the right is the one from CircleCI. There is a clear colour difference (its worth noting that we aren't doing anything custom here, this is a pretty stock SwiftUI navigation view). Could this be a colour profile problem?

Using Digital Color Meter shows the original header background color is on average rgb(250,250,253) and the one on CircleCI is rgb(255,255,255).

[lukeredpath]

Now I have a better idea of what's happening this seems to be a dupe of https://github.com/pointfreeco/swift-snapshot-testing/issues/313

[lukeredpath]

I've done a bit more testing and it seems this problem only seems to occur when drawHierarchy(in:afterScreenUpdates:) to true, which as I understand uses drawHierarchy(in:afterScreenUpdates:) under the hood. Being GPU-based, this is probably the reason for the differences across different machines.

You can see the difference on a very basic view - this is just a Text inside a NavigationView - the rendering of the navigation bar on CircleCI is very odd - not only is it a darker grey but it seems to have weird blotches too.

|||

[pikaboo]

My app is very blue, and does not have the stock SwiftUI elements in it at all. Why do I get this color diff ? Is there a way to actually fix it ?

[Nikoloutsos]

Facing the same problem and I have used drawHierarchy() as well. Has anyone tried keeping the same ColorSync profiles???

[donnywdavis]

@lukeredpath Sorry, I don't have an answer for your initial answer but wanted to share a solution that we came up with to see the failed snapshots on CircleCI, if you still have that problem. Whenever a test fails we are copying the xcresult bundle into the artifacts on CircleCI. Then we can download that bundle and open it in Xcode. This is really nice as it shows the log from the tests and allows you to see all of the failing snapshots and to go through them and view the attachments to see the snapshots and the failures.

In the CircleCI config add a step to run on fail.

- run:
    name: Copy Test Results Bundle to Artifacts Folder
    when: on_fail
    command: scripts/copy_xcresult_for_tests.sh

That script just runs the zip command to save the xcresult. We have a custom derived data path to make it easier and the $test_results_bundle_name is just a constant that I defined to always have the same bundle name.

cd $derived_data_path
zip -r -X /tmp/circleci-artifacts/test_results_bundle.zip $test_results_bundle_name

Then also add that bundle name to the xcodebuild command.

-resultBundlePath "${derived_data_path}/${test_results_bundle_name}"

[lukeredpath]

@donnywdavis thanks for the tip - we do in fact capture the xcresult bundle which is very useful. In addition to this we also use https://github.com/TitouanVanBelle/XCTestHTMLReport to generate an HTML test report from the test result bundle - this gives you output very similar to Xcode and even includes inline snapshot failure comparisons.

We also zip up the failed snapshots and store them as an artifact too:

- run:
          name: "Compress snapshots"
          command: "zip -r Snapshots.zip /tmp/__SNAPSHOT_ARTIFACTS__ || true"
          when: always
      - store_artifacts:
          path: Snapshots.zip

[niallwatchorn]

Is there a fix for this? I'm getting a similar issue. Tests passing on the local machine, but failing on CI. Also have some reference images on the local machine which have custom colours, but are failing because the tests create the elements intermittanly without the custom colours and uses stock UIColors, (stock green UISwitch instead of custom green). Any ideas on why this might be?

[aduuub]

You can export the failed snapshots by setting the environment variable for the "Test" method of your scheme

key: value "SNAPSHOT_ARTIFACTS": "/tmp/SnapshotArtifacts"

In your Circle CI (or other CI) upload the contents of that directory so you can see the resulting failed screenshots.

I am also getting issues where Circle CI doesn't seem to render the content correctly on their VM's