iOS Day 9: UI Testing

[Abhiek187]

Nothing feels more festive than finishing up the iOS build on Christmas day! Hopefully the UI tests will pass in the GitHub pipeline (they probably won't knowing my luck). I had some issues getting the tests to work locally, despite the UI tests passing within Xcode, and Fastlane wasn't generating any output. And since each test uses up quota, I don't want to spend too much time debugging these issues. I only created one UI test to fetch a random recipe and inspect all the UI elements, aside from the ones randomized from the recipe. The only annoying thing is that I can't reuse the String constants from the app's main target in the UI testing target since UI tests focus on what the user sees, rather than the internal app state. But thankfully, the end result wasn't too complicated to set up.

Along the way, I made a couple minor changes to refactor the code, improve the formatting of the UI, and improve the accessibility of the app. So everything is just a little more polished.

[Abhiek187]

I switched from xcpretty to xcbeautify as recommended by Fastlane. xcpretty hasn't been updated since 2018, while xcbeautify continues to receive ongoing support. It claims to be 2x faster than xcpretty, but I still get flaky results on my local machine. At least I can see which test is failing now thanks to better multi-device support, but it doesn't show which exact assertion is failing.

According to GitHub's docs, the macos-latest VMs run on a 3-core CPU. While it's not high-end, I'm surprised it's 10x slower than my local machine (which can run all UI tests in 2-3 minutes on 8 cores).

[Abhiek187]

Despite claiming to be 2x faster, this workflow took slightly longer than the previous attempt. But now that I can see the timings of all the tests, it looks like each test takes around the same amount of time as my local machine. But it's taking 12 minutes to start the simulators and begin testing. To break it down:

Step	Time
Build code	2m 35s
Start simulators	12m 47s
Run tests (and shut down simulators)	5m 22s
Total	20m 44s

It seems to be failing at the last device tested, no matter which device it is. I'm tempted to split each device to a separate workflow (similar to how the Android tests are set up), so if one of the tests fail, I can restart the workflow for that device only.

[Abhiek187]

That's 43% faster than testing in one workflow. Both the 8 and iPad sims booted and terminated in 3m, while the 14 booted in 4m and terminated in 6m. So I guess when I tried running in parallel before, it booted each device one at a time. But now all the tests are failing 😔. Let's see if I can get more info on which assertions are failing.

[Abhiek187]

I added custom messages to each assertion so we can see which line these tests are failing on.

[Abhiek187]

✖ testFindMeARecipe, XCTAssertTrue failed - Error line 96: The ProgressView isn't visible

Bingo!

[Abhiek187]

✖ testFindMeARecipe, XCTAssertTrue failed - Error line 80: Fiber is missing from the nutrition label

Unless there are recipes without a Fiber nutrient, I don't like how flaky these tests are getting.