cashapp / AccessibilitySnapshot

Easy regression testing for iOS accessibility
Apache License 2.0
541 stars 69 forks source link

CI builds for iOS 13 are flaking #29

Closed NickEntin closed 3 years ago

NickEntin commented 3 years ago

The CI build on iOS 13 seems to be flaking on a few specific tests:

testTabBars()
- Item B: 3 items. Tab. 2 of 4.
+ Item B. Tab. 2 of 4.
- Item C: A. Tab. 3 of 4.
+ Item C. Tab. 3 of 4.

testStepper()
- Decrement. Button.
+ minus. Button.
- Increment. Button.
+ plus. Button.

testStepperAtMin()
- Decrement. Dimmed. Button.
+ minus. Dimmed. Button.
- Increment. Button.
+ plus. Button.

This started after bumping the minimum iOS version to 12.0 (#27). I've seen this flake twice now.

NickEntin commented 3 years ago

Based on some testing in #30, it looks like this is specifically caused by bumping the framework's minimum version from iOS 10 to iOS 12. More specifically, the change to the sample app's deployment target is not what caused the flakes, and therefore I do not expect this is an issue that consumers are currently seeing.

This means that we need to dig into the root cause of these flakes before moving forward with the 0.4.0 release.

NickEntin commented 3 years ago

From #31, it looks like changing the minimum supported iOS version to 11.0 is a viable option.

NickEntin commented 3 years ago

My theory for the testStepper and testStepperAtMin cases is that _UIStepperButton isn't marking itself as an element, so VoiceOver is treating the image views instead the buttons as the elements. The accessibilityLabels for those image views are "minus" and "plus".

NickEntin commented 3 years ago

From some debugging in #36, it seems like the problem is in the test target, not the framework itself. I think at this point we're okay to move forward with other PRs and continue with the 0.4 release, since it seems that any other change to the test target will fix the tests.

I'm going to go ahead and close this issue for now. If we see more flaky tests on a subsequent PR, let's reopen this and continue investigating.