Revert bump to iOS 12 deployment target

cashapp / AccessibilitySnapshot

Easy regression testing for iOS accessibility

Apache License 2.0

541 stars 69 forks source link

Revert bump to iOS 12 deployment target #30

Closed NickEntin closed 3 years ago

NickEntin commented 3 years ago

Testing this out to see if it fixes #29, since the flaky builds were first seen after this change.

Running tally:	Passed	Failed
4	0

NickEntin commented 3 years ago

4 successful builds in a row is making me think this is definitely what broke it. Still no idea why though. 😕

NickEntin commented 3 years ago

I'm guessing this is from the deployment target change, but it's possible that I subtly changed something accidentally when removing the availability checks. Testing that now...

Running tally:	Passed	Failed
1	1

NickEntin commented 3 years ago

Looks like the same tests are flaking

Failing tests:
    SnapshotTests:
        AccessibilitySnapshotTests.testTabBars()
        DefaultControlsTests.testStepper()
        DefaultControlsTests.testStepperAtMin()

NickEntin commented 3 years ago

Testing with the framework's minimum version set to iOS 10, but the demo app's deployment target (which is shared with the tests) to iOS 12.

Running tally:	Passed	Failed
4	0

NickEntin commented 3 years ago

The build seems pretty stable. This implies that the framework's minimum supported version is what matters, not the app's deployment target.

I'm not sure why that version change makes a difference, but unfortunately it means that bumping the version will decrease test stability for consumers.