search

bitrise-steplib / steps-xcode-test

Xcode Test step
https://www.bitrise.io/cli
MIT License
33 stars 37 forks source link

Xcode 13 test repetition support #167

Closed godrei closed 3 years ago

godrei commented 3 years ago

Feature Request description

https://discuss.bitrise.io/t/support-test-retries-in-the-xcode-test-step/17134/2

Xcode 13 adds test repetitions support for XCTest. Test repetition can be configured with xcodebuild flags (-test-iterations <number>, -retry-tests-on-failure and -run-tests-until-failure) or in a Test Plan.

Technical Plan

Retry on failure

Xcode Test Step has a should_retry_test_on_fail Step Input, which controls if the whole xcodebuild test command should be retried in case of any failure.

This solution is sub-optimal as not just the failing test cases are retried, but the whole test command.

The retry logic could be tweaked by utilizing the new -retry-tests-on-failure xcodebuild flag.

Test Plan support

For better Test Plan support a dedicated Step Input could be introduced for specifying the Test Plan to run:

    - xcode-test:
        inputs:
        ...
        - test_plan: FlakyTests
godrei commented 3 years ago

should_retry_test_on_fail utilizes the new test repetition if Xcode version >= 13 and Test Plan support introduced via the new test_plan Step Input. The changes are available in version 2.7.0.