Leverages CoreSimulator's private spawn method to run the xctest executable on individual logic test bundles. Doing so requires adjustments to various inputs (e.g. the xctest executable expects the list of tests in a different format) as well as different error handling (based on the nuances of how errors/crashes surface from the spawn method.
A number of critical pieces of functionality were added to support this:
Logic tests run using a different command than traditional unit or scenario tests (xcrun simctl spawn -s xctest).
Reverse engineering of Apple's CoreSimulator framework was required to use the private spawnAsyncWithPath: method to spawn a logic test execution.
Implementation details can be found in BPSimulator's executeLogicTestsWithParser method.
The xctest API does not support a test skip-list, and instead only provides an opt-in list. Bluepill has long had a hacky, broken solution for identifying all tests in a test bundle.
The fix is to inject a dylib directly into the xctest execution to pipe out test data. Full context on the problem and solution are detailed in this document
The injected library is BPTestInspector, a new project in the BP workspace. It is injected during BPSimulator's collectTestSuiteInfoWithCompletion method.
Making this dylib available and linked to both the Bluepill and xctest executions requires some additional env setup, both in objc code (for xctest) and Bazel (for Bluepill)
When running Xcode in Rosetta mode (as is the case for many consumers), there is a weird interaction with spawned executions -- they will try to run universal binaries in arm64 rather than x86.
As a result, we have to adapt the xctest executable to the desired architecture before running it.
Impl details are in BPUtils's lipoExecutableAtPath:
Also fixes a bug where retrying tests after a crash was being dictated by the wrong config variable (onlyRetryFailed rather than retryAppCrashTests).
Tests were added to validate:
happy path testing
sad paths (failure/timeout/crashes)
timeouts for both individual test cases + execution timeouts
retries
swift vs objc tests (requires name transformation for swift)
testing multiple test bundles
Logic test execution time data was collected and available here.
Additionally, a new logic test batch was created in a consuming app (Voyager iOS), and tests passed. Notably, the new BPTestInspector code correctly omitted hundreds of other legacy false positive test method names, and only the desired tests were treated as tests.
Adds logic test support to Bluepill, so that unit tests can be run without an app host. Resolves https://github.com/MobileNativeFoundation/bluepill/issues/531.
Leverages CoreSimulator's private spawn method to run the xctest executable on individual logic test bundles. Doing so requires adjustments to various inputs (e.g. the xctest executable expects the list of tests in a different format) as well as different error handling (based on the nuances of how errors/crashes surface from the spawn method.
A number of critical pieces of functionality were added to support this:
Logic tests run using a different command than traditional unit or scenario tests (
xcrun simctl spawn -s xctest).executeLogicTestsWithParsermethod.The xctest API does not support a test skip-list, and instead only provides an opt-in list. Bluepill has long had a hacky, broken solution for identifying all tests in a test bundle.
collectTestSuiteInfoWithCompletionmethod.When running Xcode in Rosetta mode (as is the case for many consumers), there is a weird interaction with spawned executions -- they will try to run universal binaries in arm64 rather than x86.
lipoExecutableAtPath:Also fixes a bug where retrying tests after a crash was being dictated by the wrong config variable (onlyRetryFailed rather than retryAppCrashTests).
Tests were added to validate:
Additionally, a new logic test batch was created in a consuming app (Voyager iOS), and tests passed. Notably, the new BPTestInspector code correctly omitted hundreds of other legacy false positive test method names, and only the desired tests were treated as tests.