search

muter-mutation-testing / muter

🔎 Automated mutation testing for Swift 🕳️
MIT License
502 stars 39 forks source link

[276] unlock xcodebuild with SPM #282

Open aim2120 opened 7 months ago

aim2120 commented 7 months ago

Description

As muter stands today, SPM projects cannot use xcodebuild as a test executable. This is unfortunate, since any SPM projects that targets iOS, watchOS, tvOS, or visionOS must rely on xcodebuild to run tests with an available simulator.

Build Path Changes

The biggest blocker for using xcodebuild in muter's current state is the use of -showBuildSettings to get the build path. This is because -showBuildSettings does not work the same for SPM projects as it does for Xcode projects. It requires a scheme to be passed, and it also does not output the BUILD_DIR line needed to get the build path.

As a solution to the build path issue, I've decided to utilize the ability to pass an explicit build path to xcodebuild with the flag -derivedDataPath. To keep parity, I'm also using --build-path with the swift executable. To allow consumers to specify a custom build path, I've added buildPath as a value in the muter config, defaulting to .build.

With this change, we can now run muter on an SPM project with xcodebuild without issue. 🎉

SPM Muter Config Init

Since xcodebuild is now available for SPM, I've added configuration generation logic that detects if an SPM project is targeting iOS. If iOS is detected as a platform, then the xcodebuild executable is added, along with the flags necessary to run the correct scheme and simulator.

Since muter only currently supports iOS simulators, I have not added anything related to the other simulators that might be available.

Acceptance Test Additions

I've added two new example repositories for acceptance tests:

The test script runs the same acceptance test on ExampleiOSPackage as is currently run on ExampleApp. It also runs the same acceptance test on ExampleMacOSPackage as is currently run on ExampleMacOSApp.

To be able to differentiate between the output for app vs. package, I've added the suffix spm and xcodeproj to the test output files. This is utilized in the AcceptanceTests suite to verify the output of running the acceptance tests.

I also changed the acceptance tests to run in a temp directory, which to me seems cleaner than running in the AcceptanceTests directory. However, I'm open to undoing this change if the project maintainers prefer the old way, as it's not necessary for this MR.

rakaramos commented 6 months ago

hey @aim2120 thanks a lot for the time and effort! the code looks great 👍, the only thing I would suggest is to rely more on the unit test and maybe remove the ios tests from the acceptance test, since iOS will need to boot up the simulador and run the tests, this adds up time to the acceptance tests. do you foresee any issue in having only macos tests in acceptance tests? also, does it work if users are using hardcoded-clonedSourcePackagesDirPath in the config? I'm thinking about CI usages where it is usually a good idea to cache the dependencies.