Option to use build_for_testing to prepare UITests builds

[mikhailmulyar]

Is there any option to build *Test-Runner.app for ios_ui_test target for device? Currently when trying to build ios_ui_test target as a result you will get *.ipa file of main application + *Tests.zip archive containing only *Tests.xctest file.

There is an option to get *Test-Runner.app using xcodeproj generated with rules_xcodeproj. When building test target with Xcode for testing or using xcodebuild build_for_testing *Test-Runner.app is actually built. But there is a problem that it is building always for simulator. Test target generated with rules_xcodeproj supports only simulator. When building it is always creating *Test-Runner.app for simulator even when main app target is built for device.

[keith]

You can directly run tests on device https://github.com/bazelbuild/rules_apple//commit/cc872ba7a78f6643467b39039631c87367f6a104 but you can't just build and then test later like xcodebuild does

[mikhailmulyar]

Yes I know that. But I need to prepare build, because I need to upload it to AWS Device Farm and run tests there.

[keith]

how does the xcode flow for that work (maybe there's a doc somewhere I can just read?). I imagine we could produce some bundle with similar stuff, but it would probably require a decent number of changes

[mikhailmulyar]

I haven't seen any apple doc about that. Here is aws doc https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest-ui.html. But it is pretty straightforward. In Xcode you simply select Product > Build For > Testing. Using command xcodebuild it can be achieved by passing build_for_testing option.

xcodebuild -scheme MyUITests -project ./MyApp.xcodeproj -derivedDataPath my_folder -destination 'generic/platform=iOS' DEVELOPMENT_TEAM='Team_id' PROVISIONING_PROFILE_SPECIFIER='Provising_name' build-for-testing

It will build both MyApp.app and MyUITests-Runner.app (assuming MyApp target is test host for MyUITests)

[keith]

and what do you upload in that case? those files specifically with the xctestrun and all?

[mikhailmulyar]

I need to upload 2 .ipa files. First is main app MyApp.ipa and second is test runner app - MyUITests-Runner.ipa

[keith]

im surprised you don't have to upload the xctestrun too, since the details in there could vary from what is easily derivable, but yea makes sense. In the meantime if that's all you need you can do your build like normal to get your app's IPA, and then you can get the runner directly from Xcode (which is what the rules do too) for device in /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Agents/XCTRunner.app and copy that. The only issue is you might have to manually sign it

[mikhailmulyar]

I don't think it is that simple MyUITests-Runner.app contains inside .xctest file for tests to run and maybe some other info about app in binary. It is build specially for test target, it is not just default agent app. Here is file structure in it

[keith]

yea, you can see how it's setup for us in the test runner itself, you'd have to mimic that https://github.com/bazelbuild/rules_apple/blob/c6e58e642447420fdbd2dd3f7c29e4faee69d1c9/apple/testing/default_runner/ios_xctestrun_runner.template.sh#L125-L152

[mikhailmulyar]

ok got it now, thanks I will try it out

[keith]

I do think this is a reasonable request though, that might just be a short term workaround that's worth doing

[tylervick]

This would be a great feature to have, especially with the new "xctestproducts" bundles that xcodebuild can generate: https://keith.github.io/xcode-man-pages/xcodebuild.1.html#testProductsPath

[aaronsky]

I wanted to document some of my findings in this area, as this is something I needed at the day job and the question has come up a few times here and in Bazel Slack.

If you are using rules_xcodeproj to generate an Xcode project for your tests, one option you have instead of relying on Bazel to produce a well-structured bundle (preferable as that may be) is to have xcodebuild do this work for you. Using the xctestproducts feature introduced in Xcode 13.3 and mentioned in the comment above, you can create a bundle that can be zipped up and run wherever you need, so long as that environment knows how to use the -testProductsPath setting.

Here is an approximation of one solution I'm testing using to create a distributable test bundle:

# //:xcodeproj is a xcodeproj target from rules_xcodeproj with iOSUITests as a target and generated scheme. It produces ./xcodeproj.xcodeproj as an output.
bazel run //:xcodeproj

# xcodebuild can be used to build the test bundle and XCTRunner.app, just like it can in Xcode. And with the xctestproducts bundle, you can move it around to whatever node(s) are going to run your tests, without needing to build.
xcodebuild build-for-testing \
    -project xcodeproj.xcodeproj \
    -scheme iOSAppUITests \
    -testProductsPath iOSAppUITests.xctestproducts \
    -quiet

# You can use -enumerate-tests to list the tests in the xctestproducts bundle, if this is required by your sharding system
xcodebuild test-without-building \
    -testProductsPath iOSAppUITests.xctestproducts \
    -destination "platform=iOS Simulator,name=iPhone 15,OS=17.5" \
    -enumerate-tests \
    -test-enumeration-style flat \
    -test-enumeration-format json \
    -quiet >tests.json

# Finally, you can perform the tests using the xctestproducts bundle
xcodebuild test-without-building \
    -testProductsPath iOSAppUITests.xctestproducts \
    -destination "platform=iOS Simulator,name=iPhone 15,OS=17.5"

I was surprised and excited at how straightforward this was to do with the tools that already exist. I am not 100% confident this works in all cases yet, but I have managed to prove it out using the rules_xcodeproj integration example project and I intend to test it at my job soon. Hopefully this helps someone!