search

actions / runner-images

GitHub Actions runner images
MIT License
9.17k stars 2.84k forks source link

New macOS 13 runners default to iOS 17 beta simulators for Xcode 14.3.1 CI jobs. #8023

Closed mluisbrown closed 5 months ago

mluisbrown commented 9 months ago

Description

The latest macOS 13 runners enable the iOS 17 beta simulators that come with Xcode 15.

This means that any Xcode test action which specifies the device name only (without an OS specifier) will run using the iOS 17 simulator by default instead of the latest iOS 16 simulator. This will cause many CI actions to fail as code has not yet been adapted for iOS 17.

Moreover, it is a failure that is not obvious and very hard to debug, as it's completely unexpected that an Xcode 14.3.1 CI job would use beta iOS 17 simulators from Xcode 15.

Platforms affected

Runner images affected

Image version and build link

Current runner version: '2.307.1'
Operating System
macOS
13.5
22G74
Runner Image
Image: macos-13
Version: 20230801.2
Included Software: https://github.com/actions/runner-images/blob/macos-13/20230801.2/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-13%2F20230801.2
Runner Image Provisioner
2.0.264.1

Unfortunately I cannot include a link to a build as it's a private repo.

Is it regression?

20230611.2

Expected behavior

iOS 17 beta simulators should only be selected if explicitly specified, instead of by default.

Actual behavior

iOS 17 beta simulators are selected by default, if no other OS specifier is used.

Repro steps

Run any Xcode CI job which selects a test device (eg using fastlane run_tests and selecting iPhone SE (3rd generation) as the test device.

For example, in our project we have the following Fastlane lane which we use for CI:

lane :tests do
    run_tests(project: "<redacted>.xcodeproj",
        scheme: "<redacted>",
        devices: ["iPhone SE (3rd generation)"],
        result_bundle: true
    )
end

This is now selecting an iOS 17 version of the simulator when previously it was selecting an iOS 16.4 simulator. The potential workaround is to specify the device as "iPhone SE (3rd generation) (16.4)" to fix the iOS version at 16.4, but that should not be necessary.

joshuapoq commented 9 months ago

This probably explains why our builds are now reaching an hour and timing out. It's taking from 2 to 11 minutes of time between each test bundle for our frameworks (24 test bundles). It was previously taking between 3 and 15 seconds.

mluisbrown commented 9 months ago

I have been able to confirm that adding the OS version in the device string, eg: "iPhone SE (3rd generation) (16.4)" is a workaround for this problem.

ilia-shipitsin commented 9 months ago

@mluisbrown , we added simulators for XCode-15 yesterday. MacOs-13 currently is shipped with simulators for XCode-14 and XCode-15 as well. Whichever is "default", I'm afraid someone will complain.

I'm not sure what would be an approach to make everyone happy, some people may want to be "iOS 17" as default, other "iOS 16.4".

Maybe, the best approach would be to specify explicitly, what do you think ?

Vyazovoy commented 9 months ago

I think this is the problem:

Screenshot 2023-08-02 at 10 34 35

iOS 17 simulators hava to be installed only for Xcode 15. So if it is true that the default selected Xcode version is 14.2 then it should not see any iOS 17 simulators. The runtime should be marked as "unavailable". You can see that if you open Xcode 14.2 settings. So it is definitely a regression.

mluisbrown commented 9 months ago

@ilia-shipitsin as @Vyazovoy points out, the defaults for Xcode 14.3.1 should be the iOS 16 runtimes and the defaults for Xcode 15 should be the iOS 17 runtimes. It makes no sense at all to use the iOS 17 runtimes with Xcode 14.3.1.

It's a regression as absolutely no one expects that Xcode 14.x would have iOS 17 runtimes installed.

Baza207 commented 9 months ago

This does seem to be a regression, but it feels like it's maybe an Apple/Xcode issue. You can actually use the iOS 17 sim from Xcode 14.3.1, I tried it locally and it works fine. However because iOS 17 is still beta, it's buggy. 🙃 It's a bad default and should be fixed, but we should probably submit Apple feedbacks for it. Also, I could be wrong but I don't think it's possible to install sims for just one version of Xcode anymore. Locally for me at least they seem to sync between 14 and 15 of Xcode (add/delete in one place, it's available/gone from the other) and I'm guessing having a macOS-13 agent per Xcode major version would also not be on anyones list of requests.

The above mentioned work-around of adding (16.4) to the end of the sim name didn't work, but checking out the docs you can do something like iPhone 14,OS=16.4 and it seems to be working for us now. (Converts to -destination platform=iOS Simulator,name=iPhone 14,OS=16.4 for xcodebuild.) Look for destinationSimulators here.

Really not the greatest, but I guess that's what happens when we work with a beta. Better case would be macOS wasn't still in beta, and this could be tested in a separate beta. But I'd much prefer having Xcode 15 beta available now, rather than after it's released. 🙈

Let's hope the next Xcode beta sorts things out. 🤞😅

ilia-shipitsin commented 9 months ago

@Vyazovoy , @mluisbrown I need your help.

do you have (locally) installed both XCode-14.X and XCode-15.X and all working in the expected way ? if so, please share details how did you get it working.

for image generating we use https://github.com/actions/runner-images/blob/main/images/macos/provision/core/xcode.ps1#L30-L39

which invokes xcodebuild -downloadAllPlatforms in loop. if you have other steps, we'll adjust our steps

mluisbrown commented 9 months ago

@ilia-shipitsin I just installed Xcode 15 beta to check this. Now all my iOS 16.4 simulators have (16.4) appended to the end of their description. So apparently with both versions installed you must specify the OS version. It's not clear what determines the default if no OS version is specified.

Nevertheless, I imagine this will be an issue for a lot of projects. At this moment I can't think of a good solution 😞

2sem commented 9 months ago

Suddenly Archiving Not working 스크린샷 2023-08-03 오후 2 41 41

Working 스크린샷 2023-08-03 오후 2 44 23

How can I downgrade with run-on?? like run-on: macos-13.4 ???

ilia-shipitsin commented 9 months ago

Suddenly Archiving Not working 스크린샷 2023-08-03 오후 2 41 41

Working 스크린샷 2023-08-03 오후 2 44 23

How can I downgrade with run-on?? like run-on: macos-13.4 ???

@2sem , looks like your issue is not related to XCode simulators. Please open dedicated issue and provide repro steps

joshuapoq commented 9 months ago

The above mentioned work-around of adding (16.4) to the end of the sim name didn't work, but checking out the docs you can do something like iPhone 14,OS=16.4 and it seems to be working for us now.

Thank you @Baza207! This was the solution for us.

mluisbrown commented 9 months ago

The above mentioned work-around of adding (16.4) to the end of the sim name didn't work, but checking out the docs you can do something like iPhone 14,OS=16.4 and it seems to be working for us now.

The suggestion of adding (16.4) to the end of the sim name is what works if you are using Fastlane with the run_tests command, where you have to pass a literal simulator name, including the OS version, exactly as it would appear in the run destination at the top of the Xcode window.

@joshuapoq @Baza207 it looks like you are using a different tool?

mluisbrown commented 9 months ago

@ilia-shipitsin I think a potential solution would be to have 2 distinct runner images for macOS 13, one with only the currently shipping production version of Xcode (currently 14.3.1) and another which also includes the current beta version (currently Xcode 15).

I realise that may be too much to ask, and have other issues, so I'm just throwing it out there as a potential option.

Baza207 commented 9 months ago

@joshuapoq @Baza207 it looks like you are using a different tool?

Using Azure DevOps pipelines, which uses Fastlane under the hood (which in turn uses xcodebuild), but as MS own Azure and GitHub both run off of the same agents. 🙃

Vyazovoy commented 9 months ago

My previous comment is not fully correct. I've made some experiments and here is what I've found: (precondition: Xcode 14 and Xcode 15 installed and iOS 17 sim runtime installed)

Taking into an account everything mentioned above looks like there is a bug/feature of xcodebuild/simctl that despite promising to use "the most recent version of iOS supported by this version of Xcode." actually uses literally the most recent version that is visible to the tool. But this weird behavior gets more problematic after Xcode 15 stopped providing a default iOS simulator runtime.

So it is not a runner image regression but more like a new discovered bug/feature of Xcode.

mluisbrown commented 9 months ago

So it is not a runner image regression but more like a new discovered bug/feature of Xcode.

Thanks for the very detailed explanation @Vyazovoy 🙏

Whilst this does appear to be a bug with xcodebuild, it is still also a runner image regression, even if the problem is not directly in the runner itself. CI jobs which were working fine with the old runner are now broken.

As I suggested above, the best fix would be to have two separate macOS runners, one with only released versions of Xcode and one with only the current beta version of Xcode. That may be too much to ask though.

Vyazovoy commented 9 months ago

the best fix would be to have two separate macOS runners

Totally agree with you, @mluisbrown. Ideally it should be a runner with the macOS 14 Sonoma Beta with Xcode 15 Beta and iOS 17 Beta 😊 But unlikely that we will have it especially taking into an account that we are ~ one month from the betas will become GM.

Vyazovoy commented 9 months ago

@ilia-shipitsin, I think this issue will happen with all Xcode 15+ unless Apple fix the behavior. Imagine that Xcode 15.3 will install iOS 17.3 runtime globally and it will be used by default even if you have selected Xcode 15.1 with xcode-select. Maybe in some cases it won't cause any issues but at least it won't be what people expect. So consider investigating potential solutions. At least it worth to report to Apple.

Steve-Glass commented 9 months ago

👋 @Vyazovoy This is something we can bring to Apple. In the past, I've found it helpful to bring feedback directly from customers to Apple. Would you be open to submitting an issue via Feedback Assistant and providing us with the issue number?

lucascorrea commented 9 months ago

The above mentioned work-around of adding (16.4) to the end of the sim name didn't work, but checking out the docs you can do something like iPhone 14,OS=16.4 and it seems to be working for us now. (Converts to -destination platform=iOS Simulator,name=iPhone 14,OS=16.4 for xcodebuild.)

We have the same problem in CI, this Xcode issue of allowing Xcode 14.3.1 to run the simulator with iOS 17 is a critical issue.

Let's try this solution. Thanks

stephencelis commented 9 months ago

The new Xcode images seem to have a lot of strange issues. After xcode-selecting a particular version, xcodebuild doesn't resolve:

https://github.com/pointfreeco/swift-dependencies/actions/runs/5823359023/job/15790061504

$ sudo xcode-select -s /Applications/Xcode_14.3.app
$ xcodebuild …
error: unable to locate xcodebuild, please make sure the path to the Xcode folder is set correctly!
error: You can set the path to the Xcode folder using /usr/bin/xcode-select -switch
mluisbrown commented 9 months ago

@stephencelis I think you need to specify /Applications/Xcode_14.3.1.app (add the .1) for it to work. Or .0 if you want 14.3.

stephencelis commented 9 months ago

@mluisbrown I bumped versions coincidentally, but the path to 14.3.0 should simply be 14.3:

image

And this worked in the past. I had to change xcodebuild to xcrun xcodebuild to get things working again.

mluisbrown commented 9 months ago

@stephencelis it's true that 14.3 without the 0 should work. It's really weird that you needed to add xcrun 😕

ilia-shipitsin commented 9 months ago

Hello,

XCode 15 Beta 6 was deployed.

Brett-Best commented 9 months ago

@stephencelis

The new Xcode images seem to have a lot of strange issues. After xcode-selecting a particular version, xcodebuild doesn't resolve:

https://github.com/pointfreeco/swift-dependencies/actions/runs/5823359023/job/15790061504

$ sudo xcode-select -s /Applications/Xcode_14.3.app
$ xcodebuild …
error: unable to locate xcodebuild, please make sure the path to the Xcode folder is set correctly!
error: You can set the path to the Xcode folder using /usr/bin/xcode-select -switch

You actually need to point to the developer directory within the Xcode you want to use. So try:

$ sudo xcode-select -s /Applications/Xcode_14.3.app/Contents/Developer
tristangrichard commented 8 months ago

I am experiencing the same issue.

None of the Xcode-select steps work for me either

UI tests are failing since August. They simply fail to launch.

Any way I can specify simulator running iOS 16.4 without using fastlane

amilton-camargo-ext commented 7 months ago

Does anybody know if Xcode 15 is already available?

$ sudo xcode-select -s /Applications/Xcode_15.app/Contents/Developer

JUSTINMKAUFMAN commented 7 months ago

Does anybody know if Xcode 15 is already available?

$ sudo xcode-select -s /Applications/Xcode_15.app/Contents/Developer

This command (above) selects Xcode 15 Beta, which can not be used to deploy to App Store Connect.

Is there a different location for the release version?

Do we need to use macOS 14 runners for this? And if so, are they already available (because there is no mention of them in the docs yet)?

Hopefully the docs are updated soon since Xcode 15 has been out of beta for a few weeks now.

amilton-camargo-ext commented 7 months ago

I did set the system to macos-13 and Xcode to sudo xcode-select -s /Applications/Xcode_15.0.app/Contents/Developer and it worked. But I'm not deploying to the Store, just running the tests.

JUSTINMKAUFMAN commented 7 months ago

I did set the system to macos-13 and Xcode to sudo xcode-select -s /Applications/Xcode_15.0.app/Contents/Developer and it worked. But I'm not deploying to the Store, just running the tests.

It will work fine for running tests but App Store Connect will complain that it is in an invalid toolchain for production.

Vyazovoy commented 7 months ago

@JUSTINMKAUFMAN, take a look at #8282 please

dk53 commented 7 months ago

Bumping this, as build using iOS 17 simulators are enormously long

patrickhartling commented 7 months ago

I've been testing Xcode 15.0 and iOS 17.0 simulators, and the simulator is very slow. Everything else is about the same speed as using Xcode 14.3.1 and iOS 16.4. Execution of tests with an iOS 17.0 takes about twice as long as with an iOS 16.4 simulator.

mluisbrown commented 7 months ago

I've been testing Xcode 15.0 and iOS 17.0 simulators, and the simulator is very slow. Everything else is about the same speed as using Xcode 14.3.1 and iOS 16.4. Execution of tests with an iOS 17.0 takes about twice as long as with an iOS 16.4 simulator.

I have the same issue. See https://github.com/actions/runner-images/issues/8434

stephencelis commented 7 months ago

@mluisbrown @patrickhartling This is a known issue in Xcode 15 that was fixed in 15.0.1 and 15.1. From the release notes:

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)

mluisbrown commented 7 months ago

@stephencelis I saw that issue and wondered if it was the cause. It's just weird that running the test suite locally on my M1 MacBook Pro doesn't seem to have these issues.

But I hope this is the issue, as it means there's a solution in sight 👍

mluisbrown commented 7 months ago

It's just weird that running the test suite locally on my M1 MacBook Pro doesn't seem to have these issues.

Turns out I did... a full test suite was taking 6-7 minutes and with Xcode 15.0.1 and iOS 17.0.1 simulators it only takes 2-3 minutes 😄

svenmuennich commented 7 months ago

with Xcode 15.0.1 and iOS 17.0.1 simulators it only takes 2-3 minutes

@mluisbrown Does that mean that with Xcode 15.0.1 execution time is roughly the same as with Xcode 14 for your test suite?

mluisbrown commented 7 months ago

@svenmuennich on my local machine, yes.

erik-bershel commented 5 months ago

Hello @mluisbrown! Unfortunately, it is now impossible to associate specific runtimes with a specific Xcode in image directly. Thus, no matter what basic simulator we choose, such changes will break workflows without pre-build settings. In this regard, I am going to close this issue.