iOS Simulators no longer launching

[ds8k]

Description

Something changed within the past couple weeks that is preventing any iOS simulators from launching successfully, which is breaking our E2E Detox tests. They used to successfully run, but since Xcode 15.3 was released it appears the entire process has broken.

I've tried running tests on macos-13 and macos-14 with the same results.

Platforms affected

• [ ] Azure DevOps
• [X] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] macOS 11
• [ ] macOS 12
• [X] macOS 13
• [x] macOS 13 Arm64
• [X] macOS 14
• [x] macOS 14 Arm64
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

Image: macos-14-arm64 Version: 20240219.1 Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/2024021[9](https://github.com/axioscode/axios-app/actions/runs/8475906614/job/23224707831#step:1:10).1/images/macos/macos-14-arm64-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20240219.1

Is it regression?

https://github.com/axioscode/axios-app/actions/runs/8209405392/job/22454863523

Expected behavior

Detox tests are able to launch the designated simulator and successfully run

Actual behavior

iOS Simulator never launches and tests time out

Repro steps

1. Attempt to run Detox tests on macos-13 or macos-14
2. Detox CLI attempts to launch an iOS simulator
3. Simulator never launches - tests fail

[shamil-mubarakshin]

Hey @ds8k, Please provide repro steps or public sample project with steps for us to troubleshoot.

[kirillzyusko]

@shamil-mubarakshin @ds8k I believe I have the same problem. You can have a look on https://github.com/kirillzyusko/react-native-keyboard-controller repository and iOS e2e action/failed jobs.

I am observing such problem from Mar 19. Basically some tests are failing because they exceed a timeout of 2 minutes (120000ms).

I've tried to increase timeout to 13 mins but still tests are very flaky. On my local machine everything works fine.

Another my observation - after 1 or 2 failing tests everything becomes to work faster again and subsequent tests can be executed successfully. As a temporary workaround I'm using --retry 2 flag. But ideally I would like to have a single successful run. Let me know if I can help somehow 🙌

[tongyuze]

@shamil-mubarakshin I think we have the same problem. We are using "Image: macos-12" in the Azure pipeline to run our e2e tests on the ios simulator.

Since the week before last week, everything starts to fail because of this error: ❌ xcodebuild: error: Unable to find a destination matching the provided destination specifier:

The destination id we set is the "iPhone 14 pro" device id.

Here is a link to our pipeline: https://identitydivision.visualstudio.com/IDDP/_build/results?buildId=1278003&view=logs&j=f71b8be3-a86a-5c85-a350-6d71c1f97c14&t=c67dfeb8-f426-5ec6-237b-6331a4e8e3ff

This is blocking our release currently. Could you please help with high priority?

[ericswpark]

Same issue here, using macos-14 runner running Detox tests. The tests were flaky but they sometimes worked previously (example run: https://github.com/Purdue-CS307-tsundoku/tsundoku/actions/runs/8437619619/job/23107830553). Now they keep failing as the emulator won't launch (example run: https://github.com/Purdue-CS307-tsundoku/tsundoku/actions/runs/8512335611)

[freakboy3742]

@shamil-mubarakshin I'm seeing this as well - although in our case, the iOS simulator is starting, but it takes a lot longer than it once did.

We're only testing using macOS-14 runner.

As recently as Monday, our full iOS test suite took 3 minutes to complete, including building the app, starting the simulator, and running the full test suite. At this point, the test was running on macOS 14.2.1, with an image that was tagged 20240116.1 (although the hyperlink in the log doesn't point at a release - the tag exists, but the release doesn't appear to exist any more, and the hyperlinks are broken).

As of the update, it can take 10 minutes or longer to start the iOS simulator (example 1, example 2). The slowdown is then causing other problems that appear to cause the test suite to crash (fairly consistently at the same place) after about 20% of the tests. These two tests are running on macOS 14.4, with an image tagged 20240325.1. I've also seen this problem with macOS 14.4.1, with the image tagged 20240402.2 (example 1, example 2); on the more recent image, it appears that the simulator might be slightly faster, but still much slower than before.

One manifestation of the slowdown can be observed around line 270 of the of the "Test App" logs. In the old configuration, there would be 2-3 repeats of the "... still waiting" indicator (printed 10 seconds apart). In the new configuration, there are 20 or more iterations of "... still waiting".

The "Testbed (macOS-arm64)" tests take the same time to run, so the issue doesn't appear to be a general problem with the macOS-14 runner; it's specific to the iOS simulator running on the macOS-14 runner.

The test suite is using a tool called briefcase to run the test suite; internally, briefcase is starting the simulator using:

xcrun simctl boot 9531516F-3EE1-4463-AFCF-9BE73B199E7B
open -a Simulator --args -CurrentDeviceUDID 9531516F-3EE1-4463-AFCF-9BE73B199E7B
xcrun simctl uninstall 9531516F-3EE1-4463-AFCF-9BE73B199E7B <app UDID>
xcrun simctl install 9531516F-3EE1-4463-AFCF-9BE73B199E7B <path to app>
xcrun simctl launch 9531516F-3EE1-4463-AFCF-9BE73B199E7B <app bundle ID>

where 9531516F-3EE1-4463-AFCF-9BE73B199E7B is the UDID of the iPhone SE (3rd generation) simulator running iOS 17.4. On the older (successful) runs, the UDID is E33AE7FB-692F-4481-B3CB-23A7BE8F17DE, as the simulator is running iOS 17.2.

I have also tried running the test suite on the new configuration, but selecting an iOS 17.2 simulator, and running after explicitly selecting Xcode 15.2; neither approach restores the historical simulator speed.

[Kyome22]

Hello. We are encountering the same problem. We had a similar problem here with Xcode 15.0. For the sake of it, I used yeetd, the workaround from that time, and it seems to have improved a bit.

[kirillzyusko]

@freakboy3742 have you tried to run simulator 16.x (for example 16.4)? I've heard it can give back full speed 👀

For me it looks like new iOS 17 simulators do a heavy CPU work for a first run. I thought to cache simulator and restore for all subsequent runs, but on my local machine the size of simulator folder is about 4 GB and with limit of 10GB cache it's very unreasonable to try to cache it 😔

@Kyome22 I used yeetd but it didn't give any noticeable improvements 😔

[freakboy3742]

@kirillzyusko Thanks for that suggestion - it appears you're correct. Reverting to Xcode 14.3.1 and an iOS 16.4 simulator restores the previous testing speed.

That isn't a long term fix, if only because I'd predict Xcode 14.3.1 will be dropped from the runner image in the not-too-distant-future - but it at least it unsticks my immediate CI issue.

[kirillzyusko]

@freakboy3742 you can add iOS 16.4 in XCode 15 too, I believe. But you'll need to define additional steps 👀

But I totally agree with you - it should be fixed on image level somehow 👍

[freakboy3742]

@kirillzyusko I did try that; but it looks like the 16.4 API is only included with the older Xcode. It's probably possible to download, but pulling a 7GB image in CI is a bit of a tall order.

[kirillzyusko]

@freakboy3742 direct downloading will take ~5 mins, you're right. But you can also create a symbolic link https://github.com/actions/runner-images/issues/8402#issuecomment-1797918209

I haven't tried it and again you're right - if you go to symbolic link approach then you still depends on XCode 14.3.1 (which is not good).

And downloading 7GB and further installation is also an expensive operation.

[erik-bershel]

Hey @sssuourabh! Seems to be your case more likely.

[freakboy3742]

It appears that the 20240405.1 runner image has resolved this issue.

[freakboy3742]

...or maybe I've spoken too soon - this job passed, but when I tried to apply the same change to another PR, it fails.

[sssuourabh]

Same thing happening with me. Sometimes it passes and next Time it fails.

Sourabh Shekhar Singh +91-7838135034 https://www.linkedin.com/in/sourabhshekhar/

On Fri, 12 Apr 2024 at 1:21 PM, Russell Keith-Magee < @.***> wrote:

...or maybe I've spoken too soon - this job passed https://github.com/beeware/toga/actions/runs/8658180101/job/23741567362, but when I tried to apply the same change to another PR, it fails https://github.com/beeware/toga/actions/runs/8658471154/job/23743197182.

— Reply to this email directly, view it on GitHub https://github.com/actions/runner-images/issues/9591#issuecomment-2051210559, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWIRLLDUGUKKPJFPUO44STY46HAJAVCNFSM6AAAAABFOVDOUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRGIYTANJVHE . You are receiving this because you were mentioned.Message ID: @.***>

[PSchmiedmayer]

We are running into the same issues with our build infrastructure using GitHub hosted runners: https://github.com/StanfordBDHG/SwiftPackageTemplate/actions/runs/8680699610/job/23803319639

Some of the tests timeout due to missing resources and sometimes the simulator does not even start, e.g., using the encountered an error (The test runner failed to initialize for UI testing. (Underlying Error: Timed out waiting for AX loaded notification)) error message.

[scottohara]

Just adding our anecdata to this, we have a workflow that first boots a named simulator with the following command:

DEVICE_TYPE_ID="com.apple.CoreSimulator.SimDeviceType.iPhone-14-Pro-Max"
RUNTIME_ID="com.apple.CoreSimulator.SimRuntime.iOS-17-2"
SIMULATOR_NAME="simulator name goes here"

xcrun simctl bootstatus $(xcrun simctl create $NAME $DEVICE_TYPE_ID $RUNTIME_ID) -b

...then once the simulator is finished booting, loads a specific URL into Mobile Safari with the following command:

URL="url goes here"

xcrun simctl openurl booted $URL

We have observed the following timings:

Runner version	OS	Image version	Xcode version	Boot time	Open URL time	Workflow result
2.314.1	macOS 14.2.1 (23C71)	macos-14-arm64 20240116.1	15.2	36s	15s	✅
2.314.1	macOS 14.4 (23E214)	macos-14-arm64 20240325.1	15.2	1m 34s	> 20s	❌
2.315.0	macOS 14.4.1 (23E224)	macos-14-arm64 20240405.1	15.3	1m 0s	> 20s	❌

(Note: The "Open URL time" for the last 2 rows is "> 20s" because 20s is the arbitrary timeout we have on this command; hence those workflow runs failed and we cannot confirm how long it actually takes to load the URL)

(Also worth noting that on my local machine (M1 Macbook Pro), the xcrun simctl openurl booted command takes ~10s to open the URL after the simulator boots).

[Kyome22]

The simulator may be able to start stably with a combination of Xcode 15.1 and iOS 17.2.

[freakboy3742]

The 20240415.6 image hasn't resolved the issue.

Is there any possibility that someone from Github might respond? I've been advised via a ticket raised on an enterprise support contract that I should monitor the public ticket for updates, which would be great, if only there were updates of any kind from Github.

[shamil-mubarakshin]

Hey @freakboy3742. issue is currently being investigated. There is an issue with ios-17.4 simulator and CPU consumption. Checking if it is relevant and there is a command line equivalent to disabling Thread Performance Checker. I have also run a few workflows and after initial CPU spike, it drops down to ~30%, which we currently investigate. I have also tried running workflows on macos-14-xlarge that comes with 6CPU instead of 3, which allows tests to complete, but it is still slower than when using xcode 14.3.

[kirillzyusko]

I don't know, what happened, but today I discovered that tests are executing fast again 🚀 (under 5 minutes).

[scottohara]

Confirming the same.

To further add to my table from above:

Runner version	OS	Image version	Xcode version	Boot time	Open URL time	Workflow result
2.314.1	macOS 14.2.1 (23C71)	macos-14-arm64 20240116.1	15.2	36s	15s	✅
2.314.1	macOS 14.4 (23E214)	macos-14-arm64 20240325.1	15.2	1m 34s	> 20s	❌
2.315.0	macOS 14.4.1 (23E224)	macos-14-arm64 20240405.1	15.3	1m 0s	> 20s	❌
2.316.0	macOS 14.4.1 (23E224)	macos-14-arm64 20240422.3	15.3	44s	16s	✅

It appears that the latest image version has almost (but not quite) returned to the same level of performance as the 20240116.1 image version.

It would be great to understand what the change was that resulted in this improvement.

[kirillzyusko]

It would be great to understand what the change was that resulted in this improvement.

Agree. From what I observed - it's again not possible to programmatically set time for iOS simulator (was a known issue startig from XCode 14). So just curious what actually was changed in latest runner version 🤔

[freakboy3742]

The iOS 17.5 simulator that is the default on 20240422.3 definitely appears to run better in my experience, but it's still significantly slower than before - my test suite used to run to completion in 3-4 minutes; I'm now getting a test timeout 15 mins. The "Time to first test" has almost doubled - previously the suite would build and start running in <2 minutes; it now takes 4-5 minutes to do the same build. The test appears to be stalling on one particular test; it takes almost 8 minutes to get to that point in the suite.

[sssuourabh]

still facing the issue Testing failed: TestApp encountered an error (Failed to prepare device 'Clone 3 of iPhone 14' for impending launch. (Underlying Error: Unable to boot the Simulator. launchd failed to respond. (Underlying Error: Failed to start launchd_sim: could not bind to session, launchd_sim may have crashed or quit responding))) Test session results, code coverage, and logs: TestAppUITests-Runner (55863) encountered an error (The test runner failed to initialize for UI testing. (Underlying Error: Timed out while loading Accessibility.)) /Users/runner/Library/Developer/Xcode/DerivedData/TestApp-edrmvopwamjcnzaqgkvnmiwdeywz/Logs/Test/Test-TestApp-2024.04.27_15-57-24-+0000.xcresult ** TEST EXECUTE FAILED **

[dreampiggy]

Found also a similar problem:

The macos14-arm image seems broken after April. The media decoder of HEVC/HEIF on iPhone simulator does not run, while macos13 works

[ericswpark]

@dreampiggy:

The media decoder of HEVC/HEIF on iPhone simulator does not run, while macos13 works

I think this should be filed as a separate issue.

[ds8k]

This issue is still happening. It appears that maybe the simulator is running, but all of my tests are still failing. The same tests pass locally on MacOS 14 with Xcode 15.3 and on the macos-12 image using Xcode 14.