XCode simulators failing when booting or using openurl

[fpkamp]

Description

Hi, I have been redirected from GitHub Support to describe our use case and perhaps influence future performance of your runner for macOS. We have a use case where we need to use XCode 14.3.1 and XCode 15.0 on the macOS13 runner. We create simulators with iPhones and navigate to a url and then we dispose the simulator. All of the above works pretty well locally, but it fails when executing in GitHub Actions. The behavior between XCode versions is different, with XCode 14 timing out on booting or navigating, however XCode 15 fails on other functions (perhaps an effect of changes to XCode itself) like binding launchd_sim. An example failure message I get would be:

An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=60):
Unable to boot the Simulator.
launchd failed to respond.
Underlying error (domain=com.apple.SimLaunchHostService.RequestError, code=4):
    Failed to start launchd_sim: could not bind to session, launchd_sim may have crashed or quit responding

In the ticket that has originally redirected me here, Arthur says he was successful with adding a cache clean (~/Library/Caches/ and ~/Library/Developer/CoreSimulator/Caches/) and waiting 60-120 seconds, however upon trying that I observed that it only works for a single simulator and subsequent simulators would fail (at least using a rinse and repeat approach).

Platforms affected

• [ ] Azure DevOps
• [X] GitHub Actions - Standard Runners
• [X] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] macOS 11
• [ ] macOS 12
• [X] macOS 13
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

version: 20230611.2 workflow run: https://github.com/fingerprintjs/fingerprintjs-pro/actions/runs/5646633166/job/15295123030

Is it regression?

no

Expected behavior

Simulators work smoothly and boot / open urls without crashing.

Actual behavior

Attempts to boot simulators and navigate to a URL fail very frequently.

Repro steps

Use the following script in a macOS 13 runner workflow: open /Applications/Xcode_14.3.1.app/Contents/Developer/Applications/Simulator.app/ phone1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl create iPhone-hzso25lt6h9 com.apple.CoreSimulator.SimDeviceType.iPhone-14 com.apple.CoreSimulator.SimRuntime.iOS-16-4) echo "${phone1}" boot1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl boot ${phone1}) echo "${boot1}" phone2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl create iPhone-sios1839ti com.apple.CoreSimulator.SimDeviceType.iPhone-14 com.apple.CoreSimulator.SimRuntime.iOS-17-0) echo "${phone2}" boot2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl boot ${phone2}) rm -r ~/Library/Caches/* rm -r ~/Library/Developer/CoreSimulator/Caches/* sleep 120 nav1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl openurl booted 'https://google.com') echo "${nav1}" echo "${boot2}" rm -r ~/Library/Caches/* rm -r ~/Library/Developer/CoreSimulator/Caches/* sleep 120 nav2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl openurl ${phone2} 'https://google.com') echo "${nav2}"

[sergei-pyshnoi]

Hello @fpkamp . Thanks for your report. MacOS 13 in beta state for now so it can have some issues like this. We will investigate problem and return with feedback as we get any result.

[ilia-shipitsin]

@fpkamp , we are preparing new macos-13 image with xcode update. we are not certain on time framing, probably 2 weeks or so.

can you provide repro steps so we can test your scenario on updated image ?

[ilia-shipitsin]

@fpkamp , we deployed new image today (macos-13.5 + xcode-15.0 Beta2), please try

[joshuapoq]

Crossposting my message here as it might relate: https://github.com/actions/runner-images/issues/8023#issuecomment-1661916389

Our builds are now reaching an hour and timing out. It's taking from 2 to 11 minutes of time between each test bundle for our frameworks (24 test bundles). It was previously taking between 3 and 15 seconds.

I think this new deploy might be to blame as there were no issues on Monday. This is a considerable amount of added time.

EDIT: This is for a private repo (with extensive tooling set up). Repro steps are to create a Package with multiple test targets and include the same empty test in each of them. Then set up CI to run all tests and it should take well over an hour to test 0 code. This could be an Apple issue (not at all ruling that out).

SOLUTION: Set the simulator to iPhone 14,OS=16.4 as mentioned here as the issue is it's defaulting to iOS 17.

[ilia-shipitsin]

@joshuapoq , can you help with repro steps (or link to repo/build) ?

[fpkamp]

@ilia-shipitsin sorry, I missed your ping from last week. The repro steps are in the original ticket - to clarify, the part I pasted in the repro steps is a bash script you should execute in your workflow. You may need to install the simulators before, so you want to first do xcodebuild -downloadPlatform iOS

[ilia-shipitsin]

hmm, is open a bash command ?

[fpkamp]

Sorry, I meant just using a normal run action, feel free to just use this ` - name: Open devices 14.3 run: | xcodebuild -downloadPlatform iOS open /Applications/Xcode_14.3.1.app/Contents/Developer/Applications/Simulator.app/ phone1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl create iPhone-hzso25lt6h9 com.apple.CoreSimulator.SimDeviceType.iPhone-14 com.apple.CoreSimulator.SimRuntime.iOS-16-4) echo "${phone1}" boot1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl boot ${phone1}) echo "${boot1}" phone2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl create iPhone-sios1839ti com.apple.CoreSimulator.SimDeviceType.iPhone-14 com.apple.CoreSimulator.SimRuntime.iOS-17-0) echo "${phone2}" boot2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl boot ${phone2}) nav1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl openurl booted 'https://google.com')

      echo "${nav1}"
      echo "${boot2}"
      nav2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl openurl ${phone2} 'https://google.com')
      echo "${nav2}"`

14.3.1 because it worked better, but obviously you can also use 15.0, I am currently running a build on the 13.5 image so I'll update soon as I get the results.

[ilia-shipitsin]

starting with yesterday we included simulators for XCode-15 as well, i.e. xcodebuild-downloadAllPlatforms, probably xcodebuild -downloadPlatform iOS can be omitted

[fpkamp]

Just to confirm - macos-13.5 is the expected input? I see that my job has been stuck in waiting for queue for the past 4 hours, but running macos13-xl was immediate (just to rule out something on my org end I haven't been notified about)

[ilia-shipitsin]

Just to confirm - macos-13.5 is the expected input? I see that my job has been stuck in waiting for queue for the past 4 hours, but running macos13-xl was immediate (just to rule out something on my org end I haven't been notified about)

sorry, what do you mean by "input" ?

[fpkamp]

Just to confirm - macos-13.5 is the expected input? I see that my job has been stuck in waiting for queue for the past 4 hours, but running macos13-xl was immediate (just to rule out something on my org end I haven't been notified about)

sorry, what do you mean by "input" ?

is runs-on: macos-13.5 the correct way to specify I want my workflow to run on the image you deployed this week ?

[ilia-shipitsin]

Just to confirm - macos-13.5 is the expected input? I see that my job has been stuck in waiting for queue for the past 4 hours, but running macos13-xl was immediate (just to rule out something on my org end I haven't been notified about)

sorry, what do you mean by "input" ?

is runs-on: macos-13.5 the correct way to specify I want my workflow to run on the image you deployed this week ?

no.... and yes ))

labels supported:

macos-11 macos-12 macos-13 macos-latest (alias for macos-12).

if you specify "macos-13.5", GHA will try to find your attached self hosted runner with that label.

[fpkamp]

Just to confirm - macos-13.5 is the expected input? I see that my job has been stuck in waiting for queue for the past 4 hours, but running macos13-xl was immediate (just to rule out something on my org end I haven't been notified about)

sorry, what do you mean by "input" ?

is runs-on: macos-13.5 the correct way to specify I want my workflow to run on the image you deployed this week ?

no.... and yes ))

labels supported:

macos-11 macos-12 macos-13 macos-latest (alias for macos-12).

if you specify "macos-13.5" GHA will try to find your attached self hosted runner with that label.

Alright, I see, so can you please ELI5 how to use the aforementioned image?

[ilia-shipitsin]

aforementioned image is used automatically if you specify runs-on: macos-13, currently deployed image is used for that label. no way to choose another (or previously deployed) image.

[fpkamp]

@ilia-shipitsin okay so I finally ran it on the new version and looks pretty much the same in terms of stability.

[fpkamp]

hi there, was wondering if there's any updates? Is there a new version of the runner coming, maybe? Anything I can do to help?

[ilia-shipitsin]

Hey, @fpkamp , my access to test farm was restored today, I'll have a look

[stancho-stanchev]

@ilia-shipitsin Since last week the builds have increased in time so much causing all sort of issues, I hope you guys find some sort of solution very soon!!!

using macOS-13 and Xcode 14.3.1 😢

[ilia-shipitsin]

@stancho-stanchev , in current issue we investigate "Attempts to boot simulators and navigate to a URL fail very frequently."

if you observe another regression, please open an issue.

[ilia-shipitsin]

I acknowledge issue, @fpkamp , thank for repro steps. I'm able to run simulators using your commands. iphone is starting, even it opens www.google.com, but everything is slow. not yet sure on which side is an issue, is it simulator itself or something with our vm.

[fpkamp]

@ilia-shipitsin cool, thank you. Maybe Arthur's observations here can help: Ticket 2248137. And just to confirm - did you get any exceptions or errors while issuing the commands? If that helps, I can also say that I have recently noticed that Safari driver started failing with a lot higher rate with some exotic error suggesting the server was already closed, but I wasn't able to find any more details. So I think the problem is in the VM. The simulator worked quite well for me on my mac as well, it was only the GH runner where I started experiencing issues.

[ilia-shipitsin]

@fpkamp , XCode 15 Beta 6 was just deployed, please test.

I did some dive on local installation. No conclusion yet, it does not work as I expect, but I'm not sure whether it is bug or feature.

[fpkamp]

@ilia-shipitsin I've noticed there are two XCode 15's in the /Applications folder: Xcode_15.0.0.app Xcode_15.0.app Which one should I use? Which one is the most current beta?

[ilia-shipitsin]

those are identical. I guess that Xcode_15.0.app is a symlink

[ilia-shipitsin]

so far...

1) updating XCode 15 to Beta 6 - no visible change 2) warming up cache - no visible change 3) updating macos 13.5 --> 13.5.1 - no visible change

still investigating

[fpkamp]

sorry I have been meaning to update on my side, but I was fighting a different problem. I ran the same workflow and it looked the same except for failing on deleting the caches, hence I was going to run a workflow tomorrow where I wouldn't remove those, I'll do that and let you know, but not holding my breath.

[ilia-shipitsin]

it's utilizing all 8 cores when booting iPhone

[fpkamp]

I can confirm I see no improvement.

[ilia-shipitsin]

I narrowed it a little.

2023-08-18 22:06:46.181524+0000 0x4a7a     Default     0x0                  1316   0    ReportCrash: (OSAnalytics) CFBundle URL refined <private> to <private>
2023-08-18 22:06:46.188679+0000 0x47e8     Default     0x0                  1316   0    ReportCrash: (OSAnalytics) CFBundle URL refined <private> to <private>
2023-08-18 22:06:46.189954+0000 0x4a7a     Default     0x0                  1316   0    ReportCrash: (OSAnalytics) CFBundle URL refined <private> to <private>
2023-08-18 22:06:46.274019+0000 0x4dd2     Default     0x0                  0      0    kernel: promotedcontentd[1895] Corpse failure, too many 6
2023-08-18 22:06:46.274028+0000 0x4dd2     Default     0x0                  0      0    kernel: Process[1895] crashed: promotedcontentd. Too many corpses being created.

"promotedcontentd" is a part of simulator. something makes it crashing.

good news - it does not crash on clean machine. I'll try to find what makes it crash

[fpkamp]

hi @ilia-shipitsin just wondering if you had more time to look into what was causing the crash?

[ilia-shipitsin]

it appeared to be more complex.

for some versions of XCode (like 15.0.0 Beta 5) it crashed, but for later (like Beta 8) it does not crash. however, resource consumption is very high in all cases.

it does open url, but it takes quite a lot of time to complete. we run on anka vm on top of mac mini, I'm trying to figure out what is bottleneck and whether we are supposed to be able to run 2-3 simulators at a time or not

[christiangal-indi]

Hi I have a different issue running my tests on the simulator, but it might be related to the slowness of the simulator as the same tests on Xcode14 run ok and it is random on the Xcode15.

UITests-Runner (9588) encountered an error (The test runner failed to initialize for UI testing. (Underlying Error: Timed out while loading Accessibility.))

[ilia-shipitsin]

@christiangal-indi , your issue may be either related or unrelated to this issue.

feel free to report your issue separately with repro steps. we can mention this issue if they are related

[joshuapoq]

@christiangal-indi we're seeing the same thing. Upgraded to Xcode 15 (but it's the same problem described by this thread) that the iOS 17.0 runners are completely unusable. They're taking around 10m to launch and then either fail the build or run incredibly slowly causing our builds to timeout (which has a huge knock on effect).

This might be on Apple and related to: https://developer.apple.com/forums/thread/737695

We'll likely have to downgrade to Xcode 14 for validation lanes but use Xcode 15 for releases. Perhaps this will become more stable when the MacOS 14 images are released.

EDIT: Apologies for duplicate messages - looks like GitHub / Safari has a cache bug that's not showing them even after reloading.

[NorseGaud]

Just wanted to include my 2-cents here: it's not even using any resources to make sense of why it's taking forever to boot.

Lots of this log spam from the Console for the simulator:

Connection for <APSConnection: 0x6000002802a0> differs from current queue!!
Unable to copy activation identity: _APSCopyActivationIdentity() failed. Device is probably unactivated. Do you need to hacktivate? {error: (null)}
<APSKeychainClientIdentityProvider: 0x60000332a280> SecItemDelete() failed: -25300
<APSUserCourier 0x1595061a0 development 0>: Terminating connection, deleting identity and clearing public token.
apsd (CFNetwork) Connection 4820: missing error, so heuristics synthesized error(1:53)
apsd (CFNetwork) Connection 4818: encountered error(1:53)

Seems like Xcode 15 was a little untested by Apple. Seeing a few other issues too.

[NorseGaud]

I've also enabled verbose logging for simulators with xcrun simctl logverbose 4E32F7BB-8248-46BA-93DA-9571E0FB9F3F enable and then ran xcrun simctl diagnose --udid=12ABA5FC-2F7C-4C9E-B6BD-B6E8BECE78B6 -X which dumps a ton of useful logs for Apple to use and diagnose.

[wtto00]

It seems like it's an issue with Xcode 15. I ran a sample example: https://github.com/wtto00/test-github-action/actions/runs/6397195336

sudo xcode-select -s "/Applications/Xcode_15.0.app"
IPHONE_MODEL="CA968C7D-0661-4517-9E8F-06A54215C5FD" # iPhone 15 with iOS 17.0 on xcode_15_macos_13

# IPHONE_MODEL="E8C11E27-784B-4FC0-AB92-D79C0307B337" # iPhone 14 with iOS 16.4 on xcode_14_macos_13
# IPHONE_MODEL="92DF7F17-8DA7-400E-975F-7AA7C5004EA0" # iPhone 14 with iOS 16.2 on xcode_14_macos_12

rm -rf ~/Library/Caches/*
rm -rf ~/Library/Developer/CoreSimulator/Caches/*

open -a Simulator

xcrun simctl list

xcrun simctl boot "$IPHONE_MODEL"

echo 'wait for booted'
while true; do
  status=$(xcrun simctl list | grep "Phone:" | grep "$IPHONE_MODEL" | awk '{print $5}')
  echo $(xcrun simctl list | grep "Phone:" | grep "$IPHONE_MODEL")
  echo "status: $status"
  if [[ "$status" == "(Booted)" ]]; then
    break
  fi
  sleep 1
done

echo 'openurl'
xcrun simctl openurl $IPHONE_MODEL "https://google.com"

When running iPhone 15 with iOS 17.0 on xcode_15_macos_13, it will throw an error like this:

An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=60):
Simulator device returned an error for the requested operation.
Operation timed out
Underlying error (domain=NSPOSIXErrorDomain, code=60):
    The operation couldn’t be completed. Operation timed out
    Operation timed out

But the other two cases will not throw an error.

[NorseGaud]

Apple just released iOS 17.0.1 Simulator Runtime so maybe they've included some fixes for these problems.

[NorseGaud]

Xcode 15.1 release notes

Resolved Issues

Fixed: Creating a new visionOS App target sets the Application Scene Manifest (Generation) target build setting to Yes by default. Changing to Application Scene Manifest using the Info dictionary editor doesn’t remove the build setting “Application Scene Manifest (Generation)” as expected, which will override changes with the default value. (109428090)

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776)

Fixed: Swift macro definitions from the macOS/iOS/tvOS/watchOS SDKs are not available in Swift Playgrounds. (112122752)

Fixed: When building with the visionOS SDK, TargetConditionals.h incorrectly sets TARGET_OS_IOS to 1 instead of 0. (112716373)

Fixed: The simulator may crash when opening Settings or Action Button settings on iPhone 15 Pro devices. (115388496)

[NorseGaud]

Just ran a test on Xcode 15.1 beta 15C5028h AND 17.0.1, but Simulator boot times are still super slow or don't load at all :(

#!/usr/bin/env bash
set -exo pipefail
ID="${RANDOM}"
git clone https://github.com/veertuinc/SwiftVoxel.git "SwiftVoxel${ID}"
cd "SwiftVoxel${ID}"
xcrun simctl list --json devices available; sleep 20 # fix a weird bug where xcrun simctl list --json devices available is empty the first run
SIM_VER="$(xcrun simctl list --json devices available | grep name | grep Pro | head -1 | cut -d'"' -f4)"
mkdir -p "/tmp/${ID}"
xcodebuild -workspace SwiftVoxel.xcworkspace -derivedDataPath "/tmp/${ID}" -scheme SwiftVoxel -destination "platform=iOS Simulator,name=${SIM_VER}" build
SIMID=$(xcrun simctl create ${ID} "com.apple.CoreSimulator.SimDeviceType.$(echo ${SIM_VER} | sed 's/ /-/g')")
cleanup() {
    xcrun simctl shutdown "${SIMID}" || true
    xcrun simctl delete "${SIMID}"
}
trap cleanup EXIT
xcrun simctl boot "${SIMID}"
#xcrun simctl logverbose "${SIMID}" enable
sleep 120
open /Applications/Xcode.app/Contents/Developer/Applications/Simulator.app
xcrun simctl install "${ID}" "/tmp/${ID}/Build/Products/Debug-iphonesimulator/SwiftVoxel.app"
BUNDLE_ID="$(defaults read /tmp/${ID}/Build/Products/Debug-iphonesimulator/SwiftVoxel.app/Info.plist CFBundleIdentifier)"
xcrun simctl launch "${ID}" "${BUNDLE_ID}"
sleep 120

Completely fine on Xcode 14 :\

[ilia-shipitsin]

hello, @NorseGaud , @wtto00 , @joshuapoq , @fpkamp , @stancho-stanchev

I've made some investigation, so far it looks like

1. simulator is started
2. it creates additional multicast targets
3. mDNSResponder on the host is happy to log those messages
4. IO is wasted by logging/mDNS

mDNSResponder on the host seem to be nonneccessary and it is a leftover os using desktop mac mini. in my tests I gained some speedup by

   sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSresponder.plist
   sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSresponderHelper.plist
   sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.airportd.plist

can you please try those commands in your pipelines to check

1. whether it will bring some side effect
2. whether it provides speedup or not

thanks!

[mikehardy]

@ilia-shipitsin - I tried that as I'm very interested in restoring performance to the macOS-12 levels, but it appeared to make DNS in general non-functional. Test results indicate that after running that it is no longer possible to resolve anything so git checkout fails.

Perhaps it is possible to carefully order when these commands are run such that the only activity after running them is machine-local but that seems like a fragile idea so I won't pursue it.

Runs linked to failure line, "macOS-13-runner speedup" steps are just above them: https://github.com/invertase/react-native-firebase/actions/runs/6969529848/job/18965658862#step:3:58 https://github.com/invertase/react-native-firebase/actions/runs/6969529845/job/18965654159#step:6:59

[ilia-shipitsin]

@mikehardy , thank for your feedback. can you try unloading diagnosticd instead ?

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.diagnosticd.plist

(also, I've found that simulator has its own diagnosticd, I'm looking at it)

[mikehardy]

@ilia-shipitsin with that one line unloading diagnosticd.plist my workflow was functional (a good thing of course) but performance did not seem to be greatly enhanced if enhanced at all. Performance testing usually requires a large number of pre/post runs which I don't have the time to do, so I can't tell you with any confidence if it was some small percentage faster or not, but I can say that the difference between macos12 and macos13 runners (and/or Xcode15/ios17 vs behavior) was kind of a "two or more times slower" performance hit, and your change was not enough to restore that large of a difference - it was marginal if anything

Hopefully that's useful. I won't have time to do much more testing today as I was just piggy-backing on a PR that was running workflows anyway and I need to move on

[ilia-shipitsin]

@mikehardy , I'll try different things on https://github.com/invertase/react-native-firebase

[ilia-shipitsin]

@mikehardy , is there a branch for macos 12 ? (I would like to compare things between macos-12 and macos-13 using that repo)

[mikehardy]

@ilia-shipitsin hmm not easily. I think you could change one line that specifies the simulator to something available on both and it would work though?

https://github.com/invertase/react-native-firebase/blob/main/tests/package.json#L66

https://github.com/actions/runner-images/blob/main/images/macos/macos-12-Readme.md#installed-simulators https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.md#installed-simulators

"iPhone 14" is allegedly available for the value of Xcode that is "xcode-current" on macOS12 and macOS13, although that wasn't working for me for some reason on Xcode 15 - I don't recall trying too hard to investigate or fix it though

If that doesn't work, could easily add a matrix check in the workflow yaml you are working with such that you execute a "tests:ios:test-cover" for macOS-13 and a new "tests:ios:test-cover-macos12" script you add here that references a new "ios.sim.debug.macos12" stanza you add here with iPhone 14 for macOS12

a little bit of a detour vs your goal here but I think that would work well and be quick to do

[ilia-shipitsin]

@mikehardy , thank for the tip. I'm able to run "iPhone 14" on both macos-12 and macos-13. give me some time to compare results.

actually, macos-12 runs test in 7 min while macos-13 need ~50 min even with "yeetd" enabled. no idea why, but we can try to find out

[mikehardy]

@mikehardy , thank for the tip. I'm able to run "iPhone 14" on both macos-12 and macos-13. give me some time to compare results.

actually, macos-12 runs test in 7 min while macos-13 need ~50 min even with "yeetd" enabled. no idea why, but we can try to find out

macos-13 has been about that painful, yes 😂. Appreciate that you're examining, I believe everyone will appreciate any progress!