search

actions / runner-images

GitHub Actions runner images
MIT License
9.17k stars 2.84k forks source link

Xcode 14.3.1 is missing from macos-13 runner image #8341

Closed einar-notland closed 7 months ago

einar-notland commented 8 months ago

Description

Recently Xcode 14.3.1 went missing from the macos-13 runner image.

We use maxim-lobanov/setup-xcode to switch to our defined xcode version, and it started failing today.

Switching Xcode to version '14.3.1'...
Available versions:
0 │ '14.3.0' │  '14E222b'   │    'GM'     │  true  │ '/Applications/Xcode_14.3.app' │
1 │ '14.2.0' │   '14C18'    │    'GM'     │  true  │ '/Applications/Xcode_14.2.app' │
2 │ '14.1.0' │  '14B47b'    │    'GM'     │  true  │ '/Applications/Xcode_14.1.app' │

Error: Could not find Xcode version that satisfied version spec: '14.3.1'

According to the macos-13 readme, the runners should contain Xcode 14.3.1 (14E300c) ...And they should NOT contain Xcode 14.3.0 (14E222b)

Platforms affected

Runner images affected

Image version and build link

Version: 20230417.1 (It is not a public build)

Is it regression?

Yes, Version: 20230903.1 is working

Expected behavior

Expected Xcode 14.3.1 to be present in macos-13 runners.

Actual behavior

Xcode 14.3.1 is not present in macos-13 runner image. (Although Xcode 14.3.0 is...).

Repro steps

Use https://github.com/maxim-lobanov/setup-xcode to specify Xcode 14.3.1 on build job running on macos-13.

einar-notland commented 8 months ago

I noticed now that the image version I experienced this on (20230417.1) is older then where I see it working (20230903.1). So I guess I was unlucky to get a runner with an old image or something?

I dont really know how this image stuff works...so it might just be normal?

If this was a one-of, I guess this issue can be closed....

erik-bershel commented 8 months ago

Hey @einar-notland! You are exactly right. Old image happened to you. Please drop a link to the run. No mater public or private. Generally I couldn't say it's normal behaviour. But since macOS 13 is in beta some issues could persist.

einar-notland commented 8 months ago

I see, thx for the quick response.

It is an build job on an enterprise account, so you wont be able to access it. But good to know this was a random mishap rather then the images being wrong.

I can just close this issue. Hopefully you will be able to see it occur elsewhere so you can investigate further.

erik-bershel commented 8 months ago

@einar-notland good, I already found issue source. Will fix it next couple hours.

TellowKrinkle commented 7 months ago

I think I may have just gotten this as well, though I don't have a list of what Xcode versions were there, just that 14.3.1 wasn't https://github.com/PCSX2/pcsx2/actions/runs/6307790831/job/17124951431

einar-notland commented 7 months ago

I think I may have just gotten this as well, though I don't have a list of what Xcode versions were there, just that 14.3.1 wasn't https://github.com/PCSX2/pcsx2/actions/runs/6307790831/job/17124951431

Yes, seems like you also were unlucky and got the Image with version: 20230417.1

@erik-bershel I still see this error intermittently on our builds as well. Any news on the fix you were talking about here?

Valpertui commented 7 months ago

Same issue here: https://github.com/Spendesk/spendesk-ios/actions/runs/6310596987/job/17133656251

pepicrft commented 7 months ago

We are also experiencing this issue here

erik-bershel commented 7 months ago

Hey @pepicrft @Valpertui and @einar-notland. Thanks for the report. Should take couple hours to fix/update runners.

erik-bershel commented 7 months ago

Should be okay now.

pepicrft commented 7 months ago

Thanks @erik-bershel for the quick turnaround.

Mittchel commented 7 months ago

Experiencing this issue as well, as of 10 minutes ago:

xcode-select: error: invalid developer directory '/Applications/Xcode_14.3.1.app'

[error]Bash exited with code '1'.

This also happened on our macOS-12 agent by the way.

Jjagg commented 7 months ago

Hi, I'm running into the same issue. Getting image version 20230417.1 rather than 20230924.1 which previous builds used.

cho-trackman commented 7 months ago

We face this issue now in our Azure Pipelines. Anybody knows where can I see the status (in Azure) for the fix @erik-bershel did?

honkmaster commented 7 months ago

We have the same issue with Xcode 15 in Azure Pipelines since around an hour. xcrun: error: missing DEVELOPER_DIR path: /Applications/Xcode_15.0.app/Contents/Developer When can we expect a fix?

volovely commented 7 months ago

Happens to me as well. Image version is 20230417.1. Swift tools 5.8.0 are not available, the same as Cocoapods 1.13. I can not build the app

mortenkleveland commented 7 months ago

Also experiencing the same with Xcode 15 in Azure Pipelines. We are using multiple pipelines, and Xcode 15 is not available on macos-13 in all of them.

Image: macos-13
Agent: Azure Pipelines 3

Image version: 20230417.1

In this pipeline /Applications/Xcode_15.0.app cannot be found

Image: macos-13
Agent: Azure Pipelines 2

Image version: 20230924.1

Here Xcode 15 is found as expected

Edit: added image versions.

erik-bershel commented 7 months ago

Hey there! Got it. Will take about three hours to fix it. Thanks for the report. cc: @Mittchel @Jjagg @cho-trackman @honkmaster @volovely @mortenkleveland

radko93 commented 7 months ago

Seems like currently the macOS-13 image has Xcode 15 beta 8 build instead of the final version.

erik-bershel commented 7 months ago

Hi @radko93! May I ask you to provide more info? Screenshoot/log or whatever you have. As I see it should be 15A240d build which is the same as 15.0 RC1 or 15.0 release but not 15A5229m which is 15.0_beta8.

radko93 commented 7 months ago

@erik-bershel this is a run from last 30 minutes. It's using the previous Macos-13 image it seems

image
erik-bershel commented 7 months ago

@radko93 it's the same issue - old image. We are working on solving it right now.

wAuner commented 7 months ago

The macOS 13 image seems broken to me when it comes to Xcode. We're using azure pipelines. Up until now we were using /Applications/Xcode_13.4.1.app which now is gone. The path in the documentation does no longer exist.

xcode-select: error: invalid developer directory '/Applications/Xcode_14.3.1.app'

When I perform an ls /Applications/Xcode* in an agent I get:

/Applications/Xcode.app:

/Applications/Xcode_14.1.0.app:

/Applications/Xcode_14.1.app:

/Applications/Xcode_14.2.0.app:

/Applications/Xcode_14.2.app:

/Applications/Xcode_14.3.0.app:

/Applications/Xcode_14.3.app:

What's that? No 14.3.1 but 14.3.0 and 14.3? That just all looks very broken. We tried to use /Applications/Xcode.app but that is version 14.2.

Please fix this asap.

honkmaster commented 7 months ago

One could really ask oneself why such a serious mistake can happen in a service for which some people also pay quite a bit of money. We were unable to process any pull requests (as UI tests were not running) or create builds this working day. The status page for azure still does not show any issue.

https://status.dev.azure.com

Baza207 commented 7 months ago

We've started seeing the same. Agents versioned 20230417.1 seem to have started appearing in the last 4 hours or so instead of the expected 20230924.1 that was previously working with Xcode 14.3.1.

AndriiZakhliupanyi commented 7 months ago

We have a similar problem with not being able to use Xcode 15 on macOS-13. With image version: 20230924.1 all works.

volovely commented 7 months ago

@radko93 Any news here?

erik-bershel commented 7 months ago

Should be fine by now. But inform me please if it happen again. We are working to eliminate the causes. I'll keep this issue open until we resolved the root cause.

volovely commented 7 months ago

Xcode 14.3.1 works fine on macOS-13 image. Cocoapods 1.13 available as well. All seems to be working fine. Thank you @erik-bershel

swhitty commented 7 months ago

@erik-bershel I have just seen this on Azure pipelines again today 20230417.1

weisunOW commented 7 months ago

I'm also seeing the same issue on Azure pipeline just now.

Current image version: '20230417.1'

christiangal-indi commented 7 months ago

@erik-bershel We are seen the same issue again. Since friday some of my teams are reporting it. On Azure DevOps. Could you please take a look.

erik-bershel commented 7 months ago

Hello there! I see some small amount of 20230417.1 agents. Will update it in about two hours.

cc: @christiangal-indi @weisunOW @swhitty

jfversluis commented 7 months ago

Just piling on here! I guess I'm not adding much of value here anymore, but just this this 30 minutes ago: https://dev.azure.com/dotnet/CommunityToolkit/_build/results?buildId=99002&view=logs&j=7f63e264-4998-5012-ab3d-c5232e82c0a0&t=bc1d15fd-d770-4696-a970-ea36f283e3cc

jfk056 commented 7 months ago

Hi @erik-bershel ,

For us the issue was gone for some jobs, but the issue is again back and all the jobs now failing. Although when the agents were working with Xcode 15 each job took 2x more time to compile application code.

erik-bershel commented 7 months ago

Hey @jfk056! May I ask you to send a link to the failed run no matter private or public? At the time of writing your comment, the problem should have already disappeared - I want to “synchronise watches”.

jfk056 commented 7 months ago

So the most recent failed build is: https://dev.azure.com/RingierAG/blick-ios/_build/results?buildId=7733&view=logs&j=8975e814-cf16-59b9-15d2-8cf11ca16ccd&t=8e25fc47-10cb-50bf-00d9-c58194a402de

BohdanKubkin commented 7 months ago

Hello, so the 20230921.4 has the same issue. Missing both Xcode 14.3.1 and 15.0.

[14:20:19]: $ ls -d /Applications/Xcode* [14:20:19]: ▸ /Applications/Xcode.app [14:20:19]: ▸ /Applications/Xcode_13.1.0.app [14:20:19]: ▸ /Applications/Xcode_13.1.app [14:20:19]: ▸ /Applications/Xcode_13.2.1.app [14:20:19]: ▸ /Applications/Xcode_13.2.app [14:20:19]: ▸ /Applications/Xcode_13.3.1.app [14:20:19]: ▸ /Applications/Xcode_13.3.app [14:20:19]: ▸ /Applications/Xcode_13.4.1.app [14:20:19]: ▸ /Applications/Xcode_13.4.app [14:20:19]: ▸ /Applications/Xcode_14.0.1.app [14:20:19]: ▸ /Applications/Xcode_14.0.app [14:20:19]: ▸ /Applications/Xcode_14.1.0.app [14:20:19]: ▸ /Applications/Xcode_14.1.app [14:20:19]: ▸ /Applications/Xcode_14.2.0.app [14:20:19]: ▸ /Applications/Xcode_14.2.app

kode54 commented 7 months ago

I got a runner that didn't even have 14.3.anything, just up to 14.2.x. I need Xcode 15.

erik-bershel commented 7 months ago

We have carried out some work and eliminated the source of the problem with the emergence of new agents with an old, unsupported version of the image for macOS-13 and macOS-12-XL. At the moment, according to the logs, not a single outdated agent should be publicly available. I plan to close this issue upon completion of the current deployment of the new image for both of these OSes.

erik-bershel commented 7 months ago

Hi @kode54! You used wrong agent spec. Please change it from macos-latest which means macOS-12 to macOS-13.

erik-bershel commented 7 months ago

Closed as fixed.

gtbuchanan commented 7 months ago

I might have just had bad timing, but I may have gotten an old agent yesterday missing the mono framework (details here and pipeline run here). Could this be related?

christiangal-indi commented 7 months ago

Hi @erik-bershel, we are seen this issue again, this time with the macOS12. We are running a matrix and some of the jobs has one version and others has another version: Both versions below.

Agent name: 'Azure Pipelines 14' Agent machine name: 'Mac-1697530904413' Current agent version: '3.227.2' Operating System macOS 12.7 21G816 Runner Image Image: macos-12 Version: 20230921.4

Agent name: 'Azure Pipelines 13' Agent machine name: 'Mac-1697530931460' Current agent version: '3.227.1' Operating System macOS 12.6.9 21G726 Runner Image Image: macos-12 Version: 20230921.1

Could you please take a look? using the old version is making the job failed.

erik-bershel commented 7 months ago

Hey @christiangal-indi! Seems weird. May I ask you to send a link to the failed job no mater if private?

christiangal-indi commented 7 months ago

@erik-bershel Any news on the investigation?

erik-bershel commented 7 months ago

Hey @christiangal-indi! It's mainly because of the difference between Anka-based and VMware-based runners. Some macOS 12 runners are a bit different. We are working to reduce this difference now. But I can't even offer an approximate time frame. In general, some of the tasks fell on runners, which are built on the basis of an image similar to macOS-12-XL. That's where you got into trouble with it. It's likely that the next update to this image, which is happening right now, should fix the issue, but we need more information to be sure.

christiangal-indi commented 7 months ago

@erik-bershel Let me know once the change is done and I will gladly test it. I assume there is no way to only point to specific versions of a runner correct?

On the picture below, you can see that we are getting a random image, the ones that failed are the 3.227.1 and the ones that are running are 3.227.2 Screenshot 2023-10-18 at 17 50 13

erik-bershel commented 7 months ago

@christiangal-indi yep, it is not possible currently to pin specific version of a runner. If you provided more information about the error in the second version of the runner, then we would be able to work on the causes of the error itself. It is likely that this is a removable issue since it works fine on other version.

christiangal-indi commented 7 months ago

@erik-bershel Sure. The task get stuck on starting Android Emulator. And times out after a long time.

Success - it takes 10 min to start: 2023-10-18T14:58:38.9629880Z --- Start Emulator --- 2023-10-18T14:58:39.8882810Z daemon not running; starting now at tcp:5037 2023-10-18T14:58:40.2092040Z daemon started successfully 2023-10-18T15:10:50.6462120Z --- Emulator up and running in background --- 2023-10-18T15:10:50.6577110Z ##[section]Finishing: Create/Start Android-32 Emulator

Failed - Failed after 30min: 2023-10-18T14:56:39.2468540Z --- Start Emulator --- 2023-10-18T14:56:39.4133740Z daemon not running; starting now at tcp:5037 2023-10-18T14:56:39.7774750Z daemon started successfully 2023-10-18T15:29:48.4593270Z ##[error]The task has timed out. 2023-10-18T15:29:48.4734830Z ##[section]Finishing: Create/Start Android-32 Emulator