search

nodejs / node

Node.js JavaScript runtime ✨🐢🚀✨
https://nodejs.org
Other
107.16k stars 29.37k forks source link

Heads-up: Potential Delay in macOS Infrastructure for Node.js v23 #55181

Open UlisesGascon opened 1 week ago

UlisesGascon commented 1 week ago

TL;DR:

We need to confirm it first, but so far, it seems like the macOS infrastructure won't be ready for the Node.js v23 release planned on 2024-10-15 (see Release Issue #1034). This is mainly due to a delay in the Orka migration (see Build Issue #3686).

Original discussion in Slack: Direct Link

Context

How did we arrive at this?

For a long time, the macOS infrastructure has been a challenge for us. Many collaborators and releasers have suffered from it (lack of space, downtimes, etc.). Due to recent events, we ended up having a single provider for the macOS machines, which offered us the opportunity to rethink the infrastructure and mitigate much accumulated technical debt.

After extensive planning for a suitable alternative, we decided to re-engineer our test and release machines using Orka (see Build Issue #3686), thanks to a revamped agreement and a fantastic sponsorship opportunity from MacStadium (read more).

We decided to move from static VMs to ephemeral VMs. This offered us the opportunity to solve all the historical issues we had and allowed us to rebalance the workload more easily. As a result, we are using our resources more efficiently.

What is missing?

In general, we have already finalized the most critical parts of the new infrastructure (new cluster, VPN connections, Orka deployments using Jenkins, image generation using Packer, etc.), but we need more time to finalize the actual testing and migration of the workloads in Jenkins to the new Orka cluster (including the setup of new nodes).

More details: Issue Comment #2278119351

The remaining tasks might take us a bit longer, as we need to wait for more resources to be deployed, and we need to test and patch all the images (macOS version and architecture combinations) in all the pipelines. This is due to the limited availability of the team. I was able to dedicate several weekends in a row to prepare the infrastructure, but currently, I am dealing with some personal things in the following weeks that will limit my availability.

More details: Issue Comment #2352851401

Why does the current infrastructure not support Node.js v23?

Well, Node.js v23 requires a newer version of XCode. This means that we need at least macOS (13.x) in order to support GCC 12 (see Node Pull Request #53668). Our current infrastructure only supports macOS 10.15 and 11.x, which are already obsolete (see). The new infrastructure includes support for macOS 13.x.

What is the impact in terms of the releases?

In practical terms:

Next Steps

Potential Options/Mitigations

TBD/TBC

mhdawson commented 1 week ago

Well, Node.js v23 requires a newer version of macOS (13.x) in order to support GCC 12 (see https://github.com/nodejs/node/pull/53668). Our current infrastructure only supports macOS 10.15 and 11.x, which are already obsolete. The new infrastructure includes support for macOS 13.x.

Is GCC12 already required to build v23? Want to understand if it is an option to build on the existing GCC level initially until we get the new machines. This does add the risk that we'll break people later when we move to the new gcc level, but we can weigh that risk versus not shipping macOS in the initial v23 release as an option.

richardlau commented 1 week ago

Well, Node.js v23 requires a newer version of macOS (13.x) in order to support GCC 12 (see https://github.com/nodejs/node/pull/53668).

This isn't quite correct as on macOS we use xcode which is clang based (nothing to do with GCC). However there did seem to be general agreement from those more familiar with macOS that we should be on a later version of xcode.

mhdawson commented 1 week ago

We won't be able to generate the signed binaries for macOS. As an example, check Node.js Latest Releases.

I think @ryanaslett was fixed the release machine, ignoring the version of the compiler used would there be any other blockers for generating the signed binaries?

richardlau commented 1 week ago

Is GCC12 already required to build v23?

On platforms that use gcc, 12 is the new minimum supported version for Node.js 23. When last checked main still built with gcc 10 but there was at least one PR that was failing to build with gcc 10.

That doesn't apply to macOS as xcode is clang based (see above).

mhdawson commented 1 week ago

I think the options that we want to confirm are possible and then choose from include:

1) Ship 23 on time with macOS based on limited testing in GitHub Actions 2) Ship 23 on time with macOS based on testing in GitHub actions and a single run on the test CI (not sure if this is possible or not) 3) Ship 23 on time without macOS 4) Delay 23 for ??? long

Please add any other options, and identify concerns if some of the options are not possible (for example 2 may not be in the current state).

I don't believe we should consider 4) as I don't think we can guarrantee how long might be ??? and its important to ship on our consistent schedule.

UlisesGascon commented 1 week ago

Please add any other options, and identify concerns if some of the options are not possible (for example 2 may not be in the current state).

The biggest issue will be signing and uploading the binaries. This is something we always handle using Jenkins and agents, not on our local machines. Not having signed releases would be worse than shipping version 23 without macOS support, as this would break many users and systems. macOS will flag the binaries when you try to install them if they are not properly signed.

richardlau commented 1 week ago

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

joyeecheung commented 1 week ago

One thing to note is that many developers use macOS anyway, so we should already get good coverage from local testing (or I think if macOS gets broken people would notice very soon). It's unknown what versions of macOS people are using, though.

mhdawson commented 1 week ago

The biggest issue will be signing and uploading the binaries. This is something we always handle using Jenkins and agents, not on our local machines. Not having signed releases would be worse than shipping version 23 without macOS support, as this would break many users and systems. macOS will flag the binaries when you try to install them if they are not properly signed.

I'm not quite understanding the issue here. I think @ryanaslett fixed the existing release machine and I don't think we've stopped doing existing releases so I'm not sure how it will be different for 23 in terms of signing and uploading the binaries?

UlisesGascon commented 1 week ago

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

I can make it!

UlisesGascon commented 1 week ago

I'm not quite understanding the issue here. I think @ryanaslett fixed the existing release machine and I don't think we've stopped doing existing releases so I'm not sure how it will be different for 23 in terms of signing and uploading the binaries?

The problem is the MacOS version that we use in the existing release Machine(s) is old (11.x), we will need at least 13.x in order to use a newer version of xcode that can support the release process for Node@23.

AFAIK we need to build the binary in that machine in order to sign it, but maybe this process is different in the Pipelines and we grab the binaries from a different place in order to sign them :thinking:

mhdawson commented 1 week ago

@UlisesGascon can you remind me why the release process is different for node 23, assume apple changed something and we have only incorporated the changes into the main/23 line so far?

targos commented 1 week ago

The main question is: would we be fine with releasing v23.0.0 with the current toolchain, and changing it (potentially breaking users on older macOS in a non-major release) later?

aduh95 commented 1 week ago

2. Ship 23 on time with macOS based on testing in GitHub actions and a single run on the test CI (not sure if this is possible or not)

Would it be possible to disable the Jenkins macOS jobs only on main? In that case, I think we could disable caching (assuming that would solve the disk space issue), meaning macOS builds on Jenkins would take a long time, but that would only affect release lines – and the relatively low frequency at which we need Jenkins builds on release lines, the jobs would be much less likely to build up.

ryanaslett commented 1 week ago

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

I can attend as well

mhdawson commented 1 week ago

Would it be possible to disable the Jenkins macOS jobs only on main? In that case, I think we could disable caching (assuming that would solve the disk space issue), meaning macOS builds on Jenkins would take a long time, but that would only affect release lines – and the relatively low frequency at which we need Jenkins builds on release lines, the jobs would be much less likely to build up.

I think we could do that through the version selector - https://github.com/nodejs/build/blame/main/jenkins/scripts/VersionSelectorScript.groovy although I'm not 100% sure that would catch all CI jobs but it should catch the main PR test jobs.

richardlau commented 1 week ago

Looks like the V8 13.0 update will require the new infra and won't compile on the existing machines.

Refs: https://github.com/nodejs/node/pull/55014#issuecomment-2389054494

This PR requires newer macOS versions in CI.