search

erlef / build-and-packaging-wg

Build and Packaging Working Group
19 stars 5 forks source link

OTP macOS builds #80

Open wojtekmach opened 1 month ago

wojtekmach commented 1 month ago

I'd like to propose setting up a central place where macOS binary builds can be downloaded from.

The main use cases are:

Existing Solutions

Worth mentioning that the Hex team maintains OTP builds for Ubuntu: https://github.com/hexpm/bob?tab=readme-ov-file#erlang-builds. This is what erlef/setup_beam uses on Linux. At the moment, Bob (an Elixir app) regularly checks the OTP repo for new releases and builds them. We'd like to eventually move from that process to using GitHub actions so there's less infrastructure to maintain for the Hex team, more visibility into the builds, and people can more easily contribute (they can test their changes in a fork, for example).

Dynamic vs Static Linking Tradeoffs

OTP depends on third party software like OpenSSL and wxWidgets and can be dynamically or statically linked to these.

Dynamic linking results in smaller binaries, a given version of, say, OpenSSL, is shared between other programs that need it. If there is a vulnerability and a new version out, we can do brew upgrade openssl and all programs are patched (so to speak). On the other hand, an automatically updated OpenSSL (say jumping from 1.1.1 to 3.0.0) could break OTP crypto app.

The biggest benefits of static linking is the things work out of the box without installing additional dependencies. Additionally, it's a rare use case in the grand scheme of things, but Livebook ships one click installers for macOS and Windows and we REQUIRE an OTP with statically linked OpenSSL.

The main focus here is for development so lack of easily upgradable OpenSSL is perhaps not a great concern.

The OTP team maintains binary releases for Windows and both OpenSSL and wxWidgets are statically linked.

Goals

Non-Goals

Proposed Solution

Create https://github.com/erlef/otp_builds.

The repository would contain three things:

  1. A .github/workflows/build.yml that builds OTP. The build would be triggered using workflow_dispatch event. This way we can trigger the build manually from GitHub UI or using GitHub REST API. After the build is built, it is uploaded to GitHub releases mentioned below. One nice thing is the build log would be public, if something went wrong, anyone could jump in and investigate.

  2. Git tags and GitHub releases that follow OTP releases, e.g. OTP-27.0, OTP-27.0.1, etc. We could additionally have tags for master, maint, and maint-<release> which would be built, say, nightly. This is slightly unusual in that all git tags would point to a commit in this repo (with mostly just build scripts) as opposed to the source code the build was made from. On the other hand, using GitHub releases is very practical. They are discoverable in the UI, the release assets are easily downloaded (https://github.com/{user}/{repo}/releases/download/{tag}/{file}) and managed (REST API, gh CLI, etc) programatically. If builds are downloaded for GitHub Actions CI runs, it would make sense downloading from GitHub releases would be faster and more reliable than from, say, builds.hex.pm.

  3. A builds/{os}-{arch}.txt file in the git repository. It would be automatically updated by build job. The format could be similar to https://builds.hex.pm/builds/otp/amd64/ubuntu-24.04/builds.txt. Tools like asdf and erlef/setup-beam could easily fetch this file, https://github.com/{user}/{repo}/raw/{branch}/{file}, to know all available builds.

The builds would ideally be triggered by github.com/erlang/otp which is very easy to do.

Hosting this under https://github.com/erlef is not strictly necessary, all compute and storage can be done on GitHub free plan. Given the non-profit status, it mighe be easier to ask for more compute and storage, if needed.

Alternative Solutions

Possible Future Work

Proof of Concept

https://github.com/wojtekmach/otp_builds

wojtekmach commented 1 month ago

I previously submitted a PR to OTP: https://github.com/erlang/otp/pull/5723. This is another attempt at a community maintained solution.

josevalim commented 1 month ago

👍 this type of project makes perfect sense to be hosted here, since it benefits a big part of the community (and if we were to host Ubuntu/Debian builds, that’s a huge majority).

paulo-ferraz-oliveira commented 1 month ago

@wojtekmach, great initiative. Me and @starbelly started https://github.com/jelly-beam/otp-macos a while ago, with the goal of eventually (after testing and preliminary experimental integration into setup-beam) moving it to erlef. It runs on GitHub, where the images are released, and makes considerations around OpenSSL. We also include a builds.txt (setup-beam compatible) and would generate images every x hours. We have the goal (check the issues) to replace as much 3rd party as possible, to make it as independent of dependencies as possible. Would some of it fit this issue's goals? Or could we even discuss moving this to erlef and build on top of it?

rvirding commented 1 month ago

A very brief comment is that if possible maybe support older OTP versions. I sometimes need to run different versions. I don't know how much effort this would be.

ferd commented 1 month ago

Something interesting to at least specify here is that since it'll use static linking, you should specify what your policy will be with regards to upgrading openssl versions in images after security advisories. The reason for this is that if you keep old images static, you're probably fine for CI purposes, but it's going to be a risk for releases installed on user systems because vulnerable libraries may be linked into the app.

So either the builds would need to be re-built, or a clear warning explaining this limitation should be mentioned.

starbelly commented 1 month ago

Something interesting to at least specify here is that since it'll use static linking, you should specify what your policy will be with regards to upgrading openssl versions in images after security advisories.

Indeed and this was something I wanted to figure out before anything was finalized. In the worst scenario every build has to be invalidated, and a massive re-build must happen, which means service interruption if you will. I think we need to ask the question again : Why do we want to statically link? The main reason I thought about the time was so you that your build was portable, but that's not exactly going to be true is it? Even if you statically link, you're still going to have other dependencies (libc and friends). Playing devil's advocate to myself here, but yeah.

paulo-ferraz-oliveira commented 1 month ago

In the worst scenario every build has to be invalidated

You can probably build a new release, with a newer OpenSSL, and replace the reference in the list of builds. Consumers just need to know they shouldn't cache endpoints to releases. e.g. if Hex/Bob replaces a build in builds.txt, today, used by setup-beam (results may vary for asdf) service is not interrupted.

And are any of these considerations done, at present, for the builds setup-beam uses already, for Ubuntu (Hex/Bob) and Windows (erlang/otp)?

wojtekmach commented 1 month ago

My understanding is macOS ships with LibreSSL but without header files and there is a recommendation to bring your own SSL library anyway: https://rentzsch.tumblr.com/post/33696323211/wherein-i-write-apples-technote-about-openssl-on.

In my testing the build is quite portable. I didn't do a thorough testing but I tried it on a few laptops and fresh VMs and it worked. My understanding is there's no glibc/musl on macOS, there's one libc (libSystem) that Apple maintains.

One of the goals I listed is:

Statically linked OpenSSL and wxWidgets.

And to be fair that's not quite what I am after, or rather, that's not a goal but a solution to reach a goal. I want to download an OTP on my Mac that just works, I don't have to install anything else (no homebrew, macports, etc). I'd personally be OK with OTP wx app not working out of the box (e.g. you need to install hombre and brew install wxwidgets) but I think it'd unnecessary friction. A lot of learning resources mention observer.

Does anyone know what is OTP security policy for Windows builds?

wojtekmach commented 1 month ago

@rvirding how far would you prefer to go back? I would consider OTP 25 as the bare minimum. Pertinent to builds, it adds JIT for arm64, OpenSSL 3.0 support, and relocatable installation directory.

wojtekmach commented 1 month ago

I have updated the proposal with a proof of concept implementation, currently under my personal account:

https://github.com/wojtekmach/otp_builds

paulo-ferraz-oliveira commented 1 month ago

Does anyone know what is OTP security policy for Windows builds?

I have no specific knowledge about this, but erlang/otp is a good place to ask (or maybe the Erlang Forum?) and information would be more acessible for future reference.

garazdawi commented 1 month ago

Does anyone know what is OTP security policy for Windows builds?

We don't update installers after they have been released. Since we only depend on libcrypto its very rare for there to be vulnerabilities that effect Erlang.

cocoa-xu commented 3 weeks ago

@rvirding how far would you prefer to go back? I would consider OTP 25 as the bare minimum. Pertinent to builds, it adds JIT for arm64, OpenSSL 3.0 support, and relocatable installation directory.

In my otp-build repo, it can go back to OTP 24.0 for x86_64 macOS and 24.3 for arm64 macOS. I also have support for some BSD systems.

wojtekmach commented 3 weeks ago

@cocoa-xu I didn’t know you maintain OTP binary builds too but can’t say I’m surprised. This is awesome!

I noticed you have Linux builds per libc, not per distribution, which is something @tsloughter was keen on.

Your solution is much more comprehensive than what’s proposed here. If our goals align should we use your implementation instead? How do you approach OpenSSL and wxWidgets?

wojtekmach commented 3 weeks ago

I have added beamup to the use case list in this proposal, congrats on the release @tsloughter! :)

cocoa-xu commented 3 weeks ago

I noticed you have Linux builds per libc, not per distribution, which is something @tsloughter was keen on.

Yeah, and part of the reason for this is for containers, especially for people who want to use alpine ones which are using musl libc.

How do you approach OpenSSL and wxWidgets?

For OpenSSL I also have a repo that's monitoring the latest OpenSSL version and does the precompilation workflow here, https://github.com/cocoa-xu/openssl-build (although iirc on macOS it might not be able to do a full statically-linked build); As for wxWidgets, it's not included yet (as I was mainly targeting for CI) but I believe it won't be too hard to statically link that.

Your solution is much more comprehensive than what’s proposed here. If our goals align should we use your implementation instead?

I think our goals definitely would align, and I'll be happy to add support for wxWidgets and generate a Hex/Bob compatible builds.txt file!

tsloughter commented 3 weeks ago

@wojtekmach thanks! still needs to have support for Erlang binary release install but that is up their on my priority list after finishing up the missing commands to be feature equivalent to erlup.

@cocoa-xu ah, your per-libc for alpine makes sense.

I was thinking about just setting up builds per-distro and contributing that since it wouldn't be too bad with docker. Starting with Fedora since that is what I use.

I doubt I'll be able to catch for a user when an install breaks because they upgraded distro versions though. Usually they won't, even though library versions installed changed, it'll keep working. But when there is a change that breaks the build it would be so nice to not just crash but be able to say, "maybe you should reinstall since OTP-27.0 was built for Fedora 39 not your current 42".

But I'm execve'ing the executable of commands like erl, not going to be able to recover a crash there.

Maybe just a command of upgrade-possibly-outdated-installs so I don't have to actually keep track of what versions of what cause breakage and leave it up to the user to run if they think it might help, hehe.

wojtekmach commented 3 weeks ago

Your solution is much more comprehensive than what’s proposed here. If our goals align should we use your implementation instead?

I think our goals definitely would align

@cocoa-xu after thinking about it some more, I think the end goal is something like your project where we support as many targets as possible, but I think it's better to start small after all. As was mentioned here previously, with macOS support we will serve a good chunk of the community and down the road with Ubuntu probably a majority. I'm committing to maintaining macOS builds as I am a user. I'd commit to Ubuntu too (since it's moving responsibilities away from Hex/Bob) but that's about the extent I'm comfortable with at the moment.

wojtekmach commented 2 days ago

I've just opened the PR with the implementation of this proposal: https://github.com/erlef/otp_builds/pull/1