nodejs / unofficial-builds

Unofficial binaries for Node.js
https://unofficial-builds.nodejs.org
262 stars 55 forks source link

Node.js unofficial-builds project

https://unofficial-builds.nodejs.org/

This project is experimental: its output is not guaranteed to remain consistent and its existence is not guaranteed into the future. This project is in need of a community of maintainers to keep it viable. If you would like to join, please submit pull requests to improve the work here.

About

The unofficial-builds project aims to provide Node.js binaries for some platforms not made available officially by the Node.js project at nodejs.org. Node.js is used on a large variety of platforms, but the Node.js project, in consultation with the Node.js Build Working Group, maintains a limited set of platforms that it tests code on and produces binaries for.

This list of officially supported platforms is available in the Node.js BUILDING.md, where you can also find details in the official nodejs.org binaries section. Some platforms are "supported" in that they are tested by the Node.js test infrastructure, but they don't have binaries produced for nodejs.org. Other platforms receive minimal or no official support.

unofficial-builds attempts to provide basic Node.js binaries for some platforms that either not supported or only partially supported by Node.js. This project does not provide any guarantees and its results are not rigorously tested. Builds made available at nodejs.org have very high quality standards for code quality, support on the relevant platforms and for timing and methods of delivery. Builds made available by unofficial-builds have minimal or no testing; the platforms may have no inclusion in the official Node.js test infrastructure. These builds are made available for the convenience of their user community but those communities are expected to assist in their maintenance.

Builds

"Experimental" status for Node.js is defined as:

Experimental: May not compile or test suite may not pass. The core team does not create releases for these platforms. Test failures on experimental platforms do not block releases. Contributions to improve support for these platforms are welcome.

Therefore, it is possible that unofficial-builds may occasionally fail to produce binaries and fixes to support these platforms may need to be contributed to Node.js.

How

This project makes use of a server provided by the Node.js Build Working Group to compile and host binaries. Currently all binaries are produced on that server within specialized Docker containers. The possibility of future expansion to platforms that require alternative infrastructure to build is not excluded.

The server is configured according to the Ansible unofficial-builds role in the nodejs/build repository. This is executed via the create-unofficial-builds.yml playbook.

The build process operates as the nodejs user and in /home/nodejs which has the following layout:

The build process can be described as:

  1. This repository is cloned onto the unofficial-builds server whenever it is updated (triggered via github-webhook) and Docker images contained within the /recipes directory are built by means of the /bin/deploy.sh script which in turn calls the /bin/prepare-images.sh script.
  2. A periodic service runs every 5 minutes via systemd on the server which calls /bin/periodic.sh script.
  3. periodic.sh calls /bin/check-releases.sh for each release line being checked ("release", "rc", etc.). Any new versions that check-releases.sh finds are added to the build queue via /bin/queue-push.sh (the build queue uses a locking mechanism to prevent concurrent changes).
  4. periodic.sh calls /bin/build-if-queued.sh which will execute a build if there is at least one build in the queue and no builds are currently running. /bin/queue-pop.sh is used to atomically remove the next build from the queue. Note that only zero or one build per periodic run is executed. If the queue has more than one build, these will be deferred until later periodic runs.
  5. When build-if-queued.sh encounters a build in the queue that it can execute, it calls /bin/build.sh to perform the build. This script iterates through the images that have been pre-built from the /recipes directory, starting with the /recipes/fetch-source recipe that fetches the source file for the given version and validates official releases using GPG keys. Optionally, a recipe might have a should-build file which is used to determine if the recipe should run for a specific Node.js version. Each recipe is passed this source and is given a staging directory to place its binaries in. After all recipes are finished, builds are promoted to the https://unofficial-builds.nodejs.org/download/ directory along with a SHASUMS256.txt file and the index.tab and index.json files for that release type are updated.

How to add new target

  1. Add target dir in recipe, and ensure that the necessary functions are implemented according to the above process description.
  2. Add target to the bottom of the recipes list in bin/_config.sh. If the build should be prioritized place the recipe higher up and make note of it in the comments of the PR so there is a record of why the build should happen earlier.
  3. In order for the index.dat and index.json to index the new target, you will likely need to modify nodejs-dist-indexer so it understands the new filenames.
  4. After a nodejs-dist-indexer release, the new version will need to be listed in bin/_config.sh.
  5. Add or modify the README if necessary.

Manual build triggers

Admins with access to the server can manually trigger a build using the /bin/queue-push.sh command. e.g.

su nodejs # perform the action as the "nodejs" user so as to retain proper queue permissions
cd ~
unofficial-builds/bin/queue-push.sh -v v16.4.0 # queue a new build for "v16.4.0" - the "v" in the tag is necessary

Optionally it is possible to (re)build recipes for historical versions that are already hosted.

Important: Be aware the re-building historical releases will change the digest in the SHASUMS. A consistent digest is required by some consumers of builds, so certain recipes should not be rebuilt. Notably those that are used by the docker-node project, such as musl. A change in digest will lead to verification errors downstream. If you are unsure, check with other team members.

This can be done by adding the -r flag to the queue-push.sh command. e.g.

su nodejs # perform the action as the "nodejs" user so as to retain proper queue permissions
cd ~
unofficial-builds/bin/queue-push.sh -v v16.4.0 -r x64-debug -r musl # will only build the `x64-debug` and `musl` recipes for "v16.4.0"

This places "v16.4.0" into ~/var/build_queue which will be read on the next invocation of the build check timer. It may take up to 5 minutes for the build to start, at which point the log should be visible at https://unofficial-builds.nodejs.org/logs/.

The same process can be used to queue rc or test builds.

Local use

This repository is primarily intended for use on the unofficial-builds server but it can be used locally for testing purposes. The bin/local_build.sh script is designed to mirror the server build process but with local trigger and for one specific recipe at a time.

Setup for local builds

On deploy, this repository is placed within a the unofficial-builds home directory, it is intended to operate from a subdirectory of where the assets are build, it's $workdir is the parent directory of wherever it is located. The local_build.sh script will create some directories within its $workdir so it's best to create a new directory for it to operate in. So the steps for local build will be in general:

e.g. Login as normal user and clone this repository using the following commands to place it within an unofficial-builds-home directory:

rm -fr ~/Devel/unofficial-builds-home
mkdir -p ~/Devel/unofficial-builds-home
cd ~/Devel/unofficial-builds-home
git clone https://github.com/nodejs/unofficial-builds

In the script presented, the $workdir will be ~/Devel/unofficial-builds-home and it can be customized with a -w <newdir> argument to local_build.sh, with the limitation that this script, although they build Nodejs binaries for other platforms, the commands presented will only execute on a 64-bit Linux environment based on x86 cpus. All of those commands are running as a normal user.

Building

Once you have cloned this repository, you can build a specific recipe by running bin/local_build.sh with the recipe (an existing one or one you create within the recipes/ subdirectory) name and the Node.js version you want to build. e.g. (following previous example commands)

cd ~/Devel/unofficial-builds-home/unofficial-builds
bin/local_build.sh -r musl -v v21.0.0

A successful build will place the source in $workdir/staging/src/ and binaries in $workdir/staging/release/v21.0.0/ (where $workdir currently is ~/Devel/unnofficial-builds-home). All of those commands are running as a normal user.

You must erase all dockers layers before run a new recipe. Take in considerations that not all recipes can be built for all versions.

Team

unofficial-builds is maintained by:

Emeritus