Set up community build bot for Linux/Aarch64 builds

[bvibber]

Per recent email list discussion I've proposed setting up a community-run build bot for Linux/Aarch64 (ARM64) builds of LLVM + binaryen that can be downloaded by emsdk for direct installation on compatible machines.

(This is a niche platform today but we can expect it to become big as Apple is planning to convert their entire Mac line over to ARM64 CPUs in the next two years. Macs will eventually want native macOS/ARM64 builds too but Linux/ARM64 builds will also be used there in things like Docker or other virtualized Linux environments too, and it's easier to bootstrap now with easy availability of Linux server VMs)

I'm seeing a small regression in building with emsdk on ARM64 that I'll check over, then I'll see if I can build using the same tool that's used for the CI builds that emsdk downloads and make sure we've got something compatible.

[otterley]

I've verified that on arm64 the output of normal.js and jsmath.js are identical. Not sure why, but I guess that's good? I can update the test case to suit.

[dschuff]

Slightly unrelated: Can you tell me what platform.machine() returns on python3 on arm64 Linux?

[otterley]

$ python3 -c 'import platform; print(platform.machine())'
aarch64

[otterley]

For reference: https://github.com/emscripten-core/emscripten/pull/16562

[kripken]

@sbc100

do you remember the rational behind this test?

Basically what you said. This tests the JS_MATH option which replaces musl tan() etc. with Math.tan() in JS, so it is testing that the results are close enough, allowing for differences on the JS side.

Any difference in JS is probably fine - I'm not sure how much the JS spec allows for variation, but it does have some nan leeway (to allow nan canonicalization in some VMs, for example).

I think its fine to change the test do assertLessThan(diff, 5) even just skip that test on arm64 for now.

Both options sgtm. I don't think there's anything to worry about here as we have fewer differences on ARM and not more.

[sbc100]

The first build of arm64/linux was published and is available via emsdk install 3.1.7.

[jsrz]

Hi All,

I'd like to propose shifting this build bot away from what @otterley's hosted setup, towards machines hosted through the works on arm program (https://github.com/WorksOnArm). The idea being that emscripten maintainers could manage this HW and build bot rather than @otterley.

Works on arm is an initiative to help in situations like this. Basically, we give access to Arm based HW for projects to setup their CI/CD loops on. Let me know what you think. FYI, I'm a employee at Arm (and can help with the works on arm program access if interested).

[PiotrSikora]

Another option is using QEMU's user mode (e.g. via Docker's new --platform flag). There is some overhead compared to a native build, but it pretty much "just works".

[otterley]

For a project the size of Emscripten, QEMU emulation of aarch64 on x86_64 isn't really viable - that emulation penalty is enormous in terms of build times. Even on a native aarch64 CPU, build times from scratch are 10-20 minutes long, and that's with multiple CPUs on hand. This would easily translate to hours under software emulation.

[otterley]

I'm happy to support @jsrz 's effort and can contribute my code to him. My work was intended to be a stopgap solution anyway.

[PiotrSikora]

For a project the size of Emscripten, QEMU emulation of aarch64 on x86_64 isn't really viable - that emulation penalty is enormous in terms of build times. Even on a native aarch64 CPU, build times from scratch are 10-20 minutes long, and that's with multiple CPUs on hand. This would easily translate to hours under software emulation.

If this is a "fire & forget" build that's done as part of CI/CD pipeline for releases, then I don't see an issue with 2-3 hour builds... but I'm working with projects that kick-off tests taking 2-3 hours on each PR, so I might be used to those numbers.

[jsrz]

Here is the works on Arm free cluster access information. Basically involves opening an issue. Linking this issue to it should explain things to them. We can also help push this along once the issue is opened.

https://github.com/WorksOnArm/equinix-metal-arm64-cluster

[sbc100]

Done: https://github.com/WorksOnArm/equinix-metal-arm64-cluster/issues/306

[otterley]

I think we can close this, now that Emscripten is being built automatically for Arm64 on EC2.

[kramarb]

Where can I find the new arm images?

[sbc100]

You can install them with ./emsdk install <version> or ./emsdk install latest-arm64-linux. Sadly its not always up-to-date but there are a few versions published now.

[sbc100]

Re-opening since many folks seem to be running into issues with the fact that we still only have some arm64 linux releases.

[danieloneill]

Coming up on the 3 year anniversary of this issue and it still doesn't seem fully resolved. What is needed (or what could a motivated contributor do) to ensure builds for each release are available for aarch64 in a timely manner? At the time of posting, the latest available release is 3.1.33 which isn't too far behind 3.1.41, but far enough to feel second-class. Is this an issue of builds for releases failing for aarch64 on the buildserver?

[reinhrst]

For people coming here hoping to find a Arm64 Docker image, I'm happy to share this gist

[jamsinclair]

Hey @otterley, if you're still working in related areas, is there any flexibility to increase the frequency of the emscripten arm64 builds on AWS? I understand this is entirely thanks to your (or Amazon's) generosity, so I understand if the bandwidth isn't there.

I'm sure many of us would be super thankful for more regular builds for arm64 🙇

---

Also on this topic, now that CircleCI has arm64 machines that can be easily used in pipelines, would it be too resource intensive to build and upload the binaries from the project itself? (Either here or over in emscripten-core/emscripten). This could allow for more seamless releases and easy multi-arch docker builds.

[sbc100]

I think the problem is that our current builders (the ones that actually produce the SDK binaries) run on google chrome infrastructure, and they don't current support arm64 machines: https://ci.chromium.org/p/emscripten-releases/g/main/console.

To have some releases build by github/circleci we would likely need to redesign the whole system. The simplest way might be to setup an automatic mirror of the emscripten-releases repository hosted on github.

[dschuff]

Another possibility might be if we could cross-build ARM64 binaries on our x86-64 Chromium builder. I think that would maybe just require a sysroot?

[jamsinclair]

Just a heads up it seems like the public S3 bucket that the linux arm64 EC2 builds were being shared from has either been made private or existing assets have expired.

I guess this may mean that there are no longer any automated builds for the foreseeable future?

[otterley]

Hi @jamsinclair, I had to disable public access to the S3 bucket. It was meant to share assets only with the Emscripten build team to place in their official repository, not to share assets with the public.

The current status of aarch64 support is that the last build created was the last one that worked. There were subsequent changes to various dependencies that broke the build, and I haven't had the spare cycles to work with the Emscripten teams to get it working.

Ideally the Emscripten team can produce a fully-working aarch64/arm64 build on their own. Unfortunately I don't have the resources right now to contribute to its ongoing maintenance.

[dschuff]

I'm experimenting with a cross-build for aarch64-linux based on Chrome's aarch64-linux sysroot. It's straightforward to create, but I have no way to test it right now. If it works, maybe we can check the code in, and someone can use it to build and run their own tests, or we can make it an unofficial bot (or set up some kind of circleCI testing on the emsdk side).

[dschuff]

I've uploaded an experimental aarch64-linux build using Chromium's sysroot at https://storage.googleapis.com/webassembly/wasm-binaries-aarch64.tar.xz but I have no way to test it. Can y'all tell me if the binaries work? It should be possible to just drop them in as a replacement for the emsdk binaries and libs from another platform.

[bvibber]

I've uploaded an experimental aarch64-linux build using Chromium's sysroot at https://storage.googleapis.com/webassembly/wasm-binaries-aarch64.tar.xz but I have no way to test it. Can y'all tell me if the binaries work? It should be possible to just drop them in as a replacement for the emsdk binaries and libs from another platform.

Unfortunately they do not; they are built for x86-64:

wasm-as: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped

Note you should be able to load up an emulated aarch64 Debian VM using Docker like this:

docker pull --platform linux/aarch64 debian:bookworm
docker run --platform linux/aarch64 --name debian-bookworm -h bookworm -e LANG=C.UTF-8 -it debian:bookworm /bin/bash -l

On a native arm64 machine this will be fast, on an x86-64 machine it should run the binaries via QEMU emulation. (It might still try to run x86-64 binaires if you give it to them but they'd fail without a glibc to link to, probably)

[sbc100]

BTW I think it should be possible to test there binaries using docker + qemu on your linux x84_64 machine. (See https://github.com/WebAssembly/wabt/pull/2380 for an example how to do this).

[dschuff]

Oops, I think I uploaded the wrong archive. I think I fixed, you can try again.

[bvibber]

Oops, I think I uploaded the wrong archive. I think I fixed, you can try again.

Good news is the binaries run, bad news is emscripten seems to want a much newer version of clang:

#include <stdio.h>

int main(int argc, const char **argv) {
    printf("Hello, world\n");
    return 0;
}

$ emcc -o test.wasm test.c
emcc: warning: LLVM version for clang executable "/home/brooke/src/emsdk/upstream/bin/clang" appears incorrect (seeing "13.0", expected "18") [-Wversion-check]
cache:INFO: generating system asset: symbol_lists/d84e367b9259bd8a3e5b31a38b0b4bd18fea7ae4.json... (this will be cached in "/home/brooke/src/emsdk/upstream/emscripten/cache/symbol_lists/d84e367b9259bd8a3e5b31a38b0b4bd18fea7ae4.json" for subsequent builds)
cache:INFO:  - ok
wasm-ld: error: unknown argument: --table-base=1
wasm-ld: error: unknown file type: /tmp/tmpz4o8dd9blibemscripten_js_symbols.so
emcc: error: '/home/brooke/src/emsdk/upstream/bin/wasm-ld -o test.wasm /tmp/emscripten_temp_u1q1h_0q/test_0.o -L/home/brooke/src/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten /home/brooke/src/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/crt1.o -lGL -lal -lhtml5 -lstandalonewasm-nocatch -lstubs-debug -lc-debug -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-debug-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmpz4o8dd9blibemscripten_js_symbols.so --strip-debug --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export-table -z stack-size=65536 --initial-memory=16777216 --max-memory=16777216 --stack-first --table-base=1' failed (returned 1)

[dschuff]

Sorry, another misconfiguration on my local machine... Should be good now.

[bvibber]

[updated: worked around this, it was bad matching version of emscripten]

We're getting warmer! :D Now llvm/clang and binaryen are too new ;)

emcc: warning: LLVM version for clang executable "/home/brooke/src/emsdk/upstream/bin/clang" appears incorrect (seeing "19.0", expected "18") [-Wversion-check]

emcc: warning: unexpected binaryen version: 117 (expected 115) [-Wversion-check]

[bvibber]

[update: yep, that was my fault lol]

Hmm, that may be an artifact of the previous emsdk linux/arm version being installed [for the emscripten components]. But I don't know how to get another version except by installing from source or going over to an x86 machine.

[bvibber]

WE HAVE LIFTOFF :D

Ok I manually copied over the latest release emscripten from an x86_64 VM, plus the latest version of the aarch64 wasm tools build, and it works :D

brooke@bookworm:~/src/test$ uname -a
Linux bookworm 6.6.12-linuxkit #1 SMP Thu Feb  8 06:36:34 UTC 2024 aarch64 GNU/Linux
brooke@bookworm:~/src/test$ emcc --version
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.56 (cf90417346b78455089e64eb909d71d091ecc055)
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

brooke@bookworm:~/src/test$ emcc -o test.js test.c && node ./test.js
Hello, world

[sbc100]

This is awesome news! Great works @dschuff

It would be good to some very minimal testing the of the arm64 binaryen on the x86_64 build host. If that is not possible we can also take are of that on the emsdk and emscripten builders which have access that arm64 circleci bots.

[dschuff]

If this is something it would be easy to do in a github action on the emsdk side, that's probably going to be easier in the short run that setting it up in Chromium. I'm not sure Chromium's infrastructure has a convenient emulator install.

[otterley]

Maybe something like https://github.com/uraimo/run-on-arch-action ?

On Fri, Mar 29, 2024 at 9:04 AM Derek Schuff @.***> wrote:

If this is something it would be easy to do in a github action on the emsdk side, that's probably going to be easier in the short run that setting it up in Chromium. I'm not sure Chromium's infrastructure has a convenient emulator install.

— Reply to this email directly, view it on GitHub https://github.com/emscripten-core/emsdk/issues/547#issuecomment-2027425444, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAXHJKT6GI4HSVDSSO5A33Y2WGJDAVCNFSM4OXMT3PKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBSG42DENJUGQ2A . You are receiving this because you were mentioned.Message ID: @.***>

[sbc100]

We already have linux arm64 testing as part of emsdk (via circleci): https://github.com/emscripten-core/emsdk/blob/e10826f9196bb0e704014af2252ef58fb4ba261b/.circleci/config.yml#L91-L98

[plasticalligator]

[sbc100]

This is now fixed and we build linux/arm64 binaries continuously on the main waterfall now: https://ci.chromium.org/p/emscripten-releases/g/main/console