Set up community build bot for Linux/Aarch64 builds

[bvibber]

Per recent email list discussion I've proposed setting up a community-run build bot for Linux/Aarch64 (ARM64) builds of LLVM + binaryen that can be downloaded by emsdk for direct installation on compatible machines.

(This is a niche platform today but we can expect it to become big as Apple is planning to convert their entire Mac line over to ARM64 CPUs in the next two years. Macs will eventually want native macOS/ARM64 builds too but Linux/ARM64 builds will also be used there in things like Docker or other virtualized Linux environments too, and it's easier to bootstrap now with easy availability of Linux server VMs)

I'm seeing a small regression in building with emsdk on ARM64 that I'll check over, then I'll see if I can build using the same tool that's used for the CI builds that emsdk downloads and make sure we've got something compatible.

[kripken]

Talking to @brion , wasm2c may be able to help here. While we don't like the idea of end users needing to build C builds, if the main infra produces C builds then those could be built by @brion on an ARM machine, which as a single C file is much simpler than LLVM with the whole build system there. And then those builds would be uploaded, so end users get fully compiled builds, and wasm2c is just a step in the middle.

This will still have some downsides (like there is no simd or pthreads in wasm2c builds atm), so it would be best to get normal builds up. But I think it's worth seeing about wasm2c builds on the main bots - just one more build target, and it doesn't need a special machine to do so (the fast Linux machines can do it).

[sbc100]

The problem not that that target machine can't build llvm from source (its full linux/arm installation). He can build the whole SDK just fine. The problem @brion wants to solve is having a pre-built set of binaries uploaded by the bots. So it mostly about infa maintenance. Converting to wasm2c doesn't seem to solve this specific problem AFAICT.

[kripken]

@sbc100 See https://twitter.com/brionv/status/1283165113248133121 and earlier tweets right before it - my understanding is that there are some difficulties with building from source.

It would be much easier to just build a C file - clang clang.c and that's it.

[sbc100]

Bit wait, I though the original discussion for this issue came of out request to have pre-built binaries because building from source was too slow? Am I remembering wrong?

If the problem is "I can't build from source on arm linux", then that seems like something that should be solvable. I'm sure llvm itself must be buildable on arm linux.

[kripken]

Maybe @brion you can elaborate on the specific issues you ran into, that you referred to in the tweet?

In general, I think it is useful to make things simpler. If we emit a C build that makes setting up community bots as easy as "create a VM, clang clang.c, clang wasm-opt.c etc." then that's helpful - no build systems, no need to install anything. As we have more platforms (not just ARM Linux but also ARM MacOS etc.) the easier we make this the more helpful it will be.

[bvibber]

The original problem is roughly:

• on Linux/arm64, ready-to-run release binaries cannot be installed with emsdk, requiring slow memory and disk-intensive local compilation of LLVM and binaryen before one is able to use emcc

Proposed solution is:

• provide community-buillt Linux/arm64 binaries alongside the official Linux/x86_64 and other binaries, so emsdk can download and install them without local compilation

There seem to be two ways to compile emscripten's dependencies:

• compile through emsdk (not used by the production releases, so may not be tested or reliable or compatible with binary downloads)
• compile through waterfall (used by the production releases, so is tested and reliable and will produce builds configured the same way as the release builds)

I've previously done patches to emsdk to allow building on Linux/arm & Linux/arm64 so once I fixed a small recent regression, building with emsdk does seem to work. However I'm not certain if it's the recommended way to be building for release packaging.

I made an attempt to build with waterfall as well, however quickly discovered that it hardcodes x86_64 as the architecture to build for, so when it fetches prebuilt dependencies like cmake or virtualenv they fail to execute on a Linux/arm64 system.

So there are three ways forward I think:

1. go ahead and use emsdk as a meta-build tool, at the risk of divergence from the release builds
2. modify waterfall to be aware of multiple architectures and either download native binaries for deps or require them to be installed in the environment
3. modify waterfall to produce a wasm2c build of all the output binaries and then on the community-run build bots do an intermediate stage build from the C source files into native Linux/arm64 (etc) binaries

There would additionally need to be agreement on how to store & distribute the community binaries and change emsdk to point at them.

I hope this clarifies what I'm thinking. Thanks!

[sbc100]

I think (2) would b easiest of all those options (and depending on the system tools seeems find to me).

Side note: I'm not sure what you mean by virtualenv.. as far as I know we don't use that in the waterfall. Do do download cmake binaries though.

I wish (1) was easier and it make be pretty sad that we have two completely different ways to build the SDK and the results look quite different. I would actually love to go back to shipping separate packages for llvm, binarien and emscripten like the emsdk does, but its fair bit of work, and probably not worth it right now.

[CarloCattano]

Also unable to build emscripten on pi 64 bit

Linux raspberrypi 5.10.63-v8+ #1496 SMP PREEMPT Wed Dec 1 15:59:46 GMT 2021 aarch64 GNU/Linux

./emsdk install 2.0.33
Resolving SDK version '2.0.33' to 'sdk-releases-upstream-cef8850d57278271766fb2163eebcb07354018e7-64bit'
error: tool or SDK not found: 'sdk-releases-upstream-cef8850d57278271766fb2163eebcb07354018e7-64bit'

[otterley]

@sbc100 @brion @kripken Hi! I'm with the Graviton team at AWS. We'd like to provide resources to the project so that it can be built on arm64. I'm happy to carry water and chop wood as needed.

What's the best way to get started?

You can also reach me at my email address, is the first 4 letters of my last name (fisc), followed by the first 2 letters of my first name (mi), at amazon.com.

[sbc100]

Awesome! Thanks for the offer to help out.

As far as I understand these are roughly the steps that need to happen:

1. Setup and arm64 linux machine that is somehow triggered by commits to https://chromium.googlesource.com/emscripten-releases/
2. On each commit run the ./src/build.py script from that repo
3. Upload the results to some publicly accessible storage bucket
4. Point emsdk at the uploaded assets

For (2) you might need to do some hacking to get the ./src/build.py to run on your infra

For (4) we might need/want to modify the emsdk/scripts/create_release.py script, or create a separate process for injecting the arm64 binaries into the emscripten-releases-tags.json file.

@brion does that sounds about right to you?

[otterley]

The build.py script appears to download a number of prebuilt binaries including CMake, Node.JS, and Java from https://wasm.storage.googleapis.com/ -- I assume this is to ensure reproducible builds vs. relying on whatever the distro may provide. These prebuilt binaries don't seem to be available for Linux/arm64, so the script terminates quickly.

1. Is this strictly required, as opposed to, say, using a Docker image?
2. If it's strictly required, how do we make those prerequisites available?

[sbc100]

We could modify the build.py such that it assume the presence of system installs of all of those things.

We run this script on chromium infrastruce over which we have limited control so its usefull for us that it is hermetic in the way. I don't believe using docker is an option for us, and also need to continue to run on both macOS and windows.

But modifying the script to also run on your infrastructure is fine... including avoiding those specific versions of the tools.

[sbc100]

BTW, would you be prepared to setup and instance that emscripten/emsdk developers could have SSH access too?

[sbc100]

(BTW I think we no longer need the Java install at all: https://chromium-review.googlesource.com/c/emscripten-releases/+/3529671)

[otterley]

modifying the script to also run on your infrastructure is fine... including avoiding those specific versions of the tools.

👍🏻

BTW, would you be prepared to setup and instance that emscripten/emsdk developers could have SSH access too?

I think we can get you remote access somehow. Not sure whether that'll be via SSH or SSM Session Manager, but some way.

[dschuff]

The other option here could be to do a cross-build, as we do for MacOS. If you have a Linux ARM64 sysroot, then an Intel build of clang that has the ARM64 backend enabled should be able to do cross-builds. That would mean that build.py wouldn't need to be updated.

Having said that, I don't think updating build.py would be all that hard, and we'd be happy to take patches. There are 2 ways you might go about it:

1. Use the same tool (depot_tools, specifically gclient) we and Chrome use to sync the build dependencies and sources. My guess is that it would probably actually work on ARM64 Linux since I know that it works at least on ARM32 Linux and ARM64 macOS (and it's all just Python). In that case you could modify the logic for syncing prebuilt CMake and node to point to your ARM64 linux versions (If those prebuilts exist, I could probably just add them to our storage bucket).

2. Handle syncing the sources yourself (DEPS files are easy to parse), and fix build.py to use CMake and node from the system rather than prebuilt.

Either way I think you want to bypass all the logic that tries to use Chromium's build package (clang, SDKs, etc) since I'm not sure they support Arm64 linux. build.py has an option --no-host-clang that does this already (although since we don't test it much, it's possible that it has issues you'd need to fix). You probably also want --no-sysroot since you probably don't want to use Chrome's build sysroot.

Once you have the build dependencies and sources, you can build: src/build.py --build-include=llvm,binaryen,emscripten

[sbc100]

I think given that we have some arm64 linux hardware available we might as well do a native build.. then we can run some tests too.

Yes, I should have mentioned that. You will need to use gclient to sync the emscripten-releases repo.. that is how you get all the dependencies (similar to git submodules).

[dschuff]

I updated my comment above. I'm not really sure whether gclient or manual sync would be easier. It might depend on whether they want to depend on our build deps or not.

[sbc100]

Given that we are proposing to using these binaries in emsdk releases I think it would be confusing to do anything but follow the DEPS. In fact I think its a pretty hard requirement. Version X of emsdk should produce the same output whatever platform one runs on.

[dschuff]

Yes, we have to follow the DEPS. It's just a matter of whether they want to run gclient to do that, or do it manually.

[sbc100]

Right, I guess may folks are not familiar with depot_tools or gclient. The basic steps are:

1. Install depot_tools in your PATH: https://www.chromium.org/developers/how-tos/install-depot-tools/
2. mkdir && cd
3. gclient config https://chromium.googlesource.com/emscripten-releases
4. gclient sync

Now you have emscripten-releases checked out along with all of its dependencies. Each time you modify DEPS to checkout a release with different DEPS you would re-run gclient sync to keep the deps in sync.

[otterley]

Here's how far I've gotten today.

This works so far:

# Install Depot tools as per https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up

$ gclient config https://chromium.googlesource.com/emscripten-releases
$ gclient sync

This results in emscripten_releases, install, and v8 repositories being downloaded. No problems there.

The auto-downloaded version of CMake was for x86. Replacing that was pretty easy by downloading the tarball from https://github.com/Kitware/CMake/releases/tag/v3.21.3, and updating build.py to suit.

I decided to follow how the build bot is configured, by looking at bots.json to see the build steps, and replicate them in my shell.

(It looks like ninja needed to be installed, so I took care of that.)

Then I ran into the following almost immediately.

$ src/build.py --no-sync --no-test --build-include=libcxx --use-lto=auto
Python version 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0]
sys.executable = /usr/bin/python3
EMSDK_PYTHON = /usr/bin/python3
subprocess.check_output(`/usr/bin/git status --porcelain /instance-store/em/emscripten-releases/DEPS.tagged-release`, cwd=`/instance-store/em/emscripten-releases`)
/instance-store/em/emscripten-releases/DEPS.tagged-release git status: (unchanged)
subprocess.check_output(`/usr/bin/git rev-parse HEAD`, cwd=`/instance-store/em/emscripten-releases`)
subprocess.check_output(`/usr/bin/git rev-list -n1 HEAD /instance-store/em/emscripten-releases/DEPS.tagged-release`, cwd=`/instance-store/em/emscripten-releases`)
Last rev modifying /instance-store/em/emscripten-releases/DEPS.tagged-release is b'd0e637fe48197587d981f79e8114757731d0c2a9', HEAD is b'da522a861fff8e70db2c412a894b2cbfae1a2385'
Change directory to: /instance-store/em/emscripten-releases/src
adding to path: /instance-store/em/emscripten-releases/cmake-3.21.3-linux-aarch64/bin
adding to path: /instance-store/em/emscripten-releases/node-v14.15.5-linux-x64/bin

@@@BUILD_STEP libcxx@@@
subprocess.check_call(`/instance-store/em/emscripten-releases/cmake-3.21.3-linux-aarch64/bin/cmake -G Ninja -DPython3_EXECUTABLE=/usr/bin/python3 -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/instance-store/em/install -DCMAKE_C_COMPILER=/instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang -DCMAKE_CXX_COMPILER=/instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang++ /instance-store/em/emscripten-releases/llvm-project/runtimes -DLLVM_ENABLE_RUNTIMES=libcxx;libcxxabi -DLIBCXX_ABI_VERSION=2 -DLIBCXX_HAS_ATOMIC_LIB=OFF -DLIBCXX_ENABLE_SHARED=ON -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF -DLIBCXXABI_ENABLE_SHARED=OFF -DLIBCXX_INCLUDE_TESTS=OFF -DLIBCXXABI_INCLUDE_TESTS=OFF -DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON -DLIBCXX_INSTALL_STATIC_LIBRARY=OFF -DLIBCXXABI_INSTALL_STATIC_LIBRARY=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON`, cwd=`/instance-store/em/emscripten-releases/build/libcxx-out`)
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- The ASM compiler identification is unknown
-- Found assembler: /instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang
-- Check for working C compiler: /instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang - broken
CMake Error at /instance-store/em/emscripten-releases/cmake-3.21.3-linux-aarch64/share/cmake-3.21/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler

    "/instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /instance-store/em/emscripten-releases/build/libcxx-out/CMakeFiles/CMakeTmp

    Run Build Command(s):/instance-store/ninja/ninja cmTC_e2284 && [1/2] Building C object CMakeFiles/cmTC_e2284.dir/testCCompiler.c.o
    FAILED: CMakeFiles/cmTC_e2284.dir/testCCompiler.c.o 
    /instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang    -o CMakeFiles/cmTC_e2284.dir/testCCompiler.c.o -c /instance-store/em/emscripten-releases/build/libcxx-out/CMakeFiles/CMakeTmp/testCCompiler.c
    /bin/sh: 1: /instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang: Exec format error
    ninja: build stopped: subcommand failed.

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:3 (project)

-- Configuring incomplete, errors occurred!
See also "/instance-store/em/emscripten-releases/build/libcxx-out/CMakeFiles/CMakeOutput.log".
See also "/instance-store/em/emscripten-releases/build/libcxx-out/CMakeFiles/CMakeError.log".
Exception thrown in build step.
Traceback (most recent call last):
  File "src/build.py", line 1548, in run
  File "src/build.py", line 1282, in BuildRepos
  File "src/build.py", line 1211, in Run
  File "src/build.py", line 1005, in LibCXX
  File "/instance-store/em/emscripten-releases/src/proc.py", line 86, in check_call
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/instance-store/em/emscripten-releases/cmake-3.21.3-linux-aarch64/bin/cmake', '-G', 'Ninja', '-DPython3_EXECUTABLE=/usr/bin/python3', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/instance-store/em/install', '-DCMAKE_C_COMPILER=/instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang', '-DCMAKE_CXX_COMPILER=/instance-store/em/v8/third_party/llvm-build/Release+Asserts/bin/clang++', '/instance-store/em/emscripten-releases/llvm-project/runtimes', '-DLLVM_ENABLE_RUNTIMES=libcxx;libcxxabi', '-DLIBCXX_ABI_VERSION=2', '-DLIBCXX_HAS_ATOMIC_LIB=OFF', '-DLIBCXX_ENABLE_SHARED=ON', '-DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF', '-DLIBCXXABI_ENABLE_SHARED=OFF', '-DLIBCXX_INCLUDE_TESTS=OFF', '-DLIBCXXABI_INCLUDE_TESTS=OFF', '-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON', '-DLIBCXX_INSTALL_STATIC_LIBRARY=OFF', '-DLIBCXXABI_INSTALL_STATIC_LIBRARY=OFF', '-DCMAKE_POSITION_INDEPENDENT_CODE=ON']' returned non-zero exit status 1.

@@@STEP_FAILURE@@@

@@@BUILD_STEP Summary@@@
Failed steps: 1.
    libcxx
Warned steps: 0.

@@@STEP_FAILURE@@@
Completed in 0.39360880851745605s

So right now, I'm stuck because the build appears to be an x86 version of LLVM under the v8 subrepository. (v8/third_party/llvm-build/Release+Asserts/bin/clang).

As an experiment, I also tried re-running with src/build.py --no-sync --no-test --build-include=libcxx --use-lto=auto --no-host-clang --no-sysroot, but it crashed and burned pretty hard with lots of compilation errors.

[otterley]

After installing clang (1:10.0-50~exp1, which comes with Ubuntu 20.04), and setting CC=/usr/bin/clang and CXX=/usr/bin/clang++, I was able to get the libcxx build to complete successfully.

[sbc100]

I think you probably will want to modify host_toolchains.GetPrebuiltClang() to just return tools from your PATH rather than using our prebuilt clang. It shouldn't really matter what version of clang you use to actually build the toolchain. As long as its recent enough to support building ToT clang.

So you will want to skip the SyncToolchain step and instead add some kind of option (either command line or environment variable) that allows system toolchain to be used.

[sbc100]

Oh wait.. we already have --no-host-clang command line option to do exactly what you want.

[otterley]

✅ binaryen build also completed successfully via src/build.py --no-sync --no-test --build-include=binaryen --use-lto=auto --no-host-clang --no-sysroot.

❌ llvm build is not successful:

-- Performing Test CXX_SUPPORTS_CUSTOM_LINKER
-- Performing Test CXX_SUPPORTS_CUSTOM_LINKER - Failed
CMake Error at cmake/modules/HandleLLVMOptions.cmake:308 (message):
  Host compiler does not support '-fuse-ld=lld'
Call Stack (most recent call first):
  CMakeLists.txt:774 (include)

[sbc100]

You can either install lld or somehow disable the -DLLVM_ENABLE_LLD=ON line in build.py.

[otterley]

✅ llvm built successfully after installing lld.

I had to also install the Node.js binary distribution from upstream.

🛑 Building emscripten fails because the google-closure-compiler-linux module is not blessed for arm64:

$ src/build.py --no-sync --no-test --build-include=emscripten --no-host-clang --no-sysroot 
Python version 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0]
sys.executable = /usr/bin/python3
EMSDK_PYTHON = /usr/bin/python3
Change directory to: /instance-store/em/emscripten-releases/src
adding to path: /instance-store/em/emscripten-releases/cmake-3.21.3-linux-aarch64/bin
adding to path: /instance-store/em/emscripten-releases/node-v14.15.5-linux-arm64/bin
Removing /instance-store/em/install/emscripten
Installing emscripten into /instance-store/em/install/emscripten
subprocess.check_call(`/usr/bin/python3 /instance-store/em/emscripten-releases/emscripten/tools/install.py /instance-store/em/install/emscripten`, cwd=`/instance-store/em/emscripten-releases/emscripten`)
Running npm install ...
subprocess.check_call(`/instance-store/em/emscripten-releases/node-v14.15.5-linux-arm64/bin/npm ci --production --no-optional`, cwd=`/instance-store/em/install/emscripten`)
added 48 packages in 0.654s
subprocess.check_call(`/instance-store/em/emscripten-releases/node-v14.15.5-linux-arm64/bin/npm install --production --no-optional google-closure-compiler-linux`, cwd=`/instance-store/em/install/emscripten`)
npm ERR! code EBADPLATFORM
npm ERR! notsup Unsupported platform for google-closure-compiler-linux@20220301.0.0: wanted {"os":"linux","arch":"x64,x86"} (current: {"os":"linux","arch":"arm64"})
npm ERR! notsup Valid OS:    linux
npm ERR! notsup Valid Arch:  x64,x86
npm ERR! notsup Actual OS:   linux
npm ERR! notsup Actual Arch: arm64

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ubuntu/.npm/_logs/2022-03-17T02_04_49_227Z-debug.log
Exception thrown in build step.
Traceback (most recent call last):
  File "src/build.py", line 1554, in run
  File "src/build.py", line 1288, in BuildRepos
  File "src/build.py", line 1217, in Run
  File "src/build.py", line 1070, in Emscripten
  File "src/build.py", line 1065, in InstallEmscripten
  File "/instance-store/em/emscripten-releases/src/proc.py", line 86, in check_call
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/instance-store/em/emscripten-releases/node-v14.15.5-linux-arm64/bin/npm', 'install', '--production', '--no-optional', 'google-closure-compiler-linux']' returned non-zero exit status 1.

@@@STEP_FAILURE@@@

@@@BUILD_STEP Summary@@@
Failed steps: 1.
    None
Warned steps: 0.

@@@STEP_FAILURE@@@
Completed in 2.8284225463867188s

👉🏻 There might be a workaround to install google-closure-compiler instead of the -linux prebuilt module. I'm investigating that. Separately, it's probably worth asking the module's developers to produce a ready-made arm64 build.

[sbc100]

BTW, I don't think we need a particularly recent version of node so just apt-get install node should work fine.

[sbc100]

Yes, the problem you have now is the the binary version of closure compile in the npm package is not available for arm64.

See the "Native Binary" section of https://www.npmjs.com/package/google-closure-compiler.

I would just make the npm install --production --no-optional google-closure-compiler-linux part conditional, and if running on arm just do npm ci .. on its own that should work and be sufficient.

You might want to talk to the closure compile npm folks about getting an arm64 linux binary version, but it should fall back to Java and/or JavaScript if that native version is missing (which should be good enough for now).

[otterley]

The following change to InstallEmscripten() in build.py does the trick:

    native = None
    if IsMac():
        native = 'google-closure-compiler-osx'
    elif IsWindows():
        native = 'google-closure-compiler-windows'
    elif IsLinux() and platform.machine() in ('i386', 'x86_64'):
        native = 'google-closure-compiler-linux'
    if native:
        proc.check_call(['npm', 'install', '--production', '--no-optional', native],
                        cwd=em_install_dir)

✅ emscripten now builds. ✅ archive now builds.

We now have a build!

Next steps:

1. I'll submit a PR with changes
2. Let's work out-of-band to determine the best way to get you the arm64/Linux build resource that your bot can use. Can one of you email me at the address above?

[sbc100]

Awesome news!

[naqvis]

Awesome 👏 . Thank you @otterley and @sbc100 for making it happened. I was struggling with getting this compiled for arm64 arch. Looking forward to seeing arm64 precompiled SDK made available to folks who are running things on ARM64 based systems.

[sbc100]

@otterley, if you are able to do a build today could do one of the 3.1.7 release (emscripten-release DEPS from 88f0cab4e7db846e171cbbbbf20cc1a51b8c779f). If you can attach it this issue or upload it somewhere I can take care of uploading it to google storage.. and we can add it to the emsdk repo so folks can try it out. Once we know it works we can think more about how to automate the publishing process

[otterley]

@sbc100 I don't see a refid for emscripten-release that matches that value, nor any branches or tags that might match a 3.1.7 release. Can you kindly walk me through the checkout procedure?

[sbc100]

We bless certain revisions in emscripten-releases and mark them as released here in the emsdk repository: https://github.com/emscripten-core/emsdk/blob/main/emscripten-releases-tags.json

For example 3.1.7 is here:

https://github.com/emscripten-core/emsdk/blob/858f73d26057cbbce972ed90f12db53627685dac/emscripten-releases-tags.json#L12-L13

I recommend using the 3.1.7-asserts tag which is 88f0cab4e7db846e171cbbbbf20cc1a51b8c779f. You can see the revision here: https://chromium.googlesource.com/emscripten-releases/+/88f0cab4e7db846e171cbbbbf20cc1a51b8c779f. Not sure why you wouldn't be able to see the revision?

[sbc100]

Our (rather convoluted) release process is documented here: https://github.com/emscripten-core/emscripten/blob/main/docs/process.md#release-processes

[otterley]

My fault - I did not look back far enough in the commit log and so I missed the refid. I was assuming, incorrectly, that it was recently made, but it is a 12-day-old commit. Building now.

[otterley]

🛑 Although the build succeeded, some tests in the test suite are failing, so I don't think it's quite ready to ship yet.

I ran: src/build.py --no-sync --no-build --test-include=emtest --test-param=cores

FAILED (failures=7, errors=2, skipped=79)

Some samples from the output:

ERROR: test_bullet_autoconf (test_core.cores)
-- configure stdout --
checking build system type... 
-- end configure stdout --
-- configure stderr --
configure: sh ./configure --disable-shared --host=i686-pc-linux-gnu --disable-demos --disable-dependency-tracking
configure: WARNING: If you wanted to set the --build type, don't use --host.
    If a cross compiler is detected then cross compile mode will be used.
./config.guess: unable to guess system type

This script, last modified 2006-07-02, has failed to recognize
the operating system you are using. It is advised that you
download the most up to date version of the config scripts from

  http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/config/config/config.guess
and
  http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/config/config/config.sub

If the version you run (./config.guess) is already up to date, please
send the following data and any information you think might be
pertinent to <config-patches@gnu.org> in order to provide the needed
information to handle your system.

config.guess timestamp = 2006-07-02

uname -m = aarch64
uname -r = 5.13.0-1017-aws
uname -s = Linux
uname -v = #19~20.04.1-Ubuntu SMP Mon Mar 7 12:55:31 UTC 2022

/usr/bin/uname -p = aarch64
/bin/uname -X     = 

hostinfo               = 
/bin/universe          = 
/usr/bin/arch -k       = 
/bin/arch              = aarch64
/usr/bin/oslevel       = 
/usr/convex/getsysinfo = 

UNAME_MACHINE = aarch64
UNAME_RELEASE = 5.13.0-1017-aws
UNAME_SYSTEM  = Linux
UNAME_VERSION = #19~20.04.1-Ubuntu SMP Mon Mar 7 12:55:31 UTC 2022
configure: error: cannot guess build type; you must specify one
emconfigure: error: 'sh ./configure --disable-shared --host=i686-pc-linux-gnu --disable-demos --disable-dependency-tracking' failed (returned 1)

-- end configure stderr --

FAIL: test_cube2hash_standalone (test_core.cores)
error: unable to create target: 'No available targets are compatible with triple "aarch64-unknown-linux-gnu"'

ERROR: test_freetype (test_core.cores)
-- configure stdout --

FreeType build system -- automatic system detection

The following settings are used:

  platform                    unix
  compiler                    /instance-store/em/install/emscripten/emcc
  configuration directory     ./builds/unix
  configuration rules         ./builds/unix/unix.mk

If this does not correspond to your system or settings please remove the file
`config.mk' from this directory then read the INSTALL file for help.

Otherwise, simply type `make' again to build the library,
or `make refdoc' to build the API reference (the latter needs python).

Generating modules list in ./objs/ftmodule.h...
* module: truetype  (Windows/Mac font files with extension *.ttf or *.ttc)
* module: type1     (Postscript font files with extension *.pfa or *.pfb)
* module: cff       (OpenType fonts with extension *.otf)
* module: cid       (Postscript CID-keyed fonts, no known extension)
* module: pfr       (PFR/TrueDoc font files with extension *.pfr)
* module: type42    (Type 42 font files with no known extension)
* module: winfnt    (Windows bitmap fonts with extension *.fnt or *.fon)
* module: pcf       (pcf bitmap fonts)
* module: bdf       (bdf bitmap fonts)
* module: sfnt      (helper module for TrueType & OpenType formats)
* module: autofit   (automatic hinting module)
* module: pshinter  (Postscript hinter module)
* module: raster    (monochrome bitmap renderer)
* module: smooth    (anti-aliased bitmap renderer)
* module: smooth    (anti-aliased bitmap renderer for LCDs)
* module: smooth    (anti-aliased bitmap renderer for vertical LCDs)
* module: psaux     (Postscript Type 1 & Type 2 helper module)
* module: psnames   (Postscript & Unicode Glyph name handling)
done.
cd builds/unix; ./configure  '--disable-shared' '--without-zlib'
checking build system type... 
-- end configure stdout --
-- configure stderr --
configure: sh ./configure --disable-shared --without-zlib
./config.guess: unable to guess system type

This script, last modified 2010-09-24, has failed to recognize
the operating system you are using. It is advised that you
download the most up to date version of the config scripts from

  http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD
and
  http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD

If the version you run (./config.guess) is already up to date, please
send the following data and any information you think might be
pertinent to <config-patches@gnu.org> in order to provide the needed
information to handle your system.

config.guess timestamp = 2010-09-24

uname -m = aarch64
uname -r = 5.13.0-1017-aws
uname -s = Linux
uname -v = #19~20.04.1-Ubuntu SMP Mon Mar 7 12:55:31 UTC 2022

/usr/bin/uname -p = aarch64
/bin/uname -X     = 

hostinfo               = 
/bin/universe          = 
/usr/bin/arch -k       = 
/bin/arch              = aarch64
/usr/bin/oslevel       = 
/usr/convex/getsysinfo = 

UNAME_MACHINE = aarch64
UNAME_RELEASE = 5.13.0-1017-aws
UNAME_SYSTEM  = Linux
UNAME_VERSION = #19~20.04.1-Ubuntu SMP Mon Mar 7 12:55:31 UTC 2022
configure: error: cannot guess build type; you must specify one
make: *** [builds/unix/detect.mk:84: setup] Error 1
emconfigure: error: 'sh ./configure --disable-shared --without-zlib' failed (returned 2)

-- end configure stderr --
subprocess.CalledProcessError: Command '['/instance-store/em/install/emscripten/emconfigure', 'sh', './configure', '--disable-shared', '--without-zlib']' returned non-zero exit status 1.

FAIL: test_autodebug_wasm_standalone (test_core.cores)
error: unable to create target: 'No available targets are compatible with triple "aarch64-unknown-linux-gnu"'
AssertionError: subprocess exited with non-zero return code(1): `/instance-store/em/install/bin/clang src.wasm.c -o src.exe`

FAIL: test_longjmp_standalone_standalone (test_core.cores)
error: unable to create target: 'No available targets are compatible with triple "aarch64-unknown-linux-gnu"'
AssertionError: subprocess exited with non-zero return code(1): `/instance-store/em/install/bin/clang test_longjmp.wasm.c -o test_longjmp.exe`

FAIL: test_posixtime_standalone (test_core.cores)
error: unable to create target: 'No available targets are compatible with triple "aarch64-unknown-linux-gnu"'
AssertionError: subprocess exited with non-zero return code(1): `/instance-store/em/install/bin/clang test_posixtime.wasm.c -o test_posixtime.exe`

FAIL: test_wasm2c_sandboxing_full (test_core.cores)
error: unable to create target: 'No available targets are compatible with triple "aarch64-unknown-linux-gnu"'
AssertionError: subprocess exited with non-zero return code(1): `/instance-store/em/install/bin/clang test_hello_world.wasm.c -o test_hello_world.exe`

FAIL: test_wasm2c_sandboxing_mask (test_core.cores)
error: unable to create target: 'No available targets are compatible with triple "aarch64-unknown-linux-gnu"'
AssertionError: subprocess exited with non-zero return code(1): `/instance-store/em/install/bin/clang test_hello_world.wasm.c -o test_hello_world.exe`

FAIL: test_wasm2c_sandboxing_none (test_core.cores)
error: unable to create target: 'No available targets are compatible with triple "aarch64-unknown-linux-gnu"'
AssertionError: subprocess exited with non-zero return code(1): `/instance-store/em/install/bin/clang test_hello_world.wasm.c -o test_hello_world.exe`

I also notice a bunch of warnings like:

/instance-store/em/install/emscripten/node_modules/google-closure-compiler-linux/compiler: 2: Syntax error: Unterminated quoted string

Looks like that pesky precompiled module might be a problem again.

I see a bunch of skipped tests (test_exceptions_*_wasm) with the reason skipped 'test requires v8 and EMTEST_SKIP_V8 is set' - is that a problem?

[sbc100]

There seem to three different issues here:

1. wasm2c related failures. This is because the test expects to be able to build host binaries using the version of clang that we built. To fix these you will want to modify LLVM_TARGETS_TO_BUILD in build.py to use the host arch rather than hardcoding "X86".
2. closure compiler failures. This is a little worrying. It looks like maybe we are trying to run the x86 binary here. Can you see what time of file /instance-store/em/install/emscripten/node_modules/google-closure-compiler-linux/compiler is? (what does file say about it?)
3. autoconf related failures. No idea her you will be to debug. I imagine it will be some kind of local configuration issue. I don't think it matters that these tests pass. emconfigure and autoconf stuff is already a non-portable feature since it doesn't run on windows.

I would say that only (2) is really required to ship an useful version of emsdk on arm64.

[otterley]

@sbc100

/instance-store/em/install/emscripten/node_modules/google-closure-compiler-linux/compiler: ELF 64-bit LSB shared object, 
x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=0c2f44faeb696c4e23138882583460568590100c, for GNU/Linux 3.2.0, with debug_info, not stripped

We discussed earlier that google-closure-compiler-linux lacked a precompiled arm64 binary. To overcome this during the build process, I changed the code in build.py so that it doesn't install this on arm64, and falls back to google-closure-compiler (without the -linux suffix). But that seems not to have propagated to the emscripten installation. Any idea why?

[sbc100]

We probably just need to figure out which npm command to run to get to the state that you need. Do which npm commands are you running in build.py? Maybe you should paste the log from the running of build.py showing the npm commands and the output?

BTW you can experiment locally with this without actually running build.py:

$ rm -rf node_modules
$ npm install/ci ....
$ ./node_modules/.bin/google-closure-compiler --help   # <-- verify that the installed closure compiler runs

[sbc100]

BTW, if you they don't supply a native arm build of closure compiler, then we probably want to make sure we run the JAVA one as opposed to calling all the way back to the pure JS one (cross compiled from java). I forgotten how that all works I'm afraid. The JS one will work fine for the initial release anyway.

[otterley]

I've traced down the issue to some code in the google-closure-compiler JS module that incorrectly assumes that the prebuilt binary can be used on arm64. Associated PR

A quick workaround is to pass --platform java to the google-closure-compiler invocation when running on Linux arm64.

[sbc100]

It you can arrange to not install the native package that would probably be better since it will also save you a bunch of disk space.

I think if its not installed it won't try to use it.

[otterley]

I duplicated the revised npm installation logic that I previously applied to the InstallEmscripten() function to the ExecuteEmscriptenTestSuite() function. The latter was installing all the prebuilt closure compiler binaries; with my changes, it skips that step on Linux/arm64.

Next, some of the config.guess files in the emscripten repo are very outdated - as far back as 2006. Specifically, tests/third_party/freetype/builds/unix/config.guess and tests/third_party/bullet/config.guess. These should be updated to a more recent version that knows about arm64 Linux.

There is also arch-specific code in tests/test_other.py that needs to be changed (test_dot_a_all_contents_invalid):

-    self.run_process([CLANG_CC, 'native.c', '-target', 'x86_64-linux', '-c', '-o', 'native.o'])
+    self.run_process([CLANG_CC, 'native.c', '-target', '%s-linux' % platform.machine(), '-c', '-o', 'native.o'])

Finally, I set LLVM_TARGETS_TO_BUILD in the LLVM() function as follows:

-        '-DLLVM_TARGETS_TO_BUILD=X86;WebAssembly',
+        '-DLLVM_TARGETS_TO_BUILD=%s;WebAssembly' % 'AArch64' if IsArm64() else 'X86',

The only remaining test not passing is test_jsmath:

======================================================================
FAIL: test_jsmath (test_other.other)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
    yield
  File "/usr/lib/python3.8/unittest/case.py", line 676, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
    method()
  File "/instance-store/em/install/emscripten/tests/test_other.py", line 9365, in test_jsmath
    self.assertEqual(diff, 4)
  File "/usr/lib/python3.8/unittest/case.py", line 912, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.8/unittest/case.py", line 905, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 0 != 4

----------------------------------------------------------------------

Any thoughts on this last one? We're very close!

[sbc100]

Looking at the test_jsmath test it looks like its testing the agreement level between math operations of JS vs WebAssembly. On x86 node is seems that they disagree in 4 cases. Logging those lines I see:

float acos => -nan float acos => nan double acos => -nan double acos => nan float asin => -nan float asin => nan double asin => -nan double asin => nan

I suspect that this might be some kind difference in behaviour that is visible between ARM and X86, but only in JS. Here is look like ARM is agreeing with WebAssembly in all cases but x86 differs in the signedness of the trig functions when the result is nan.

I think it should be reasonable to change that assertion to assertLessThan(diff, 5), but we should do some more research first. The test should also probably be checking that things diverge only in the signedness of nan, not any 4 tests could fail in any way like it is. @kripken do you remember the rational behind this test?

[sbc100]

Can you see what your version of node prints when you run Math.asin(2.0).. on x86 node (even the latest version) I see NaN but it seems like the wasm version (and maybe the node-on-arm) version is producing -NaN.

[otterley]

./node-v14.15.5-linux-arm64/bin/node  -e 'console.log(Math.asin(2.0))'
NaN

[sbc100]

I think its fine to change the test do assertLessThan(diff, 5) even just skip that test on arm64 for now.