Add ARM64 assembly?

[DonaldKellett]

Please complete the following information about the language:

• Name: ARM64 assembly
• Website: TBD
• Language Version: TBD

The following are optional, but will help us add the language:

• Test Frameworks: TBD
• How to install: TBD
• How to compile/run: TBD

@nomennescio wrote under channel #asm on Discord:

Reminds me that I need to open a request for ARM, which was part of my original request for additional assembly languages

While discussing with @nomennescio about native vs. non-native assembly and the pain points of running an emulator for the latter, it struck me that maybe, just maybe, it could be possible for Codewars to support ARM64 assembly natively, alongside x86_64. In recent years, ARM instances have been gaining popularity among cloud providers; for example, GCP now offers pre-GA Arm VMs on Compute. An initial idea could be then to run ARM64 assembly workloads on ARM nodes and everything else on x86_64 nodes.

So here are some questions to consider, which official Codewars staff and the team at Qualified would probably in the best position to answer, as they know best the details of the cloud service(s) being used:

• Are ARM offerings on GCP (or whatever cloud provider Codewars is using) considered mature enough for production usage?
• If so, how easy (or difficult) is it to mix together x86_64 nodes and ARM nodes, if at all possible?

---

:+1: reaction might help to get this request prioritized.

[kazk]

• Are ARM offerings on GCP (or whatever cloud provider Codewars is using) considered mature enough for production usage?

It's currently in Preview which means:

Unless stated otherwise by Google, Preview offerings are intended for use in test environments only. https://cloud.google.com/products#product-launch-stages

If so, how easy (or difficult) is it to mix together x86_64 nodes and ARM nodes, if at all possible?

I'm assuming it's just a new machine type, so you should be able to add a new instance group.

This doesn't mean the runner can utilize them easily.

---

I won't say it's impossible because I haven't looked yet, but we don't have enough reason to support ARM64 natively considering the amount of work required and demand.

In addition to figuring out how to run/test safely, we need to at least:

• configure and maintain a separate infrastructure for testing it
• configure and maintain an additional instance group dedicated to it (VMs are also more expensive)
• change how the runner processes requests

See my answer to non-native assembly request for some points I consider when adding support. https://github.com/codewars/runner/issues/149#issuecomment-1023859927

[kazk]

To emulate ARM, it should be possible to run ARM images with Docker+QEMU emulation. I can't find the docs, but https://docs.docker.com/build/buildx/#build-multi-platform-images mentions it.

[Bubbler-4]

I have a bit of experience with cross-compiling and emulating arm32 assembly on x86 linux.

The instructions for compiling and running a single .S file:

• install gcc-arm-linux-gnueabihf toolchain and qemu-user
• compile with arm-linux-gnueabihf-as main.S -o main.o && arm-linux-gnueabihf-ld main.o -o main
• run with qemu-arm main

It shouldn't be hard to modify this to multi-file cases (including C files) and to aarch64. One problem I can imagine is how large the image containing qemu would be, but apparently an image for single emulated arch can be made quite small...?

[kazk]

I don't think the image need to include qemu.

With /proc/sys/fs/binfmt_misc/aarch64 like the following with qemu-aarch64-static installed on the host:

$ cat /proc/sys/fs/binfmt_misc/aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: OCF
offset 0
magic 7f454c460201010000000000000000000200b7
mask ffffffffffffff00fffffffffffffffffeffff

I was able to run ARM64 images with platform flag (it runs without it too, but shows a warning):

$ docker run --rm --platform linux/arm64 arm64v8/alpine uname -m
aarch64

$ docker run --rm --platform linux/arm64 arm64v8/ubuntu uname -m
aarch64

So, I think we can build an image with binutils for ARM64, then use it like:

W=/workspace
I=codewars/arm64
C=$(docker container create --rm --platform linux/arm64 -w $W $I sh -c "as main.s -o main.o && ld main.o -o main && ./main")

docker container cp main.s $C:$W
docker container start --attach $C

[kazk]

Using example from https://peterdn.com/post/2020/08/22/hello-world-in-arm64-assembly

main.s

```text .data /* Data segment: define our message string and calculate its length. */ msg: .ascii "Hello, ARM64!\n" len = . - msg .text /* Our application's entry point. */ .globl _start _start: /* syscall write(int fd, const void *buf, size_t count) */ mov x0, #1 /* fd := STDOUT_FILENO */ ldr x1, =msg /* buf := msg */ ldr x2, =len /* count := len */ mov w8, #64 /* write is syscall #64 */ svc #0 /* invoke syscall */ /* syscall exit(int status) */ mov x0, #0 /* status := 0 */ mov w8, #93 /* exit is syscall #93 */ svc #0 /* invoke syscall */ ```

Dockerfile:

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y binutils

WORKDIR /workspace

Build with:

docker buildx build --platform linux/arm64 -t arm64-test .

Run with:

W=/workspace
I=arm64-test
C=$(docker container create --rm --platform linux/arm64 -w $W $I sh -c "as main.s -o main.o && ld -s main.o -o main && ./main")

docker container cp main.s $C:$W
docker container start --attach $C

Outputs:

Hello, ARM64!

[DonaldKellett]

I just read up on QEMU user-mode emulation. Fascinating stuff! It seems we could support assembly language for a wide range of architectures this way, ARM and MIPS included, though RISC-V seems to be missing.

I also did a quick search on the differences between @Bubbler-4 's approach of installing the emulator within the container, and @kazk 's approach of installing qemu-user-static on the container host and running multi-arch containers directly. One concern I have for the latter approach is whether it will work properly for more complex projects such as those including Criterion unit tests. For example multiarch/qemu-user-static#172 mentions a case where building a non-trivial project in a multi-arch container failed with ldd exiting with code 139, and QemuUserEmulation in the Debian Wiki provides a possible explanation under section "Running dynamically linked executables":

With the instructions above, you should be able to run statically linked target executables. To be able to run dynamically linked binaries, QEMU needs to have access to the target ELF interpreter.

[kazk]

I don't think MIPS is supported by Docker. I have /proc/sys/fs/binfmt_misc/qemu-mips enabled, but docker buildx inspect doesn't list it.

[kazk]

RISCV64 is supported (#199)

[kazk]

We also need a base image in the architecture. See https://github.com/docker-library/official-images#architectures-other-than-amd64 for supported architectures.

[nomennescio]

@nomennescio wrote under channel #asm on Discord:

Reminds me that I need to open a request for ARM, which was part of my original request for additional assembly languages

Except I was thinking about ARM32, as that is most widely known, still has the very interesting conditional opcode execution, as well as program counter as a directly modifiable register, and possibly the use of Thumb instructions, and ARM64 is only a VERY recent and VERY breaking change to the ARM ISA.

[nomennescio]

I'm not sure if it would be possible or makes sense to support both ARM32 and ARM64. If it would, I can make an additional request, if not, I would like to discuss which one to use.

[DonaldKellett]

@nomennescio wrote under channel #asm on Discord:

Reminds me that I need to open a request for ARM, which was part of my original request for additional assembly languages

Except I was thinking about ARM32, as that is most widely known, still has the very interesting conditional opcode execution, as well as program counter as a directly modifiable register, and possibly the use of Thumb instructions, and ARM64 is only a VERY recent and VERY breaking change to the ARM ISA.

https://github.com/docker-library/official-images#architectures-other-than-amd64 mentions ARM32 as a supported architecture. I guess we could open a separate issue for ARM32 support and track the progress there. In fact, since #199 demonstrates the feasibility of supporting RISC-V with unit testing, I suppose adding support for additional architectures afterwards will only become much easier as long as both Docker and QEMU support it, and the only remaining factor of consideration becomes the operational cost of maintaining one more container image versus popular demand.

[DonaldKellett]

For the icon, I guess we could use a cropped variant of this?

For CodeMirror mode, there's a GAS mode upstream supporting ARM assembly syntax highlighting that should be directly applicable should we decide to use the GNU toolchain: https://github.com/codemirror/codemirror5/blob/master/mode/gas/gas.js

I'll probably look into adapting the work in RISC-V to aarch64 tonight - it should be as simple as replacing the assembly files and tweaking the docker build command to build for aarch64 instead of riscv64. IMHO we should prioritize ARM64 support over ARM32 support since the former is already used in most modern consumer smartphones, in Apple's recent M1 / M2 SoC and in recent developments like Qualcomm Snapdragon, and is therefore more relevant.

[Bubbler-4]

A minor nitpick: I doubt you can say "ARM64 is more relevant than ARM32" - yes, ARM64 is pretty common these days for high-end consumer devices (smartphones and Raspberry Pi series), but there is also a whole separate market of non-consumer microprocessor devices which mainly use ARM32. See STMicroelectronics and one of its main product lines STM32 for example.

[DonaldKellett]

@kazk Created https://github.com/DonaldKellett/codewars-arm64v8 and initiated transfer, please review and accept if there are no outstanding issues

[kazk]

Let's focus on finishing #199 first. There's no need to rush this.