[NativeAOT] linux-arm bring up

[filipnavara]

This is tracking issue for the known problems that need to be resolved to get working NativeAOT support on linux-arm platform.

Known issues:

• [x] Inline TLS access emits incorrect code for Optimized=true which results in stack overflow
• [ ] Some code paths around Align8 / FEATURE_64BIT_ALIGNMENT may be unhandled (ref: https://github.com/dotnet/runtime/pull/97269#issuecomment-1909219996)
• [x] https://github.com/dotnet/runtime/issues/97730
• [x] https://github.com/dotnet/runtime/issues/97748
• [x] https://github.com/dotnet/runtime/issues/97749
• [x] https://github.com/dotnet/runtime/issues/97750
• [x] InWriteBarrierHelper returns incorrect answer on Release builds
• [ ] RhpInitialInterfaceDispatch is missing AV value check (likely not needed or we have a test gap)
• [x] https://github.com/dotnet/runtime/issues/97912
• [x] https://github.com/dotnet/runtime/issues/97922
• [ ] RhGetCodeTarget doesn't handle PC-relative movw/movt correctly

Failing runtime tests:

• [x] JIT/Directed/Directed_1 https://github.com/dotnet/runtime/issues/95517
• [x] Loader/classloader/Casting/punning https://github.com/dotnet/runtime/issues/98493
• [x] https://github.com/dotnet/runtime/issues/99079

Other things requiring clean up:

• [x] https://github.com/dotnet/runtime/pull/97996
• [x] Run the smoke tests to prevent regressions

[sonatique]

Thanks again @am11 !

Now that I better understand what I am doing and that I have a working setup, I am curious about one of the things you wrote earlier. You said you "installed lld, it was either that or apt install binutils-arm" and later you wrote the same about llvm.

So you appear to have made efforts to avoid "binutils-arm", though you wrote "binuitls/bfd is slightly better at size optimization".

So I am wondering: what would be wrong in "simply" using binutils? I have to admit I tried for some time to install binutils and remove or replace -p:LinkerFlavor=lld and -p:ObjCopy=llvm-objcopy but to no avail, so it's probably not trivial (at least to me).

Do you have any pointer regarding how to use binutils? I wish I could compare binary produced by both lld/llvm and binutils.

Thanks in advance if you have time for this low priority question.

[am11]

gcc toolchain (gcc, binutils etc.) are architecture-specific, while llvm toolchain is multiarch. Meaning to cross-compile stuff, lld from llvm toolchain of host architecture will do the job, while gcc requires target arch specific package, e.g. apt install -y binutils-arm-linux-gnueabi.

See the previous attempt of using gcc toolchian in crosscompile: https://github.com/dotnet/runtime/issues/78559, the conclusion was it's best to stick with llvm toolchain for cross-compilation. The difference in size is a few KBs and it's not meaningful in grand scheme of things.

[sonatique]

Hi @am11 , OK I see, thanks a lot!

[tunger]

Thanks a lot for the Dockerfile, @am11, and for asking the questions, @sonatique!

With the provided Dockerfile, we can utilize vscode dev containers to develop net9.0 apps in the IDE and compile it AOT for linux-arm. It works great!

Awesome work on the native AOT for linux-arm. It speeds up my application a lot.

Set it up like this:

.devcontainer/devcontainer.json

{
  "build": { "dockerfile": "Dockerfile" },

  "customizations": {
    "vscode": {
      "extensions": ["ms-dotnettools.csdevkit"]
    }
  }
}

.devcontainer/Dockerfile

FROM --platform=$BUILDPLATFORM ubuntu:latest AS builder

RUN apt update && apt install -y clang debootstrap curl lld llvm

RUN mkdir /dev/arm; \
  curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/arm/sources.list.jammy -o /dev/arm/sources.list.jammy; \
  curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/build-rootfs.sh |\
    bash /dev/stdin arm jammy llvm15 lldb15

RUN mkdir -p "$HOME/.dotnet" "$HOME/.nuget/NuGet";
RUN curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --quality daily --channel 9.0;
RUN cat > "$HOME/.nuget/NuGet/NuGet.Config" <<EOF
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="dotnet9" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/index.json" />
</packageSources>
EOF

ENV DOTNET_NOLOGO=1
ENV PATH "$PATH:/root/.dotnet"

Then you can use vscode tasks to publish the app, like this:

.vscode/tasks.json

{
  "version": "2.0.0",
  "tasks": [
    {
      "command": "dotnet",
      "args": [
        "publish",
        "application.csproj",
        "-r",
        "linux-arm",
        "-c",
        "Release",
        "--self-contained",
        "true",
        "-o",
        "${workspaceFolder}/out",
        "-p:PublishSingleFile=false",
        "-p:EnableCompressionInSingleFile=true",
        "-p:PublishAot=true",
        "-p:LinkerFlavor=lld",
        "-p:ObjCopy=llvm-objcopy",
        "-p:SysRoot=\"/.tools/rootfs/arm\""
      ],
      "options": {
        "cwd": "${workspaceFolder}"
      },
      "group": "build",
      "label": "dotnet9 publish release AOT"
    }
  ]
}

[am11]

@tunger, very nice! Thanks for sharing. After #101213 is merged, we can use the same mechanism for linux-musl-arm targeting Alpine Linux. Basically, in the Dockerfile:

- bash /dev/stdin arm jammy llvm15 lldb15
+ bash /dev/stdin arm alpine llvm15 lldb15

and in tasks.json:

        "-r",
-        "linux-arm",
+        "linux-musl-arm",

Once .NET 9 is shipped, we would be able to use the prebuilt official docker images which will exempt setting up cross environment.

One reason of recommending official images is slightly(⚓) important because the more people start using this kind of experimental solution, the more confusion it's going to cause. For instance; this dockerfile (for linux-arm and not for linux-musl-arm) requires 'nested virtualization' support for chroot (fakechroot has some issues so fakeroot fakechroot debootstrap ... -variant=fakechroot also ends up requiring this the real chroot.. which - according to google results - is a known issue actively being investigated in Debian world). Nested virtualization needs to "at least" EL0 (aka Non-hardware Assisted Nested Virtualization; the kind which is available on M1 macs, which does not support hardware assisted virtualization). Alpine's crossbuild setup, OTOH, doesn't require chroot etc., and it is lightening faster than debootstrap (yet to find any package manager faster than Alpine's apk(1) 😁).

I tested this docker on a few public CI systems, here is the support situation:

• appveyor: no EL0 support, so debootsrap throws fatal exit code at chroot step
• cirrus-ci: same as appveyor
• GitHub Actions: only Linux supports it. So it's best to publish your base layer to something like Docker or GitHub Package Registry to avoid the surprises.

⚓ It is only about "building" the image, dotnet-publish does not use chroot.

[dabbinavo]

@am11, I just followed your instructions from your comment https://github.com/dotnet/runtime/issues/97729#issuecomment-2050701956 step by step, but unfortunately, docker build fails with the output below.

I tried your steps inside a Ubuntu 22.04 WSL (with docker installed) on a Windows 11 x64 host machine.

Is it possible that this file changed in the meantime?: https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/build-rootfs.sh

I get the same error at the exact same step if i try to run each command directly in my WSL without using docker at all.

/tmp/arm-builder$ docker build . -t armv7-nativeaot-webapi
[+] Building 26.6s (7/8)                                                                                                                     docker:default
 => [internal] load build definition from Dockerfile                                                                                                   0.2s
 => => transferring dockerfile: 1.30kB                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                       1.0s
 => [internal] load .dockerignore                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                        0.0s
 => [1/5] FROM docker.io/library/ubuntu:latest@sha256:3f85b7caad41a95462cf5b787d8a04604c8262cdcdf9a472b8c52ef83375fe15                                 0.0s
 => CACHED [2/5] RUN apt update && apt install -y clang debootstrap curl lld llvm                                                                      0.0s
 => CACHED [3/5] RUN mkdir -p "$HOME/.dotnet9" "$HOME/.nuget/NuGet";   curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --quality dai  0.0s
 => ERROR [4/5] RUN mkdir /dev/arm;   curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/arm/sources.list.jammy -o /dev  25.4s
------
 > [4/5] RUN mkdir /dev/arm;   curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/arm/sources.list.jammy -o /dev/arm/sources.list.jammy;   curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/build-rootfs.sh |    bash /dev/stdin arm jammy llvm15 lldb15:
1.016 I: Retrieving InRelease
1.406 I: Checking Release signature
1.413 I: Valid Release signature (key id F6ECB3762474EDA9D21B7022871920D1991BC93C)
1.834 I: Retrieving Packages
2.035 I: Validating Packages
2.159 I: Resolving dependencies of required packages...
2.301 I: Resolving dependencies of base packages...
3.377 I: Checking component main on http://ports.ubuntu.com...
3.634 I: Retrieving adduser 3.118ubuntu5
4.366 I: Validating adduser 3.118ubuntu5
4.381 I: Retrieving apt 2.4.5
4.589 I: Validating apt 2.4.5
........
24.28 I: Extracting usrmerge...
24.29 I: Extracting util-linux...
24.33 I: Extracting zlib1g...
24.66 W: Failure trying to run: chroot "/crossrootfs/arm" /bin/true
24.66 W: See /crossrootfs/arm/debootstrap/debootstrap.log for details
------
Dockerfile:20
--------------------
  19 |
  20 | >>> RUN mkdir /dev/arm; \
  21 | >>>   curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/arm/sources.list.jammy -o /dev/arm/sources.list.jammy; \
  22 | >>>   curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/build-rootfs.sh |\
  23 | >>>     bash /dev/stdin arm jammy llvm15 lldb15
  24 |
--------------------
ERROR: failed to solve: process "/bin/sh -c mkdir /dev/arm;   curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/arm/sources.list.jammy -o /dev/arm/sources.list.jammy;   curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/build-rootfs.sh |    bash /dev/stdin arm jammy llvm15 lldb15" did not complete successfully: exit code: 1

[filipnavara]

On WSL I had to manually update the binfmts configuration to register the QEMU emulators. You may want to check update-binfmts --display and update-binfmts --enable.

[dabbinavo]

Still the same failure (docker build, as well directly inside WSL) after executing your suggested commands inside WSL

sudo update-binfmts --display
sudo update-binfmts --enable

[filipnavara]

enbale is a typo, right? update-binfmts --display is supposed to list all the various QEMU user configurations, does it?

[dabbinavo]

Yes, it was a typo and the display switch outputs:

~$ update-binfmts --display
llvm-14-runtime.binfmt (enabled):
     package = llvm-14-runtime
        type = magic
      offset = 0
       magic = BC
        mask =
 interpreter = /usr/bin/lli-14
    detector =
python3.10 (enabled):
     package = python3.10
        type = magic
      offset = 0
       magic = \x6f\x0d\x0d\x0a
        mask =
 interpreter = /usr/bin/python3.10
    detector =

[filipnavara]

That doesn't list any of the QEMU user packages. There would be an entry similar to:

qemu-arm (disabled):
     package = qemu-arm
        type = magic
      offset = 0
       magic = \x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00
        mask = \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
 interpreter = /usr/bin/qemu-arm-static
    detector =

That means the QEMU user emulators are not installed properly and thus the chroot binaries in the debootstrap process cannot execute. I don't remember anymore how I fixed this but hopefully this points in the right direction to Google and fix the problem.

[am11]

I was using this docker https://github.com/am11/CrossRepoCITesting/blob/master/linux-arm-aot/Dockerfile and this workflow https://github.com/am11/CrossRepoCITesting/blob/master/.github/workflows/docker-naot-arm32.yml. I found that running docker run --privileged --rm tonistiigi/binfmt --install all before hand (as I did in the workflow yml) made things whole lot easier. Also, using that command fixed Appveyor CI and Cirrus-CI builds.

[dabbinavo]

Okay, thanks for all the information. My specific issue from comment https://github.com/dotnet/runtime/issues/97729#issuecomment-2112294469 was resolved by doing sudo apt install qemu-user-static qemu qemu-system-arm qemu-efi inside my WSL. For now, i am unsure, which of the packages was neccessary.

Will do more tests on Tuesday and post a final solution in the case i find one for me.

[dabbinavo]

I finally was able to compile an app for an embedded Debian 11 (bullseye) system by following these setup steps on the build machine (Azure Devops Services, Microsoft Hosted Agent, vmImage: ubuntu-20.04):

sudo apt install -y clang debootstrap curl lld llvm qemu-user-static binfmt-support
mkdir -p "$HOME/.dotnet9" "$HOME/.nuget/NuGet"
curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --quality daily --channel 9.0 --install-dir "$HOME/.dotnet9"
cat > "$HOME/.nuget/NuGet/NuGet.Config" <<EOF
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="dotnet9" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/index.json" />
</packageSources>
</configuration>
EOF

export DOTNET_NOLOGO=1
export ROOTFS_DIR=/crossrootfs/arm

sudo mkdir /dev/arm
curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/arm/sources.list.focal -o /dev/arm/sources.list.focal
curl -sSL https://raw.githubusercontent.com/dotnet/arcade/main/eng/common/cross/build-rootfs.sh | sudo -E bash /dev/stdin arm focal llvm10 lldb10

Then running the cross compile build with:

/home/vsts/.dotnet9/dotnet publish /path/to/csharp-project --property:RuntimeIdentifier="linux-arm" --property:TargetFramework="net9.0" --property:PublishAot="true" --property:Configuration="Release" --property:LinkerFlavor="lld" --property:ObjCopy="llvm-objcopy" --property:SysRoot="/crossrootfs/arm" -o /desired/output/directory

As mentioned in this comment, I had to use Ubuntu 20.04 and focal rootfs with llvm10 and lldb10 in order to run it on the bullseye system with "old" glibc.

[dabbinavo]

Another question from my side: Does anyone know if there will be "official" support for natively cross-compiling to linux-arm for .NET 9 and maybe also newer versions? The question comes up, as we are currently evaluating if it will be safe to use this functionality in a consumer product.

I mean will .NET 9 be internally testet against NativeAOT support for linux-arm and will it be maintained and bug-fixed after .NET 9 is released?

Unfortunately, I could not find any announcement or anything else about this.

[am11]

As it happened, there already are tags for arm32 with .NET 9 SDK https://hub.docker.com/_/microsoft-dotnet-nightly-sdk/. 👌😎

# run a throw-away-after-use (--rm) container interactively for linux/arm/v7,
# while mounting the current-working-directory to /myapp 
$ docker run --rm --platform linux/arm/v7 -v$(pwd):/myapp -w /myapp -it \
      mcr.microsoft.com/dotnet/nightly/sdk:9.0-preview

# inside the container
$ uname -a
Linux ea9e301095ba 6.6.26-linuxkit #1 SMP Sat Apr 27 04:13:19 UTC 2024 armv7l GNU/Linux

$ dotnet --info
Linux ea9e301095ba 6.6.26-linuxkit #1 SMP Sat Apr 27 04:13:19 UTC 2024 armv7l GNU/Linux
root@ea9e301095ba:/# dotnet --info
.NET SDK:
 Version:           9.0.100-preview.4.24266.28
 Commit:            75a08fda5c
 Workload version:  9.0.100-manifests.2c9affbd
 MSBuild version:   17.11.0-preview-24225-01+bd0b1e466

Runtime Environment:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-arm
 Base Path:   /usr/share/dotnet/sdk/9.0.100-preview.4.24266.28/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      9.0.0-preview.4.24260.3
  Architecture: arm
  Commit:       2270e3185f

.NET SDKs installed:
  9.0.100-preview.4.24266.28 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 9.0.0-preview.4.24260.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 9.0.0-preview.4.24260.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

On macOS arm64, I run into qemu assertion https://github.com/dotnet/runtime/issues/97729#issuecomment-2050507728.

If you are on the x64 host with docker installed, you can use one of these tags with prerequisites for cross compilation: https://github.com/dotnet/versions/blob/main/build-info/docker/image-info.dotnet-dotnet-buildtools-prereqs-docker-main.json

e.g.

$ docker run -e ROOTFS_DIR=/crossrootfs/arm --rm -it mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm
# install dotnet in it

You can also create a Dockerfile to make it ready. e.g.

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm

# install dotnet (dotnet-script install)

# now warm up NativeAOT so ilc packages are ready to use
RUN dotnet new webapiaot -n warmupapp && dotnet publish --project warmupapp && rm -rf warmupapp

then build this image and tag it docker build . -t my-dotnet9-linux-arm-builder. Usage dotnet run --rm -v$(pwd):/myapp -w /myapp my-dotnet9-linux-arm-builder dotnet publish -c Release -o dist.

However, if you are on non-x64 machine like arm64, or you wanted the bleeding-edge daily build then you can build the builder image from scratch as discussed above.