ARM32 native AOT fails to link on Raspberry Pi

[MichalStrehovsky]

Latest dotnet/installer builds already have arm32 support so I gave it a try.

    /usr/bin/ld.bfd : error : bin/Release/net9.0/linux-arm/native/aottest uses VFP register arguments, obj/Release/net9.0/linux-arm/native/aottest.o does not [/home/michals/aottest/aottest.csproj]
    collect2 : error : ld returned 1 exit status [/home/michals/aottest/aottest.csproj]
    /home/michals/.nuget/packages/microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/build/Microsoft.NETCore.Native.targets(369,5): error MSB3073: The command ""gcc" "obj/Release/net9.0/linux-arm/native/aottest.o" -o "bin/Release/net9.0/linux-arm/native/aottest" -Wl,--version-script=obj/Release/net9.0/linux-arm/native/aottest.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libbootstrapper.o /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libRuntime.WorkstationGC.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libeventpipe-disabled.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libstandalonegc-disabled.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libstdc++compat.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/framework/libSystem.Native.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/framework/libSystem.IO.Compression.Native.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/framework/libSystem.Net.Security.Native.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/framework/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lz -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/home/michals/aottest/aottest.csproj]

Looks like what it's complaining about might be related to:

https://github.com/dotnet/runtime/blob/bee4602cd5974c9abb2d83cc58d5342b75b4cdbe/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/ObjectWriter/ElfObjectWriter.cs#L449-L469

Do we miss setting Tag_ABI_VFP_args? A hello world compiled with gcc produces an object file with this aeabi:

Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6"
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_ABI_optimization_goals: Aggressive Debug
  Tag_CPU_unaligned_access: v6

[ghost]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas See info in area-owners.md if you want to be subscribed.

Issue Details

---

Latest dotnet/installer builds already have arm32 support so I gave it a try. ``` /usr/bin/ld.bfd : error : bin/Release/net9.0/linux-arm/native/aottest uses VFP register arguments, obj/Release/net9.0/linux-arm/native/aottest.o does not [/home/michals/aottest/aottest.csproj] collect2 : error : ld returned 1 exit status [/home/michals/aottest/aottest.csproj] /home/michals/.nuget/packages/microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/build/Microsoft.NETCore.Native.targets(369,5): error MSB3073: The command ""gcc" "obj/Release/net9.0/linux-arm/native/aottest.o" -o "bin/Release/net9.0/linux-arm/native/aottest" -Wl,--version-script=obj/Release/net9.0/linux-arm/native/aottest.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libbootstrapper.o /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libRuntime.WorkstationGC.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libeventpipe-disabled.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libstandalonegc-disabled.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/sdk/libstdc++compat.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/framework/libSystem.Native.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/framework/libSystem.IO.Compression.Native.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/framework/libSystem.Net.Security.Native.a /home/michals/.nuget/packages/runtime.linux-arm.microsoft.dotnet.ilcompiler/9.0.0-preview.2.24115.1/framework/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lz -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/home/michals/aottest/aottest.csproj] ``` Looks like what it's complaining about might be related to: https://github.com/dotnet/runtime/blob/bee4602cd5974c9abb2d83cc58d5342b75b4cdbe/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/ObjectWriter/ElfObjectWriter.cs#L449-L469 Do we miss setting `Tag_ABI_VFP_args`? A hello world compiled with gcc produces an object file with this aeabi: ``` Attribute Section: aeabi File Attributes Tag_CPU_name: "6" Tag_CPU_arch: v6 Tag_ARM_ISA_use: Yes Tag_THUMB_ISA_use: Thumb-1 Tag_FP_arch: VFPv2 Tag_ABI_PCS_wchar_t: 4 Tag_ABI_FP_denormal: Needed Tag_ABI_FP_exceptions: Needed Tag_ABI_FP_number_model: IEEE 754 Tag_ABI_align_needed: 8-byte Tag_ABI_align_preserved: 8-byte, except leaf SP Tag_ABI_enum_size: int Tag_ABI_VFP_args: VFP registers Tag_ABI_optimization_goals: Aggressive Debug Tag_CPU_unaligned_access: v6 ```

Author:	MichalStrehovsky
Assignees:	-
Labels:	`arch-arm32`, `area-NativeAOT-coreclr`
Milestone:	-

[MichalStrehovsky]

FWIW I also tried with -p:LinkerFlavor=lld and got ld.lld : error : undefined symbol: __start___modules. Then I tried -p:LinkerFlavor=gold but that gave me bus error.

Cc @filipnavara

[filipnavara]

What Raspberry Pi version was it? We can certainly add the Tag_ABI_VFP_args property but since we target armv7 (RPi 3+) I am not sure whether it makes sense.

[MichalStrehovsky]

What Raspberry Pi version was it? We can certainly add the Tag_ABI_VFP_args property but since we target armv7 (RPi 3+) I am not sure whether it makes sense.

Raspberry pi 4 with a 32bit Raspbian bookworm.

[filipnavara]

Raspberry pi 4 with a 32bit Raspbian bookworm.

Interesting, that matches one of my environments and I didn’t hit the issue there. Hopefully the EABI header change resolves it.

[MichalStrehovsky]

We can link fine on the default bookworm config now with BFD.

LLD still complains about the thing it complained before.

Gold now hangs instead of crashing. Progress, I guess?

[am11]

Maybe we should explicitly set a few smoke tests projects to exercise certain linkers: bfd,lld,gold,mold, just enough to ensure basic coverage. Currently we are almost, always testing with lld in the CI: https://github.com/dotnet/runtime/blob/79dd9bae9bb881eb716b608577c4cedc6c9cba72/src/tests/Directory.Build.targets#L633 because the script which ultimately resolves _LDFLAGS favors lld; which, in realworld, is not very meaningful (majority of users end up with bfd linker while we are testing lld 24/7-365). Ideally, prereq images for CI (one per architecture) could have various linker flavors installed for testing.

[MichalStrehovsky]

Maybe we should explicitly set a few smoke tests projects to exercise certain linkers: bfd,lld,gold,mold, just enough to ensure basic coverage. Currently we are almost, always testing with lld in the CI:

We had discussion about the testing aspect when the official builds were being switched to use a rootfs that doesn't even have bfd. I can't find the discussion on Github so maybe it happened internally, but the conclusion is basically in https://github.com/dotnet/runtime/pull/85478#issuecomment-1531778436. It's not just about the flavor of the linker, but also the version. We'd be using the latest because that's necessary for our shipping criteria, but end users would be using whatever stale thing landed in their years old LTS distribution. To really test this, we'd want to pick some popular distros and versions and use whatever ships with them. But we're not equipped for that in dotnet/runtime repo and it hasn't been an issue we'd be getting user reports on.

What we do get user reports on is Apple platform linkers but that's a completely separate pit of pain an suffering.

[am11]

not just about the flavor of the linker, but also the version

Yup, I remember adding lld v13 detection in BuildIntergration and linker script which goes with it. :)

But we're not equipped for that in dotnet/runtime repo and it hasn't been an issue we'd be getting user reports on.

Manual testing keep those issues from popping up. My point was CI is not helping in this regard. We can use one of the existing Ubuntu 22.04 image (we have plenty of them), which have lld and bfd already installed, we can install gold and mold as well to complete test apparatus. (we install GBs of optional stuff on those images, so few KBs of additional linker is not gonna hurt the size).