NativeAOT CppComilerAndLinker uses system linker (cross compiling)

[hilpara]

I'm trying to use NativeAOT with linux-arm64 target, but it always stops on linking, because dotnet wants to use system linker instead of custom one. I have a custom build made with yocto and I have sourced the development environment where all compiler flags are defined ($CC, $LD etc). To publish I have tried to use: dotnet publish -r linux-arm64 -o publish -c Release -p:StripSymbols=true -p:CppComilerAndLinker=aarch64-oe-linux-gcc -p:SysRoot=$SDKTARGETSYSROOT -p:TargetTriple=aarch64-oe-linux but even with CppComilerAndLinker, it will try to use system /usr/bin/ld which of course doesn't support aarch64linux /usr/bin/ld.bfd: unrecognised emulation mode: aarch64linux Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386pep i386pe elf64bpf

Is there a way to use the linker from $LD (aarch64-oe-linux-ld)? I tried with clang and lld, but it can't find libgcc or crtbeginS.o.

[jkotas]

NativeAOT build does not invoke the ld linker directly. It invokes the C/C++ compiler that in turn invokes the linker.

If you see the system ld being called, it means that aarch64-oe-linux-gcc invokes system ld. You may need to set LinkerFlavor (https://github.com/search?q=repo%3Adotnet%2Fruntime%20LinkerFlavor&type=code) property or pass additional arguments to aarch64-oe-linux-gcc using LinkerArg.

[MichalStrehovsky]

Quick way to check if the cross toolchain is configured correctly would be to run aarch64-oe-linux-gcc --sysroot=$SDKTARGETSYSROOT test.c where test.c is some hello world C program. I assume the linker issue will repro there as well. Native AOT doesn't do much else in terms of toolchain configuration. Toolchain needs to be configured correctly.

[hilpara]

Quick way to check if the cross toolchain is configured correctly would be to run aarch64-oe-linux-gcc --sysroot=$SDKTARGETSYSROOT test.c where test.c is some hello world C program. I assume the linker issue will repro there as well. Native AOT doesn't do much else in terms of toolchain configuration. Toolchain needs to be configured correctly.

aarch64-oe-linux-gcc --sysroot=$SDKTARGETSYSROOT test.c -I usr/include will compile helloworld app.

file a.out
a.out: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=e8070b15e5c3cb2996425a2106d142b797647b97, for GNU/Linux 3.14.0, with debug_info, not stripped

[hilpara]

NativeAOT build does not invoke the ld linker directly. It invokes the C/C++ compiler that in turn invokes the linker.

If you see the system ld being called, it means that aarch64-oe-linux-gcc invokes system ld. You may need to set LinkerFlavor (https://github.com/search?q=repo%3Adotnet%2Fruntime%20LinkerFlavor&type=code) property or pass additional arguments to aarch64-oe-linux-gcc using LinkerArg.

Adding -p:LinkerFlavor=aarch64-oe-linux-ld produces: lang : error : invalid linker name in argument '-fuse-ld=aarch64-oe-linux-ld'

Adding -p:LinkerArgs=$CONFIGURE_FLAGS will still launch /usr/bin/ld

echo $CONFIGURE_FLAGS   
--target=aarch64-oe-linux --host=aarch64-oe-linux --build=x86_64-linux --with-libtool-sysroot=/home/hilpara/Sources/MYC-YM6252/arago-2023.10/sysroots/aarch64-oe-linux

[jkotas]

If you run the build with verbose logging (dotnet publish /v:diag), you should see the command line used for the lining. For example, this command is used for linking in the default linux x64:

"clang" "obj/Release/net8.0/linux-x64/native/test.o" -o "bin/Release/net8.0/linux-x64/native/test" -Wl,--version-script=obj/Release/net8.0/linux-x64/native/test.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /home/jkotas/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.6/sdk/libbootstrapper.o /home/jkotas/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.6/sdk/libRuntime.WorkstationGC.a /home/jkotas/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.6/sdk/libeventpipe-disabled.a /home/jkotas/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.6/sdk/libstdc++compat.a /home/jkotas/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.6/framework/libSystem.Native.a /home/jkotas/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.6/framework/libSystem.IO.Compression.Native.a /home/jkotas/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.6/framework/libSystem.Net.Security.Native.a /home/jkotas/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.6/framework/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lz -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections

Then you can execute this command directly and tweak command line options until you get a working one. For example, you can try deleting -fuse-ld=bfd to see whether it helps. Once you have a working command, we can figure out how to modify the .csproj file to produce it.

[hilpara]

All I needed to do is to change the target to --target=aarch64-oe-linux. I already tried to change the LinkerArgs to change the target as you can see from above. Is there any other way to do that? Or did I do that wrong?

Edit: I added quotes to (and used only the target parameter to LinkerArgs) and it did work. BUT it doesn't work directly to another project. I had to "compile" it first with the command I got from diag paramter and had to change the --target parameter. Afterwards I can use to normal dotnet publish command with LinkArgs to be able to publish it. Is this how it should work?

[jkotas]

It sounds like a problem with incremental build. You can try deleting obj and bin directories to see whether it helps. The changes in the project file do not always work well with leftovers in obj and bin directories.

[hilpara]

Sorry for the delayed comment. I did delete obj and bin folders and it won't compile (the target is still shown as aarch64-linux-gnu). So the -p:LinkerArgs="--target=aarch64-oe-linux" Is not used with clang command afterwards. I have to compile myself with the command I get from diag parameter to get it work.

[jkotas]

-p:LinkerArgs="--target=aarch64-oe-linux"

It needs to be LinkerArg (without 's'). Also, I am not sure it is going to work from the command line since LinkerArg is ItemGroup.

Try to add this to your .csproj file:

<ItemGroup>
    <LinkerArg Include="--target=aarch64-oe-linux" />
</ItemGroup>