ILCompiler in .NET9 for FreeBSD does not produce usable binaries

[Thefrank]

Description

Under FreeBSD, .NET9 the ILCompiler (/p:PublishAOT=true) does not generate usable binaries. Even a simple "Hello, World!"

Discovered https://github.com/dotnet/runtime/issues/104497#issuecomment-2227749323 as a part of https://github.com/dotnet/runtime/issues/104497

Reproduction Steps

Copied in part from above issue:

./dotnet --version
9.0.100-preview.5.24307.3
---

# ./dotnet new console -o /root/test
The template "Console App" was created successfully.

Processing post-creation actions...
Restoring /root/test/test.csproj:
Restore succeeded.
---

# ./dotnet publish -c Release /root/test/test.csproj /p:PublishAot=true -t:Rebuild && /root/test/bin/Release/net9.0/freebsd-x64/publish/test
Restore complete (4.4s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  test succeeded (18.8s) → /root/test/bin/Release/net9.0/freebsd-x64/publish/

Build succeeded in 25.6s
Segmentation fault (core dumped)

Expected behavior

Hello, World!

Actual behavior

Segmentation fault (core dumped)

Regression?

Yes. This works in .NET8

# ./dotnet --version
8.0.100

---
# ./dotnet new console -o /root/testnet8/
The template "Console App" was created successfully.

Processing post-creation actions...
Restoring /root/testnet8/testnet8.csproj:
  Determining projects to restore...
  Restored /root/testnet8/testnet8.csproj (in 1.62 sec).
Restore succeeded.
---
./dotnet publish -c Release /root/testnet8/testnet8.csproj /p:PublishAot=true -t:Rebuild && /root/testnet8/bin/Release/net8.0/freebsd-x64/publish/testnet8
MSBuild version 17.8.3+195e7f5a3 for .NET
  Determining projects to restore...
  Restored /root/testnet8/testnet8.csproj (in 3.75 sec).
  testnet8 -> /root/testnet8/bin/Release/net8.0/freebsd-x64/testnet8.dll
  Generating native code
  testnet8 -> /root/testnet8/bin/Release/net8.0/freebsd-x64/publish/
Hello, World!

Known Workarounds

No known workarounds

Configuration

Versions used: .NET8 (8.0.100) and .NET9 (9.0.100-preview.5.24307.3) OS: FreeBSD 13.3 & 14.1 Arch: (x64)

Other information

No response

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas See info in area-owners.md if you want to be subscribed.

[am11]

@Thefrank, could you please share the binary build (with https://github.com/dotnet/runtime/pull/105004 applied)?

[am11]

NVM, I'll let GHA upload the debug artifacts https://github.com/am11/CrossRepoCITesting/commit/0e515ed15392d30f22327fe05e1dad3502b5b7d0 🤞

[am11]

Ran into https://github.com/dotnet/runtime/pull/105417, will rebuild with that change.

[am11]

After a few asserts suppression (https://github.com/am11/runtime/commit/37940bfe7da248be5e3f9a012a4dd12e140f4fba and https://github.com/am11/runtime/commit/3b700e42385f2a06acdc9b2bbe085ff415ef5910), I get runtime's debug build to publish the aot app. It is failing at the very start:

backtrace:

(lldb) bt
* thread #1, name = 'foo', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x000003fa95530f10 libc.so.7`__guard_setup at stack_protector.c:70:27
    frame #1: 0x000011de40623bfd ld-elf.so.1`objlist_call_init(list=0x000003fa91c825c8, lockstate=0x000003fa91c823d8) at rtld.c:3089:7
    frame #2: 0x000011de406228c9 ld-elf.so.1`_rtld(sp=<unavailable>, exit_proc=0x000003fa91c82640, objp=0x000003fa91c82648) at rtld.c:979:5
    frame #3: 0x000011de4061fb89 ld-elf.so.1`.rtld_start at rtld_start.S:40

Tried commenting out -fstack-protector-strong in runtime repo build and verified that none of '.note.GNU-stack|.eh_frame|.eh_frame_hdr' sections and '__stack_chk|__stack_chk_fail|__stack_chk_guard' symbols are present but still the same backtrace. I can try with explicitly disabling it -fno-stack-protector etc. next but since dotnet(1) with sdk are also built with the same flags, and the fact that .net9 sdk is publishing AOT apps running on freebsd, so stack-protector might not be the issue after all?

@janvorli, do you have any ideas? This is the (constructor) function in question: https://github.com/freebsd/freebsd-src/blob/56ecc8a93221c16b6ea15d3dbea706a2c8bb33b4/lib/libc/secure/stack_protector.c#L40-L62

[am11]

PublishAot clang command is:

~/.dotnet9/dotnet publish -p:PublishAot=true -p:StripSymbols=false -o dist --packages pkgs -v:diag ... "clang" "obj/Release/net9.0/freebsd-x64/native/foo.o" -o "bin/Release/net9.0/freebsd-x64/native/foo" -Wl,--version-script=obj/Release/net9.0/freebsd-x64/native/foo.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=lld /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/sdk/libbootstrapper.o /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/sdk/libRuntime.WorkstationGC.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/sdk/libeventpipe-disabled.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/sdk/libRuntime.VxsortEnabled.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/sdk/libstandalonegc-disabled.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/sdk/libstdc++compat.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/framework/libSystem.Native.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/framework/libSystem.Globalization.Native.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/framework/libSystem.IO.Compression.Native.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/framework/libSystem.Net.Security.Native.a /root/foo/pkgs/runtime.freebsd-x64.microsoft.dotnet.ilcompiler/9.0.0-ci/framework/libSystem.Security.Cryptography.Native.OpenSsl.a -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lz -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -L/usr/local/lib -lgssapi_krb5 -L/usr/local/lib -linotify -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections -Wl,-T,"obj/Release/net9.0/freebsd-x64/native/sections.ld"

also tried -p:PositionIndependentExecutable=false which removes pie from:

$ file dist/foo dist/foo: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, FreeBSD-style, BuildID[sha1]=ebbd7de68c6bc779758413456db31a78b6ea3022, with debug_info, not stripped

as-in:

dist/foo: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, FreeBSD-style, BuildID[sha1]=48445d4f676fcda202762332e402f032f86670b3, with debug_info, not stripped

to no avail. 😕

[am11]

I installed binutils and used -p:LinkerFlavor=bfd which showed the error about two symbols, exporting them fixed the issue. https://github.com/dotnet/runtime/blob/7e429c2393a002065b641c3817fff62145c926db/src/native/corehost/apphost/static/singlefilehost_freebsdexports.src#L16-L18 need to incorporate them in buildintegration. Will push a PR after another round of testing.

[am11]

https://github.com/dotnet/runtime/pull/105587 hello world is working. 👍 @Thefrank please run smoke tests when you get around a new release build. The debug build is here: https://github.com/am11/CrossRepoCITesting/releases/tag/freebsd_10121426228. It took bunch hacks to make the SDK work (especially your README and the special case for ILLink which is stuck at preview.1 for some reason).

[Thefrank]

@am11 I think --test covers all tests outside of PAL but I am unsure. Test results: https://dev.azure.com/IFailAt/freebsd-dotnet-runtime-nightly/_testManagement/runs?_a=runCharts&runId=1052

EDIT: If you want a debug build. I can build/run that too

[am11]

I don’t seem to have the access.

[Thefrank]

ah! Even as a public repo, non-members can't access test results by default.

The build itself should have the tests and this works without being logged in: https://dev.azure.com/IFailAt/freebsd-dotnet-runtime-nightly/_build/results?buildId=1603&view=results

[am11]

Thanks. Not sure if -test supersedes native aot smoke tests, but that pipeline can probably run them explicitly as a separate step? ./src/tests/build.sh -nativeaot [Debug|Release] -tree:nativeaot (more info: https://github.com/dotnet/runtime/blob/bffe34e0c7ff8a05e79d884ed8447426aae17bfb/docs/workflow/building/coreclr/nativeaot.md#running-tests)