Define libc targeting plan

[richlander]

We need a new libc targeting plan. We've been able to skip this need for some time since we've used CentOS 7 as our primary build OS. For Arm64, we've used Ubuntu 16.04. Both are not feasible as long term options.

More context: https://github.com/dotnet/core/pull/7437

The proposed plan is:

• Use a recent OS as the build host. We're planning on using Mariner 2, but it could be anything like RHEL 9 or Ubuntu 22.04.
• Acquire an old glibc that we can compile with.
• Do the same for any other dependencies establish a compatiblity contract for .NET with users.

From my perspective, it would be ideal if we could acquire these dependencies from a trusted server in the RHEL ecosystem. The strongest concerns have come from that ecosystem, so it makes sense to orient the solution in that direction. I don't see any downsides to such an approach to other ecosystems.

/cc @omajid @jkotas @MichaelSimons

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/area-meta See info in area-owners.md if you want to be subscribed.

Issue Details

---

We need a new libc targeting plan. We've been able to skip this need for some time since we've used CentOS 7 as our primary build OS. For Arm64, we've used Ubuntu 16.04. Both are not feasible as long term options. More context: https://github.com/dotnet/core/pull/7437 The proposed plan is: - Use a recent OS as the build host. We're planning on using Mariner 2, but it could be anything like RHEL 9 or Ubuntu 22.04. - Acquire an old glibc that we can compile with. - Do the same for any other dependencies establish a compatiblity contract for .NET with users. From my perspective, it would be ideal if we could acquire these dependencies from a trusted server in the RHEL ecosystem. The strongest concerns have come from that ecosystem, so it makes sense to orient the solution in that direction. I don't see any downsides to such an approach to other ecosystems. /cc @omajid @jkotas @MichaelSimons

Author:	richlander
Assignees:	-
Labels:	`area-Meta`, `untriaged`
Milestone:	-

[Clockwork-Muse]

Would the Python community's work on the manylinux project be relevant here? This is solving a similar problem, no?

Possibly some of their work on the problem may be relevant; For instance, it turning out that manylinux_2_24 had problems, which they're taking into consideration for manylinux_2_28.

[richlander]

Excellent question. We actually looked at the manylinux solution. See my comment: https://github.com/dotnet/core/pull/7437#issuecomment-1121236518.

Our plan is to do something very similar, except we already have infrastructure that allows us the same outcome w/o quite so much ceremony. Each .NET TFM will target a different libc version. That's the contract. It ends up being the same as manylinux_x_y. Make sense?

[jkoritzinsky]

As we already have the rootfs targeting plan for our cross-architecture builds, it might be worthwhile using a similar model for targeting a down-level libc version (basically cross-compiling from x64 to x64-with-older-libc).

[richlander]

What does rootfs targeting look like with containers? The rootfs instructions I have seen have all been oriented on bare metal. Does it work the same in containers? I think we want to continue to user containers as our primary build environment.

[jkoritzinsky]

We currently use it in containers for our arm32 and arm64 builds (as well as our FreeBSD build).

[richlander]

Cool. Maybe that will work. It's a question of whether we can acquire the desired rootfs. I'd look to @omajid for that.

Got a link to a Dockerfile that does that today?

[hoyosjs]

Image: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/18.04/cross/arm64/Dockerfile

The tar gets produced by the hook: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/18.04/cross/arm64/hooks/pre-build

That uses this script: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/build-scripts/build-rootfs.sh

[omajid]

cc @tmds

[ghost]

Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.

Issue Details

---

We need a new libc targeting plan. We've been able to skip this need for some time since we've used CentOS 7 as our primary build OS. For Arm64, we've used Ubuntu 16.04. Both are not feasible as long term options. More context: https://github.com/dotnet/core/pull/7437 The proposed plan is: - Use a recent OS as the build host. We're planning on using Mariner 2, but it could be anything like RHEL 9 or Ubuntu 22.04. - Acquire an old glibc that we can compile with. - Do the same for any other dependencies establish a compatiblity contract for .NET with users. From my perspective, it would be ideal if we could acquire these dependencies from a trusted server in the RHEL ecosystem. The strongest concerns have come from that ecosystem, so it makes sense to orient the solution in that direction. I don't see any downsides to such an approach to other ecosystems. /cc @omajid @jkotas @MichaelSimons

Author:	richlander
Assignees:	-
Labels:	`area-Infrastructure-coreclr`, `untriaged`
Milestone:	-

[normj]

Any updates on this issue for either .NET 7 or 8 so distributions with lower GLIBC versions can continue to work?

[jkotas]

Any updates on this issue for either .NET 7 or 8 so distributions with lower GLIBC versions can continue to work?

We have been investigating how to create a build environment that would fulfill all requirements.

@sbomer Anything you can share about our progress?

cc @janvorli

[sbomer]

Yes, so far it looks like we have a path forward to fixing this for .NET 7 and 8, by cross-building for an OS with the older glibc. I'll describe the setup we intend to use for .NET 7 arm64 linux:

• Ubuntu 20.04 as the host os. This is in support for longer than the current 18.04 build machines, and still has llvm 9 available.
• Ubuntu 16.04 as the target os. This will lower the glibc requirements. We have existing infrastructure that can create a rootfs for cross-compilation.
  • Llvm 9 is not available for 16.04, but we need a specific llvm 9 library to be built for the target. We will build this from source (similar to how we build llvm 9 in centos 7 for our x64 builds today).

@janvorli kindly gave me some pointers and I am working on this. I hope to have a fix soon for .NET 7.

For .NET 8 there might be more changes, including using later Ubuntu releases as the host, and/or using Mariner instead for the official build - but the general idea is the same (using cross-compilation to support a low-enough glibc).

[normj]

@sbomer That is great news. Sorry to ask the annoying next question but any guess on the timeline for these changes?

[sbomer]

@normj it's a fair question :)

I'm aiming to get it fixed in one of the next servicing releases for .NET 7 - probably 7.0.4 (sounds like it is too late to make 7.0.3). And for .NET 8, probably Preview 2. This is assuming there aren't any big unforeseen blockers. I'll post any updates here.

[normj]

Great, so if I understand correctly by targeting Ubuntu 16.04 that would set the minimum GLIBC version to 2.23. Is that correct?

[janvorli]

@normj yes, that's correct.

[normj]

Please feel free to ping me if you want any early tests of new builds on Amazon Linux 2 which is currently stuck because of its usage of GLIBC 2.26.

[richlander]

When you make this change, can you update the .NET 7 release notes @sbomer?

https://github.com/dotnet/core/blob/main/release-notes/7.0/supported-os.md#libc-compatibility

[sbomer]

Will do!

[sbomer]

@normj we have a CI build of the .NET 7 runtime binaries with the fix: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/142699/artifacts?artifactName=CoreCLRProduct___Linux_arm64_release&api-version=7.0&%24format=zip.

Would you be able to test using these bits? You can use the instructions at https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/using-your-build-with-installed-sdk.md, but use the .NET 7 SDK together with the binaries from this build.

[normj]

@sbomer Yes definitely I'll give the bits a test through.

[omajid]

@richlander this change might have an impact on third-party .NET projects that carry native libraries on arm64. If they were targeting net7.0 on arm64, until now they could also include shared libraries which needed glibc 2.26 to run. With this change, they might now be expected to provide libraries that run on a lower glibc version. Do you think this deserves a larger announcement and notice to the .NET ecosystem?

[normj]

@sbomer I'll keep testing but I wanted to give some initial positive feedback. I was able to deploy an ARM based AWS Lambda function using a self contained publish and substituting in the CoreCLR binaries from the link above. The Lambda function worked perfectly compared to before which failed immediately with the GLIBC 2.27 warning. This was our key scenario that was blocked and is now unblocked. Thank you all for your hard work making this change!

[normj]

@sbomer based on doing various deployments, including ASP.NET Core apps, the build looks good for Amazon Linux 2. I was surprised the patch only required updating 3 files (libclrjit.so, libcoreclr.so and System.Private.CoreLib.dll).

[richlander]

Plan for .NET 8:

https://github.com/dotnet/core/issues/8133#issuecomment-1432611595

[am11]

Regarding musl-libc, Alpine 3.13 indicates two things:

1. distro support
2. musl-libc support

In the second form, Alpine 3.13 represents baseline compatibility with musl-libc v1.2.2 for entire set of musl distros: https://wiki.musl-libc.org/projects-using-musl.html. The portable linux-musl builds were enabled in .NET Core 3.0, so it has some mileage beyond Alpine Linux; such as Void Linux (musl), OpenWRT etc.

Now that Alpine 3.13 has reached EOL, can we make it clear that we are keeping this version for the purpose of "testing" with baseline musl-libc v1.2.2 and not for the distro support? Otherwise, we can find another distro for musl-libc baseline testing in CI.

Compared to glibc 2.23 (release on February 19, 2016), musl 1.2.2 is very recent (January 15, 2021). If anything, we should instead try to lower the requirement to v1.2.0 (February 20, 2020) to match their "Stable vs. EOL" series https://musl.libc.org/releases.html rather than Alpine Linux release cycle.

[richlander]

That is interesting. We've been targeting Alpine since (perhaps naively so) we thought that the only folks using .NET with musl were Alpine users. We're definitely open to adopting a different plan for musl targeting if it helps folks.

For glibc, we are planning to target Ubuntu 16.04 for .NET 8.

[smhmhmd]

@sbomer @richlander @normj

We still see the issue on Amazon Linux 2 arm64 while compiling for NativeAOT. Thanks to @Beau-Gosse-dev and @mrkdeng for finding it.

[ec2-user@ip-10-0-0-147 t]$ dotnet publish

MSBuild version 17.5.0+6f08c67f3 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  t -> /home/ec2-user/t/bin/Debug/net7.0/linux-arm64/t.dll
  Generating native code
  /home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc)
  /home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc)
/home/ec2-user/.dotnet/sdk/7.0.202/Sdks/Microsoft.DotNet.ILCompiler/build/Microsoft.NETCore.Native.targets(278,5): error MSB3073: The command ""/home/ec2-user/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/7.0.4/tools/ilc" @"obj/Debug/net7.0/linux-arm64/native/t.ilc.rsp"" exited with code 1. [/home/ec2-user/t/t.csproj

[ec2-user@ip-10-0-0-147 t]$ dotnet --list-sdks
7.0.202 [/home/ec2-user/.dotnet/sdk]
[ec2-user@ip-10-0-0-147 t]$ dotnet --list-runtimes
Microsoft.AspNetCore.App 7.0.4 [/home/ec2-user/.dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 7.0.4 [/home/ec2-user/.dotnet/shared/Microsoft.NETCore.App]

[ec2-user@ip-10-0-0-147 t]$ cat t.csproj 
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <PublishAot>true</PublishAot>
    <Nullable>enable</Nullable>
  </PropertyGroup>

</Project>

[jkotas]

We use .NET SDK 7.0.X for building .NET SDK 7.0.(X+1). It is how the old GLIBC reference got into the ilc binary. This should fix itself in the next servicing update (needs to be verified).

[agocke]

@smhmhmd opened a new issue for this https://github.com/dotnet/runtime/issues/84183

[wasabii]

Hello. I am the present maintainer of IKVM. I wanted to add my two cents to this, since we're encounting a similar issue with IKVM.

IKVM, as many of you know, compiles Java to MSIL. We distribute a full JDK, based on OpenJDK. Up until now we've been rewriting the native C parts in C#. This has worked "fine". But, we've decided to actually just build and distribute the C as is. Will remove a lot of work keeping C# copies of stuff up to date.

So, we have the same issue: building a dozen different native libraries targeting every combination of OS/arch that .NET itself supports. Since, let's say, IKVM supports .NET 8, we should be runnable on every RID .NET 8 supports (we also have to deal with backwards compat, since we need to maintain back to .NET Core 3.1 (and Framework) back to whatever it supports).

That's a lot of OS' and architectures. A lot of different versions of glibc. And musl. And whatever.

The path we're taking is to build SDKs for each of the .NET target RIDs, with the headers and libraries we require back to the version the various TFMs we support advertised. Including for Windows and Mac OS X. And then use clang to cross compile from whatever the host OS/platform is, against those headers and libraries.

This, so far, has worked in tests for every OS/arch we've encountered.

We're going to have a separate project called ikvm-native-sdk or some such that produces these SDKs. Windows and Mac are easy. Those are just single bundles that the user has to acquire. But Linux is not easy. But, I've pretty much got it working.

So we've got scripts to build a cross compile toolchain. And then use that cross compile toolchain to rebuild GCC, GLIBC, etc. This will give us something like a 'netcoreapp3.1-linux-x64' /include and /lib directory, which we can then tar up.

We then use a clang MSBuild project type, which supports a Inner Build structure (just like TargetFramework). For each inner build of this, we point clang to the proper SDK directory. So we can have a single MSBuild project file that builds a dozen or so different targets.

Anyways, cool.

The key take away here is we're not going to be using old versions of RedHat or Ubuntu, or containers. Because we want our devs to be able to build the entire product, including all of the native libraries for every supported RID, from whatever machine they work from. We can produce Linux shared objects while building on Windows, or Mac, or Windows DLLs while building on Linux or Mac, etc.

And we'll have a nice little library of tars that represent the snapshot of the supported APIs at that moment in time.

Food for thought.

[agocke]

Hey @wasabii, thanks for the write-up!

Not sure I fully understood your flow. When you say you build a cross-compile toolchain, is this a cross-compile toolchain on your users' machines? Or a toolchain on your own platform? If the latter, does that mean you're producing, say, dozens of versions of your package for each different Linux distro?

[wasabii]

It is the latter. Just the developers need the toolchains. Just a bunch of tars for them to download. And the answer is no, we're producing the set we need to target the various distros .net supports. Careful selection of glibc version, etc. There will be overlaps. There might be a lot eventually. Unknown.

So far is a half dozen. But will probably end up more in the future.

[wasabii]

https://github.com/ikvmnet/ikvm-native-sdk/releases/tag/20230613.2

This is a new approach for us. So we haven't gone too deep down into the various distros yet. But the approach seems sound.

It will let our developers build everything for all platforms from whatever machine they're on. Including nuget packages with all the native libs in one place. They just need to download a bunch of tars. No docker. No WSL.

And it lets us put the C in msbuild.

[jkotas]

Did you get lawyers to confirm that everything in your approach is compliant with all respective licenses? The kind of problems that this approach can run into is touched on in https://github.com/ikvmnet/ikvm-native-sdk/blob/main/macosx/README .

[wasabii]

For the Linux related stuff there is no concern at all.

Obviously the Windows and Mac stuff differ in that regard. Windows should be fine, since they don't have a hardware restriction. Apple is a different matter.

Regardless, different subject.

[agocke]

It sounds like your end state is very similar, but you're taking on building glibc -- I think that's the part we don't want to do. Keeping glibc up-to-date and supported is something we want to rely on distro maintainers for, in particular for potential security issues.

By building against a supported copy of glibc from a trusted maintainer, we avoid maintaining glibc ourselves.

[wasabii]

Perhaps, except we don't distribute glibc as part of our application. It exists only to link against. Nothing in these SDKs is distributed as part of the application.

It's C headers and .so files for lld at build time. It links to whatever the user is running at runtime.

It's also why we're not likely to need different SDKs per distro. All that matters to us is that the ABI surface is correct. The only time we'd upgrade glibc is if .NET starts requiring a newer version or something. Something I'm sure you guys avoid.

Same way you guys get away building against a super old Ubuntu.

[wasabii]

Also... don't overestimate how difficult it is to actually build a bunch of linux libraries and install them into a tar. Yeah. It's a thing. But it's also not a super complicated thing.

It takes a lot of time to build them, but that's CI/CD's problem.

[agocke]

Note that it's theoretically possible for a vulnerability to exist in those header files. By relying on distro support, we avoid exposure to those files. Understandably, you may have a different risk profile.

Also, note that we only use the Ubuntu base libraries (glibc, etc). The actual build is happening in an up-to-date mariner distribution with a current toolchain.

[agocke]

This work should now be complete