Yubico.NativeShims doesn't build as expected with AOT

SheepReaper commented 1 year ago

When building an executable the native AOT compiler, Yubico.NativeShims.dll is output next to the executable instead of being trimmed away. There are also no compiler warnings about dependencies having trimming issues, which is odd. But I suspect it has to do with native shims being pre-built/native or shared vs static, it's not IL. It probably needs to be IL or static, or maybe static with a p/invoke wrapper class, in order for trimming to work (not sure haven't done one of these myself before).

Expected output

single executable with no libraries around it

What's actually happening

Yubico.NativeShims.dll in the output dir.

Steps to reproduce

dotnet new console
dotnet add package Yubico.YubiKey
Add <PublishAot>true</PublishAot> to the project file
dotnet publish -o ./out

Sample with expected output

Instead of adding the YubiKey reference add one to Newtonsoft.Json You should get a ton of warnings about trimming, but the output is as expected absent of any Newtonsoft libs.

Why do I want this?

if you take a peek at my latest builds: https://github.com/SheepReaper/yk-csr-generator/releases/tag/v1.0.0 I'm building 80MB assemblies because of the runtime bits included. Granted, if I didn't do single file, it would be way smaller, but then the user needs to have a runtime installed. I just want to build a single executable with no external dependencies, so I'm fine with the bloat. However, in testing AOT builds, I was able to achieve a 1.5KB binary plus the Shim lib next to it with almost 3KB. Which is super nice, but we're back at the original problem of external dependencies.

Additional references

native static library: https://github.com/dotnet/samples/blob/main/core/nativeaot/NativeLibrary/README.md
aot compatible library limitations: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/aot/native-aot-tutorial?view=aspnetcore-8.0&tabs=netcore-cli#libraries-and-native-aot
declaring your libs to be aot compatible enables the aot-compatibility analyzers (compiler warnings): https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/?tabs=net8plus%2Cwindows#aot-compatibility-analyzers
preparing libs for AOT: https://aka.ms/dotnet-illink/libraries

I don't know enough about the Shims lib to determine what about it is preventing AOT from working nicely, but worst case it's hitting one of the many limitations of AOT compatibility, which might be too much work to overcome. In which case, I'll be sticking to my thicc binaries...

GregDomzalski commented 1 year ago

Phew - clearly there has been a lot of advancements on the NativeAOT front that I need to catch up on!

We have some of our own internal motivations to get the SDK more NativeAOT friendly - however I'm not sure what the timeline on this will be. The SDK was just not designed with NativeAOT in mind. We architected it back in the .NET Core 3.1 era when Microsoft still treated the AOT runtime as a science experiment and had zero commitment to the project.

I think our goals do align here. I'm just not sure that I can commit to a timeline. Is this a showstopper for your project? Or something that you would simply like for us to improve over time? Left to our own devices, we would likely bake in NativeAOT support over the next 6 months or so. But again, that's not a commitment. The trickiest part being that we need to maintain .NET Framework 4.7.2+ compatibility, which means we're stuck on .NET Standard 2.0. I've noticed some of the new source generators that help with AOT target .NET Standard 2.0 (such as the JSON one), but others like the LibraryImport/PInvoke one do not.

I really appreciate the links. Your reports have been super enlightening and helpful to increase my own knowledge 😄

As an aside to this issue: If you know of any links or sample projects that would serve as a better structure for our NativeShims project, I would love to learn. I've never been happy with how that project has worked out. Since one of the goals of the SDK has been to be usable "out of the box" - that's meant we've needed to (NuGet) package our own native dependencies. We've taken a lot of ideas from how the dotnet/runtime project itself does it (at least how they did things 3 years ago) but without taking a dependency on their very complicated build system. It seems to work for 90% of our consumers, but there always seems to be special cases (package.config, PowerShell, etc.) that have just been difficult if not impossible to figure out.

I'm not sure we'll be able to use the DirectPInvoke stuff, though that looks amazing. I'm still reading through your resources, so maybe I'll find more insights in there. Remember that whatever we use, our SDK needs to be consumable/compatible with .NET Framework applications. I have no problem taking a build dependency on the .NET 8 SDK, though.

Any additional insights would be greatly appreciated!

SheepReaper commented 1 year ago

Definitely not a show-stopper for me, my little tool has no adoption as of yet. I did make a typo though, my assemblies are only 20MB. modern .NET has me sold on the native cross-platform capabilities, but I hear the nay-sayers screaming about binary size, and (insert argument invented in the 90s about memory measured in Kilobytes). AOT just seems like the thing that, if it works, why not? So yeah, no hurry. It's never as simple as specifying multiple target frameworks. If I knew how to do it, I'd share, but I don't do much direct native development these days.

Yubico / Yubico.NET.SDK