Support LibraryImport with non-C# languages

[DemiMarie]

DllImport is no longer recommended as it is unfriendly to tree shaking and requires extra runtime support. The replacement is LibraryImport, but that requires a source generator that is only available to C#. Therefore, languages like F# do not support it, and it is not clear how they should support it.

This is a request to document what the actual tree-shaking-friendly runtime primitive is, so that source language compilers know what code they need to emit. This is a prerequisite for tree-shaking-friendly FFI in languages other than C#.

[ghost]

Tagging subscribers to this area: @agocke, @sbomer, @vitek-karas See info in area-owners.md if you want to be subscribed.

Issue Details

---

`DllImport` is no longer recommended as it is unfriendly to tree shaking and requires extra runtime support. The replacement is `LibraryImport`, but that requires a source generator that is only available to C#. Therefore, languages like F# do not support it, and it is not clear how they should support it. This is a request to document what the actual tree-shaking-friendly runtime primitive is, so that source language compilers know what code they need to emit. This is a prerequisite for tree-shaking-friendly FFI in languages other than C#.

Author:	DemiMarie
Assignees:	-
Labels:	`untriaged`, `area-Tools-ILLink`, `needs-area-label`
Milestone:	-

[PaulusParssinen]

Using DllImport for the native interop logic itself is not the issue for trimming (term more commonly used for tree shaking in .NET) but the runtime generated marshalling logic that is enabled by default. The LibraryImport actually uses the DllImport -attribute under the hood in the source generated code but with blittable^1 signature which requires no runtime marshalling.

You can also disable the runtime marshalling for DllImport with DisableRuntimeMarshalling -attribute or at project level in the .csproj with <DisableRuntimeMarshalling>true</DisableRuntimeMarshalling>^2

I recommend checking out following article for more information on the topic: https://learn.microsoft.com/en-us/dotnet/standard/native-interop/pinvoke-source-generation

[EgorBo]

Still a fair point that features like this only improve C#. And F# still has to force runtime to generate the pinvoke stub in run time (for non-blittable signatures).

[DemiMarie]

@EgorBo an alternative approach would be a generic solution that every source language could use.

[tannergooding]

A good portion of the actual marshalling stuff that the LibraryImport generator uses is reusable by other languages

Someone particularly interested could write a tool that uses the relevant F# features to do the same thing there (such as using a type provider, which may not be the best way but is certainly a way).

Similarly, someone could extend existing tools, like https://github.com/dotnet/clangsharp, to support other language targets such F# (it currently supports C# and an XML based output format, and is loosely designed to support other languages if that were desired).

There isn't really such a thing as a generic solution here. Every language is pretty unique and while some core information might be reusable, the actual logic required to generate code per language can get quite specialized. Even in cases where you have a shared interface like Roslyn between C#/VB, you can still end up with a lot of non-reusable logic or handling to support the various quirks of each language.

[tannergooding]

Note, most of the reusable stuff exists here now: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.marshalling?view=net-8.0

Each language still has to do some basics around recognizing the LibraryImport attribute, generating the corresponding DllImport and the basic marshalling logic. But the actual marshalling logic basically boils down to using these publicly exposed helper types and following the fairly straightforward logic to map from the user-defined signature to the blittable signature.

[DemiMarie]

@tannergooding one alternative that works with any language would be an IL ⇒ IL translator, which does the same thing the runtime would do except at compile time.

[tannergooding]

IL => IL translator itself can have issues and may be non-portable.

Not all IL features are supported by all languages. Transforming existing IL can subtly break sequence points or debugging, can easily lead to de-optimizations if they use patterns that aren't what one of the standard compilers emit, can break signing or other downstream tooling, may not integrate cleanly with the general build system or expectations, can increase the risk of bugs, increase build times, etc.

Not to mention that it is functionally a "black box", meaning that you can no longer trivially view the generated code, step into it, modify it if necessary, learn from it as easily, etc.

Source Generators do have their downsides, but overall they are one of the best directions you can go when you look at and consider all the upsides. The main downside is that each language may need to support the thing independently, but that is really a necessity for most features when you actually sit down and think about it. Almost any new or core runtime feature requires some kind of language integration, and source generators are really no different.

[tannergooding]

And notably, if the fact that source generators are going to be used is accounted for, one can provide reusable components (like the System.Runtime.InteropServices.Marshalling namespace) which makes it simpler for a language to support the functionality.

Enough so that the primary concern they have it one around doing the basic recognition of the pattern and the selection of the language specific code they want to generate.

[neon-sunset]

DllImport is no longer recommended as it is unfriendly to tree shaking and requires extra runtime support.

@tannergooding one alternative that works with any language would be an IL ⇒ IL translator, which does the same thing the runtime would do except at compile time.

The LibraryGenerator-based P/Invokes are completely orthogonal to tree-shaking aka trimming done for AOT and certain JIT-based applications and/or libraries.

The reason it was replaced was two-fold: 1) runtime marshalling stub generation was old, brittle and resistant to (performance) improvements, effectively tech debt; 2) runtime marshalling relies on JIT and is therefore cannot be done during AOT. Therefore, [LibraryImport] that relies on source generation was made. Languages which target .NET can simply opt to generate P/Invokes in C# and easily call them directly without any subsequent work (except perhaps certain F# constructs) - after all, because everything is compiled down to IL, such methods can be transparently called from F#, VB.NET and smaller projects. In fact, there are many existing P/Invoke generators and nothing restricts anyone from consuming them from other .NET languages because of the common type system.

In addition, as others noted, some languages like F# have more advanced type/code generation features like type providers which can be extended to define [DllImport]s with blittable arguments and return types (signatures which do not require marshalling) to customize and tune the interop experience to their needs if there is a need to completely bypass using C#-related tooling.

Even if some "generic" solution is introduced where the IL sequences are expanded by some build post-processing phase, the marshalling semantics can and will differ in each language resulting in a requirement for significant upfront work (for each language) to lower the representation to some simpler common denominator in IL, because IL does not and cannot know about types like F# unions or nullable structs in C# (both can be arguable special-cased, at the cost of violating the abstraction).

[MichalStrehovsky]

I'll move this to area interop and change the title. There's no trimming issues with DllImport that LibraryImport would fix. DllImports are trim-safe and in cases when they're not, LibraryImport doesn't help (things like AsAny marshalling that LibraryImport doesn't support). LibraryImport improves startup time and makes it possible to evolve interop support independently of runtime.

I don't think the interop team has plans to abandon the advantages that source generators provide (ability to step through the generated code to troubleshoot incorrect p/invoke declarations) in favor of an IL to IL transformer.

[jkotas]

There's no trimming issues with DllImport that LibraryImport would fix. DllImports are trim-safe and in cases when they're not, LibraryImport doesn't help (things like AsAny marshalling that LibraryImport doesn't support).

LibraryImport helps with detecting cases that do not work. If you use LibraryImport, you can be sure that it is trim and AOT compatible. If you use DllImport (w/o DisableRuntimeMarshalling), you have to be careful.

[MichalStrehovsky]

There's no trimming issues with DllImport that LibraryImport would fix. DllImports are trim-safe and in cases when they're not, LibraryImport doesn't help (things like AsAny marshalling that LibraryImport doesn't support).

LibraryImport helps with detecting cases that do not work. If you use LibraryImport, you can be sure that it is trim and AOT compatible. If you use DllImport (w/o DisableRuntimeMarshalling), you have to be careful.

IIRC we generate publish-time warnings whenever problematic cases are hit (which pretty much just means COM in the trimming case; I think even AsAny marshalling is going to "just work" with trimming).

Then there are some DllImports that are problematic for AOT but not for trimming. Generating warnings for those is tracked in #74697.