[LibraryImport] Support generating static Marshalling stub for unmanaged function pointer calls.

[teo-tsirpanis]

Background and motivation

The upcoming DllImport source generator will decouple the job of marshalling P/Invoke calls from the runtime, but in its current form it can't do much when native code is invoked via Marshal.GetDelegateForFunctionPointer. Function pointers can be used to call unmanaged code, but the built-in marshaller cannot be avoided if their arguments and return type are not blittable.

I propose a counterpart attribute to LibraryImportAttribute that instructs the generator to only marshal the parameters and the return type of the function, and call a user-specified function pointer instead of a P/Invoke. This attribute will provide the source-generated successor of Marshal.GetDelegateForFunctionPointer.

API Proposal

Updated to match the approved shape of LibraryImportAttribute.

namespace System.Runtime.InteropServices
{
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)]
    public sealed class NativeFunctionMarshalAttribute : Attribute
    {
        /// <summary>
        /// Constructor.
        /// </summary>
        public NativeFunctionMarshalAttribute();

        /// <summary>
        /// Indicates how to marshal string parameters to the method.
        /// </summary>
        /// <remarks>
        /// If this field is specified, <see cref="StringMarshallingCustomType" /> must not be specified.
        /// </remarks>
        public StringMarshalling StringMarshalling { get; set; }

        /// <summary>
        /// Indicates how to marshal string parameters to the method.
        /// </summary>
        /// <remarks>
        /// If this field is specified, <see cref="StringMarshalling" /> must not be specified.
        /// The type should be one that conforms to usage with the attributes:
        /// <see cref="System.Runtime.InteropServices.MarshalUsingAttribute"/>
        /// <see cref="System.Runtime.InteropServices.NativeMarshallingAttribute"/>
        /// </remarks>
        public Type? StringMarshallingCustomType { get; set; }

        /// <summary>
        /// Indicates whether the callee sents an error (SetLastError on Windows or errorno
        /// on other platforms) before returning from the attributed method.
        /// </summary>
        /// <see cref="System.Runtime.InteropServices.DllImportAttribute.SetLastError"/>
        public bool SetLastError { get; set; }
    }
}

The attribute resembles LibraryImportAttribute, with the difference that it lacks the constructor parameter for the library name and the EntryPoint property.

When it is applied to a partial method, the method's first parameter must be an IntPtr (otherwise it is an error) and is not passed to the native call; instead it contains the pointer to the unmanaged function; the generated method casts the IntPtr to the appropriate function pointer type and marshals and passes the other parameters to it.

In all cases that don't involve importing a DLL, the samantics of NativeFunctionMarshalAttribute are identical with LibraryImportAttribute and these two attributes will evolve in tandem. If LibraryImportAttribute gets a relevant new property or behavior, this attribute will get it as well.

API Usage

[NativeFunctionMarshal]
[UnmanagedCallConv(CallConvs = new[] {typeof(CallConvSuppressGCTransition)})]
public partial int QueryPerformanceCounter_wrapper(IntPtr nativeFunc, out long counter);

var kernel32 = NativeLibrary.Load("kernel32.dll");
var qpc_func = NativeLibrary.GetExport(kernel32, "QueryPerformanceCounter");

_ = QueryPerformanceCounter_wrapper(qpc_func, out long counter);

Console.WriteLine($"QPC returned {counter}");

Alternative Designs

• We could use a different attribute name.
• Instead of adding one function pointer argument to the method at the beginning, we could create source-generated specializations of Marshal.GetDelegateFromFunctionPointer<TDelegate>, for various TDelegates. While it would ease migration, it is less efficient and less flexible. And if somebody wants to create such delegates they can still be done with the proposed API.
• The first parameter could have been either of UIntPtr, nint, nuint and void* but there is no reason to be and it would be better to keep things simple. The functions in the NativeLibrary function return an IntPtr either way.

Risks

This attribute brings a conceptual change when dealing with P/Invokes; not all arguments on the managed function correspond to arguments on the native function. There might be some confusion, but this API is not intended for everyday use either way.

[jkoritzinsky]

We talked over this in the interop team meeting, and we think the idea is interesting and the scenario is something we want to support. We're unsure about the proposed attribute shape as we want to keep it in sync with GeneratedDllImport, which is still in flux.

We'll tentatively mark this for .NET 7 in the case that we have time, but it's likely that the implementation might slip to .NET 8.

[teo-tsirpanis]

Thanks for the feedback. I forgot to explicitly mention it, but the new attribute's shape will evolve in tandem with GeneratedDllImportAttribute, except for the properties that have to do with importing a DLL (for now the library name, EntryPoint and ExactSpelling). I will update the proposal to make it more clear, and once GeneratedDllImportAttribute's shape gets finalized, feel free to update this one's, to move the proposal forward.

As for the release it will land, I won't have a problem with either .NET 7 or 8; it's not an API I am expecting to use myself.

[jkoritzinsky]

While working on #64279, I've found a few places where this would be useful in dotnet/runtime:

• https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Sockets/src/System/Net/Sockets/DynamicWinsockMethods.cs
• https://github.com/dotnet/runtime/blob/main/src/libraries/System.Data.OleDb/src/OleDbWrapper.cs
• https://github.com/dotnet/runtime/blob/main/src/libraries/System.Management/src/System/Management/ManagementScope.cs
• https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Internal/MsQuicApi.cs

[AaronRobinsonMSFT]

Good suggestion. This isn't something we are going to get to in 7. Moving to Future.

[teo-tsirpanis]

I don't see how that would help Reverse P/Invokes. There could be a generator mode that creates UnmanagedCallersOnly wrappers that unmarshal arguments and call a normal method but this is unrelated to what I proposed.

[AaronRobinsonMSFT]

I don't see how that would help Reverse P/Invokes. There could be a generator mode that creates UnmanagedCallersOnly wrappers that unmarshal arguments and call a normal method but this is unrelated to what I proposed.

Oops. My mistake. I completely misread this proposal. I will revert the title.

[AaronRobinsonMSFT]

@jkoritzinsky and @elinor-fung I think we should reprioritize this proposal for .NET 8. There is a scenario in the runtime that would benefit from this feature.

/cc @joperezr

[jkoritzinsky]

We aren't going to get to this before feature-complete in .NET 8. Moving to .NET 9.

[ceztko]

Recently we discussed about common interop scenarios for [UnmanagedFunctionPointer] delegates that could be handled better in case of managed exceptions throwing. @jkotas suggested that implementing the API in this issue may be instrumental to plug a managed exception handler for methods that get marshaled to native code as native function pointers. This would remove the need for boiler plate code, like trivial repetitive try-catch blocks, like in my example. Looking at the API here I can't see anything that would help for that specific use case. Unless I'm missing something I would like to suggest how it could be done. Basically NativeFunctionMarshalAttribute could also provide the following property:

/// <summary>
/// Specify how to handle exceptions when the method
/// gets wrapped to be marshaled to native code
/// </summary>
/// <remarks>
/// The type must supply a single static function with
/// signature "void HandleException(Exception ex)"
/// </remarks>
public Type? UnmanagedFunctionPointerExceptionHandlerType { get; set; }

This would not cover more complex scenarios, like surrounding the method call with custom code: it would just handle the exception. I could not grasp if you Jan was thinking something more flexibile, but for me handling exceptions would be more than enough. Do you think this can happen for the .NET 9/10 timeframe as well?