varargs and __arglist on different platforms

[msedi]

Description

We are using MKL (Math Kernel Library) that unfortunately has an external call with var args (see here: https://www.intel.com/content/www/us/en/develop/documentation/onemkl-developer-reference-c/top/fourier-transform-functions/fft-functions/fft-descriptor-manipulation-functions/dfticreatedescriptor.html) and here is the function prototype:

MKL_LONG DftiCreateDescriptor(DFTI_DESCRIPTOR_HANDLE * pHandle,
     enum DFTI_CONFIG_VALUE precision,
     enum DFTI_CONFIG_VALUE domain,
     MKL_LONG dimension, ... /* length/lengths */ );

on Windows it was easy to do, since the following worked:

[SuppressUnmanagedCodeSecurity, SuppressGCTransition]
[LibraryImport(dllName, SetLastError = false)]
[UnmanagedCallConv(CallConvs = new Type[] { typeof(System.Runtime.CompilerServices.CallConvCdecl) })]
public static unsafe partial DFTI_Error DftiCreateDescriptor(out IntPtr desc, DFTI_CONFIG_VALUE precision, DFTI_CONFIG_VALUE domain, int ndim, int[] lengths);

Now, when we wanted to also enable this on linux (docker container), we are getting into troubles since the varsargs (...) is treated differently. So we came up with the idea of using __arglist.

[SuppressUnmanagedCodeSecurity, SuppressGCTransition]
[LibraryImport(dllName, SetLastError = false)]
[UnmanagedCallConv(CallConvs = new Type[] { typeof(System.Runtime.CompilerServices.CallConvCdecl) })]
public static unsafe partial DFTI_Error DftiCreateDescriptor(out IntPtr desc, DFTI_CONFIG_VALUE precision, DFTI_CONFIG_VALUE domain, int ndim, __arglist);

here, the first problem is, that the LibraryGenerator obviously does not support it. Nevertheless when using DllImport we are getting a BadImageFormatException.

ML.NET solves this by creating a native DLL with explicit dimensions, so only 2 or 3 dimensions are supported currently in ML.NET, although the library supports more which is not supported. Of course this would be a way out, but for this sole reason we have to go native and add an additional native library without the possibility of varargs.

Reproduction Steps

Its a bit hard to provide an example, we will try, but I think the behavior is obvious.

Expected behavior

Enable __arglist to work with different platforms as a varargs substitute.

Actual behavior

I think it has never worked, and is for sure a rare case.

Regression?

No response

Known Workarounds

Create multiple native libraries with explicit method signatures.

Configuration

.NET 7.0.2, tested on windows and linux.

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/interop-contrib See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Description We are using MKL (Math Kernel Library) that unfortunately has an external call with var args (see here: https://www.intel.com/content/www/us/en/develop/documentation/onemkl-developer-reference-c/top/fourier-transform-functions/fft-functions/fft-descriptor-manipulation-functions/dfticreatedescriptor.html) and here is the function prototype: ```cpp MKL_LONG DftiCreateDescriptor(DFTI_DESCRIPTOR_HANDLE * pHandle, enum DFTI_CONFIG_VALUE precision, enum DFTI_CONFIG_VALUE domain, MKL_LONG dimension, ... /* length/lengths */ ); ``` on Windows it was easy to, since the following worked: ```cs [SuppressUnmanagedCodeSecurity, SuppressGCTransition] [LibraryImport(dllName, SetLastError = false)] [UnmanagedCallConv(CallConvs = new Type[] { typeof(System.Runtime.CompilerServices.CallConvCdecl) })] public static unsafe partial DFTI_Error DftiCreateDescriptor(out IntPtr desc, DFTI_CONFIG_VALUE precision, DFTI_CONFIG_VALUE domain, int ndim, int[] lengths); ``` Now, when we wanted to tranmsport this to linux, we are getting into troubles since the varsargs (...) is treated differently. So we came up with the idea of using __arglist. ```cs [SuppressUnmanagedCodeSecurity, SuppressGCTransition] [LibraryImport(dllName, SetLastError = false)] [UnmanagedCallConv(CallConvs = new Type[] { typeof(System.Runtime.CompilerServices.CallConvCdecl) })] public static unsafe partial DFTI_Error DftiCreateDescriptor(out IntPtr desc, DFTI_CONFIG_VALUE precision, DFTI_CONFIG_VALUE domain, int ndim, __arglist); ``` here, the first problem is, that the LibraryGenerator obviously does not support it. Nevetheless when using `DllImport` we are getting a `BadFormatImageException`. ML.NET solves this by creating a native DLL with explicit dimensions, so only 2 or 3 dimensions are supported currently in ML.NET, althought the library supports more. ### Reproduction Steps Its a bit hard to provide an example, we will try, but I think the behavior is obvious. ### Expected behavior Enable __arglist to work with different platforms as a varargs substitute. ### Actual behavior I think it has never worked, and is for sure a rare case. ### Regression? _No response_ ### Known Workarounds Create multiple native libraries with explicit method signatures. ### Configuration .NET 7.0.2, tested on windows and linux. ### Other information _No response_

Author:	msedi
Assignees:	-
Labels:	`area-System.Runtime.InteropServices`
Milestone:	-

[AaronRobinsonMSFT]

Support has been discussed multiple times. At present ideas are being discussed at https://github.com/dotnet/runtime/issues/48796. This issue ends in the same result but using existing language features instead of a new mechanism so we should leave it open and connect the two for the debate.

/cc @dotnet/interop-contrib @lambdageek

[msedi]

@AaronRobinsonMSFT. Thanks for pointing to the link. According to what I can read is that there is currently no possibility to use varargs on Linux, right?

[AaronRobinsonMSFT]

@msedi At present, yes. The work around unfortunately is to create a wrapper for the API that doesn't use varargs and instead has specific arguments. We acknowledge this is cumbersome. The varargs support in .NET has a MIcrosoft specific history and is historically tailored for C++/CLI. That doesn't mean we can't improve things, but the relative need for it to other things is low and it has a high cost to design and implement.

The referenced link would make things possible but require declaring all desired overloads in C# and then calling them without varargs expressed in C#, the JIT would do the right thing though via codegen. This is cheaper in some ways but still has high cost relative to community need - that is why it is in "Future". We track these issues for up votes so as interest increases that will help us prioritize.

[msedi]

@AaronRobinsonMSFT. Alright, thanks for the info.