LibraryImport was working properly with NET8 but not anymore with NET9

[RomainAn]

Description

Hi everyone.

Big congratulation for NET9 release ! We appreciate your hard work and we are looking forward for the future of dotnet.

We recently updated our stack from NET8 to NET9, and most of the transition went well. However one service that we have is now failing to work and the only difference is the version of .NET.

        [LibraryImport(DllName, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(AnsiStringMarshaller)), SuppressUnmanagedCodeSecurity()]
        [DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)]
        [UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
        internal static partial int tpcall(string svc, IntPtr idata, long ilen, out IntPtr odata, out long olen, long flag);

     int tpcall(char *svc, char *idata, long ilen, char **odata, long *olen, long flags);

The service is a wrapper around a C library (Oracle Tuxedo).

We are now getting InvalidOperation Exception : TPEINVAL error 4 (which is about parameters we feed the method invocation. We did check breaking changes about interop but we only found this : https://learn.microsoft.com/en-us/dotnet/core/compatibility/interop/9.0/cet-support

But this doesn't seems to be applying to us.

Is there any other changes that might affect the Interop part between NET8/9 ?

Best regards.

Reproduction Steps

I don't think it's possible to provide any minimal repro code.

Expected behavior

The method call should be working as expected.

Actual behavior

The Interop call is now failing

Regression?

This was working fine in NET8 but is now failing in NET9

Known Workarounds

No response

Configuration

No response

Other information

No response

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/interop-contrib See info in area-owners.md if you want to be subscribed.

[huoyaoyuan]

You can inspect the generated code by LibraryImport generator to see if there's any difference, and use a minimal C library to dump the values accepted at native side.

[karakasa]

The generated code are essentially same.

.NET 8

// <auto-generated/>
internal static unsafe partial class PInvoke
{
    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Interop.LibraryImportGenerator", "8.0.11.1707")]
    [global::System.Runtime.CompilerServices.SkipLocalsInitAttribute]
    internal static partial int tpcall(string svc, nint idata, long ilen, out nint odata, out long olen, long flag)
    {
        global::System.Runtime.CompilerServices.Unsafe.SkipInit(out odata);
        global::System.Runtime.CompilerServices.Unsafe.SkipInit(out olen);
        byte* __svc_native = default;
        int __retVal = default;
        // Setup - Perform required setup.
        scoped global::System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller.ManagedToUnmanagedIn __svc_native__marshaller = new();
        try
        {
            // Marshal - Convert managed data to native data.
            __svc_native__marshaller.FromManaged(svc, stackalloc byte[global::System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller.ManagedToUnmanagedIn.BufferSize]);
            // Pin - Pin data in preparation for calling the P/Invoke.
            fixed (long* __olen_native = &olen)
            fixed (nint* __odata_native = &odata)
            {
                // PinnedMarshal - Convert managed data to native data that requires the managed data to be pinned.
                __svc_native = __svc_native__marshaller.ToUnmanaged();
                __retVal = __PInvoke(__svc_native, idata, ilen, __odata_native, __olen_native, flag);
            }
        }
        finally
        {
            // CleanupCallerAllocated - Perform cleanup of caller allocated resources.
            __svc_native__marshaller.Free();
        }

        return __retVal;
        // Local P/Invoke
        [global::System.Runtime.InteropServices.DllImportAttribute("a.dll", EntryPoint = "tpcall", ExactSpelling = true)]
        [global::System.Runtime.InteropServices.UnmanagedCallConvAttribute(CallConvs = new global::System.Type[] { typeof(global::System.Runtime.CompilerServices.CallConvCdecl) }), global::System.Runtime.InteropServices.DefaultDllImportSearchPathsAttribute((global::System.Runtime.InteropServices.DllImportSearchPath)4096)]
        static extern unsafe int __PInvoke(byte* __svc_native, nint __idata_native, long __ilen_native, nint* __odata_native, long* __olen_native, long __flag_native);
    }
}

.NET 9

// <auto-generated/>
internal static unsafe partial class PInvoke
{
    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Interop.LibraryImportGenerator", "9.0.11.2809")]
    [global::System.Runtime.CompilerServices.SkipLocalsInitAttribute]
    internal static partial int tpcall(string svc, nint idata, long ilen, out nint odata, out long olen, long flag)
    {
        odata = default;
        olen = default;
        byte* __svc_native = default;
        int __retVal = default;
        // Setup - Perform required setup.
        scoped global::System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller.ManagedToUnmanagedIn __svc_native__marshaller = new();
        try
        {
            // Marshal - Convert managed data to native data.
            __svc_native__marshaller.FromManaged(svc, stackalloc byte[global::System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller.ManagedToUnmanagedIn.BufferSize]);
            // Pin - Pin data in preparation for calling the P/Invoke.
            fixed (long* __olen_native = &olen)
            fixed (nint* __odata_native = &odata)
            {
                // PinnedMarshal - Convert managed data to native data that requires the managed data to be pinned.
                __svc_native = __svc_native__marshaller.ToUnmanaged();
                __retVal = __PInvoke(__svc_native, idata, ilen, __odata_native, __olen_native, flag);
            }
        }
        finally
        {
            // CleanupCallerAllocated - Perform cleanup of caller allocated resources.
            __svc_native__marshaller.Free();
        }

        return __retVal;
        // Local P/Invoke
        [global::System.Runtime.InteropServices.DllImportAttribute("a.dll", EntryPoint = "tpcall", ExactSpelling = true)]
        [global::System.Runtime.InteropServices.UnmanagedCallConvAttribute(CallConvs = new global::System.Type[] { typeof(global::System.Runtime.CompilerServices.CallConvCdecl) }), global::System.Runtime.InteropServices.DefaultDllImportSearchPathsAttribute((global::System.Runtime.InteropServices.DllImportSearchPath)4096)]
        static extern unsafe int __PInvoke(byte* __svc_native, nint __idata_native, long __ilen_native, nint* __odata_native, long* __olen_native, long __flag_native);
    }
}

[MichalPetryka]

C long corresponds to CLong, not C# long.