Support setting GCC attributes ms_abi and sysv_abi on x86-64

[nattthebear]

I'm currently working on porting a Windows only application to Linux and the lack of ability to express the equivalent of __attribute__((ms_abi)) or __attribute__((sysv_abi)) is a major pain point. I'm having to write my own custom wrappers around GetFunctionPointerForDelegate / GetDelegateForFunctionPointer that allocate their own trampolines to switch calling conventions. The whole thing works, but I feel like a fully functional foreign call interface should support this when targeting amd64. A lot of the CLR interop stuff feels very slanted towards x86-32 and Windows APIs.

[AaronRobinsonMSFT]

@nattthebear With respect to the scenario in question, I was a bit confused as to why anyone would want to use the ms_abi on a non-Windows scenarios. I asked some of the other .NET team members and the most common response was having a collection of hand written assembly that was targeted for Windows and is now being ported to a non-Windows platform. Is this the scenario in question? If there is another scenario were this is also common please feel free to share that as well.

I am interested in the type of code being ported over to infer perhaps how common the scenario is and how many .NET developers would experience the described issue.

but I feel like a fully functional foreign call interface should support this when targeting amd64.

I agree in principle with this notion. The .NET FFI is lacking in some areas and they do need to be improved. For example, support for vectorcall. In this case support for a non-target platform's calling convention feels like the beginning of a platform emulation layer (e.g. Wine). I am unclear how desirable this is from a .NET runtime point of view since it puts a lot of pressure on the runtime to support precise non-target platform semantics on the target platform. There is no doubt there is value, but this comes down to a cost/benefit debate. Better understanding scenarios like yours would help in that debate.

A lot of the CLR interop stuff feels very slanted towards x86-32 and Windows APIs.

That is generally true and is of course historical. Specific API suggestions are welcome. The details on how to proceed can be found at here. The CallingConvention.Winapi is poorly named, but represents the default calling convention on the OS (e.g. 32-bit Windows stdcall, 64-bit Windows win-x64, 32-bit Linux cdecl, 64-bit Linux sysv-x86-64).

[nattthebear]

@AaronRobinsonMSFT Thank you for your response! The more I think about it, I suspect my use case isn't very common at all. I have native binaries that are architecture dependent but OS independent: They're compiled as x86-64 using the msabi calling convention, but all syscalls are through an abstraction layer. My C# application hosts those binaries, providing the OS abstraction layer and other related services and then interop to the native code within.

The native binaries are under my control and I could recompile them targeting sysv, but I also want memory dumps from them to be sharable cross platform, which means the exact same binaries on all supported systems, so either the Windows host would have to support sysv, or the Linux host would have to support msabi.

With a bit of spit and wire it all works, and the custom trampolines code (which is limited to my use case and doesn't support much beyond what I need right now) is only a few hundred lines, so maybe it's not a "huge pain point" for me after all.

Specific API suggestions are welcome.

I don't think I have the experience and expertise with .NET interop needed to make specific proposals.

[AaronRobinsonMSFT]

I have native binaries that are architecture dependent but OS independent: They're compiled as x86-64 using the msabi calling convention, but all syscalls are through an abstraction layer.

That is really impressive and sounds like some interesting tech.

The native binaries are under my control and I could recompile them targeting sysv, but I also want memory dumps from them to be sharable cross platform, which means the exact same binaries on all supported systems, so either the Windows host would have to support sysv, or the Linux host would have to support msabi.

Yep, makes sense.

With a bit of spit and wire it all works, and the custom trampolines code (which is limited to my use case and doesn't support much beyond what I need right now) is only a few hundred lines, so maybe it's not a "huge pain point" for me after all.

Thanks for the links, very informative. I agree this seems like a rather niche scenario. As I mentioned being able to support this does have value, but the cost seems a bit high for the few advanced users that would require the feature.

I don't think I have the experience and expertise with .NET interop needed to make specific proposals.

Proposals can be very simple. In this case suggesting that the ms_abi and sysv_abi options be added to the CallingConvention enum with a list of reasons would be enough. It isn't expected that the proposer is also the implementer. If the community shows enough interest someone from the .NET team would probably wind up adding support, otherwise someone from the community would need to do the work.

I've marked this for Future and will update the title of the issue. We can leave this open for a while to see if anyone else in the community has thoughts as well.

[DaZombieKiller]

Just commenting to show my support for this feature. My scenario is very similar to the one expressed here, so many of the same factors apply.

Ideally in addition to adding ms_abi and sysv_abi to the CallingConvention enum, these types could be exposed so they can be used with function pointers:

namespace System.Runtime.CompilerServices
{
    public class CallConvMsabi { }
    public class CallConvSysv { }
}

[Perksey]

We've just hit this on dotnet/Silk.NET in trying to interoperate with the vkd3d library, which uses ms_abi to mirror the Windows implementation of d3dcompiler. The scenario is we're trying to enable native portability of applications written against DirectX by using the Vulkan translation layer, and we're getting a segfault when calling into the library using plain stdcall because of this ms_abi issue.

cc @tannergooding @Beyley looked like there was an issue for this already

[Bargest]

I've just hit the same problem. I have OS-independent code that takes callbacks from caller and invokes them using ms abi. On dotnet side I have an array of delegates and pass them to native code. Since I pass result of GetFunctionPointerForDelegate() and on linux the native function generated by dotnet is SYSV-oriented, library crashes trying to invoke the callback. The same problem when calling library callbacks from dotnet using GetDelegateForFunctionPointer. I've ended up writing back and forth trampolines on assembly language, converting ABI between MS<->SYSV, building them into raw byte array and wrapping each GetFunctionPointerForDelegate() result with such trampolines. Generating a generic forward trampoline for GetDelegateForFunctionPointer() was a great pain and this code does not look stable and secure at all.

If GetFunctionPointerForDelegate and GetDelegateForFunctionPointer could take an extra argument, directly specifying ABI and CallingConvention, it would be very handy. Something like

var nativeForDelegate = Marshal.GetFunctionPointerForDelegate<T>(func, CallingConvention.StdCall | CallingConvention.MsAbi);
var delegateForNative = Marshal.GetDelegateForFunctionPointer<T>(ptr, CallingConvention.StdCall | CallingConvention.MsAbi);

Implementing this on dotnet side does not seem to be a big problem. As far as I understand current implementation of delegate marshalling it already uses precompiled native stubs as a kind of trampoline, expands them with info about delegate and stores them into some cache. In this case dotnet could have not just one stub, but one for each specified calling convention and abi combination.

Another option is to expand delegate's UnmanagedFunctionPointerAttribute to include ABI, but this is much less convenient. In case of two instances of OS-independent native code with same interface, one with MS ABI and other with SYSV ABI, if we need dotnet code to support both versions on both platforms, we would have to duplicate all delegate declarations in dotnet application with different attribute sets.

[nattthebear]

Glad to see there's still interest in this issue :+1:.

In my case, we did switch from ms to sysv for unrelated reasons, but as I detailed in my previous post that just means we need to marshal in the other direction (and since we have callbacks back into managed, still in both directions somewhere.) These days, we're handling it with these trampolines written in assembly: https://github.com/TASEmulators/BizHawk/blob/master/ExternalProjects/LibBizAbiAdapter/msabi_sysv.s#L143-L147. They include full unwind information so SEH stack scanning can pass through them, and expect a JITed stub to provide the function pointer in rax which we do directly in C#. They don't handle any more than 6 integer args or 4 fp args, and no mix of the two, so still are much cruder than a real solution.

[Bargest]

Thanks, maybe I'll use it later. At least this looks more stable than my approach. I don't have control over native part though. Since I didn't need fp args and didn't want to build a native proxy dll, I compiled the following and saved this just as 3 byte arrays, allocating mem for each .net callback and saving GetFunctionPointerForDelegate() result at .dq 0. Trailer was the same for all created callbacks. .netToNative was used directly passing native address as first arg.

use64

; trampoline to call .net sysv callback from msabi native code on linux 
.nativeToNet:
dq 0 ;  <store trailer addr here>
dq 0 ;  <store function addr here>
call $+5
pop rax
; save registers
; SystemV: preserves           rbx, rsp, rbp, r12, r13, r14, r15
; MSABI:   preserves rdi, rsi, rbx, rsp, rbp, r12, r13, r14, r15
push rdi
push rsi
; translate arguments
; first save 7th argument to stack
mov rdi, [rsp + 24 + 32 + 8 * 2] ; saved  rsi + rdi + ret + shadow(32)
push rdi
; then all others
mov rdi, rcx
mov rsi, rdx
mov rdx, r8
mov rcx, r9
mov  r8, [rsp + 32 + 32 + 8 * 0] ; saved  arg7 + rsi + rdi + ret + shadow(32)
mov  r9, [rsp + 32 + 32 + 8 * 1]
; call managed handler
pushq [rax - 5 - 16]   ; push trailer addr (one for all trampolines)
mov rax, [rax - 5 - 8] ; call handler
jmp rax

.trailer:
add rsp, 8 ; skip saved arg7
; restore registers
pop rsi
pop rdi
ret

; trampoline to call msabi callback from sysv .net code on linux
.netToNative:
; translate sysv abi to msabi
; function addr is in first argument (rdi)
; SystemV: preserves           rbx, rsp, rbp, r12, r13, r14, r15
; MSABI:   preserves rdi, rsi, rbx, rsp, rbp, r12, r13, r14, r15
; msabi saves everything sysv expects to be saved, so no need to save extras
; translate stack arguments
mov rax, [rsp + 16] ; skip ret, arg7
push rax    ; save arg7
mov rax, [rsp + 16] ; skip ret, saved arg, get arg6
push rax    ; save arg 6
push r9     ; save arg 5
sub rsp, 32 ; allocate shadow space
; translate reg arguments
mov r9, r8
mov r8, rcx
mov rdx, rdx ; just nothing
mov rcx, rsi
; call unmanaged handler
call rdi
add rsp, 32 + 24 ; skip shadow and saved args
ret