Process crashes if CloseHandle is Hooked

[Drobor]

If .NET 5\6\7 is used and CloseHandle hook is activated inside 32bit app like that

private static IHook<CloseHandle_Delegate> s_closeHandleHook;

[UnmanagedFunctionPointer(CallingConvention.StdCall, SetLastError = true)]
[Function(CallingConventions.Stdcall)]
public delegate int CloseHandle_Delegate(int hObject);

public static void Main(string[] args)
{
    var address = GetProcAddress(LoadLibraryW("kernel32.dll"), "CloseHandle");            
    s_closeHandleHook = ReloadedHooks.Instance.CreateHook<CloseHandle_Delegate>(CloseHandle_Hook, (long)address);
    s_closeHandleHook.Activate();
    Console.ReadLine();
}

private static int CloseHandle_Hook(int hObject)
{
    return s_closeHandleHook.OriginalFunction(hObject);
}

Process crashes on Win10x64.

Depending on .NET version following happens: NET5.0: Silently crashes NET6.0: Crashes with message to stderr "Fatal error while logging another fatal error." NET7.0: Crashes with message to stderr "Fatal error. Failed to setup new thread during reverse P/Invoke"

If i run it on my Win7x86 Vmware VM NET5.0: Appears to be working correctly NET6\7 Crashes with message to stderr "Fatal error. Invalid Program: attempted to call a UnmanagedCallersOnly method from managed code."

This issue affects 32bit netcore apps only, 64 bit apps work correctly. .NET Framework 4.8 32bit apps are working too,

Here is code that i used to reproduce this issue: https://github.com/Drobor/Reloaded.Hooks/tree/CloseHandleCrash/source/CloseHandleTest

[Sewer56]

I've ran into this issue before, and the culprit is actually .NET's native<=>managed transition. More specifically you might notice it if you follow JIT_PInvokeEndRarePath in .NET Source code.

What can happen is it's possible for a CloseHandle hook to infinitely recurse itself, causing a stack overflow to occur. This is because part of the native<=>managed transition can itself trigger a CloseHandle call (to close a thread, related to GC), and that would go back into your hook again, and trigger itself again and back into your hook etc.

How you really wish to go about this is up to you, in one of my projects, for example, I just wrote assembly code to filter out only handles I wanted before passing them to my .NET callback function.

If you want to go crazy, feel free to try Thread Local Storage from ASM, and use variable to detect recursion; I considered that approach at some point but it was a bit overkill for my purpose.