Reloaded-Project / Reloaded.Injector

C# DLL Injection Library capable of injecting x86 DLLs to x86 process from x64 processes.
GNU Lesser General Public License v3.0
159 stars 32 forks source link

Allow creating process with injected DLL #7

Open chyyran opened 2 years ago

chyyran commented 2 years ago

I am trying to hook Vulkan in an application, and due to how the Vulkan loader works, it doesn't seem possible to hook after the application calls vkCreateInstance.

[X64.Function(X64.CallingConventions.Microsoft)]
[X86.Function(X86.CallingConventions.Stdcall)]
[UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode, SetLastError = true)]
public unsafe delegate Result vkQueuePresentKHR(Queue queue, PresentInfoKHR* pPresentInfo);```

...

var handle = Kernel32.GetModuleHandle("vulkan-1");

// QueuePresentKHRFn is never called.
this.VkQueuePresentKHRHook  = ReloadedHooks.Instance.CreateHook<vkQueuePresentKHR>(QueuePresentKHRFn, (long)handle.GetProcAddress("vkQueuePresentKHR")).Activate();

I realize that the canonical way of doing Vulkan hooking is via layers, but doing it via a validation layer complicates .NET interop, so if possible it would be nice to do it the 'traditional' way.

To be able to hook Vulkan properly, I need to be able to obtain a handle to the created VkInstance, and thereafter I can use vkGetInstanceProcAddr. vkCreateInstance is called directly via vulkan-1.dll exports, so if I can hook the process before the target application creates its instance, I can obtain a handle to the VkInstance and resolve the other functions via vkInstanceGetProcAddr.

Detours provides DetourCreateProcessWithDllEx which does injection before the process is fully loaded, so I can hook vkCreateInstance as early as possible. I was wondering if this functionality could also be provided in Reloaded.Injector.

Alternatively maybe I'm doing things completely wrong and there's a much easier way to hook Vulkan calls without needing early-hooking, in which case I would be happy to do instead.

Sewer56 commented 2 years ago

A few ways to approach this.

One that doesn't involve modifying the target folder whatsoever is as follows:

Only problem is this approach deadlocks during one of the initialisation steps with this specific library, so I wound up creating a simpler injector for another project that needed this sort of behaviour.

Was something related to grabbing the address of kernel32 IIRC.

chyyran commented 2 years ago

I tried creating the process suspended and indeed ran into the deadlocking issues mentioned.

Rather than having to create another injector library, for my use case specifically I can probably 'inject' my CLR loader as a Vulkan layer and just pass in pfnNextGetInstanceProcAddr to the entry point, then do hooking as usual. Theoretically I think that should work though I haven't tried it out yet. Just a bit disappointing that I can't do it the same way as the other graphics APIs.