Hooks fail to run under RenderDoc/apitrace

[BlueAmulet]

When a game, Muck in this case, is run through RenderDoc or apitrace, doorstop's hooks fail to activate. I confirmed that winhttp.dll loads in the game's process, but no environment variables from doorstop are created. I built x64 VerboseRelease 3.4.1.0 and the resulting log is the same as a normal run except it just ends at "Hook installed!!", instead of continuing on with "Got mono_jit_init_version at XX" and so forth.

[ghorsington]

Greetings!

Doorstop makes use of DLL proxying and import address table editing to hook some mono runtime functions. Any external tools that use the same approach to "hijack" the process without allowing other DLLs to hook as well will not work with Doorstop. For example, API Monitor provides multiple injection approaches, out of which only attaching via remote thread and context switching works in a way that allows other proxy DLLs to load.

At the moment there are no plans to switch up the hooking approach, so this will likely go unadressed. I'll keep the issue open since this also likely concerns Doorstop 4 but it's unlikely to be fixed in the near future.

[PJB3005]

I ran into this and found that you can modify BepInEx (or whatever you're loading with Doorstop) to LoadLibraryW("renderdoc.dll") first thing. This runs before Unity even initializes the game window, so RenderDoc can attach without stopping Doorstop from loading. Then you can attach to running instance in RenderDoc.

e.g. changing BepInEx's preloader entrypoint:

        [DllImport("KERNEL32.DLL", SetLastError = true)]
        private static extern IntPtr LoadLibraryW([MarshalAs(UnmanagedType.LPWStr)] string lpLibFileName);

        /// <summary>
        ///     The main entrypoint of BepInEx, called from Doorstop.
        /// </summary>
        public static void Main()
        {
            LoadLibraryW(@"C:\Program Files\RenderDoc\x86\renderdoc.dll");
            // ...