Steam Overlay (+ controller) compatibility issue

We went out of our way to make 3DMigoto and the Steam overlay work together ages back, but something has changed and nowadays the Steam overlay sometimes works and sometimes does not. This is a problem since if the Steam overlay fails it won't load configuration for the Steam controller (which will be left in the default keyboard + mouse emulation mode). This is impacting me in Team Sonic Racing, but I'm certain that I've noticed the Steam overlay failing in other games for quite some time now. I only care about it now, because this is meant as a party game and my 4th controller is a Steam controller and I need that to work reliably - in other games I just use keyboard & mouse or an xbone controller.

I can't see any obvious pattern as to when it succesfully loads or fails, nor is there any obvious difference in the 3DMigoto log files. With 3DMigoto out of the way it works 100% of the time, with 3DMigoto in place it's maybe more like 20%. 3D Vision status does not matter. The first sign that something is wrong is the Steam overlay startup notification not appearing, and the second sign is that the game uses keyboard icons ("SPACE") rather than controller icons.

d3d11_log~steam_overlay_bad.txt d3d11_log~steam_overlay_good.txt nvapi_log~steam_overlay_bad.txt nvapi_log~steam_overlay_good.txt

My gut feeling is that this will turn out fo be some sort of issue hooking the swap chain, since both 3DMigoto and Steam hook that - maybe there's a race and the behaviour depends on which tool gets in first.

Same startup sequence in both good and bad log cases. No secondary SwapChain or Device created. I was looking for a secondary creation like Steam creating anything new for its overlay.

This type of problem is actually the reason I was trying to emphasize the wrapper approach, because if we act as the d3d11.dll wrapper, and happen to be hard-linked to a game, we'll get loaded and called in DLLMain before anything else happens. If we get a force loaded or hard linked load, it will be a consistent launch without race timing loopholes.

Being hard linked is not super common, but that is also the premise behind the dxgi loader, where a game is hard linked to dxgi, which allows it to force load the d3d11 early. There are a whole series of these that are possible, including Dinput8.dll. Will depend upon the game. If you look at Sonic with DependencyWalker you can see what it has hard-linked with, and possibly force an early or at least consistent load.

That's not a bad idea... This is the dependency list for this game:

R:\Steam\steamapps\common\Team Sonic Racing>dumpbin /dependents GameApp_PcDx11_x64Final.exe
Microsoft (R) COFF/PE Dumper Version 14.20.27508.1
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file GameApp_PcDx11_x64Final.exe

File Type: EXECUTABLE IMAGE

  Image has the following dependencies:

    advapi32.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-environment-l1-1-0.dll
    api-ms-win-crt-filesystem-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll
    api-ms-win-crt-utility-l1-1-0.dll
    crypt32.dll
    dwmapi.dll
    gdi32.dll
    kernel32.dll
    msvcp140.dll
    ole32.dll
    oleaut32.dll
    setupapi.dll
    shell32.dll
    shlwapi.dll
    steam_api64.dll
    user32.dll
    vcruntime140.dll
    wininet.dll
    wldap32.dll
    ws2_32.dll

I don't see anything there that we already have a loader for (unless I've forgotten any?), though we could certainly add a new loader for one of those.

It is worth noting that steam_api64.dll is among that list and that Steam therefore does have a chance to load their overlay (GameOverlayRenderer64.dll) before us, though I'm not positive if that is actually used for that purpose, given that Steam can still inject their overlay into other games without that.

Looks like the Steam overlay keeps a log file in the Steam directory.

GameOverlayRenderer~good.log GameOverlayRenderer~bad.log GameOverlayRenderer~no_3Dmigoto.log

There's a number of differences between the logs, so I'm just noting those that seem potentially relevant to the current problem. In the bad log we see this message:

  Wed Jun 05 15:40:32 2019 UTC - Failure waiting for hooking mutex!
  Wed Jun 05 15:40:33 2019 UTC - Failure waiting for hooking mutex!
  Wed Jun 05 15:40:33 2019 UTC - Failure waiting for hooking mutex!

That might suggest that Steam could be using CreateMutex with the same name as Deviare, or indeed could themselves be using Deviare. Despite that the bad log file seems to continue, so it may have retried once more and gained the mutex (and if so that might suggest our hooks on DXGI could have already been installed by this point) or that may indicate that something has failed but it is continuing with other work.

In all three log files we do see it hook DXGI (some small differences between them):

Wed Jun 05 15:44:37 2019 UTC - Game is using dxgi (dx10/dx11), preparing to hook.
...
Wed Jun 05 15:44:41 2019 UTC - hookCreateDXGIFactory1 called
Wed Jun 05 15:44:41 2019 UTC - Hooking vtable for factory
Wed Jun 05 15:44:41 2019 UTC - DXGIFactory2_CreateSwapChain already hooked via IDXGIFactory or IDXGIFactory1
Wed Jun 05 15:44:41 2019 UTC - hookCreateDXGIFactory2 called
Wed Jun 05 15:44:41 2019 UTC - Hooking vtable for factory
Wed Jun 05 15:44:41 2019 UTC - DXGIFactory2_CreateSwapChain already hooked via IDXGIFactory or IDXGIFactory1
Wed Jun 05 15:44:41 2019 UTC - hookCreateDXGIFactory called
Wed Jun 05 15:44:41 2019 UTC - Hooking vtable for factory
Wed Jun 05 15:44:41 2019 UTC - DXGIFactory_CreateSwapChain already hooked via IDXGIFactory1 or IDXGIFactory2
Wed Jun 05 15:44:41 2019 UTC - DXGIFactory2_CreateSwapChain already hooked via IDXGIFactory or IDXGIFactory1

In the no 3DMigoto log file we see Factory1::CreateSwapChain being used:

Wed Jun 05 15:44:41 2019 UTC - IWrapDXGIFactory1::CreateSwapChain called
Wed Jun 05 15:44:41 2019 UTC - Hooking vtable for swap chain
Wed Jun 05 15:44:41 2019 UTC - Trying to detour d3d11 device
Wed Jun 05 15:44:41 2019 UTC - Hooking vtable for device
Wed Jun 05 15:44:41 2019 UTC - Tracking new device: 7442e08
Wed Jun 05 15:44:41 2019 UTC - Tracking new swap chain: 7674020 (with device: 7442e08)
Wed Jun 05 15:44:41 2019 UTC - Creating D3D11 renderer

In the 3DMigoto good case we see a near identical log, except Factory0 instead of Factory1:

Wed Jun 05 15:39:24 2019 UTC - IWrapDXGIFactory::CreateSwapChain called
...

In the 3DMigoto bad case we don't see CreateSwapChain going through Steam at all. This log would seem to suggest that both 3DMigoto and Steam have hooked in, but we somehow inadvertently bypassed it when creating a swap chain. It's worth noting that this game uses the CreateDeviceAndSwapChain call, which we have re-implemented in terms of separate CreateDevice and CreateSwapChain calls, and we use internal unhookable calls for both of those. It would be interesting to see if this issue still occurs if we were to revert the CreateDeviceAndSwapChain to something closer to the older implementation, ~~but I'm not positive that would actually help~~ (Edit: I'm now pretty convinced it would solve this issue).

It's worth noting that I don't see anything in the Steam log file to suggest that they have hooked the CreateDevice[AndSwapChain] calls - it looks like they are only hooking CreateSwapChain and once that is called they take the opportunity to install hooks on the d3d11 device. If that is true, than it would suggest Steam is relying on hooking an internal DirectX call (internal for the CreateDeviceAndSwapChain path, external when CreateDevice and CreateSwapChain are called separately) and our unhookable internal calls might therefore be the culprit.

Earlier I did an experiment commenting out the DXGI hook in our DLLMain, which seemed to work for this game and the Steam overlay loaded every time in about a dozen attempts, but obviously that is not an acceptable general solution (and more testing would be needed to confirm that wasn't a fluke).

I'm now pretty convinced I know why this is happening. The flow would be:

Steam and 3DMigoto both hook DXGI in a random unpredictable order
Game calls CreateDeviceAndSwapChain(), which goes into 3DMigoto
3DMigoto reimplements this call in terms of separate CreateDevice (uninteresting) and CreateSwapChain calls
3DMigoto employs multiple (and in some cases redundant) strategies to avoid games of hot potato that we have previously hit with other third party tools, one of which is to directly call an internal UnhookableCreateSwapChain as opposed to calling the hooked (by both Steam and 3DMigoto) DXGIFactory::CreateSwapChain
UnhookableCreateSwapChain calls straight through to the trampoline / original CreateSwapChain. Depending on whether Steam or 3DMigoto got in first this might then call into Steam's hook, or it might not.

The two possible solutions that I see are:

Remove the reimplementation of CreateDeviceAndSwapChain. DirectX will then internally call CreateSwapChain which will then go through both Steam and 3DMigoto's hooks on that call. 3DMigoto can ignore that call if it needs to via the hooking reentrancy check that it already has in place. It's worth noting that some refactoring I did around these routines a while back should make this change fairly straight forward (and I remember making a topic branch that already did this but we decided that we didn't need to go that far at the time... I'll need to see if I can dig that up and see if it still merges & is still applicable).
Have UnhookableCreateSwapChain call the hookable CreateSwapChain to guarantee that Steam's hook will be called regardless of hooking order, using our reentrancy check to avoid calling back into ourselves. Less code restructure than the above (at least, when compared to what we currently have and not when compared to the old version). I have a feeling that we didn't actually have a proven need for an unhookable version of CreateSwapChain like we did for CreateDevice, but added it for consitency and "just in case", but I'll go back and review the commit messages & related bug reports to make sure. Notably, since we don't wrap dxgi.dll there is no way for an external tool to put a hook on any internal CreateSwapChain function we may have, as it can only put the hook on the original DirectX function. Therefore, if we go down this path we could drop the "unhookable" from the name (since we now kind of do want it to be hooked) and just add a big comment reminding us why we are not using the trampoline.

I have some preference towards option 1, since we have now ran into many hooking issues related to reimplementing functions rather than calling through to the original, but it is arguably a slightly larger change than option 2 from our current code. Option 1 also completely removes the need for the internal unhookable calls.

That seems like a pretty good theory of operation for the problem.

I haven't looked at this in depth for quite some time, but my earlier analysis is that we have a fundamental error in or our architecture that causes all these problems. We are mixing both wrapping and hooking at the same time, which makes for a very complicated runtime environment and leads directly to problems like this where we expect the wrap to happen, but it doesn't because the game loads d3d11 lazily. Which leads to hooks being installed at lazy load times.

The wrapper style is more of an historical anomaly, because that's what Chiri started with, and may not be the best approach. Nearly everyone else like Reshade and SpecialK and Steam using hooking directly. On the other hand, wrapping can in fact be the most reliable, because the startup path is always the same with no chance for race windows.

We should probably have made an explicit dxgi.dll wrapper with some sort of IPC to make the dxgi side match the d3d11 side. I would add this as a third option to your list, where a dxgi.dll wrapper reimplements the functions and thus will be loaded on demand, and hooked by the other tools.

Another even further out possibility would be to create a loader dll that is consistent across the board. The Dinput8 was the closest I saw before, but that is not static-linked with Sonic here. The one that is always consistent is user32.dll, and that would provide an always first loader, at the expense of wrapping all those calls.

A fourth possibility would be to use your d3d11 hooking technique and make it the primary loading system, so that architecturally it's always hooking, not ever wrapping.

Your call. I'm not at all a fan of this heroic looking for reentrancy and checking for others hooks. This all seems like hacks to solve something that should be solved by a consistent architecture instead. Maybe there is no choice because of the complexity of the runtime and bugs/inconsistencies with other tools.

You are in charge, do what you feel is best. I'm just trying to provide some background.

Fix is in master - D3D11CreateDeviceAndSwapChain is reverted to the older proven design (with refactoring so the code is still nicer than it used to be). The unhookable routines have been removed as they are no longer required. Hooking reentrancy detection remains in place and is required (e.g. to ignore DX when it calls our CreateSwapChain hook).

Steam overlay and controller is working reliably in Sonic, regardless of hooking order (further testing on that does show that the "Failure waiting for hooking mutex" message in the GameOverlayRenderer.log is a fairly reliable indication that 3DMigoto hooked in before Steam). Regression tested with SpecialK in NieR Regression tested with ReShade in RE2, DOA6, DOAXVV

Some minor changes to upscaling were necessary to keep things clean - upscaling mode 1 was regression tested in Sonic.

On the architectural standpoint I think the only real issue we had here was that reimplementing D3D11CreateDeviceAndSwapChain meant that other tools assumptions on how DX would be initialised was broken, and that wouldn't have mattered whether we had intercepted that call via hooking or wrapping, and regardless of what loaders we might have used (in fact, I'm pretty sure that the problem occurred when 3DMigoto got in before Steam because that was the scenario that allowed us to inadvertently bypass Steam's hook, so the loaders might have actually been more problematic in this case).

I think we've learnt our lesson multiple times now that if the game calls an API that we have intercepted, we should call through to the original API to keep any other tools that may be present happy, regardless of whether we intercepted the call through hooking or wrapping.

I think there are some other good reasons to consider the merits of other approaches to loading though. One model I've been thinking about implementing is to allow the main 3DMigoto DLL to be named dxgi.dll or dinput8.dll and have it hook any functions it needs, giving us the flexbility to use whichever works for a given game without requiring additional DLLs (I believe both ReShade and SpecialK do this, though I think I saw that there may be issues with conflicting ordinals between the DLLs that we'd need to be aware of). This would also be useful for cases where ReShade must be named d3d11.dll to work around certain bugs when named dxgi.dll (some but not all DOAXVV users hit this), though I would also like to know the details of that issue before committing to this model, in case it turns out to be something that could affect us as well (AFAIK no one knows the details, and I can't reproduce this issue myself).

The multiple DLLs with IPC model would be nice to eliminate some of our hooks in favour of wrapping (and you know my feeling on hooking, so part of me would quite like this), but goes against the goal of reducing the amount of clutter we add to the game directory, and may complicate end users trying to combine 3DMigoto and ReShade (we have the proxy_d3d11 option, but a number of users I've tried to get to use this option have just met me with blank stares).

bo3b / 3Dmigoto

Steam Overlay (+ controller) compatibility issue #112