Refactoring to be done

[bo3b]

I want to start a list of some things we'd like to refactor, that we can't really justify doing piecemeal, because of the impact on too many files. When we refactor some things, we touch so many files, and so many lines of code that we essentially invalidate any prior diff style comparisons by generating too much line-noise. But if we never refactor, things keep getting cruftier and cruftier.

I expect to be able to do a big workover sometime late this year. The idea will be to make a refactoring branch that will be used for all the changes, and when the time is right becomes a new 1.3.x variant. Working in a branch will avoid impacting WIP, and allow us to choose a 'least-bad' time to switch. I won't start working on this until there is a 'lull' in feature improvements, because merging while refactoring is tedious.

I'm just writing these down as a way to remember and keep some notes. Please feel free to note more in the discussion, and I'll add them to this list.

List of desirables

1. Add logging library (most likely g4log) , and refactor all our DebugLog/DebugInfo calls.
   This is interesting because it will remove the constant pain of \n line endings, allow more levels, and give us a lot more flexibility for log output and types.

2. ~~Move out all game fixes from 3Dmigoto to a new repo. There are a lot of game fix folders in the project, from when the idea was that we would ship game fixes directly out of 3Dmigoto. Even though they are still working fixes, these make no real sense as part of the 3Dmigoto project, and it would dramatically clean up the cruft.~~

3. Unify variable formats especially in globals.h, but also elsewhere.
   This is worth doing because our current usage is a random mish-mash of ALL_CAPS, and CamelCase, and under_bar, and gInitial_Var styles. It would make sense to decide upon a strict format for future use.

4. Complete the move of FrameAnalysis to become a subclass of HackerContext. The reason this is worth considering is because performance analysis always shows FrameAnalysisLog as using measurable CPU on the level of 0.2% to 0.3% of CPU. It would be nice to not pay that price unless FrameAnalysis is actually enabled in d3dx.ini, and create a specific FrameAnalysisContext object.

5. Change any 'orig', or 'real', or 'new', or 'old' naming to something more clear and specific. These are interesting because it's high bang-for-the-buck to clarify variable names.

6. nullptr checks on C++ object creation Exception policy? As noted, these are useless and actually misleading, because any C++ instantiation that fails will throw an exception. As part of this, it would be worth considering or codifying our exception policy.

7. Since we are C++, just use std::string. We've got a lot of tweaky variants, passing around pointers, and wide strings, and wchar and char and probably every variant possible. It would be a nice-to-have to just use std::string everywhere, and do last second conversions as needed.

8. Switch our pointer use to ComPtr. This is another nice-to-have. Would be moderately risky in terms of introducing bugs, but would give us a dramatic bump in readability and ensure we are not leaking memory.

9. Update both Nektra deviare library to latest version. Update to latest version of DirectXTK. It might make sense to use git submodules for these.

[DarkStarSword]

I think you are letting your enthusiasm for recognizing that IDXGIDevice returns the same pointer as ID3D11Device overrule your normal tendencies. You've spoken many times about not counting on implementation details. And this most certainly is an implementation detail.

Not really - they would be violating COM if they changed that detail, but that is why I went back and verified that this still held true on Windows 7 without the evil update, just in case. The claims that QueryInterface was some kind of back door has always bugged me and seemed wrong, but once you read up on COM and realise that they are the same object it makes perfect sense and it is exactly how COM is supposed to work.

The documentation never, ever says the two objects are the same, that the pointers might be the same, or that they can be treated the same.

Yes, it does say it - they don't spell it out, but that is what this is saying:

You can query this Direct3D device object for the device's corresponding IDXGIDevice interface. To retrieve the IDXGIDevice interface of a Direct3D device, use the following code:

That is saying right there that the Direct3D device supports the IDXGIDevice interface, and the fact that QueryInterface works is just more proof of this - that's how COM interfaces work. It's not saying that the ID3D11Device interface supports the IDXGIDevice interface - those are separate interfaces, but it is saying that the Direct3D device supports both interfaces.

You skipped the documentation I added to that QueryInterface routine above: For any one object, a specific query for the IUnknown interface on any of the object's interfaces must always return the same pointer value. This enables a client to determine whether two pointers point to the same component by calling QueryInterface with IID_IUnknown and comparing the results. It is specifically not the case that queries for interfaces other than IUnknown (even the same interface through the same pointer) must return the same pointer value.

That's confirming exactly what I've been saying. If ID3D11Device::QueryInterface(IID_IUnknown) and IDXGIDevice::QueryInterface(IID_IUnknown) return the same pointer, then they are two interfaces of the same object/component. Other interfaces (like the ID3D11Device and IDXGIDevice) may have different pointers, and in fact you could (theoretically, but I don't believe they used tear-off interfaces in this case) have multiple ID3D11Devices all with different pointers, but that all correspond to the same device and using QueryInterface(IID_IUnknown) on any of them will return the same pointer.

If a caller asks specifically for QueryInterface(IDXGIDevice) then that documentation right there says that they cannot rely upon it being the same pointer. By the strict reading, it is not legal to pass this to CreateSwapChain.

Don't forget https://msdn.microsoft.com/en-us/library/windows/desktop/ms682521(v=vs.85).aspx:

There are four requirements for implementations of QueryInterface (In these cases, "must succeed" means "must succeed barring catastrophic failure."):

    The set of interfaces accessible on an object through QueryInterface must be static, not dynamic. This means that if a call to QueryInterface for a pointer to a specified interface succeeds the first time, it must succeed again, and if it fails the first time, it must fail on all subsequent queries.

    It must be reflexive — if a client holds a pointer to an interface on an object, and queries for that interface, the call must succeed.

    It must be symmetric — if a client holding a pointer to one interface queries successfully for another, a query through the obtained pointer for the first interface must succeed.

    It must be transitive — if a client holding a pointer to one interface queries successfully for a second, and through that pointer queries successfully for a third interface, a query for the first interface through the pointer for the third interface must succeed.

It says right there that it must be symmetric. An ID3D11Device is an IDXGIDevice is an ID3D11Device. And since "For any one object, a specific query for the IUnknown interface on any of the object's interfaces must always return the same pointer value" then they must be the same pointer if you go back to IUnknown, regardless of which interface you start with.

For a Direct3D Device, it most definitely is not an IDXGIDevice as defined in the documentation:

https://msdn.microsoft.com/en-us/library/windows/desktop/bb174527(v=vs.85).aspx

The IDXGIDevice interface is designed for use by DXGI objects that need access to other DXGI objects. This interface is useful to applications that do not use Direct3D to communicate with DXGI.

The Direct3D create device functions return a Direct3D device object. This Direct3D device object implements the IUnknown interface. You can query this Direct3D device object for the device's corresponding IDXGIDevice interface. To retrieve the IDXGIDevice interface of a Direct3D device, use the following code:

No, that's just confirming that they are interfaces of the same device - it's just saying that the IDXGIDevice interface can be used by applications that don't want to know about specific DX versions, but that doesn't mean that it's a separate object, just a separate interface.

If these are in fact the same object, and we can just slop around and use one when we mean to use the other, like in the call to CreateSwapChain... Then surely it also OK to simply call Release on whichever one we happen to be holding. i.e. It doesn't matter if it's IDXGIDevice->Release, or ID3D11Device->Release.

NO DEFINITELY NOT (although in some cases technically yes, but don't rely on that)!

Read up on COM tear-off interfaces to get a feel for this. The refcounting MIGHT be shared between all interfaces of the same object, or it MIGHT be per-interface. Knowing that IDXGIDevice and ID3D11Device share the same underlying object is not an implementation detail because QueryInterface works between them, but THIS IS AN IMPLEMENTATION DETAIL.

It's the same object, right? Are you willing to bet the ranch on that behavior?

Sure, if I had the money to spare - as long as you are clear that same object != same interface, because I'm not really sure you are quite getting that distinction.

https://msdn.microsoft.com/en-us/library/windows/desktop/ms692481(v=vs.85).aspx

From a COM client's perspective, reference counting is always done for each interface. Clients should never assume that an object uses the same counter for all interfaces.

Yep, exactly as I said above.

By them passing in the wrong object to CreateSwapChain, the ref counts are not correct.

No, no problem there either - CreateSwapChain has to QueryInterface (it was passed an IUnknown remember), so will bump and release the refcount internally. It's up to the caller to manage the refcount of whichever interface they passed in, so everything balances regardless of which interface they elected to use.

You and I know that it works, but that is not what the documentation says. Whether we want to rely upon undocumented behavior is a different question.

:)

However, I think your approach for using IUnknown for the pointer is correct, and does not break any rules I've read. Any call to IUnknown must always give the same result. Using the ID3D11Device pointer would possibly work, but would not match the documentation.

Yeah - I think this is the best strategy for now, unless we later find something that utterly violates this. I'm more concerned about the fact that we return a real IDXGIDevice and therefore violate the symmetric requirement of COM, i.e. a game could use that to get back to the real ID3D11Device and bypass us. Hopefully rare in practice, and hook=recommended will probably work around it if that ever does crop up.

No, this cannot happen. There is no legal way for them to fetch a IDXGIDevice without using an existing ID3D11Device. There is no documentation showing any way to create them directly.

That's what I'm referring to - using an ID3D11Device to get an IDXGIDevice as we see UE4 doing here, but then using that IDXGIDevice to get an unwrapped ID3D11Device. It would be really stupid if a game did that... but not impossible.

Someone might have hacked something up that we'll have to deal with, like some Factory->QueryInterface or something, but there are no extant examples in my searches.

Oh god what a horrible thought 😱

I wonder if we should start logging the IUnknowns of everything we see to look for any related interfaces we haven't thought of? Have there been any links between say an IDXGIAdatper and an IDXGIDevice? Based on our old code I guess we must have seen evidence that IDXGIFactory and IDXGIAdapter are two interfaces to the same object at some point - I hope that relationship doesn't extend to an IDXGIDevice.

One major point of doing this refactoring was simplification. If we are just going to junk it all up again with all the myriad ways it might fail, then there was no point. I really, really do not want to add more complexity.

Absolutely 100% agreed - the DXGI wrapping was a nightmare and if we can do without that I'm happy. I'm hoping that this solution with looking up the IUnknown pointers will suffice and be simpler and superior than our previous solutions, and if we do find a game that can abuse QueryInterface to escape our wrapper that hooking will save us.

[bo3b]

OK, thanks for taking the time to detail that out.

I don't think I'm smart enough to program on WinAPI.

They don't ever show me an architectural overview, and all their documentation simply says 'what', never 'why'. Like bad code documentation.

[DarkStarSword]

Yeah, I was thinking about this some more, and I figure part of the problem with the DirectX documentation is that it's not trying to teach COM, and some of this needs an understanding of COM to really "get it". The DirectX documentation shows how to use COM to achieve certain tasks, and that's probably enough for most developers. We're in a bit of a special case because we're essentially implementing COM objects ourselves to wrap the DirectX interfaces, so it pays to understand the underlying technology a little.

I've been looking at the DirectX 12 documentation lately... it might be a step backwards from the DX11 documentation - particularly all their examples are copy & paste jobs from the DirectX sample code, so they have a lot of irrelevant line noise and fail to highlight important things.

[DarkStarSword]

Looks like the CreateDeviceAndSwapChain() restructure has caused an interoperability problem with the Special K Modding System in the Nier Automata fix - every time we try to call CreateDevice() we get called back as CreateDeviceAndSwapChain() - refer to the attached log. Removing the Special K dxgi.dll shipped with the fix gets us into the game.

I think it's a good bet that they tried to do a very similar thing to us and simplify the init paths by re-implementing CreateDevice() to call CreateDeviceAndSwapChain(), but since we re-implemented CreateDeviceAndSwapChain() to call CreateDevice() we end up in a game of hot potato until the music stops and we run out of stack.

Edit: Yeah, here's the code responsible in Special K - no indication of why they did this: https://gitlab.com/Kaldaien/SpecialK/blob/0.9.x/src/render/d3d11/d3d11.cpp#L1267

Edit: Tracing it back it appears to have come from this commit: https://gitlab.com/Kaldaien/SpecialK/commit/914e5240e21aac654791f91d1ad1c42efd3c1daf No particular explanation there beyond the self-explanatory init path simplification. There is something there about working around an issue with infinite recursion with Reshade, but it is not clear if that is related.

Given the new way we lookup the HackerDevice from CreateSwapChain(), we can probably go back to our old implementation of CreateDeviceAndSwapChain() now to solve this. Edit: No, that's not quite right, since we did this restructure to get around the problem that CreateSwapChain() was being called with a device we hadn't yet seen and hadn't wrapped. We might be able to wrap the new device in CreateSwapChain() and retrieve it from CreateDeviceAndSwapChain() instead.

d3d11_log.txt

[bo3b]

Huh, weird how these things all come in groups. I'm working on this exact problem right now, but for ENB. One of our users has been trying to get ENB to work with SkyRimSE, and found the problem. Boris posted his code which lead me to our bug.

Exact same symptoms, but my current analysis doesn't show it to have anything to do with our new broken apart CreateDeviceAndSwapChain. For ENB it happens because our proxy code is completely broken.

I don't know or understand how SpecialK loads, but presumably either they or we use proxy loading.

I'll be pushing up a fix for the proxy loading problem, which will also fix Issue #49 for Reshade.

I'm going to add this to master, and to exp_single_layer. My initial fix should have no impact on master.

But for exp_single_layer I plan to rework our skip loader hook on LoadLibrary, because we have two mechanisms that do the same thing, and I want a more universal approach for ENB.

[DarkStarSword]

I don't know or understand how SpecialK loads, but presumably either they or we use proxy loading.

I'm not an expert on their code, but from what I can tell with a quick poke around they are using a sort of uber dll (no idea what the real name of this is) - they can be named d3d11.dll, dxgi.dll, d3d12.dll, d3d9.dll, ddraw, opengl, dinput, etc - whatever the game will try to load. They have exported functions they want to intercept for all of these, and they hook into the exported functions of all the other DLLs they weren't named as using MinHook. In this case they are named dxgi.dll and hook into the d3d11.dll exports.

I don't think either is using proxy loading in this case.

[bo3b]

Interesting idea to make an uber dll. Gets around the early loading problems, and force loaders like our fake dxgi.dll.

Pretty sure my change here will fix this as well. If they are doing MinHook for everything, and just happen to load after our d3d11, then it will be similar to a proxy load operation and have the same problem- which is that our hook on LoadLibraryExW will force everything back to the game folder. Edit: They'll GetProcAddress on whatever they get from LoadLibrary, which will be us.

That also suggests they could bypass the problem by using some variant that loads before we do, like dinput. If they load first, I expect it to work.

I'm adding yet another flag in the d3dx.ini, to allow any game or setup to escape our LoadLibrary hook. In master this will be off by default, so the behavior is the same as history. In exp_single_layer it will be on by default, because we are force loading in all scenarios, and mostly we don't have to. It's only some small handful of games we need to work around.

This will solve the problem for ENB for sure (tested), and I'm 95% sure it will fix the problem for Reshade. More soon.

[DarkStarSword]

Do you think we are about ready to make the master branch 1.3, or would you rather wait a little longer first? I've cherry-picked a few commits I wouldn't mind seeing in a 1.2.73 release (but equally don't mind if we skip that), but it already feels like it's in maintenance mode at the moment, so perhaps it is time to swap our branches around to reflect that?

[bo3b]

Do you think we are about ready to make the master branch 1.3, or would you rather wait a little longer first? I've cherry-picked a few commits I wouldn't mind seeing in a 1.2.73 release (but equally don't mind if we skip that), but it already feels like it's in maintenance mode at the moment, so perhaps it is time to swap our branches around to reflect that?

I'll leave this one to your judgment. Your opinion as to the state of 1.2.x branch is the determining factor. I do not have an opinion either way, as I've only focused on 1.3 recently. I do think that 1.3 is probably out of the 'experimental' stage, it seems fairly solid.

Edit: Small preference from me for a 1.2.73 release to pick up your changes for the .bin handling. That seems worthy for the ending archive, and I prefer to have top-of-tree in an archive branch be actually shipped.

Please feel free to swap these whenever you think it is right.

[DarkStarSword]

Edit: Small preference from me for a 1.2.73 release to pick up your changes for the .bin handling. That seems worthy for the ending archive, and I prefer to have top-of-tree in an archive branch be actually shipped.

Agreed. I'll do some quick testing on everything that has been backported to it and do a final 1.2.x release shortly then switch the branches around :)

[DarkStarSword]

All done - 1.2.73 is out and I've updated all the branches: archive_1.2 points to the final 1.2.73 release, in case we need to do any further backports in the future. archive_1.2_vs2015 is the final vs2015 branch for 1.2 exp_single_layer is now merged into master, and exp_single_layer has been deleted exp_vs2015 is now vs2015, and exp_vs2015 has been deleted

[DarkStarSword]

I'm knee deep getting 3DMigoto working with Windows Store apps (successfully injected via privileged system wide loader and logging to the application's temp directory) and in ROTTR I'm seeing our call to the original D3D11CreateDevice being redirected to our D3D11CreateDeviceAndSwapChain for some reason. I'm not positive why - I am hooking both calls so it is possible I might have just messed something up, but it's looking like I might need to bring back the work I was doing for SpecialK after all... I might see if I can simplify it though.

Oh BTW, we can actually log from DllMain - filesystem access is OK (MSDN confirms this). I'm not sure we could run the entire ini parser - that spawns way too much, but we could probably parse enough to find out if logging is enabled if we wanted earlier logging. I probably can't do this for Windows Store apps though - looking up the app's temp directory deadlocks in DllMain (I've hardcoded the path for debugging purposes).

[DarkStarSword]

You know, I think this must be how DirectX works internally - D3D11CreateDevice just calls D3D11CreateDeviceAndSwapChain with NULL for ppSwapChain, and since I'm hooking them in the Windows Store we get called back into our DLL. Probably why SpecialK had to do it the way they did.

Gah, I hate hooks - so damn messy. I wonder if I should just patch the game's TOC instead.

[bo3b]

I'm knee deep getting 3DMigoto working with Windows Store apps (successfully injected via privileged system wide loader and logging to the application's temp directory) and in ROTTR I'm seeing our call to the original D3D11CreateDevice being redirected to our D3D11CreateDeviceAndSwapChain for some reason. I'm not positive why - I am hooking both calls so it is possible I might have just messed something up, but it's looking like I might need to bring back the work I was doing for SpecialK after all... I might see if I can simplify it though.

Interesting case. Please do any changes of this form in an experimental branch, if you haven't already. I have a very strong preference to not complicating our startup sequence again. One of the big problems here is that we are mixing both hooking and wrapping at the same time, and that makes it much harder than it needs to be. If we absolutely need hooking, I think we should consider making separate builds or branches for the two different loading styles. Adding in a third layer for injecting into protected apps is one step too far, in my opinion.

Since I'm just stating my opinion here, I'd also like to discourage people from doing anything with the UWP and Windows Store. I think we should be a lagging support, not early support. If Microsoft wins this fight for the gaming world, then they will continue to lock it down until it's PC as console, and I strongly think we should discourage that. It is their stated goal to become that platform, and that would be bad for all modding. They will never give up the arms race of whacking the modders.

Then there is the DX12 support that I also think is a waste of time. Until we see some traction on that API, like in the console realm, it's my opinion that DX12 is going the way of DX10. No one will care. It's overly complicated and does not bring any performance value for Nvidia cards. I can't see it making any headway over DX11.

Having noted all that- it doesn't hurt to experiment and support the tiny handful of games that are in UWP or DX12 state. As long as it is done in experimental branches like the DX10 support, there is no harm to our core experience.

Oh BTW, we can actually log from DllMain - filesystem access is OK (MSDN confirms this). I'm not sure we could run the entire ini parser - that spawns way too much, but we could probably parse enough to find out if logging is enabled if we wanted earlier logging. I probably can't do this for Windows Store apps though - looking up the app's temp directory deadlocks in DllMain (I've hardcoded the path for debugging purposes).

This is only true because they already connect filesystem access in order to load DLLs. The real problem is avoiding changing the loading sequence of the DLLs, and any libraries that are auto-connected can change the loading sequence and lead to hangs.

While it is indeed possible, it's a bad idea, because it's easy to accidentally drop in stuff like c runtime string parsers, or other support DLLs that are sort of invisible. We already did this previously in fact. We would call out to the DXGIFactory to install hooks, and that would specifically call loadlibrary on d3d11.dll, which is strictly forbidden. Hard to see because of the sequence of events. It's possible to work out the details, but really, isn't it better to be clean and strict?

Again though, if you really need this, doing it in an experimental branch is never a problem. Still, it might be worth writing a sample UWP graphics app instead of trying to load a game with all their madness. That would separate the loading/security problems from game weirdness.

[bo3b]

You know, I think this must be how DirectX works internally - D3D11CreateDevice just calls D3D11CreateDeviceAndSwapChain with NULL for ppSwapChain, and since I'm hooking them in the Windows Store we get called back into our DLL. Probably why SpecialK had to do it the way they did.

That's a good thought. Entirely possible.

We don't currently have any examples of this happening, so I did not flesh it out, but it might be worth adding a HackerCreateDeviceAndSwapChain stub to avoid the hooks. Unless I'm missing something, that would make all the primary creation calls hook-safe.

Gah, I hate hooks - so damn messy. I wonder if I should just patch the game's TOC instead.

Just something notable- Detours library actually has this functionality built in, as one of their installation approaches. They have a 'sticky' installation technique that modifies the file.

It's really just a shame that Microsoft blew it with that $10K price tag for x64 support. If they'd done something sensible, everyone could have used an industry standard approach instead of all these ad-hoc tools. They can't possibly have made enough money off that to justify it. More likely to be their freakish desire to control everything though.

[DarkStarSword]

We don't currently have any examples of this happening, so I did not flesh it out, but it might be worth adding a HackerCreateDeviceAndSwapChain stub to avoid the hooks. Unless I'm missing something, that would make all the primary creation calls hook-safe.

No, not in this case - it works for the SpecialK case because it avoids us calling a function that SpecialK has hooked, but won't work here because it's DirectX calling a function that we have hooked - the only way we can avoid that is to not hook the function, or re-implement D3D11CreateDevice to call D3D11CreateDeviceAndSwapChain so that DirectX doesn't need to call it, but I'd rather not rely on knowing that kind of implementation detail. It seems the best course of action is to try to avoid re-implementing any of these functions in terms of others, and always call through to the original.

Well, there is one other option I can think of - set a thread local flag before calling into DirectX and clear it afterwards, then check it on any of our hooks and call straight through to the original if it is set.

[DarkStarSword]

Interesting case. Please do any changes of this form in an experimental branch, if you haven't already. I have a very strong preference to not complicating our startup sequence again. One of the big problems here is that we are mixing both hooking and wrapping at the same time, and that makes it much harder than it needs to be. If we absolutely need hooking, I think we should consider making separate builds or branches for the two different loading styles. Adding in a third layer for injecting into protected apps is one step too far, in my opinion.

A lot of this is code we are going to need sooner or later anyway, and I don't want to be maintaining a separate branch that diverges too far from master, so I will be backporting most of this in to keep the branches as close as possible. I'm not sure if I'm going to bother with official support in the vs2012 branch though - Windows Store encourages more recent APIs so if we aren't building against a more recent Windows SDK this might not be worth doing, so I might keep some of the more experimental work in a separate branch based off that. Do you think it's worth making an experimental 1.4.x series for the vs2015+ port? Until now it hasn't had any significant advantage over the vs2012 build for an end user, so there has been no reason to release it, but with this and the partial DX12 support that is now changing, and it is maybe a little too big of a change to call 1.3.5... But then we only just released 1.3.x, and it feels a little too early to start 1.4.x, but that's more just a feeling rather than a good reason.

The way to hook in to UWP is actually dead simple - it's little more than a one liner function in 3DMigoto and an external tool with three significant lines of code (polish will add more to that), and this will work to hook us into any game where d3d11.dll is normally loaded too late, not just UWP (actually, this is the exact same method that the Helix Mod loader uses, and their loader will even work for our 32bit DLL) - in most games we don't even need to hook d3d11.dll functions so long as we are still named d3d11.dll, but at least in ROTTR that doesn't seem to work and it's a little more predictable if we are named something else to disambiguate GetModuleHandle(), so I'm going with that.

The hard part specific to UWP was figuring out the commands to generate a code signing certificate and actually signing the external injector so it has permission to inject into UWP apps, and I still have to work out how to programatically install said certificate on an end users system (if nothing else they can use certmgr).

[bo3b]

No, not in this case - it works for the SpecialK case because it avoids us calling a function that SpecialK has hooked, but won't work here because it's DirectX calling a function that we have hooked - the only way we can avoid that is to not hook the function, or re-implement D3D11CreateDevice to call D3D11CreateDeviceAndSwapChain so that DirectX doesn't need to call it, but I'd rather not rely on knowing that kind of implementation detail. It seems the best course of action is to try to avoid re-implementing any of these functions in terms of others, and always call through to the original.

Why does this only happen in this unusual UWP case? Why don't we see this prior to that? Change in their implementation perhaps?

I really think the current implementation is a good simplification that works in 99% of our cases, and I really don't think your specialk_rework approach is the right choice.

However, if we think these games are significant and really need the added complexity, I'd strongly suggest that we need to think through the architecture, not just graft on a quick fix. As an example, the hooking code for MGSV works, but completely breaks the architecture by grafting hooks into a fundamentally wrapping architecture. I'm suggesting we'd be better served by decoupling these loading ideas somehow, and make specific versions or branches that load different ways rather than trying to mash all the variants into one.

Maybe two key variants, one a simple wrapping for the 90% case of easy games, and an injector based hooking variant for the problem children. Or, possibly, if injector+hooking is a superset of all games, we abandon the wrapping side in favor of a hook-only architecture.

Another thought here is that Deviare already supports the idea of temporarily disabled hooks. We toss the hookId on the trash, but if we store that, we can just call EnableHook(false) when active.

Edit: It's possible we should have been doing this all along. If we were to call EnableHook(false) at every hook entry, we wouldn't need the Hacker* decoupling from hooks. I'll dummy this up as an experiment. Again though, mixed metaphors of hooking and wrapping being intertwined.

[bo3b]

A lot of this is code we are going to need sooner or later anyway, and I don't want to be maintaining a separate branch that diverges too far from master, so I will be backporting most of this in to keep the branches as close as possible.

Are we? We'd like a fix for GearsOfWar4, but is there anything else? If we successfully make a one-off fix using an alternate branch, and UWP goes nowhere, then is there any harm if it's stale? Conversely, if we add complexity and the long term maintenance cost, and it winds up useful for only a single game, is that the right approach?

[DarkStarSword]

Why does this only happen in this unusual UWP case? Why don't we see this prior to that? Change in their implementation perhaps?

For UWP I'm injecting 3DMigoto via in-context hooking ( https://msdn.microsoft.com/en-us/library/windows/desktop/ms644990(v=vs.85).aspx ), and it seems in that case I can't be certain that our dll will take precedence over the system d3d11.dll - in some cases (Unity) it does if it's named d3d11.dll, but that doesn't seem to be the case for ROTTR in UWP. I sincerely doubt this is specific to UWP - I'm not certain what factors will affect this (I though maybe link time vs LoadLibrary, but neither Unity nor ROTTR lists d3d11.dll as a link time dependency), but it is pretty clear from MSDN's documentation on GetModuleHandle() that it is undefined when there are multiple modules with the same name, so better not to make guesses or rely on it.

So, to do something that is predictable and well defined I'm naming our DLL 3dmigoto.dll instead, and hooking the functions we care about in d3d11.dll - and it is that hook that is causing DirectX to be able to call back into our DLL (and why I was considering patching the TOC instead, as that would give us the same result as being the highest priority d3d11.dll loaded in the process but without the extra hooks). Again, this is not specific to UWP - that is just the case where I'm currently hitting this. We can use this same approach to inject into any game where d3d11.dll is loaded too late (as an alternative to our d3dcompiler injector), or where we can't modify the game directory, or if we want to switch to an uber DLL architecture in the future and fold our nvapi wrapper into our main DLL (which I actually do want), or allow us to be named dxgi.dll, d3d12.dll or d3d10.dll or something else.

I really think the current implementation is a good simplification that works in 99% of our cases, and I really don't think your specialk_rework approach is the right choice.

There's two approaches I can do here - either alone would work, or both combined. At the moment I've got option 1 (sans CreateSwapChain refactoring), but after some thought, I'm probably going to go with option 2 alone as the simpler solution, however the refactoring in option 1 really should happen at some point anyway for code readability and maintainability reasons regardless of this, so I will at least split out some of the common code, even if it is only called from one function without the full option 1.

1. Bring in the part of the specialk_rework that restores D3D11CreateDeviceAndSwapChain and refactors out the common code between it and D3D11CreateDevice, but not the wrapping on demand part (this is what I was saying about simplifying it) - instead, allow CreateSwapChain to cope with the case where it can't find a HackerDevice, delaying wrapping the swap chain until we are back in D3D11CreateDeviceAndSwapChain, where a common function refactored out of CreateSwapChain will wrap the swap chain. Add detection in the common wrapping routines to prevent re-wrapping an already wrapped object (unnecessary if option 2 is used in conjunction with this).

2. Detect when one of our functions is called from DirectX or another modding tool as a result of us calling a DirectX function, and skip all our processing and call straight through to the original function instead.

Another thought here is that Deviare already supports the idea of temporarily disabled hooks. We toss the hookId on the trash, but if we store that, we can just call EnableHook(false) when active.

This is essentially one way to do option 2, however it is not thread safe since while the hook is disabled any other thread that makes the disabled call will not call into 3DMigoto. This isn't theoretical either - while tracking down the Deviare bug I noticed that The Witcher 3 does exactly this where several D3D11CreateDevice calls are in flight simultaneously from different threads.

I'm thinking thread local storage is a better option to achieve this - set a TLS flag before calling into known problematic DirectX functions and check the flag when a we get called from a known problematic DirectX hook. Ideally we'd add this to every single DirectX function call, since any of them may be hooked by an external tool, and any of them could be internally implemented in terms of calling another hooked DirectX function, but there's a lot of them and only a very small number are known to actually be problematic in practice, so I'll restrict it to those and only add new ones as we come across them in practice.

However, if we think these games are significant and really need the added complexity, I'd strongly suggest that we need to think through the architecture, not just graft on a quick fix. As an example, the hooking code for MGSV works, but completely breaks the architecture by grafting hooks into a fundamentally wrapping architecture. I'm suggesting we'd be better served by decoupling these loading ideas somehow, and make specific versions or branches that load different ways rather than trying to mash all the variants into one.

The hooking problems would all still exist even if we did away with wrapping entirely, and we'd probably actually have more - DirectX and SpecialK could still call back into us when we didn't expect it. Maybe we'd get away with since we only install the hooks on the device once, but that's a fluke and doesn't change the fact that the problem still exists, so either way we have to deal with them - if we don't explicitly deal with it, we're just saying "it's a fluke, but it works so whatever". If we get rid of hooking entirely (which we unfortunately can't do) we instead have extra boilerplate and more complicated QueryInterface routines.

If we maintain separate branches for both approaches we solve neither problem and introduce new ones as conflicts arise between the two - better to have the differences visible and as obvious as possible in the code IMO and deal with occasional problems as they arise.

I want to point out something here - the UWP case has actually very little to do with hooking - that is one small complexity that arose, but is not specific to UWP at all. The complexities I've hit are:

• Injecting our DLL - not specific to UWP - we've done this since the dawn of time with one method or another. Using an external loader to do so is primarily intended for UWP, but not exclusively so.

• Bailing if we are injected into something we didn't want to be - a consequence of using the loader, so like the loader this is not specific to UWP, but is the primary intended case

• Hooking d3d11.dll functions if we are not named d3d11.dll - not specific to UWP, but is the primary intended use case for now

• Fixing bugs where we assume GetModuleHandle("d3d11.dll") will return a handle to ourselves - not specific to UWP - these are bugs relying on undefined behaviour since there are multiple modules of that name in the normal case

• Accessing our files from somewhere the container and injector has access to instead of the executable's directory - specific to UWP, but the abstraction is well worth adding anyway

• Replacing GetAsyncKeyState with something that works in the container - the requirement is specific to UWP, but the replacement code (whatever that ends up being) does not necessarily have to be specific to UWP

• Implementing CreateDXGIFactory2 - not specific to UWP as any program could call this on Windows 8+, so this is just fleshing out 3DMigoto's coverage

• Implementing CreateSwapChainForCoreWindow - specific to UWP as far as I'm aware, though might be used outside of it in some cases?

And that's all - this was really easy to get working, only one day of hacking from beginning to investigate DLL injection options that will work with UWP (all documented on MSDN - nothing particularly fancy here) to a working proof of concept in ROTTR (sans keyboard input), and most of that time was spent trying to work out how to generate a certificate and sign the injector.

Are we? We'd like a fix for GearsOfWar4, but is there anything else? If we successfully make a one-off fix using an alternate branch, and UWP goes nowhere, then is there any harm if it's stale? Conversely, if we add complexity and the long term maintenance cost, and it winds up useful for only a single game, is that the right approach?

Like I said - very little of this is actually specific to UWP, so we do need most of this regardless and what is actually specific to UWP is relatively small. This isn't some new architecture or fundamentally new approach - it is just extending our existing support, fixing bugs and missing functions and dealing with a few small quirks.

The UWP documentation makes it sound worse than it actually is - in reality it is just a container with some funky permissions, nothing more.

[bo3b]

The hooking problems would all still exist even if we did away with wrapping entirely, and we'd probably actually have more - DirectX and SpecialK could still call back into us when we didn't expect it. Maybe we'd get away with since we only install the hooks on the device once, but that's a fluke and doesn't change the fact that the problem still exists, so either way we have to deal with them - if we don't explicitly deal with it, we're just saying "it's a fluke, but it works so whatever". If we get rid of hooking entirely (which we unfortunately can't do) we instead have extra boilerplate and more complicated QueryInterface routines.

There are no hooking problems. Unless I'm confused, I already solved all the problems with SpecialK, ENB, and Reshade hooking back into us. They can now all be used on any game. Steam overlay now always works with no changes needed. And as far as I know 1.3 is working correctly in every game tested, every scenario. Am I missing something?

The only scenario not working is this new freakish UWP setup. That's OK, but again, this makes no sense to complicate everything for a single game. We don't need hooking of CreateDevice for anything except UWP, and there is only a single game there. And my expectation is that there will only ever be one game there. I do not think it is in our best interest to make game specific changes of this form.

Is it your goal that we move to a hooking-only architecture?

[DarkStarSword]

There are no hooking problems. Unless I'm confused, I already solved all the problems with SpecialK, ENB, and Reshade hooking back into us. They can now all be used on any game. Steam overlay now always works with no changes needed. And as far as I know 1.3 is working correctly in every game tested, every scenario. Am I missing something?

No, you're not missing anything - those are valid and necessary workarounds to some of the hooking problems I'm referring to. The other workaround we have is avoiding allowing DirectX to call CreateSwapChain by not ever calling D3D11CreateDeviceAndSwapChain - that's not really one we solved, that's one we avoided, and this is just another case of that same problem.

But that's the point - these problems only exist because hooking allowed the ordering between SpecialK, 3DMigoto, Steam and DirectX to be violated and these workarounds are necessary as a result - this is a fundamental problem with hooking, and the fact we even have workarounds in our code is evidence of that - and I can pretty much guarantee that we will need more workarounds in the future for cases we haven't yet discovered or don't exist yet.

Unless we extend these avoidance strategies to cover all external code we could ever call and check every entry point for layering violations, and avoid directly calling any function inside 3DMigoto that could possibly have been hooked by a third party we still have the potential for more variants of these same problems to surface down the track - adding that sort of protection is not viable and in practice it is only necessary on a very small number of functions and wasteful to do it elsewhere, but in theory that's the extreme we would have to go to to make hooking safe.

If we don't, we will just have to deal with specific cases as we come across them - and that's fine, but it's worth having strategies ready to deal with them that we can employ as we need to that don't require much effort - I don't want another case of re-implementing a function like ID3D11CreateDeviceAndSwapChain just to avoid one of these (and as we saw, doing so introduced new hooking related problems), I want a general strategy that we can employ that targets the actual problem.

Is it your goal that we move to a hooking-only architecture?

Oh hell no - if I could choose a pure architecture I'd throw out all the hooks in a heartbeat and make every other tool do the same, but realistically doing so would have too many drawbacks to be viable - I consider them a reality we can't avoid and a necessary evil to solve certain problems and simplify our code, but that doesn't mean I have to like them.

[DarkStarSword]

Is it your goal that we move to a hooking-only architecture?

I should maybe clarify what my goal here is - it is academic, and has been ever since ROTTR was released on the Windows Store and I've been wondering what it would take to make 3DMigoto work in that context in a clean way that didn't break the container architecture. At the time, my poor internet connection meant I couldn't find out, because the game kept failing to download and the store lacks support for resuming failed downloads. Then Nixxes released their own 3D Vision support and I still couldn't download the game and it became largely irrelevant, but no less interesting a challenge, just one for another day. I don't have any specific Windows Store game in mind to fix, but I know other people do. With Flugan's return and his mentioning that he was replacing the system32 DLL my interest resumed because I don't believe that is the way to go, and my process to get this working was essentially:

• So, how does the Helix Mod loader work? Oh, ok they are using this API. Ok, let me try that in 3DMigoto - three lines of code and HOLY SHIT THAT WORKED!?! Injected into Subnautica without being in the game directory and everything is just working? Oh wow, seriously Microsoft? That's all it takes to inject a DLL into every process in the system? Jesus, no wonder the hackers have it easy on this platform... Ok, if you say so...
• Let's try a Windows Store app... Nope, doesn't work. Ok, so they have some kind of protection, which I expected...
• Huh, this MSDN article tells me exactly what I need to do to make this work in Windows Store apps. Well, now I know what it will take I clearly have to try it...
• How the hell do I generate a code signing certificate? (hours later)
• Ok, I have a certificate - how the hell do I sign an executable? (hours later)
• Ok, now let's try windows store HOLY SHIT THAT WORKED - well, it's loaded anyway, just not working.
• Well, I've come this far, what do we need to make this functional?
• Filesystem probably. Ok, log to a temporary directory instead - woop, now I have logging :)
• Working through all the issues I mentioned previously, most of which are not windows store specific - this is just the first case we have hit them (in the same way Dishonored 2 was the first game that hit certain problems - that doesn't make them specific to that game, it's just the first real world case we hit).
• Overlay works, hunting works, shader replacing works. Besides a few hardcoded paths and an injector that would need some polish this is quite functional.
• Well, it would be a shame to let this work go to waste, and very little of it is actually Windows Store specific, so a bit of cleanup and into master it goes.

[bo3b]

No, you're not missing anything - those are valid and necessary workarounds to some of the hooking problems I'm referring to. The other workaround we have is avoiding allowing DirectX to call CreateSwapChain by not ever calling D3D11CreateDeviceAndSwapChain - that's not really one we solved, that's one we avoided, and this is just another case of that same problem.

Actually, my motivation there was not to decouple anything like the CreateDevice calling through to CreateDeviceAndSwapChain with null, it was trying to make sure that we did not have to do a lot of gymnastics in CreateSwapChain for random devices. The idea was to make sure that we could trust that at CreateSwapChain we had a HackerDevice, always. That of course worked great until it took a bullet from UE4 and the DXGIDevice problem.

But that's the point - these problems only exist because hooking allowed the ordering between SpecialK, 3DMigoto, Steam and DirectX to be violated and these workarounds are necessary as a result - this is a fundamental problem with hooking, and the fact we even have workarounds in our code is evidence of that - and I can pretty much guarantee that we will need more workarounds in the future for cases we haven't yet discovered or don't exist yet.

For this one, I think we were always doing it wrong anyway, and are closer now by using the Hacker* routines plus shims. Calling out to system routines that might have been hooked in order to do some task for us internally was a naive approach that only made sense if we don't take into account other modding tools.

We never worked with ENB even using proxy for example, once we connected DXGI. And we barely worked with SpecialK before, as long as we disabled our DXGI hooks. Using the Hacker*/shim approach is a lot closer to correct for handling multiply hooked routines. When we call to a system function that might be hooked, it needs to be because we want that system function, not to access some other chunk of 3Dmigoto. e.g. CreateSwapChain is better if it's HackerSwapChain, which then calls out to the system. We sort of get the same result if we just call CreateSwapChain where we want, and works as long as their are no hooks, but goes off the rails with other hook tools.

I agree this is a fundamental problem with hooking, the multiple layers and call backs becomes a real mess. It does seem like we have no real choice but to support it though.

[bo3b]

Working through all the issues I mentioned previously, most of which are not windows store specific - this is just the first case we have hit them (in the same way Dishonored 2 was the first game that hit certain problems - that doesn't make them specific to that game, it's just the first real world case we hit).

Sure, but just like Dishonored2, I absolutely do not think it is appropriate to add code for a single game. Dishonored2 languished for a long time because it was the only game we knew of to require platform_update. Once I had three games that required it, it's clearly it's a trend and something needing attention.

Again, you keep saying this is more general than it is. Your changes here are for a single game, there is no other use case that we need at present. I do not understand why this needs to be in master. If it's in an alternate branch it will never be lost. Moreover, you just said it took you less than a day to figure it out the first time. If you ever had to do it again it's hardly a giant engineering cost, especially if you have the code in a branch already.

It's a great solution for the UWP problem, and it is possible that it will be valuable one day. Today is not that day. I am completely confused as to why you are so adamant that this must be implemented in master right now.

[DarkStarSword]

I hope I don't come across as being to cranky in this post... I probably will, but I'm tired, so apologies in advance. I respect your opinion and your insights and these discussions are worth having, but I think this one has now dragged on for too long and it would be better to just finish the work and be done with it.

---

Actually, my motivation there was not to decouple anything like the CreateDevice calling through to CreateDeviceAndSwapChain with null, it was trying to make sure that we did not have to do a lot of gymnastics in CreateSwapChain for random devices. The idea was to make sure that we could trust that at CreateSwapChain we had a HackerDevice, always. That of course worked great until it took a bullet from UE4 and the DXGIDevice problem.

Sure, but the root cause there that lead to that problem in the first place was that DirectX called CreateSwapChain internally, and our hook allowed us to intercept that when we didn't actually want to. The UE4 issue was separate, since that was a call from the game, not an internal DirectX call.

Anyway, I think we're pretty much on the same page and now we have solutions to cases where DirectX can call back into a hook, and avoiding a third party intercepting an internal call we make, so if any other cases of either of these issues arise in the future it should be much easier to deal with then.

Sure, but just like Dishonored2, I absolutely do not think it is appropriate to add code for a single game. Dishonored2 languished for a long time because it was the only game we knew of to require platform_update. Once I had three games that required it, it's clearly it's a trend and something needing attention.

I disagree with that - it is absolutely fine for us to add code to support a single game, but what is less okay is to add a hack for a single game unless we absolutely need to for some reason - we've done that in the past, I've done that in the past, but my preference is that if we're going to support something I would rather we try to support it properly. We can certainly decide where to spend our time, and if we don't think something is worth our time or not a high enough priority just yet that's fine - that is our time to decide what to do with. So far I've avoided pushing up any hacks for this work, and am holding off on pushing up code that I think should be done better or may need further consideration to be sure it is the best way to solve the problem - the code I've pushed up so far is largely cleanups, refactoring work, bug fixes, a general strategy to deal with a class of hooking issues and implementations for missing DirectX functions that we need - that's pretty much all code we should have regardless - it's only a matter of when it is written.

Again, you keep saying this is more general than it is. Your changes here are for a single game, there is no other use case that we need at present. I do not understand why this needs to be in master. If it's in an alternate branch it will never be lost.

Because forks can be an absolute PITA to maintain. It is almost always better to have code in master if it is intended to be used. Look at the lackluster support for just about any random ARM device that runs ridiculously old versions of Linux and when you trace it to understand why and you will soon find it's because they didn't push their code upstream at the time, and it has become impossible to merge in the meantime so it's pretty much stuck on that old version.

I've been subtly suggesting we switch to the vs2015 branch for ages now - that's a fork I've been continuing to maintain, and had to deal with numerous merge conflicts in that time, but the only reason it is maintainable at all is because I've put most of the code changes (bar one) in master, so the diffs are limited to the project files which don't change too often. If I hadn't put that code in master, that branch wouldn't be worth using at all - we'd be throwing it away and starting again when we finally do get around to migrating, but it is still working and as far as I'm aware has no outstanding issues that would prevent us from switching to it today if we wanted. So far, we just haven't had a pressing need to do so, so I haven't pushed very hard on this point (and UWP isn't going to change that - I thought it might since they are able to insist on more recent DirectX APIs, but building from the vs2012 branch seems to work just fine).

As it is, the upscaling_for_hwnd branch will no longer be mergeable without some effort - it's doable, but it will cost extra time because it wasn't merged in when it should have been a few weeks ago as it was awaiting further testing.

Moreover, you just said it took you less than a day to figure it out the first time. If you ever had to do it again it's hardly a giant engineering cost, especially if you have the code in a branch already.

That's true, but it is still a cost - that's a day to figure things out and get a working POC, polishing it is of course extra (and this discussion is taking up that time - my OP was really only meant to bring the hooking issues to your attention as an FYI, though I do appreciate your thoughts on the matter, and the time gave me a chance to more deeply consider the best solution to the problem).

If I'm paying the cost now because I'm currently interested in the problem then why not? It may be an academic interest, but it has practical applications and I might be far less keen to pay that same cost again later, just like I'm far less keen to re-fix a game I already fixed in the past.

[bo3b]

As a general comment- my primary concern with all of these is the creeping complexity. We keep adding more and more complex solutions to different problems, because it's always easier to add a bunch of If statements, than it is to work out a good architecture. Long term, that approach is death.

Ergo, I always push back hard on creeping complexity. I really do not like one-off stuff that makes everything harder. Maintenance is extremely important, and creeping complexity grinds the maintenance down to where nothing further can be done.

Sure, but the root cause there that lead to that problem in the first place was that DirectX called CreateSwapChain internally, and our hook allowed us to intercept that when we didn't actually want to. The UE4 issue was separate, since that was a call from the game, not an internal DirectX call.

Yep. It was my hope to be able to decouple those, so that the CreateSwapChain code could always rely upon getting a HackerDevice on input, and not have to check. That dramatically simplified and cleaned up the code, and made its job much more clear. Too bad about that UE4 kick in the balls.

Given that was a failed approach, there is no reason not to refactor CreateDeviceAndSwapChain back to what it was, or change that strategy.

I'm concerned here about adding hooks to these two calls that we've never had before, because it's creeping complexity. If it were totally worth it or necessary for a class of games like Unity or UE4, or even something like MGSV that is different. In my experience, complexity for one-offs is the road to doom.

Because forks can be an absolute PITA to maintain. It is almost always better to have code in master if it is intended to be used. Look at the lackluster support for just about any random ARM device that runs ridiculously old versions of Linux and when you trace it to understand why and you will soon find it's because they didn't push their code upstream at the time, and it has become impossible to merge in the meantime so it's pretty much stuck on that old version.

No disagreement there, forks can be a lot of trouble. But are not always depending upon the approach.

If we were to make it a one-off branch to solve UWP type problems, then exactly like the DX10 branch, there is no actual need to keep it up to date. It's available if we need it or want it in the future, if at some point it becomes clearly valuable and worth the long-term maintenance cost. Just specifically keep it as a one-off, like the 3 or so fixes we have for DX10 games.

Another approach is to do the initial merge into the branch and always keep it up to date with automatic or common merges from master. As long as the branch is not actively changing, then the merges should always work because there is no conflict. Depending of course on the structure and initial merges, and avoiding other general refactoring. For example, if we think the startup sequence is working, we won't be actively modifying that section anyway, so there should be no conflicts.

This comes down to a matter of project philosophy, and that is also why I've spent much more time talking about this than I want to as well.

I've worked on giant projects in the past, and have lots of scars from those experiences that make me very skeptical of one-off type fixes and putting stuff in because it's got potential value in the future. Every single thing has a long-term maintenance cost, and I've seen a lot of examples where it ground to a halt and the only possible solution was to start over from scratch. Every single If statement in our code that is some useless feature that no one ever uses is a long-term cost for maintenance. I'm not a fan of having millions of features that no one ever uses. I always want clear ROI.

My goal always is to only add stuff that we actually need, right now. The future is unknown, we can build stuff that it turns out no one gives a shit about, or for a future that never comes to pass. We can add DX12 support today. But, really, why? Vulkan support? I like to let the market (our users) tell us what is important, and see what happens with bigger pictures like which APIs consoles use before adding stuff.

I like to be a lagging implementation, not a leading implementation- because every single we add has a long term cost. I will make exceptions for things like upscaling, because that had a larger goal of trying to pull in more programmers, but otherwise, it's a one-off for one person and should be in a branch.

However, this project is weird, and outside my normal experience because of our problem set. We don't get to choose a best architecture because we have to workaround a bunch of crazy shit in games, a wealth of other modding tools at once, and the absolutely piss poor architecture and maintenance of DX11 itself. So maybe my concerns are unfounded here because this is more unique. I really don't know what the actual best approach is, and probably there is no 'best'.

Is adding this the right choice? Take the injector piece specifically. This is very cool, and has some real potential. Emphasis on potential, not current need. Is having it part of 3Dmigoto, even if it is never used the correct choice? Why can't it be a standalone tool instead? Maybe it totally makes sense in the context of 3Dmigoto itself, and if we always use it to launch it simplifies things. I'm really not sure, I just want to be certain that you have considered the downsides as well as the upsides.

Maybe we do genuinely need to support all the ways we can launch and there is no simplification possible, because our problem set is too diverse and complicated. And that there is in fact no one clear path that will always work for everything we currently know about. My goal with 1.3 was exactly that- get the clearest and simplest approach that will work for everything we currently know. Maybe I failed.

The reason I'm expressing my concern, is because in this project, you have added a lot of complexity for one-off scenarios that we continue to pay the price for. Best example is adding hooking for MGSV. While that is a terrific solution for that game, it complicated a lot of other things like the FrameAnalysisContext, and led to long term pain with that 'realOriginal' stuff. If that wound up being used only for a single game, it would have made more sense to make a one-off branch.

Don't get me wrong- you do great stuff. The vast majority of things you've added are terrific and always useful. You've had an outsized positive influence on the project. I just want to be certain that your enthusiasm does not overwhelm your good judgment.

I've said my piece. I hope I've clearly explained my concerns. If you still genuinely feel it's worth the long term maintenance costs, then I won't disagree further.

[bo3b]

Given that a lot of these things are based around overall project philosophy, I'd like to retract my suggestions, and say that we should go with whatever you prefer. I'll leave my comments above as an alternate viewpoint, but I think you should do what you think best, or what works best for you.

I'm going to step back from the project for the time being, so you are in charge. Please feel free to do anything you judge important.

Maybe even including adding support for more general hacking, not our usual 3D centric approach. There are a lot more game modders and 2D players that could use the tool, and it might be helpful for your Patreon efforts to extend to other realms like oomek has always wanted.

[DarkStarSword]

As a general comment- my primary concern with all of these is the creeping complexity. We keep adding more and more complex solutions to different problems, because it's always easier to add a bunch of If statements, than it is to work out a good architecture. Long term, that approach is death.

I get where you are coming from with that and I do agree with you that we don't unnecessary complexity and that we manage necessary complexity by thinking about the bigger picture and long term implications - as you know I almost never rush out half arsed solutions to anything, and that I'm always thinking of the bigger picture, and always working to clean up and refactor code as I go so that in general it will be more maintainable in the future.

I do however think it is important that in principle we should not be blocking anything that increases our compatibility with any game, third party tool or app store. There can be good reasons to reject something, such as the code being unmaintainable, or that we really need to re-work some fundamental 3DMigoto functionality before we can consider supporting it (e.g. the platform update wasn't really viable before the 1.3 rework), or that something is already achievable in 3DMigoto via existing features. And we can certainly choose whether we want to spend our time on something or not - we may view compatibility with the Windows store or some specific game as a low value use of our time and that there are more important things to focus on, but if it is work that increases 3DMigoto's compatibility with anything than that is still a net positive value, even if it is low. We may not think the Windows app store will go far, but that's not really our call to make - that's between Microsoft, game publishers and consumers to decide.

Given that was a failed approach, there is no reason not to refactor CreateDeviceAndSwapChain back to what it was, or change that strategy.

At this point that could go either way - I've refactored code out of D3D11CreateDevice and CreateSwapChain into helper functions, so if we refactored it back to what it was it would now be about the same complexity as the remaining code inside HackerCreateDevice, but on the other hand it's still working as is, and we're not aware of any other cases where it could fail, so isn't necessary to change it back either.

I'm concerned here about adding hooks to these two calls that we've never had before, because it's creeping complexity. If it were totally worth it or necessary for a class of games like Unity or UE4, or even something like MGSV that is different. In my experience, complexity for one-offs is the road to doom.

If this is necessary for Windows Store apps, then that is a net positive gain and we should not block it, no matter what we think of the Windows Store. That said, I do want to go back and double check why naming 3DMigoto d3d11.dll didn't work in that case, yet does work for Unity games. I'm fairly sure it will come down to load order and will be more reliable to not use the name d3d11.dll, but it is worth checking in case there is a better solution here.

However, at some point I do want to go down the SpecialK route of using an uber DLL so that we don't have to be named d3d11.dll (this is something I had considered well before coming across SpecialK - I've been considering it for possible use with DX12 support for a while, but obviously that is still a long way off from becoming a reality), and if we do end up using that we will need these functions hooked regardless, so because of that I have no problem with adding them now and having that well debugged by the time we need it.

If we were to make it a one-off branch to solve UWP type problems, then exactly like the DX10 branch, there is no actual need to keep it up to date. It's available if we need it or want it in the future, if at some point it becomes clearly valuable and worth the long-term maintenance cost. Just specifically keep it as a one-off, like the 3 or so fixes we have for DX10 games.

But the DX10 branch proves my point - people who have tried using that have found it too old to be useful and gave up as a result (this has come up a few times now, one was pretty recent) - there is absolutely a proven need to keep these type of forks up to date. Yes, regressions may happen, and some of the newer functionality may be broken/missing on DX10 and we'll need to fix them from time to time, but that is infinitely easier than maintaining a separate branch, and we have the benefit of being able to bisect the changes, which we don't have between branches.

I like to be a lagging implementation, not a leading implementation- because every single we add has a long term cost.

We can hold off on adding support for something until we're ready, but I don't think it's our call as to whether or not we should support something - we're not working for Microsoft, and we don't get a say in the direction that DirectX is going, and we're not working for every game engine developer on the planet, and we don't get to tell them what APIs to use.

If we choose to delay or not support something, we are relying on the developers adding a fallback path, but in that case we're forcing the game to use code paths that the developers probably only lightly tested - remember the whole Unity hang on alt+tab thing? Some older versions of Unity were broken with or without 3DMigoto, but after a while they fixed the problem, yet it still occurred when 3DMigoto was in use, which I believe was because we forced them to use a fallback path where they had not fixed that bug. Now, we could point the finger at them and say "the same thing would happen on Windows 7 without the evil update", but it was on us that a bug that should have only existed on an old version of Windows still existed for everyone using 3D Vision, regardless of which version of Windows they were using.

We may also be responsible for performance bottlenecks by refusing to support newer DirectX APIs, as some of these API updates were created to help developers solve performance issues in the real world. e.g. the 1.3 update will now mean that games that want to reduce CPU->GPU overhead by using the Context1 APIs to update subsets of constant buffers can now do so.

Now we have the architecture to do so we should (at some point) consider switching to the vs2015 branch and beginning to add support for even newer APIs - e.g. Windows 8.1 added support for tiled resources to better manage vram, that we are preventing games from using. This isn't urgent by any means, and it is reasonable to expect that game developers will have a fallback path to work without that feature, but it is something worth considering, and without it we may be responsible for a game stuttering as the player travels around the map because we've forced it into an inefficient manual resource management mode instead of being able to use tiled resources - and it will not be apparent at all that we are at fault.

Is adding this the right choice? Take the injector piece specifically. This is very cool, and has some real potential. Emphasis on potential, not current need. Is having it part of 3Dmigoto, even if it is never used the correct choice? Why can't it be a standalone tool instead? Maybe it totally makes sense in the context of 3Dmigoto itself, and if we always use it to launch it simplifies things. I'm really not sure, I just want to be certain that you have considered the downsides as well as the upsides.

Exactly - this is something I want to explore and see if it does have a use outside of UWP. I know it can successfully inject 3DMigoto into other games, which may be handy to games where we currently have to use the injector DLL. The external tool doesn't have to be our own injector (even HelixModLauncher.exe works, but injects it with the name helixmod.dll) - 3D Fix Manager could probably implement it fairly easily.

But 3DMigoto may have to be aware that it has been injected - outside of UWP we might still be able to have it's files in the game directory, but it will still need to bail out from anything it didn't want to be injected into, because the external tool doesn't have control over that, so we do need some functionality inside 3DMigoto. I've held off on pushing the logic to handle that because I want to make sure the path we choose for this is the best way to go, and that it doesn't cause issues with things like proxy loading (where we also might not be named d3d11.dll, but don't want to hook the d3d11 functions) - I think this is something worth having, but it should be done right.

My goal with 1.3 was exactly that- get the clearest and simplest approach that will work for everything we currently know. Maybe I failed.

1.3 was a huge success, and a very necessary and very high value step in the right direction. UWP wouldn't even be worth pursuing without 1.3, because everything there depends on the platform update.

The reason I'm expressing my concern, is because in this project, you have added a lot of complexity for one-off scenarios that we continue to pay the price for. Best example is adding hooking for MGSV. While that is a terrific solution for that game, it complicated a lot of other things like the FrameAnalysisContext, and led to long term pain with that 'realOriginal' stuff. If that wound up being used only for a single game, it would have made more sense to make a one-off branch.

Yes, I am quite aware of the cost that had, and had there had been an alternative to using hooking I would not have added that - I think you know my general opinions on hooking. However - the reason we have paid a maintenance cost for it at all is because we tried to use newer versions of 3DMigoto with all the new features and bug fixes, which would have simply not been possible if that was on an alternate branch (and that has allowed me to identify one of the remaining problems I could never track down with the older version) - or would have needed a significant effort to merge it in anyway, which IMO would have exceeded the maintenance cost we have paid by several orders of magnitude, and that cost would increase as the two branches continue to diverge. Metal Gear Survive is recently released, and at the moment I have every reason to believe the hooking support will be necessary for it as well, although it's not a particularly good game so I'm not sure if anyone is going to bother with it (I was thinking about it, but I tried the beta and it was crap, then the reviews came out and they were crap, so I might pass or at least wait for the price to drop - but I can confirm from the beta that it does use the exact same Fox engine as MGSVTPP).

I've said my piece. I hope I've clearly explained my concerns. If you still genuinely feel it's worth the long term maintenance costs, then I won't disagree further.

Thank you.

Maybe even including adding support for more general hacking, not our usual 3D centric approach. There are a lot more game modders and 2D players that could use the tool, and it might be helpful for your Patreon efforts to extend to other realms like oomek has always wanted.

Yeah, that's always been on the back of my mind, and the command list code certainly gives 3DMigoto a lot of power in this realm.

[bo3b]

Not sure where to put this, so I'll just add it here as a sort of refactoring problem. When we move to new VS, we should move directly to VS2017, because the built-in Git plugin is now broken in all prior versions.

Just did some research on a problem that GitHub introduced on Feb 22. My VS Git plugin stopped working, and it seemed like an authentication problem, but nothing had changed. Except... for GitHub removed any use of TLS 1.0/1.1 for authentication, because security. Not sure why that matters for public open source code, but there you go.

Since they killed TLS, old versions of VS can no longer connect to GitHub for checkins or pulls, or any git use. The built-in Git plugin is broken on VS2013, VS2015, and most of VS2017 up to the one that shipped this week. That means the only option is using git on cmd line, which is not very convenient. Not a rush, but for other potential coders, this is more undesirable friction.

[DarkStarSword]

Good to know (they probably just updated the OS on their servers - IIRC old versions of TLS were disabled in system libraries in Debian and other Linux distros quite some time ago to work towards eliminating downgrade attacks across the entire Internet, so leaving it enabled would require they be running custom builds of the OS libraries, which is non-zero effort).

I'll open a vs2017 branch with a forward port and begin testing on my own system. It's worth noting that there were a number of libstdc++ changes in the vs2015 toolchain that caused some pain (for the assembler in particular), so we will need to keep an eye out for anything similar on the vs2017 branch. That said, I'm anticipating that vs2015->vs2017 should be a lot easier than the move to vs2015.

[DarkStarSword]

The vs2017 branch is now open for testing - as anticipated the existing vs2015 forward port made it much simpler to forward port up to vs2017. All configurations build successfully, and now we just need to keep an eye out for any new issues introduced by the new toolchain. I'm going to start using this for anything I'm working with on my own system to give it some general testing, and we should also compare the results of the assembler, disassembler and decompiler using cmd_Decompiler built from master with one built from vs2017 since that was a known pain area from the vs2015 port.

[DarkStarSword]

crc32c is broken under VS2017, which is a complete deal breaker until that is resolved. I've opened a separate issue to track this and any other issues that may surface: https://github.com/bo3b/3Dmigoto/issues/92