justinstenning / Direct3DHook

DirectX Capture and Overlays by using Direct3D API hooks
http://spazzarama.com/2011/03/14/c-screen-capture-and-overlays-for-direct3d-9-10-and-11-using-api-hooks
MIT License
580 stars 178 forks source link

After making inject and then in the game i injected to i'm doing Alt+Tab the game crash any idea why ? #17

Closed Csharper1972 closed 9 years ago

Csharper1972 commented 9 years ago

The game in this case is using DX9. If i just run the game and make many times Alt+Tab(to return back to windows) it's working good no problems i can keep making Alt+Tab and return to the game and so on.

But once i did inject and now im in the game when i'm doing Alt+Tab after one time or two times the game crash for example: BouT2.exe has stopped working... And it happen only after doing inject in the program.

I can make choose debug or close if i make debug using my visual studio 2012 i see:

Unhandled exception at 0x76FDF8A2 (ntdll.dll) in BouT2.exe: 0xC0000005: Access violation writing location 0x0000009C.

Why it happen only after making inject and then when doing in the game Alt+Tab to get to the windows ? And idea how to solve it ?

remcoros commented 9 years ago

The hooked functions in D3D9 (and other) hooks are not coded correctly.

For example: https://github.com/spazzarama/Direct3DHook/blob/master/Capture/Hook/DXHookD3D9.cs#L257

PresentEx always returns 'SharpDX.Result.Ok.Code'. But this is not correct behavior.

You need to return the same value as the original (hooked) function.

Look for example on this line (in my fork): https://github.com/remcoros/Direct3DHook/blob/master/Capture/Hook/DXHookD3D9.cs#L532

It returns the original (hresult) from the hooked function, instead of always 'Ok'.

The reason for the crash is that after Alt-tab, the device is in a 'lost' state, doing anything with a 'lost' d3d device result in a crash.

In the game code, there are checks to see if Present/PresentEx/EndScene/etc. return 'Ok' or something else. And when it's not 'Ok' it doesn't do anything with the device (cause it's lost).

TL;DR: Call the original function and save it's return value. Then use that return value in your hook.

justinstenning commented 9 years ago

This has been changed in my local fork, just waiting to finish a few more changes before I commit (I.e. GPU resizing for dx11).

On Monday, 15 September 2014, Remco Ros notifications@github.com wrote:

The hooked functions in D3D9 (and other) hooks are not coded correctly.

For example:

https://github.com/spazzarama/Direct3DHook/blob/master/Capture/Hook/DXHookD3D9.cs#L257

PresentEx always returns 'SharpDX.Result.Ok.Code'. But this is not correct behavior.

You need to return the same value as the original (hooked) function.

Look for example on this line (in my fork):

https://github.com/remcoros/Direct3DHook/blob/master/Capture/Hook/DXHookD3D9.cs#L532

It returns the original (hresult) from the hooked function, instead of always 'Ok'.

The reason for the crash is that after Alt-tab, the device is in a 'lost' state, doing anything with a 'lost' d3d device result in a crash.

In the game code, there are checks to see if Present/PresentEx/EndScene/etc. return 'Ok' or something else. And when it's not 'Ok' it doesn't do anything with the device (cause it's lost).

TL;DR: Call the original function and save it's return value. Then use that return value in your hook.

— Reply to this email directly or view it on GitHub https://github.com/spazzarama/Direct3DHook/issues/17#issuecomment-55588889 .

Csharper1972 commented 9 years ago

spazzarama when do you think you will publish new version of your project/solution with all the fixes ? Hours or Days ?

Csharper1972 commented 9 years ago

remcoros i didn't understand yet. And your fork code is way different then the code i'm using the original one. For example in the line you showed me for the example there is a property for Direct3DDevice_PresentHook: Direct3DDevice_PresentHook.Original i didn't find where or how you did this property.

justinstenning commented 9 years ago

The changes I've made make use of some of remcos' changes - he has used a handy wrapper that hides some of the complexities of managing the hook life cycle and simplifies calling the original method.

On Tuesday, 16 September 2014, Csharper1972 <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

remcoros i didn't understand yet. And your fork code is way different then the code i'm using the original one. For example in the line you showed me for the example there is a property for Direct3DDevice_PresentHook: Direct3DDevice_PresentHook.Original i didn't find where or how you did this property.

— Reply to this email directly or view it on GitHub https://github.com/spazzarama/Direct3DHook/issues/17#issuecomment-55674237 .

remcoros commented 9 years ago

It's here: https://github.com/remcoros/Direct3DHook/blob/master/Capture/Hook/HookData.cs

but you might want to wait for spazzarama to check in his new version as my fork only has D3D9

justinstenning commented 9 years ago

I've uploaded the work-in-progress version of the GPUResize branch. This includes most of remcoros' changes as well.

remcoros commented 9 years ago

Awesome job.

One side note though:

For my use-case, I didn't need a single screenshot sometimes, but a constant stream of 'screenshots' (video) up to 60 fps.

After fixing the alt-tab bugs and the other changes in my repo, I realized that the current implementation (no buffering / locking / IO via remoting) did not give me the performance I needed and also hit the games framerate too much.

Currently, the games main draw loop (where Present/EndScene is called) can only be as fast as the speed at which image data is processed / send. This is because, even though the processing is done on a different thread, the main drawing loop is still waiting for (locks) the render target.

This means, with the current (and old) implementation, if you try to capture screenshots at a high rate, it will directly impact the games (and app) performance.

I have studied the OBS implementation and went completely berserk in optimizing this for high speed capture (basically, video recording). Added a double buffer with off-screen surfaces, and using shared memory (memory mapped files) and mutexes instead of standard remoting.

Using that double buffer (this.surfaces and this.copySurfaces) makes it so that the processing (copy from surface to memory / sending to the client) will not lock/halt the games main draw loop (present/endscene). Using sharedmemory instead of default remoting, avoids a lot of memory I/O (from surface to stream, from stream to byte array, to ScreenshotResponse, serialized by remoting, etc, etc.). This adds some complexity (need mutexes for inter process signaling), but gives more than double performance and more than half the memory I/O.

This results in an almost lock-free high performance and low I/O image capture hook which doesn't directly impact the games main drawing loop (can easily handle 60+ fps on a decent machine).

A live and working implementation is used in my HearthstoneTracker project and can be found here:

https://github.com/HearthstoneTracker/HearthstoneTracker/blob/master/Capture/Hook/DXHookD3D9SharedMem.cs

The client code is here: https://github.com/HearthstoneTracker/HearthstoneTracker/blob/master/HearthCap.Core/GameCapture/AutoCaptureEngine.cs#L742

Although this might be out of scope to implement in the Direct3DHook project, it's a good read and a good example IF you want to go for high speed image capture.

justinstenning commented 9 years ago

Thanks remcoros, I've been meaning to look at support for streaming for a long time - just haven't managed to get to it. I'll review those changes and look at how they can be incorporated. I have been looking at a different approach for dx11 already as well.

Cheers J

On Thursday, 18 September 2014, Remco Ros notifications@github.com wrote:

Awesome job.

One side note though:

For my use-case, I didn't need a single screenshot sometimes, but a constant stream of 'screenshots' (video) up to 60 fps.

After fixing the alt-tab bugs and the other changes in my repo, I realized that the current implementation (no buffering / locking / IO via remoting) did not give me the performance I needed and also hit the games framerate too much.

Currently, the games main draw loop (where Present/EndScene is called) can only be as fast as the speed at which image data is processed / send. This is because, even though the processing is done on a different thread, the main drawing loop is still waiting for (locks) the render target.

This means, with the current (and old) implementation, if you try to capture screenshots at a high rate, it will directly impact the games (and app) performance.

I have studied the OBS implementation and went completely berserk in optimizing this for high speed capture (basically, video recording). Added a double buffer with off-screen surfaces, and using shared memory (memory mapped files) and mutexes instead of standard remoting.

Using that double buffer (this.surfaces and this.copySurfaces) makes it so that the processing (copy from surface to memory / sending to the client) will not lock/halt the games main draw loop (present/endscene). Using sharedmemory instead of default remoting, avoids a lot of memory I/O (from surface to stream, from stream to byte array, to ScreenshotResponse, serialized by remoting, etc, etc.). This adds some complexity (need mutexes for inter process signaling), but gives more than double performance and more than half the memory I/O.

This results in an almost lock-free high performance and low I/O image capture hook which doesn't directly impact the games main drawing loop (can easily handle 60+ fps on a decent machine).

A live and working implementation is used in my HearthstoneTracker project and can be found here:

https://github.com/HearthstoneTracker/HearthstoneTracker/blob/master/Capture/Hook/DXHookD3D9SharedMem.cs

The client code is here:

https://github.com/HearthstoneTracker/HearthstoneTracker/blob/master/HearthCap.Core/GameCapture/AutoCaptureEngine.cs#L742

Although this might be out of scope to implement in the Direct3DHook project, it's a good read and a good example IF you want to go for high speed image capture.

— Reply to this email directly or view it on GitHub https://github.com/spazzarama/Direct3DHook/issues/17#issuecomment-56038575 .