baldurk / renderdoc

RenderDoc is a stand-alone graphics debugging tool.
https://renderdoc.org
MIT License
9.05k stars 1.35k forks source link

[Issue]Doesn't work attached on Epsxe emulator (D3D Directx 11) #375

Closed Nightshades1 closed 8 years ago

Nightshades1 commented 8 years ago

Hello, as the title said i start the executable using renderdocui.exe all is fine but there is nothing being logged it doesn't work at all, i did use "processhacker" to see the module loaded inside Epsxe.exe there is d3d11.dll loaded (it's a plugin to play with Directx 11 in epsxe).

I did try Nvidia Nsight this work fine sadly their software doesn't allow to browse all textures from buffer lol kinda can't rip sprite with their software, So i'm wondering why your software doesn't work at all, it should since it's loaded and DX 11 So i'm kinda lost lol.

Edit: Wanted to say that i also doesn't have the in-game pop up to start the capture, there is just nothing =P.

Regards.

baldurk commented 8 years ago

Can you post the log file from the application? There should be a RenderDoc_app_<date>_<time>.log file corresponding to when it was started in the %TEMP% directory.

I see that epsxe is a free download, is there a demo or homebrew program or something I can use to test it? Is there any documentation on specifically how to set up D3D11 use so I can try to reproduce the problem?

Note, RenderDoc is a debugger only so there's only a limited degree to which I'll try to support non-debugging use cases.

Nightshades1 commented 8 years ago

Okay so it seem that the dll isn't found somehow, i'm using the "renderdocui" and i start the application from the "Capture Executable" tab.

Note: Pcsxe 2 is sometime trash and decide to crash hard with DX plugin (i guess the autor badly coded).

RenderDoc_app_2016.09.26_01.37.57.txt

RenderDoc_replay_2016.09.26_01.37.44.txt

By the way i use RenderDoc Unofficial release (v0.32 c6b31ef4778c92a2aeaa57e6820287fd6ad6ce0bb)

Edit: Also tested to inject and then "attach to runing" process i got those error. Edit2: Doesn't work with stable release as well.

RENDERDOC: [01:47:46] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38921 RENDERDOC: [01:47:46] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38922 RENDERDOC: [01:47:47] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38923 RENDERDOC: [01:47:47] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38924 RENDERDOC: [01:47:48] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38925 RENDERDOC: [01:47:48] target_control.cpp( 409) - Log - Got remote handshake: ePSXe () [4916] RENDERDOC: [01:47:48] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38926 RENDERDOC: [01:47:49] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38927

Regards.

baldurk commented 8 years ago

OK, it looks like the messages in the replay log are a red herring, it's caused by some new code I added to handle environment variable modification and it didn't account for capturing 32-bit programs from 64-bit renderdoc. It will show up on capturing any 32-bit program but it's harmless unless you're trying to modify the environment variables (very likely you are not).

So I will need more information to reproduce. I got ePSXe which seems to come with a free bios implementation, but I can't see any D3D11 plugins listed online. Can you give me specific details on how I can run the program in D3D11 mode so I can debug why the API isn't getting hooked?

Nightshades1 commented 8 years ago

Hello, I uploaded you the pack of plugins (small) with my already configured .ini

You can get it here: http://www.mediafire.com/file/5xvy05rvluehacr/plugins+ini.rar just extract it in your Epsxe folder (Go in the plugin setting, select any GSDX plugin, and select your resolution+DirectX mode) sadly you will need to get "any" game (iso) ps1 game.

Any would be fine, i was currently doing some test with Nvidia Nsight and DirectX 9 ... on Mortal Kombat Mythologies sub zero, awesome thing during the capture you can see all event, geometry generated etc... sadly can't browse the texture since it's a fucking "Capture" there is no way to "Stream" the entire API CALL and then "browse" all texture generated since the start of the fucking software, somehow when i use nvidia nsight on DirectX11 epsxe plugin there are almost no event (Badly coded plugin ?).

So i'm kinda stuck, if i need to rip sprite "capture" per "capture" it's bad, waste of time and not efficient, fun fact those guys mainly focus on "newer" directX, take DirectX 9 as exemple, you can browse all event but there is no PixelShader history, what da fuck Nvidia -.- .

The only software that almost do that is APItrace but the trash part is that you can't browse the texture like that, the software must start the "replay" and it's time waste lol.

Does at least your software allow to do that, i mean for exemple as a test if i would debug this game "Mortal kombat mythology sub zero" on this PS1 Emulator, does this will STREAM since the start of the API CALL so it's possible to see "any" texture that were drawed ? cause it's really painfull for these type of game to save each individual picture "blindly" and comparing with other frame until you have the correct set of sprite.

And more painfull if it only work like nvidia which mean you can only "capture" 1 frame and this will just have the history of that capture FRAME, i want the history of all frame since the start (they should have made that ... sadly my GPU is a trash gtx 580 and isn't supported for PixelShader".

Sorry for the wall of text, i needed to tell it, Regards.

Ps: Those trash plugin tend to crash the emulator hard, or it's my computer idk ... Edit: My .ini uploaded got a 120hz refresh rate, you might change depending on your screen.

baldurk commented 8 years ago

Do not use language like that, it's unnecessary and inappropriate for a software bug tracker.

I've had a look into ePSXe with the repro you provided, and it seems like the problem is that the root executable is packed with UPX which interferes with the library hooking. This means when the graphics plugin is loaded it's not properly intercepted and so the D3D11 device is never created as hooked.

Fixing this is not really in scope for the project as executable compressors like UPX are very complex. RenderDoc is a debugger aimed for developers, and such anyone who is debugging their own program can run an unpacked executable. This is one case where supporting debugging third party applications is not going to be explicitly supported.

zao commented 8 years ago

For future reference for when you for some obscure reason only happen to have a packed version of your executable, the UPX tool supposedly has a --decompress operation.

I have no idea if that is sufficient for RenderDoc to hook the resulting executable, but at least there's a glimmer of hope.

Nightshades1 commented 8 years ago

Well i unpacked the upx and saved the unpacked executable it doesn't work either lol. Le 27 sept. 2016 11:26, "Lars Viklund" notifications@github.com a écrit :

For future reference for when you for some obscure reason only happen to have a packed version of your executable, the UPX tool supposedly has a --decompress operation.

I have no idea if that is sufficient for RenderDoc to hook the resulting executable, but at least there's a glimmer of hope.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/baldurk/renderdoc/issues/375#issuecomment-249813367, or mute the thread https://github.com/notifications/unsubscribe-auth/ASl_gT-GqlYvXy7gJoka1FezTNWpb5twks5quOFJgaJpZM4KF_PR .

Nightshades1 commented 8 years ago

Hmm so i recompiled it for x86 Once the emulator is started and game loaded, i use the "inject into process" i don't have any hook error but i still get those error:

RENDERDOC: [18:49:07] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38921 RENDERDOC: [18:49:08] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38922 RENDERDOC: [18:49:08] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38923 RENDERDOC: [18:49:09] target_control.cpp( 409) - Log - Got remote handshake: ePSXe () [4524] RENDERDOC: [18:49:09] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38924 RENDERDOC: [18:49:09] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38925 RENDERDOC: [18:49:10] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38926 RENDERDOC: [18:49:10] win32_network.cpp( 352) - Warning - Failed to connect to localhost:38927

I don't have any "popup" asking to start capture or anything, i use the unpacked application.

baldurk commented 8 years ago

What did you recompile, ePSXe? the graphics plugin?

Injecting into the process after the plugin has been loaded, or probably even before, will not work. There is no reason not to use the standard capture workflow here and launch the program through RenderDoc to start with.

Those warnings are expected since it's just saying that it doesn't find any applications running on other ports when enumerating - but it has succeeded to connect to the one running application you expect.

I tried to unpack with upx but I got an exe that wouldn't even run standalone. It's possible that even if the unpacking works upx might not restore the IAT needed for renderdoc to hook properly, since it might have lost the information needed to restore it properly. If you can share your unpacked exe, I'll check to see.

Nightshades1 commented 8 years ago

Hey baldurk, i had just recompiled Renderdoc as an x86 executable, i just used PE Explorer with the UPX unpacker you can get the standalone unpacked exe here http://www.mediafire.com/file/sysf0762pcuziao/ePSXe.exe It's 5Mb way more than the one compressed off course.

You still need to use the GSDX plugin for directX11, by the way do you have any crash with this plugin, sometime my game (mortal kombat mythology sub zero) crash with this plugin, well I have another great plugins which is called Next3D there are much less crash but sadly it's only a directX 9 plugin, Somehow with Nvidia Nsight i can debug an entire frame, review any call and have an overview of "any" sprite on screen/texture which is pretty good.

I would like anyway to have your software working as well using that DirectX 11 plugin GSDX, I don't know much about DirectX coding and was thinking to "create" a new plugin for this emulator, sadly i'm not an advanced programmer and i'm just a medium guy just good to make some DLL injection hack & hook haha, Well thanks for your attention it's just sad that this doesn't work yet, i really want to try out your software on "live" application.

Regards.

baldurk commented 8 years ago

Yes this decompressed exe still doesn't have a functional IAT as far as I can see. It's valid as far as I understand upx's compression - it's just that the decompression doesn't decompress everything - the IAT is still mangled.

That means that RenderDoc still won't be able to hook a decompressed exe, you will need the original non-upx'd exe.

Nightshades1 commented 8 years ago

Well i tested with PCSXR (other emulator) and the same plugin... this crash like hell because this PSX DirectX11 is bugged but once i got it working RenderDoc worked, sadly i had only 1 Texture which was the "render" of the buffer, unlikely i was expecting to browse all "rendered" texture on the screen, It's due to that PSX DX 11 plugin (the way it render things).

I can browse all texture separately using directx 9 and nvidia Nsight so i think i will keep using nvidia nsight i haven't other choice but i appreciate that you guys were helping me, thanks for your time.

Actually i can get each texture and save them as a PNG, and if wanted i can reconstruct an entire 2D level hehe, sad part is getting the geometry... But it's a good start, Actually you may wonder why i would want to rip all Sprite and well, My goal is to have all Assets of Mortal kombat trilogy and Mortal kombat mythologies sub zero.

Cause i'm willing to make a remake of Mortal kombat trilogy with much more feature, inside unreal engine with steam API hehe xD, Well As a first project it can be tricky right ? =D.

Regards.

Edit: I also made a post on nvidia nsight so maybe they will code a way to have an entire "captured playback" so using the rubber we could goes back (previously in the time) then we can browse anything at anytime (would be a giant Capture file) but would be so powerfull hehe =D, Cause being forced to only inspect "a captured frame" instead of a full capture (let's say recorded) is painfull.

You guys should try this software it's extremly well made really.

zao commented 8 years ago

@baldurk That's a bummer, sounds like a bug/unfeature in UPX's depacker, as it ought to be able to reconstruct a proper IAT.

@Nightshades1 Do note that the scope of RenderDoc is not to do a full stream capture of an application, nor is it intended to do asset dumps from applications. For your magnum opus of dubious legality, I recommend that you either look into reverse-engineering the file formats from the game disc, or alter/hook the plugins responsible for rendering. If there were something like vogl for D3D I'd recommend that, but there's not to my knowledge any such thing.

Nightshades1 commented 8 years ago

@zao haha Yeah, Well For that moment i will get MAME emulator for mortal kombat 1/2 and ultimate mortal kombat 3, thanks for all the awesome MAME developers that made it open source and had their hands on A LOT of Cabinet / Motherboard hardware, and has understand how it work (that made this emulator possible).

I will just need to understand how the graphics are being processed and decoded, remember that in this old time it was poorly made, and from the Snippet of source code i have from UMK 3 (psx port) (you can find it on google) It's UGLY as hell how they coded the game lol, no doubt reversing the file format is almost impossible and good thing that MAME developers reversed the hardware or something.

Since it's an open source project it's very good, just need to take a look into Midwunit.cpp, As for the Playstation game (MK mythologies sub-zero) I know that this use Nvram, and unlikely i have no idea how to "open the graphics itself" from the file that's simply too hard and there's no way to "reverse" a file like that.

I think it is better to directly make something in the emulator source (like a new plugin) to directly load/dump the graphics from the Vram, Remember that on the Playstation1 This kind of Vram thing is annoying and specially that Midway made so much trash with Writing/Reading the file.

Because in the old time "they wanted performence" so we have lot of trash code and an ugly non reversable file, if you open these kind of file it's giant, just field of bytes and no idea how it work, it's usually Pixel,Pointer to palette or i don't know what.

Oh Well I'm only a Medium guy and not pro in programming and just started to have a look into SDL haha ^^ for info i'm only 20 (21 soon) I don't have your experience Guys but i'm on the good track.

baldurk commented 8 years ago

This is getting way off-track for the original issue, which has been resolved.