baldurk / renderdoc

RenderDoc is a stand-alone graphics debugging tool.
https://renderdoc.org
MIT License
8.9k stars 1.33k forks source link

Can't locate renderdoc.dll #171

Closed FlorianLuna closed 8 years ago

FlorianLuna commented 8 years ago

Hi,

RenderDoc raises an error while launching an executable for capture. Here is the log I have :

RENDERDOC: [10:51:18] core.cpp( 242) - Log - RenderDoc v0.26 x64 (ebe442d1d8059d4dd076deff8f35978ac580319d-official) loaded in replay application RENDERDOC: [10:51:18] core.cpp( 242) - Log - RenderDoc v0.26 x64 (ebe442d1d8059d4dd076deff8f35978ac580319d-official) loaded in replay application RENDERDOC: [10:51:24] win32_process.cpp( 307) - Log - Injecting renderdoc into process 13388 RENDERDOC: [10:51:24] win32_process.cpp( 164) - Error - Couldn't find module 'renderdoc.dll' among 37 modules RENDERDOC: [10:51:24] win32_process.cpp( 420) - Error - Can't locate renderdoc.dll in remote PID 13388

I tried RenderDoc on nvidia samples and it works fine so I guess there is something wrong with my app. I added the render doc folder to my path as well as the dx dll folder but still I cant get it work.

RenderDoc was working on our project but for reason out of my control we had to change the setup of our project/machine.

Do you have an idea of what would cause such an error ?

Thanks, Florian

baldurk commented 8 years ago

I've seen this bug rarely before but I've never been able to reproduce it locally. What happens is roughly:

  1. RenderDoc launches the process as suspended
  2. RenderDoc injects a function call to LoadLibrary() renderdoc.dll
  3. RenderDoc then enumerates dlls in the target process, locates renderdoc, so that it can inject more function calls to set the options, logfile location, etc.

Step 3 is the one failing - when enumerating dlls, it doesn't find renderdoc.dll. I don't know why that would be so I can only suggest what you could try to investigate.

Do you have any anti-virus or similar that could be blocking the injection? I guess not if the nvidia samples work but it's possible. It could maybe be some strange windows permissions thing, I don't know if there's any process permission/ownership that could block the call.

You could try using the "N seconds delay" option on the capture executable panel. This waits after step 1 for some seconds to allow you to attach a debugger. You could then either breakpoint on LoadLibrary or look at the output window in visual studio, it should list all dlls as they are loaded. You can try this on the nvidia samples to see how it should look when working.

If renderdoc.dll isn't listed at all, then maybe something is wrong with paths or permissions or something, I don't really know. If renderdoc.dll gets listed as loaded then immediately unloaded, this suggests some problem like dll dependency not being found, or an exception that fires during dllmain() when renderdoc is inserting all of its hooks, which you might be able to catch in the debugger but would fail silently when running outside of the debugger. If I had to guess, I would say this is the most likely problem, but I don't know what would cause it.

Sorry I can't suggest anything more. If you have any more questions or you find anything out let me know - you can also discuss on email (baldurk@baldurk.org) if you don't want to post publicly.

moradin commented 8 years ago

If there is a solution to this problem I would love to hear about it too because I ran into this at some point and wasn't able to resolve it.

baldurk commented 8 years ago

@moradin did it just go away on its own eventually?

If anyone is able to repro this and can work with me I'll help investigate it any way I can, I'm a bit hamstrung without being able to look at it myself.

The other theory is just a timing issue or something where the injected function call hasn't completed by the time the modules are enumerated. You could try adding a Sleep() in before the call to FindRemoteDll() in win32_process.cpp. It seems unlikely to be timing related since it happens rarely but for those who do run into it it happens consistently every time.

FlorianLuna commented 8 years ago

Hey, I just read your answers. I'll try your suggestions. Timing is a plausible reason. The machine I have now is far more powerful in term of cpu power. I will download the source and trying that during the day. I tried the 1st suggestions you made and it does not help. I keep you updated Thanks, Florian

baldurk commented 8 years ago

When you attached the debugger did you see renderdoc.dll being loaded? there should be messages something like this:

Windows has triggered a breakpoint in program.exe.

This may be due to a corruption of the heap, which indicates a bug in program.exe or any of the DLLs it has loaded.

This may also be due to the user pressing F12 while program.exe has focus.

The output window may have more diagnostic information.
...
The thread 'DbgUiRemoteBreakin' (0x2984) has exited with code 0 (0x0).
'program.exe': Loaded 'C:\Projects\renderdoc\x64\Profile\renderdoc.dll', Symbols loaded.

In the output window.

FlorianLuna commented 8 years ago

Hi, Thansk for the mail. I built the source and launched again render doc. I hit an assert corresponding to the error. However I dont have the message quoted in your previous mail. Florian

On Tue, Jan 19, 2016 at 12:37 PM Baldur Karlsson notifications@github.com wrote:

When you attached the debugger did you see renderdoc.dll being loaded? there should be messages something like this:

Windows has triggered a breakpoint in program.exe.

This may be due to a corruption of the heap, which indicates a bug in program.exe or any of the DLLs it has loaded.

This may also be due to the user pressing F12 while program.exe has focus.

The output window may have more diagnostic information. ... The thread 'DbgUiRemoteBreakin' (0x2984) has exited with code 0 (0x0). 'program.exe': Loaded 'C:\Projects\renderdoc\x64\Profile\renderdoc.dll', Symbols loaded.

In the output window.

— Reply to this email directly or view it on GitHub https://github.com/baldurk/renderdoc/issues/171#issuecomment-172928575.

FlorianLuna commented 8 years ago

Also I loop on the FindRemoteDLL( "renderdoc") until found and I get stuck in that loop.

On Tue, Jan 19, 2016 at 12:50 PM Florian Luna florian.luna@gmail.com wrote:

Hi, Thansk for the mail. I built the source and launched again render doc. I hit an assert corresponding to the error. However I dont have the message quoted in your previous mail. Florian

On Tue, Jan 19, 2016 at 12:37 PM Baldur Karlsson notifications@github.com wrote:

When you attached the debugger did you see renderdoc.dll being loaded? there should be messages something like this:

Windows has triggered a breakpoint in program.exe.

This may be due to a corruption of the heap, which indicates a bug in program.exe or any of the DLLs it has loaded.

This may also be due to the user pressing F12 while program.exe has focus.

The output window may have more diagnostic information. ... The thread 'DbgUiRemoteBreakin' (0x2984) has exited with code 0 (0x0). 'program.exe': Loaded 'C:\Projects\renderdoc\x64\Profile\renderdoc.dll', Symbols loaded.

In the output window.

— Reply to this email directly or view it on GitHub https://github.com/baldurk/renderdoc/issues/171#issuecomment-172928575.

baldurk commented 8 years ago

I hit an assert corresponding to the error

By default when building profile, error messages will debugbreak. You can disable this in globalconfig.h, comment out #define DEBUGBREAK_ON_ERROR_LOG. Otherwise, you can just continue past - but seeing this error doesn't tell us anything new

However I dont have the message quoted in your previous mail.

Just to clarify - this message will appear in the target process, the program to be captured. You must enable the 'N seconds delay' option and attach to the process before anything happens.

If you still don't see the message at all in the target process, this means the injected LoadLibrary() call is failing - can you check in the InjectDLL() call to see if anything is going wrong? maybe the thread is not created. Also maybe add calls to GetLastError() to see if anything appears.

If there are no errors here at all, could you note the path of renderdoc.dll that is used, and then separately add LoadLibrary() call to the first line of WinMain()/main() in your program? This should be pretty similar, maybe this could print an error and we'll see what goes wrong.

Also I loop on the FindRemoteDLL( "renderdoc") until found and I get stuck in that loop.

This suggests that the LoadLibrary() call is failing, so it isn't timing related - renderdoc.dll will never be found.

FlorianLuna commented 8 years ago

I got the same issue if I attached the debugger to the target process :+1: The program '[15216] .exe' has exited with code -1073741515 (0xc0000135) 'A dependent dll was not found'.

So I guess the LoadLibrary is failing so I'll check what's going on in InjectDll

FlorianLuna commented 8 years ago

The thread is well created in the InjectDll method. But calling GetLastError just after InjectDLL gives me error code 87. CreateToolhelp32Snapshot call in FindRemoteDll will then fail and returns an invalid handle (INVALID_HANDLE_VALUE).

baldurk commented 8 years ago

error code 87 is ERROR_INVALID_PARAMETER. Can you call it inside InjectDLL to see which specific function is causing the error?

CreateToolhelp32Snapshot is allowed to return INVALID_HANDLE_VALUE if e.g. the modules changed while it was enumerating or similar, which is why I have the retry loop if the last error is ERROR_BAD_LENGTH. Does it run out of retries? or is there a different error?

FlorianLuna commented 8 years ago

It doesnt retry in the loop. It breaks at first try due to a ERROR_PARTIAL_COPY (299) error. Regarding the GetLastError in the InjectDll body it's a bit weird. I can not spot any call responsible for the GetLastError. I am only able to get it just at the the end of the method body or after it returns in the InjectIntoProcess. I disabled optimizations ( #pragma optimize("",off) ) in the win32 process file but I have the samw weird result. I am not really familiar with that kind of code so I might miss something there.

baldurk commented 8 years ago

The MSDN docs say that ERROR_PARTIAL_COPY is returned "If the specified process is a 64-bit process and the caller is a 32-bit process". This should have been caught earlier in the code, but just to check is this the case?

Also while you are debugging I'd recommend using profile builds, rather than release - then you shouldn't need any #pragma optimize because optimizations are already disabled in profile mode.

FlorianLuna commented 8 years ago

ok, I switched to Profile. I am a bit surprised about the 32/64 bits issue : I built both renderdoc and the target app using x64 config.

FlorianLuna commented 8 years ago

Ok so the call failing in InjectDLL is VirtualFreeEx

baldurk commented 8 years ago

I just wanted to check there isn't something obvious overlooked, I'm sure there is a different way for the ERROR_PARTIAL_COPY to happen somehow but that's the explanation on MSDN. It seems something is definitely wrong - perhaps the same reason the InjectDLL is failing, causes the ERROR_PARTIAL_COPY.

I'm not sure what else to investigate, it seems like the code is failing for a mysterious reason :(.

I saw this from stackoverflow - "However, it seems like I found my problem. My antivirus has hooked VirtualAllocEx and for some reason it is always making it return false." I'm not sure if that makes sense for you.

It's worth noting that this kind of injection of dlls/remote allocation/etc is very close to what cheats or hacks or viruses do, so I would not be surprised if it gets caught by protections against those things.

FlorianLuna commented 8 years ago

ok. I will double-check that but I dont have the right to shut down tha antivirus. I ll need to wait for IT

FlorianLuna commented 8 years ago

okay so I still have the issue if I disabled it.

baldurk commented 8 years ago

Hmm, I'm running out of ideas of what to check.

So we've found that the InjectDLL is running fine, and there's only an error returned with the VirtualFreeEx which should be after the 'important' work. But the dll isn't loaded in the target process, and the dll doesn't show up trying to be loaded at all.

CreateToolhelp32Snapshot fails but I think this is a side-effect maybe, not the real problem, as InjectDLL has already failed. Also in your original log, there was no error message for CreateToolhelp32Snapshot failing which suggests that in a release build it might be OK.

The only loose end is this:

The program '[15216] .exe' has exited with code -1073741515 (0xc0000135) 'A dependent dll was not found'.

which is still a bit suspicious to me. Can you investigate a bit more what this error is about? And which dependent dll is failing - whether it's one for your exe, or one for another dll. I'd be surprised if it's renderdoc.dll with the dependency missing but it's maybe possible somehow. If the process is not fully creating somehow and shutting down when some dll fails to load maybe this would explain the problems with injecting dlls or creating the snapshot.

baldurk commented 8 years ago

As another test, you could try hacking out all of the injection code. Comment out the call to InjectIntoProcess() in LaunchAndInjectIntoProcess() and remove the if(ret == 0). This should launch the process as normal, but without injecting renderdoc (so you can't capture). If that fails, then there is actually something wrong with just launching the process, not the dll injecting itself.

FlorianLuna commented 8 years ago

Hey, we found out the problem. I copied the d3dcompiler_46.dll in my exe folder and it works fine now. I think the dll was in the build folder before our setup changed. Anyway, thanks for your help and quick answers it was really appreciated !

FlorianLuna commented 8 years ago

But it's weird that I had this kind of issue on the renderdoc dll. But it's fixed doing that.

baldurk commented 8 years ago

Maybe that means that when launched from renderdoc with the working directory == exe directory, the d3dcompiler wasn't found in the path?

When running from visual studio perhaps the working directory was set to your code project directory or something else, so the d3dcompiler was found in a different folder?

FlorianLuna commented 8 years ago

Yep, it has nothing to do with render doc itself. More with the dll path. However if I add the path to that dll in my system path it still cant find it. So it's still a bit weird but it's ok.

RangerWu commented 8 years ago

Happen to find reason that function VirtualFreeEx will cause wrong parameter currently (call GetLastError after the call will see it which mentioned in this thread), with current codes:

VirtualFreeEx(hProcess, remoteMem, sizeof(dllPath), MEM_RELEASE); VirtualFreeEx(hProcess, remoteMem, dataLen, MEM_RELEASE);

According to MSDN, the 3rd parameter should always be 0 if the 4th parameter is MEM_RELEASE, otherwize wrong parameter error will be thrown. Yet this seems no functional impact and not the root cause of this issue.

baldurk commented 8 years ago

Thanks for pointing this out. Like you say it's not a big issue but it's worth tidying up to not muddy the waters.

xiaoxiaowesley commented 5 years ago

I happened to me. I am using renderdoc to debug unity.but it still crash when launch unity.exe. My unity version is 2018.3.12f .And Renderdoc version is 1.2 64bit .I had tried 1.3 and 1.4 . It also crash.

baldurk commented 5 years ago

Please don't comment on 3 year old bugs. If you have a problem open a new issue following the template and including as much information as possible so that I can track and investigate the problem.