search

DynamoRIO / drmemory

Memory Debugger for Windows, Linux, Mac, and Android
Other
2.44k stars 262 forks source link

DrMemory v2.3.0-1 on Win10 x64 crashes #2247

Open SanderBouwhuis opened 4 years ago

SanderBouwhuis commented 4 years ago

I just did a portable install of Dr Memory Windows v2.3.0-1 and ran the 64-bit version of DrMemory.exe.

Dr Memory crashes with this error:

---------------------------
Dr. Memory Notice: E:\Development\Dms v5.0.1\Distribute\64\Launcher.exe(13536)
---------------------------
Application E:\Development\Dms v5.0.1\Distribute\64\Launcher.exe (13536).  Dr. Memory internal crash at PC 0x0000000071187cbb.  Please report this at http://drmemory.org/issues along with the results of running '-debug -dr_debug'.  Program aborted.
0xc0000005 0x00000000 0x0000000071187cbb 0x0000000071187cbb 0x0000000000000001 0x0000000000000025
Base: 0x0000000071000000
Registers: eax=0x0000000000000025 ebx=0x00007ff6d2b92920 ecx=0x0000000000000284 edx=0x0000000000000000
    esi=0x00000207285728a0 edi=0x0000000000000000 esp=0x0000020729f1dd48 ebp=0x0000020729ef9080
    r8 =0x0000020729f1dd80 r9 =0x0000000000000030 r10=0x0000000000000284 r11=0x0000020729f1ddb8
    r12=0x0000020729ef9080 r13=0x0000000000000000 r14=0x0000020729ef9080 r15=0x0000000000000000
    eflags=0x0000000000010202
2.3.0-1-(Feb  6 2020 06:07:09) WinVer=105;Rel=1909;Build=18363;Edition=Professional
-no_dynamic_options -disasm_mask 8 -logdir 'E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\dynamorio' -client_lib 'E:\Downloads\~DrMemory-Windows-2.3.0-1\bin64\release\drmemorylib.dll;0;-logdir `E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs` -symcache_dir `E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs
0x0000020729ef9080 0x0000000000000000
---------------------------
OK   
---------------------------

Here is the output:

E:\Downloads\~DrMemory-Windows-2.3.0-1\bin64>drmemory.exe -- "E:\Development\Dms v5.0.1\Distribute\64\Launcher.exe"
~~Dr.M~~ Dr. Memory version 2.3.0
~~Dr.M~~ Running ""E:\Development\Dms v5.0.1\Distribute\64\Launcher.exe""
~~Dr.M~~ System call information is missing for this operating system: WinVer=105;Rel=1909;Build=18363;Edition=Professional. Restarting to trigger auto-generation of system call information. Re-run with -ignore_kernel to attempt to continue instead.
WARNING: System call information is missing for this operating system version. Attempting to auto-generate system call information...
drsys_find_sysnum_libs: C:\WINDOWS\system32\ntdll.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\kernelbase.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\kernel32.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\gdi32.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\imm32.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\user32.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\win32u.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\ntdll.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\kernelbase.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\kernel32.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\gdi32.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\imm32.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\user32.dll is readable
drsys_find_sysnum_libs: C:\WINDOWS\system32\win32u.dll is readable
Symbol cache directory is "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache"
Fetching symbols for "C:\WINDOWS\system32\ntdll.dll", attempt #0
        Successfully fetched or found symbols at "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\symbols\ntdll.pdb\0C2E19EA1901E9B82E4567D2D21E56D21\ntdll.pdb"
Fetching symbols for "C:\WINDOWS\system32\kernelbase.dll", attempt #0
        Successfully fetched or found symbols at "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\symbols\kernelbase.pdb\1773CB342642E1A70A5E70D8D2A32BD31\kernelbase.pdb"
Fetching symbols for "C:\WINDOWS\system32\kernel32.dll", attempt #0
        Successfully fetched or found symbols at "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\symbols\kernel32.pdb\5A77DE8CE8D58731F0EA38F1C92F48D81\kernel32.pdb"
Fetching symbols for "C:\WINDOWS\system32\gdi32.dll", attempt #0
        Successfully fetched or found symbols at "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\symbols\gdi32.pdb\209AD405837D061EF9D34CBDC009D7711\gdi32.pdb"
Fetching symbols for "C:\WINDOWS\system32\imm32.dll", attempt #0
        Successfully fetched or found symbols at "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\symbols\imm32.pdb\A71334B42E3311D0561A8F034058292D1\imm32.pdb"
Fetching symbols for "C:\WINDOWS\system32\user32.dll", attempt #0
        Successfully fetched or found symbols at "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\symbols\user32.pdb\6EAECFDB6B079835B1A621FADEC37BC91\user32.pdb"
Fetching symbols for "C:\WINDOWS\system32\win32u.dll", attempt #0
        Successfully fetched or found symbols at "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\symbols\win32u.pdb\BC2E49ABE46D2E93B278B4DECFCA62A81\win32u.pdb"
Searching for system calls in "C:\WINDOWS\system32\ntdll.dll"
        Found 465 system calls (0 usercalls) in "C:\WINDOWS\system32\ntdll.dll"
Searching for system calls in "C:\WINDOWS\system32\kernelbase.dll"
        Found 0 system calls (0 usercalls) in "C:\WINDOWS\system32\kernelbase.dll"
Searching for system calls in "C:\WINDOWS\system32\kernel32.dll"
        Found 0 system calls (0 usercalls) in "C:\WINDOWS\system32\kernel32.dll"
Searching for system calls in "C:\WINDOWS\system32\gdi32.dll"
        Found 0 system calls (0 usercalls) in "C:\WINDOWS\system32\gdi32.dll"
Searching for system calls in "C:\WINDOWS\system32\imm32.dll"
        Found 0 system calls (0 usercalls) in "C:\WINDOWS\system32\imm32.dll"
Searching for system calls in "C:\WINDOWS\system32\user32.dll"
        Found 81 system calls (88 usercalls) in "C:\WINDOWS\system32\user32.dll"
Searching for system calls in "C:\WINDOWS\system32\win32u.dll"
        Found 1258 system calls (0 usercalls) in "C:\WINDOWS\system32\win32u.dll"
Writing to "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\syscalls_x64.txt"
Successfully wrote "E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\syscalls_x64.txt"
~~Dr.M~~ Auto-generation succeeded.  Re-launching the application.
~~Dr.M~~ Dr. Memory version 2.3.0
~~Dr.M~~ Running ""E:\Development\Dms v5.0.1\Distribute\64\Launcher.exe""
~~Dr.M~~ Using system call file E:\Downloads\~DrMemory-Windows-2.3.0-1\drmemory\logs\symcache\syscalls_x64.txt

~~Dr.M~~ Fetching 2 symbol files...
~~Dr.M~~ [1/2] Fetching symbols for C:\WINDOWS\System32\msvcp_win.dll
~~Dr.M~~ [2/2] Fetching symbols for C:\WINDOWS\System32\msvcrt.dll
WARNING: drfront_sym_exit failed 6
~~Dr.M~~ Fetched 0 symbol files successfully
~~Dr.M~~ WARNING: application exited with abnormal code 0xffffffff
derekbruening commented 4 years ago

There is not enough information here. There is a bug template which asks for needed information which it looks like you bypassed or deleted -- there is a Github bug where if you aren't signed in until you click to open a bug then it fails to show the template: waiting on them to fix it. I assume that is what happened here.

As you can imagine, without a way to reproduce there is not much that can be done. Please provide a minimized program on which this occurs. Does this happen on hello,world? Is it limited to this application? Did prior Dr. Memory versions work on this application? Did this application work on prior Win10 versions?

As the report says, "Please report this at http://drmemory.org/issues along with the results of running '-debug -dr_debug'". Also, please report the results of each step at https://github.com/DynamoRIO/drmemory/wiki/Debugging#narrowing-down-the-source-of-the-problem.

The crash address is an invalid between-instructions spot in the interception_code_array. Impossible to say how control got there. A reproducer is needed.

0:001> U 0x00000071187cbb
dynamorio!interception_code_array+0x3cbb:
00000000`71187cbb 400000          add     byte ptr [rax],al
00000000`71187cbe 006548          add     byte ptr [rbp+48h],ah
00000000`71187cc1 8b0c2548160000  mov     ecx,dword ptr [1648h]
00000000`71187cc8 e354            jrcxz   dynamorio!interception_code_array+0x3d1e (00000000`71187d1e)

0:001> U 0x00000071187cbb-5
dynamorio!interception_code_array+0x3cb6:
00000000`71187cb6 6548890c2540000000 mov   qword ptr gs:[40h],rcx
00000000`71187cbf 65488b0c2548160000 mov   rcx,qword ptr gs:[1648h]
00000000`71187cc8 e354            jrcxz   dynamorio!interception_code_array+0x3d1e (00000000`71187d1e)
00000000`71187cca 488b89c8020000  mov     rcx,qword ptr [rcx+2C8h]
<...>
00000000`71187e39 e8326af2ff      call    dynamorio!syscall_while_native (00000000`710ae870)
SanderBouwhuis commented 4 years ago

Ok, I ran drmemory like this: drmemory.exe -dr_debug -- "E:\Development\Dms v5.0.0\Distribute\64\Launcher.exe"

In the results.txt files I also added the command-line output and the messagebox text. Is that what you needed?

DrMemory-Launcher.exe.4392.000.zip

derekbruening commented 4 years ago

Is that what you needed?

No, the request was to run with both -debug -dr_debug, to see if any asserts or warnings only present in debug build are reported prior to the crash. It looks like only -debug was run.

I asked a number of questions to provide context on this crash: please see those.

Please also run the "narrowing down" commands which will isolate where this crash is, as requested above.

Again, note that without a reproducer it may not be possible to figure out the culprit.

erich666 commented 4 years ago

I'm having a similar sort of problem, a crash of Dr. Memory on startup of my application. When I run with -debug and -dr_debug, Dr. Memory didn't crash, at least on the x64 version of my program. But then running a second time with both options on, it did crash.

Please also run the "narrowing down" commands which will isolate where this crash is, as requested above.

Do I run each of these five configurations shown here in conjunction with -debug -dr_debug? The instructions are not clear - clarifying this for us newbies would be good, if you want our help. I'll assume I run with all of these options on.

Running with -debug -dr_debug -no_count_leaks gets me further - my program now starts up, showing the window and controls. I pick an operation. To say it's "slow" after this point gives new meaning to the word - I'd go with "glacially slow." I waited 20+ minutes, but nothing seemed to happen after that. I then tried running again from the start with these same options and immediately get the meta-instr faulted? warning:

crash 1

crash 2

I've now spent an hour and a half on trying to get a proper log file for you to debug your application with, and it's not going well... I've spent enough time poking, so will attach the whole kit and caboodle of log files and some of the screen messages I've seen with all these failures. I hope these help.

I understand how useful it is to get good debugging information, as I also run a popular open source project (which I was trying to use Dr. Memory on). I'm happy to try again if you give me exacting directions (I already tried "follow the debugging page" and spent enough time doing so). Or you can try it yourself: my project is Mineways and the code is open source and runs on Windows, 32 and 64 bit.

All my debug files: DrMemory_mineways.zip

wes-219design commented 3 years ago

I'm also interested about how to resolve this issue. Here's the tail end of my console output:

~~Dr.M~~ # 0 NvCameraWhitelisting64.dll!AnselEnableCheck+0x43f50  (0x00007ffd514691e0 <NvCameraWhitelisting64.dll+0x491e0>)
~~Dr.M~~ # 1 NvCameraWhitelisting64.dll!AnselEnableCheck+0x3b692  (0x00007ffd51460923 <NvCameraWhitelisting64.dll+0x40923>)
~~Dr.M~~ # 2 NvCameraWhitelisting64.dll!AnselEnableCheck+0x3aa6b  (0x00007ffd5145fcfc <NvCameraWhitelisting64.dll+0x3fcfc>)
~~Dr.M~~ # 3 NvCameraWhitelisting64.dll!AnselEnableCheck+0x39c70  (0x00007ffd5145ef01 <NvCameraWhitelisting64.dll+0x3ef01>)
~~Dr.M~~ # 4 NvCameraWhitelisting64.dll!AnselEnableCheck+0x3c204  (0x00007ffd51461495 <NvCameraWhitelisting64.dll+0x41495>)
~~Dr.M~~ # 5 NvCameraWhitelisting64.dll!AnselEnableCheck+0xf65a   (0x00007ffd514348eb <NvCameraWhitelisting64.dll+0x148eb>)
~~Dr.M~~ # 6 NvCameraWhitelisting64.dll!AnselEnableCheck+0xfd4d   (0x00007ffd51434fde <NvCameraWhitelisting64.dll+0x14fde>)
~~Dr.M~~ # 7 NvCameraWhitelisting64.dll!AnselEnableCheck+0xf3b3   (0x00007ffd51434644 <NvCameraWhitelisting64.dll+0x14644>)
~~Dr.M~~ # 8 NvCameraWhitelisting64.dll!?         +0x0      (0x00007ffd51423e10 <NvCameraWhitelisting64.dll+0x3e10>)
~~Dr.M~~ # 9 NvCameraWhitelisting64.dll!AnselEnableCheck+0x730    (0x00007ffd514259c1 <NvCameraWhitelisting64.dll+0x59c1>)
~~Dr.M~~ #10 nvwgf2umx.dll!OpenAdapter12          +0xc7c10  (0x00007ffd4d49ac81 <nvwgf2umx.dll+0x25ac81>)
~~Dr.M~~ #11 nvwgf2umx.dll!NVENCODEAPI_Thunk      +0x36ed   (0x00007ffd4d3d1bde <nvwgf2umx.dll+0x191bde>)
~~Dr.M~~ #12 nvwgf2umx.dll!NVENCODEAPI_Thunk      +0x43e5   (0x00007ffd4d3d28d6 <nvwgf2umx.dll+0x1928d6>)
~~Dr.M~~ #13 nvwgf2umx.dll!OpenAdapter10_2        +0x72     (0x00007ffd4d3d2fd3 <nvwgf2umx.dll+0x192fd3>)
~~Dr.M~~ #14 nvldumdx.dll!OpenAdapter12           +0x3b73f  (0x00007ffd7d178470 <nvldumdx.dll+0x48470>)
~~Dr.M~~ #15 d3d11.dll!D3D11CoreCreateDevice      +0x3e4b   (0x00007ffd86c3d34c <d3d11.dll+0x3d34c>)
~~Dr.M~~ #16 d3d11.dll!D3D11CoreCreateDevice      +0x1605   (0x00007ffd86c3ab06 <d3d11.dll+0x3ab06>)
~~Dr.M~~ #17 d3d11.dll!D3D11CoreCreateDevice      +0x1e2d   (0x00007ffd86c3b32e <d3d11.dll+0x3b32e>)
~~Dr.M~~ #18 d3d11.dll!D3D11CoreCreateDevice      +0x573    (0x00007ffd86c39a74 <d3d11.dll+0x39a74>)
~~Dr.M~~ #19 d3d11.dll!D3D11CoreRegisterLayers    +0x863    (0x00007ffd86c37bf4 <d3d11.dll+0x37bf4>)
~~Dr.M~~ Note: @0:36:58.882 in thread 13388
~~Dr.M~~ Note: instruction: vpcmpeqw %ymm2 (%rdx) -> %ymm1
<Application C:\Users\219user\Documents\REDACTED\frontend\build-REDACTED-Desktop_Qt_5_14_0_MSVC2017_64bit-Debug\debug\REDACTED.exe (5532) DynamoRIO usage error : meta-instr faulted?  must set translation field and handle fault!>
<Timeout expired - 1st wait, possible deadlock (or you were debugging)
2.3.18510-0-(Sep 12 2020 00:20:21) WinVer=105;Rel=1909;Build=18363;Edition=Professional
-no_dynamic_options -logdir 'C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\DrMemory-Windows-2.3.18510\drmemory\logs\dynamorio' -client_lib 'C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\DrMemory-Windows-2.3.18510\bin64\debug\drmemorylib.dll;0;-logdir `C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\
C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\DrMemory-Windows-2.3.18510\bin64\debug\drmemorylib.dll=0x00007ff686850000
C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\DrMemory-Windows-2.3.18510\bin64\debug/dbghelp.dll=0x00007ff6870a0000
C:\WINDOWS/system32/msvcrt.dll=0x000001e2d6e20000
C:\WINDOWS/system32/kernel32.dll=0x000001e3587b0000
C:\WINDOWS/system32/KERNELBASE.dll=0x000001e358870000>

This dialog is displayed: image

Then this dialog: image

Then this console output:

2.3.18510-0-(Sep 12 2020 00:20:21) WinVer=105;Rel=1909;Build=18363;Edition=Professional
-no_dynamic_options -logdir 'C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\DrMemory-Windows-2.3.18510\drmemory\logs\dynamorio' -client_lib 'C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\DrMemory-Windows-2.3.18510\bin64\debug\drmemorylib.dll;0;-logdir `C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\
C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\DrMemory-Windows-2.3.18510\bin64\debug\drmemorylib.dll=0x00007ff686850000
C:\Users\219user\Downloads\DrMemory-Windows-2.3.18510\DrMemory-Windows-2.3.18510\bin64\debug/dbghelp.dll=0x00007ff6870a0000
C:\WINDOWS/system32/msvcrt.dll=0x000001e2d6e20000
C:\WINDOWS/system32/kernel32.dll=0x000001e3587b0000
C:\WINDOWS/system32/KERNELBASE.dll=0x000001e358870000>

Then this dialog: image

parolx commented 3 years ago

I get crash of the same portable version also on Windows 10, but with 32 bit application. Everything works perfectly on Windows 7.

Here is stderr output regular

~~Dr.M~~ Dr. Memory version 2.3.0
~~Dr.M~~ Running "vision32d.exe"
~~Dr.M~~ Using system call file d:\Sources\work\bin\test\log\drmem\symcache\syscalls_wow64.txt
<Application d:\Sources\work\bin\vision32d.exe (6020).  Dr. Memory internal crash at PC 0x726365e5.  Please report this at http://drmemory.org/issues along with the results of running '-debug -dr_debug'.  Program aborted.
0xc0000005 0x00000000 0x726365e5 0x726365e5 0x00000001 0x726cac04
Base: 0x72590000
Registers: eax=0x00000001 ebx=0x1afbd7c0 ecx=0x00000001 edx=0x00000000
    esi=0x00000000 edi=0x1afbd7c0 esp=0x0e4af74c ebp=0x0e4af7ec
    eflags=0x000
2.3.0-1-(Feb  6 2020 06:09:03) WinVer=105;Rel=2004;Build=19041;Edition=Enterprise
-no_dynamic_options -disasm_mask 8 -logdir 'd:\Sources\work\bin\test\log\drmem\dynamorio' -client_lib 'd:\Compilers\DrMemory\bin\release\drmemorylib.dll;0;`-leaks_only` `-no_check_heap_mismatch` -logdir `d:\Sources\work\bin\test\log\drmem` -symcache_dir `d:\Sources\work\bin\test\log\drmem\symcache` -lib_blacklist `C:\Win
0x0e4af7ec 0x00000000>
~~Dr.M~~ WARNING: application exited with abnormal code 0xffffffff

and with -verbose 2 -debug -dr_debug

<Starting application d:\Sources\work\bin\vision32d.exe (7604)>
<Running on newer-than-this-build "Microsoft Windows 10-2004 x64">
<Early threads found>
<unknown API-MS-Win pseudo-dll api-ms-win-core-pcw-l1-1-0.dll>
<Initial options = -no_dynamic_options -logdir 'd:\Sources\work\bin\test\log\drmem\dynamorio' -client_lib 'd:\Compilers\DrMemory\bin\debug\drmemorylib.dll;0;`-verbose` `2` `-leaks_only` `-no_check_heap_mismatch` -logdir `d:\Sources\work\bin\test\log\drmem` -symcache_dir `d:\Sources\work\bin\test\log\drmem\symcache` -lib_blacklist `C:\Windows*.d??,C:\Program Files (x86)\Common Files\Microsoft Shared*.d??,C:\Program Files (x86)\Common Files\Microsoft Shared*.d??` -resfile 7604 ' -code_api -probe_api -msgbox_mask 0 -stack_size 56K -disable_traces -no_enable_traces -max_elide_jmp 0 -max_elide_call 0 -no_shared_traces -bb_ibl_targets -bb_single_restore_prefix -no_shared_trace_ibl_routine -no_enable_reset -no_reset_at_switch_to_os_at_vmm_limit -reset_at_vmm_percent_free_limit 0 -no_reset_at_vmm_full -reset_at_commit_free_limit 0K -reset_every_nth_pending 0 -vm_size 262144K -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct -no_aslr_dr >
~~Dr.M~~ Dr. Memory version 2.3.0
~~Dr.M~~ Running "vision32d.exe"
~~Dr.M~~ options are "`-verbose` `2` `-leaks_only` `-no_check_heap_mismatch` -logdir `d:\Sources\work\bin\test\log\drmem` -symcache_dir `d:\Sources\work\bin\test\log\drmem\symcache` -lib_blacklist `C:\Windows*.d??,C:\Program Files (x86)\Common Files\Microsoft Shared*.d??,C:\Program Files (x86)\Common Files\Microsoft Shared*.d??` -resfile 7604 "
~~Dr.M~~ log dir is d:\Sources\work\bin\test\log\drmem\DrMemory-vision32d.exe.7604.000
~~Dr.M~~ Using system call file d:\Sources\work\bin\test\log\drmem\symcache\syscalls_wow64.txt
~~Dr.M~~ WARNING: application exited with abnormal code 0xc0000005

attaching also global log file.

global.7604.log

64 bit application runs, but reports nothing.

~~Dr.M~~ WARNING: unable to locate results file: can't open d:\Sources\work\bin\test\log\drmem/resfile.6424 (code=2).
Dr. Memory failed to start the target application, perhaps due to
interference from invasive security software.
Try disabling other software or running in a virtual machine.