CRASH encoding in-heap xmm shadow write on Windows 8.1

[toshipiazza]

Using Dr. Memory nightly build DrMemory-Windows-1.8.16470-0xc3210bc-sfx on Windows8.1 for medium-sized pointer-heavy programs crashes Dr. Memory. No arguments are invoked for drmemory.

Example output:

[program output...]
<Application C:\Users\piazzt\Documents\School\RCOS\DrMemory Testing\hw8\hw8\hw8-
m32.exe (7040).  Dr. Memory internal crash at PC 0x550a9f31.  Please report this
 at http://drmemory.org/issues.  Program aborted.
0xc0000005 0x00000000 0x550a9f31 0x550a9f31 0x00000001 0x00000000
Base: 0x55020000
Registers: eax=0x550fe370 ebx=0x00000000 ecx=0x000000eb edx=0x00000001
        esi=0x19d196f8 edi=0x19d196f8 esp=0x19afed9c ebp=0x19afedb0
        eflags=0x000
version 5.0.16470, custom build
-no_dynamic_options -disasm_mask 8 -logdir 'C:\Users\piazzt\Documents\School\RCO
S\DrMemory Testing\DrMemory\drmemory\logs\dynamorio' -client_lib 'C:\Users\piazz
t\Documents\School\RCOS\DrMemory Testing\DrMemory\bin\release\drmemorylib.dll;0;
-logdir `C:\Users\piazzt\Documents\School\RCOS\DrMemory Testing\DrMemory\drmemor
y\logs` -symcache_dir `C:\Users\piazzt\Documents\School\RCOS\D
0x19afedb0 0x550aa171
0x19afee24 0x550a6639
0x19afee44 0x550645f0
0x19afee74 0x55064970
0x19afeed8 0x55065066
0x19afef00 0x550b34b1
0x19afefa0 0x55064206
0x19afeff4 0x19ac1c1c
0x0028f93c 0x74a43f51
0x0028fb4c 0x74a43ee9
0x0028fb6c 0x74a440fe
0x0028fbc8 0x74a449d4
0x0028fc20 0x74a4499f
0x0028fc48 0x770f9bde
0x0028fc68 0x770f9b58>

Problem does not occur when -light or -leaks_only -no_count_leaks -no_track_allocs used as argument.

[blueelvis]

This is happening with Version 4.2.2833 as well. I tried opening a simple Hello World program written in C++ and on running the executable, the below error message pops up -

---------------------------
Dr. Memory Notice: C:\Users\Blueelvis_RoXXX\Desktop\Workspace.exe(9588)
---------------------------
Application C:\Users\Blueelvis_RoXXX\Desktop\Workspace.exe (9588).  Dr. Memory internal crash  at PC 0x62505152.  Please report this at http://drmemory.org/issues.  Program aborted.
0xc0000005 0x00000000 0x62505152 0x62505152 0x00000000 0x000000c4
Base: 0x62450000
Registers: eax=0x00000000 ebx=0x00000201 ecx=0x00000000 edx=0x00000000
esi=0x00000000 edi=0x1651ee48 esp=0x1651ec00 ebp=0x00000000
eflags=0x000
version 4.2.2833, custom build
-no_dynamic_options -disasm_mask 8 -logdir 'C:\Users\Blueelvis_RoXXX\AppData\Roaming\Dr. Memory\dynamorio' -client_lib 'C:\Program Files (x86)\Dr. Memory\bin\release\drmemorylib.dll;0;-logdir 
`C:\Users\Blueelvis_RoXXX\AppData\Roaming\Dr. Memory` -symcache_dir `C:\Users\Blueelvis_RoXXX\AppData\Roaming\Dr. Memory\symcache` -lib_blacklist C:\Windows*.d?? -resfile 9588 ' -code_api -p
---------------------------

Once I close the executable, the below Results.txt file opens up - Dr. Memory version 1.8.0 build 8 built on Sep 9 2014 16:27:02 Dr. Memory results for pid 9588: "Workspace.exe" Application cmdline: "C:\Users\Blueelvis_RoXXX\Desktop\Workspace.exe" Recorded 108 suppression(s) from default C:\Program Files (x86)\Dr. Memory\bin\suppress-default.txt

Error #1: UNADDRESSABLE ACCESS: reading 0x00402000-0x00402004 4 byte(s)
# 0 COM+_Entry_Point
# 1 KERNEL32.dll!BaseThreadInitThunk +0x23     (0x77287c04 <KERNEL32.dll+0x17c04>)
Note: @0:00:00.130 in thread 12252
Note: instruction: jmp    0x00402000

I will try obtaining a dump file and see what is happening.

[zhaoqin]

Can you run debug build to see if any ASSERT? Some debugging tips can be found at https://github.com/DynamoRIO/drmemory/wiki/Debugging.

[derekbruening]

The crash reported by blueelvis looks more like #1681 as the DR address offset matches (0x62505152 and 0x64605152 from base 0x62450000 and 0x64550000).

For the crash reported by toshipiazza, we do have more information. While I could not reproduce it running the same program (at least, built from the same sources), toshi when running with -dr_debug hit:

 +135  L3              8b 01                mov    (%ecx)[4byte] -> %eax 
 +137  m4 @0x1a9180ec                       <label>
 +137  m4 @0x1a92b314  64 80 3d d0 0e 00 00 cmp    %fs:0x00000ed0[1byte] $0x00 
                       00
 +145  m4 @0x1a9158f0  0f 85 fa ff ff ff    jnz    @0x1a92726c[4byte] 
 +151  m4 @0x1a9133d4  64 8b 1d dc 0e 00 00 mov    %fs:0x00000edc[4byte] -> %ebx 
 +158  m4 @0x1a9159b0  c7 03 00 00 00 00    mov    $0x00000000 -> (%ebx)[4byte] 
 +164  m4 @0x1a92a410                       <label>
 +164  m4 @0x1a726888                       <label>
 +164  m4 @0x1a92c468  eb fe                jmp    @0x1a9164c0[4byte] 
 +166  m4 @0x1a92726c                       <label>
 +166  m4 @0x1a92906c  64 80 3d d9 0e 00 00 cmp    %fs:0x00000ed9[1byte] $0x00 
                       00
 +174  m4 @0x1a929e08  74 fe                jz     @0x1a918b64[4byte] 
 +176  m4 @0x1a914e6c  64 80 3d d0 0e 00 00 cmp    %fs:0x00000ed0[1byte] $0x55 
                       55
 +184  m4 @0x1a92e6c4  75 fe                jnz    @0x1ac714b4[4byte] 
ERROR: Could not find encoding for: mov    $0x00000000 -> %fs:0x00000edc[4byte] 

from:

interp: start_pc = 0x74a681aa
check_thread_vm_area: pc = 0x74a681aa
prepend_entry_to_fraglist: putting fragment @0x74a681aa (shared) on vmarea 0x74a41000-0x74a8d000
  0x74a681aa  8b 01                mov    (%ecx)[4byte] -> %eax 
  0x74a681ac  66 0f 6e c0          movd   %eax -> %xmm0 
  0x74a681b0  f3 0f 12 c0          movsldup %xmm0 -> %xmm0 
  0x74a681b4  66 0f 3a df c0 00    aeskeygenassist %xmm0 $0x00 -> %xmm0 
  0x74a681ba  66 0f 7e c0          movd   %xmm0[4byte] -> %eax 
  0x74a681be  89 02                mov    %eax -> (%edx)[4byte] 
  0x74a681c0  c3                   ret    %esp (%esp)[4byte] -> %esp 
mbr exit target = 0x1a6e2240
end_pc = 0x74a681c1

If I try to repro in asmtest using that exact code sequence, I have no problem. The instru is different: the extra instru in there is from in-heap processing. It's not clear why it thinks that code is in-heap, but I can reproduce the crash if I force that block in asmtest to be in-heap. The code handling that needs to call add_dstx2_shadow_write() to properly handle writing an xmm shadow.

So, I have a fix, but I would still like to know:

1) Why is it in-heap? Which library routine is that? Is it in bcryptPrimitives.dll? The DrMem -verbose 2 log will tell us.

2) Why can't I reproduce it: what is different about our systems

3) Is it related to #1681?

[derekbruening]

The crash itself should be fixed in the latest build (http://build.chromium.org/p/client.drmemory/builds/DrMemory-Windows-1.8.16482-0x0371d69-sfx.exe) but we would still like to understand further in case there are other issues. Please double-check that the latest build does not crash, and also please provide the DrMem -verbose 2 or higher log.

[derekbruening]

With a full log, analyzing why we're in-heap:

  ...
new basic block @0x770cd750 == ntdll.dll!LdrShutdownProcess+0x0
new basic block @0x770cd75c == ntdll.dll!LdrShutdownProcess+0xc
inside heap routine: adding nop-if-mem-unaddr checks
new basic block @0x770cd772 == ntdll.dll!LdrShutdownProcess+0x22
inside heap routine: adding nop-if-mem-unaddr checks
<for rest of run, inside heap routine>

Last callstack before crash:

# 0 KERNELBASE.dll!DeviceIoControl+0x51 (0x76ed2ed9 <KERNELBASE.dll+0x12ed9>) modid:0
# 1 fp=0x0028fa34 parent=0x00000000 KERNELBASE.dll!DeviceIoControl+0x50 (0x76ed2ed9 <KERNELBASE.dll+0x12ed9>) modid:0
# 2 fp=0x0028fa94 parent=0x0028fac0 KERNEL32.dll!DeviceIoControl+0x4a (0x76959a52 <KERNEL32.dll+0x19a52>) modid:0
# 3 fp=0x0028fac0 parent=0x0028fb2c bcryptPrimitives.dll!ProcessPrng         +0x31d5 (0x749f45c1 <bcryptPrimitives.dll+0x45c1>) modid:0
# 4 fp=0x0028fb2c parent=0x0028fb70 bcryptPrimitives.dll!ProcessPrng         +0x33ed (0x749f47d9 <bcryptPrimitives.dll+0x47d9>) modid:0
# 5 fp=0x0028fb70 parent=0x0028fbc8 bcryptPrimitives.dll!ProcessPrng         +0x33bd (0x749f47a9 <bcryptPrimitives.dll+0x47a9>) modid:0
# 6 fp=0x0028fbc8 parent=0x0028fc20 bcryptPrimitives.dll!ProcessPrng         +0x35e8 (0x749f49d4 <bcryptPrimitives.dll+0x49d4>) modid:0
# 7 fp=0x0028fc20 parent=0x0028fc48 bcryptPrimitives.dll!ProcessPrng         +0x35b3 (0x749f499f <bcryptPrimitives.dll+0x499f>) modid:0
# 8 fp=0x0028fc48 parent=0x0028fc68 ntdll.dll!RtlInitializeCriticalSection+0x10d (0x770a9bde <ntdll.dll+0x49bde>) modid:0
# 9 fp=0x0028fc68 parent=0x0028fcb8 ntdll.dll!RtlInitializeCriticalSection+0x87 (0x770a9b58 <ntdll.dll+0x49b58>) modid:0
#10 fp=0x0028fcb8 parent=0x0028fd58 ntdll.dll!RtlReportSilentProcessExit+0x107 (0x770cd5b8 <ntdll.dll+0x6d5b8>) modid:0
#11 fp=0x0028fd58 parent=0x0028fe20 ntdll.dll!RtlExitUserProcess+0x80 (0x770cd661 <ntdll.dll+0x6d661>) modid:0
#12 fp=0x0028fe20 parent=0x0028fe34 KERNEL32.dll!ExitProcess +0x12 (0x76967f77 <KERNEL32.dll+0x27f77>) modid:0
#13 fp=0x0028fe34 parent=0x0028fe40 msvcrt.dll!__crtExitProcess+0x16 (0x76d65736 <msvcrt.dll+0x15736>) modid:0
#14 fp=0x0028fe40 parent=0x0028fe84 msvcrt.dll!_unlockexit+0x74d6 (0x76d65472 <msvcrt.dll+0x15472>) modid:0
#15 fp=0x0028fe84 parent=0x0028fe98 msvcrt.dll!exit+0x10 (0x76d65715 <msvcrt.dll+0x15715>) modid:0
#16 fp=0x0028fe98 parent=0x0028ff68 _fu0___set_invalid_parameter_handler+0x104 [/home/gfortran/gcc-home/workshop/gcc/objdir/x86_64-w64-mingw32/32/libstdc++-v3/include/bits/locale_facets.h:1076+0x40142c] (0x0040142d <hw8-m32-CRASH.exe+0x142d>) modid:0
#17 fp=0x0028ff68 parent=0x0028ff88 mainCRTStartup+0x20 [/home/gfortran/gcc-home/workshop/gcc/objdir/x86_64-w64-mingw32/32/libstdc++-v3/include/bits/locale_facets.h:1075+0x40111e] (0x0040111f <hw8-m32-CRASH.exe+0x111f>) modid:0
#18 fp=0x0028ff88 parent=0x0028ff94 KERNEL32.dll!BaseThreadInitThunk+0xd (0x7695919f <KERNEL32.dll+0x1919f>) modid:0

So it's basicaly just LdrShutdownProcess, which unfortunately we have had to treat as in-heap due to abstraction violations in heap access by the Ldr code. As to why my machines do not hit this: perhaps just different bcryptPrimitives.dll versions or sthg.

[toshipiazza]

Testing on the new build specified by Derek here produces no crash now. The issue is indeed fixed.