simultaneous 32-bit and 64-bit library support for x64 DR controlling WOW64 app

[derekbruening]

From derek.br...@gmail.com on February 24, 2009 14:14:20

Today 32-bit DR can control the 32-bit parts of a WOW64 app but to see all of the code including the emulation layer we want 64-bit DR able to run the whole mixed-mode app. Some of the capabilities here also apply to Linux mixed-mod apps but those are much, much rarer.

here is my list of cases that will eventually be separately filed here:

• PR 240257: support 32-bit clients on WOW64? how mix 32 and 64 bit code? like Pin, give stream to separate clients? or give 32-bit code to 64-bit client?
• PR 253431: [wow64] simultaneous 32-bit and 64-bit dll support in 64-bit DR
• PR 314367: re-enable x64 DR controlling WOW64 process once it works
• PR 272553: [x64] late injection must switch from kernel32 to ntdll for wow64 children
• PR 271317: preserve cs changes from far ctis and iret
• PR 283895: [x64][correctness][performance] for x86 code use separate x86 ibl tables and compacted or separate tls
• PR 283152: support high bit preservation across mode changes
• PR 284029: [x64] support syscalls in x86 mode TODO: reg_spill_dcontext_offs(reg_id_t reg): /* Use REG_E?? instead of REG_X?? to eventually support 32-bit code spills in
  • mixed 64-bit/32-bit execution. */
• PR 269595: WOW64 context translation failing when at our own post-syscall point
• PR 254193: [x64] inject into different-architecture child: x64 to WOW64, WOW64 to x64 => long-term we'll only support 64-bit-DR in WOW64 following (PR 253431)
• PR 253943: [x64] support sysenter
• PR 255555: [x64] 32-bit drinject options for launching 64-bit exe how know if ilist is 32-bit: instr_get_x86_mode() on each instr, or can assume if 1st then whole is same? shouldn't matter for most ops: IR is rich enough and cross-platform enough

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=49

[derekbruening]

From bruen...@google.com on April 24, 2012 08:16:37

also:

• PR 282576: [x64] generate both 32-bit and 64-bit gencode routines
• PR 279094: [x64] support simultaneous 32-bit and 64-bit exit stubs (fine, separate, coarse) and prefixes
• PR 284029: [x64] support syscalls in x86 mode
• PR 283895: [x64][correctness][performance] for x86 code use separate x86 ibl tables and compacted or separate tls
• PR 271317: far cti must exit code cache to update dcontext_t.x86_mode
• PR 215393 which put in simultaneous PE32 and PE32+ parsing, though should double-check Windows as certainly the Linux code today does not support mixing the two module types

Summary: simultaneous 32-bit and 64-bit library support for x64 DR controlling WOW64 app
Labels: -Priority-Medium Priority-Low OpSys-x64

[derekbruening]

From bruen...@google.com on June 21, 2012 12:38:25

I started filing these separately:

• issue #825 /PR 314367: re-enable x64 DR controlling WOW64 process once it works
• issue #823 /PR 271317: preserve cs changes from far ctis and iret
• issue #824 /PR 283895: [x64][correctness][performance] for x86 code use separate x86 ibl tables and compacted or separate tls
• issue #822 /PR 283152: support high bit preservation across mode changes
• issue #821 /PR 284029: [x64] support syscalls in x86 mode

[derekbruening]

From rnk@google.com on July 18, 2012 08:03:03

After Yang got mixed_mode.c working under -x86_to_x64, I suggested he try Cygwin ls.exe next, since Cygwin is likely to exercise more x86 corner cases and ls.exe is a well-behaved command line app.

However, we currently crash using our x64 mixed mode support when Cygwin modifies itself.

syslog bits:

%edx %esp 0x0000000061001655 29 d0 sub %edx %eax -> %eax 0x0000000061001657 83 c0 07 add $0x00000007 %eax -> %eax 0x000000006100165a 83 ea 0c sub $0x0000000c %edx -> %edx 0x000000006100165d 89 42 01 mov %eax -> 0x01(%edx) 0x0000000061001660 ff e2 jmp %edx end_pc = 0x0000000061001662 exit_branch_type=0x12 bb->exit_target=0x000000001c3c3bc0 emit_fragment: bb use ibl <0x000000001c3c3bc0> exit_branch_type=0x12 target=0x000000001c3c3bc0 l->flags=0x9012 Fragment 15670, tag 0x0000000061001654, flags 0x1400630, 32-bit, shared, size 26: [cygwin1.dll~__assert+0xf4,~getprogname-0xb3c] Entry into F15670(0x0000000061001654).0x000000001c84a580 (32-bit)(shared) ASYNCH intercepted exception in thread 7288 at pc 0x000000001c84a589 exception code = 0x00000000c0000005, ExceptionFlags=0x0000000000000000 record=0x0000000000000000, params=2 PC 0x000000001c84a589 tried to write address 0x000000006117d611 ... check_for_modified_code: exception was write to 0x000000006117d611 check_for_modified_code: seg fault in exec-writable region @0x000000006117d611 found_modified_code: translating 0x000000001c84a589 ... SYSLOG_WARNING: writing to executable region. WARNING: Exec 0x000000006117d000-0x000000006117f000 WE written @0x000000006117d611 by 0x000000001c84a589 == app 0x000000006100165d instr not in region, flushing entire 0x000000006117d000-0x000000006117f000 FLUSH STAGE 1: synch_unlink_priv(thread 7288 flushtime 4): 0x000000006117d000-0x000000006117f000 make_writable: pc 0x000000001c3b2000-0x000000001c3b5000, currently r-x- committed make_unwritable: pc 0x000000001c3b2000-0x000000001c3b5000, currently rwx- committed ASYNCH intercepted exception in thread 7288 at pc 0x00000000152c6977 I haven't fully investigated, I'm just filing so we have a record of it. For now he's going to try translating VS tools like cl.exe and masm.

[derekbruening]

From ya...@google.com on July 23, 2012 15:00:30

Failed to run security-common.selfmod.exe in mixed mode.

In emit_utils.c, IF_X64(unlink_shared_syscall_common(SHARED_GENCODE(true))), pc = code->unlinked_shared_syscall + code->sys_unlink_offs, which is null. Then we try to dereference pc, which causes access violation.

[derekbruening]

From ya...@google.com on July 23, 2012 15:06:24

call stack for the previous failure:

 # Child-SP          RetAddr           Call Site
00 00000000`228bdf50 00000000`152d06d9 dynamorio!unlink_shared_syscall_common+0x9a [c:\src\dynamorio\trunk\core\x86\emit_utils.c @ 6998]
01 00000000`228bdfb0 00000000`150c5fbf dynamorio!unlink_shared_syscall+0x79 [c:\src\dynamorio\trunk\core\x86\emit_utils.c @ 7016]
02 00000000`228bdff0 00000000`150ca527 dynamorio!flush_fragments_synch_unlink_priv+0x90f [c:\src\dynamorio\trunk\core\fragment.c @ 6427]
03 00000000`228be1a0 00000000`15235a2c dynamorio!flush_fragments_in_region_start+0x287 [c:\src\dynamorio\trunk\core\fragment.c @ 6892]
04 00000000`228be250 00000000`1523500b dynamorio!flush_and_remove_executable_vm_area+0x3c [c:\src\dynamorio\trunk\core\vmareas.c @ 6078]
05 00000000`228be2c0 00000000`1538c212 dynamorio!app_memory_protection_change+0x1ebb [c:\src\dynamorio\trunk\core\vmareas.c @ 6533]
06 00000000`228be570 00000000`15383208 dynamorio!presys_ProtectVirtualMemory+0x4f2 [c:\src\dynamorio\trunk\core\win32\syscall.c @ 1925]
07 00000000`228be6f0 00000000`15158da4 dynamorio!pre_system_call+0x22b8 [c:\src\dynamorio\trunk\core\win32\syscall.c @ 2395]
08 00000000`228beb70 00000000`1514c951 dynamorio!handle_system_call+0x734 [c:\src\dynamorio\trunk\core\dispatch.c @ 1770]
09 00000000`228becf0 00000000`151482f9 dynamorio!dispatch_enter_dynamorio+0x10b1 [c:\src\dynamorio\trunk\core\dispatch.c @ 748]
0a 00000000`228bee70 00000000`2281313a dynamorio!dispatch+0x19 [c:\src\dynamorio\trunk\core\dispatch.c @ 143]
0b 00000000`228befe0 00000000`22843700 0x2281313a
0c 00000000`228befe8 abababab`abababab 0x22843700
0d 00000000`228beff0 00000000`00000001 0xabababab`abababab
0e 00000000`228beff8 00000000`00000000 0x1

[derekbruening]

We never got as far as running clients: for clients there are issues with static things like DR_REG_X* defines. For that, xref inside DR's IBL generation:

https://github.com/DynamoRIO/dynamorio/blob/master/core/arch/x86/emit_utils.c#L3072

        /* Rather than complicating the REG_X* defines used above we have a post-pass
         * that shrinks all the registers and all the INTPTR immeds.
         * The other two changes we need are performed up above:
         *   1) cmp top bits to 0 for match
         *   2) no trace_cmp entry points
         * Note that we're punting on PR 283152: we go ahead and clobber the top bits
         * of all our scratch registers.
         */

PR 283152 == #822