64-bit drwrap mixed-mode support for wrapping and replacing 32-bit functions

[derekbruening]

From bruen...@google.com on July 16, 2014 12:17:57

This issue covers adding support for wrapping and replacing 32-bit functions when using a 64-bit-compiled drwrap in mixed-mode (on Windows for now, until we have issue #1345):

drwrap wrap:

• Need to add code to handle 32-bit calling convention for 64-bit drwrap when the target address to be wrapped is 32-bit code. Arg access is through API routines written in C so this should be straightforward.
  • How does drwrap know whether target function is 32-bit or not? Add a flag to drwrap_wrap_flags_t Is there some other way? dcontext->x86_mode is only updated in DR code (right?) and drwrap is invoked from arbitrary points in the cache.

drwrap replace_native:

• Ditto, but needs to directly marshal args from 32-bit slots into 64-bit slots via generated code, so this is a little trickier than wrapping.
• It's up to the client to handle differences in structure field sizes and layouts.
• Plus pop;jmp instead of ret
• May need to add drwrap_replace_native_ex() that takes in flags

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=1475

[byron-hawkins]

drwrap now supports the common 32-bit calling conventions via flags: xref #1772. The 32-bit flags are available in a 64-bit build, though I don't think it has been tested in mixed mode.