search

martona / mhook

A Windows API hooking library
Other
729 stars 326 forks source link

ANOMALY: Use of REX.w is meaningless (default operand size is 64) #16

Open gsuberland opened 7 years ago

gsuberland commented 7 years ago

I get this error message for each instance of attempting to hook an API. This appears to be the same issue as #11.

[0x7FFE0A04B210] ANOMALY: use of REX.w is meaningless (default operand size is 64)
[0x7FFE0A04BC10] ANOMALY: use of REX.w is meaningless (default operand size is 64)
[0x7FFE0A04B960] ANOMALY: use of REX.w is meaningless (default operand size is 64)

The memory at the failing location appears to be an indirect jump:

48 FF 25 71 9B 05 00    jmp qword ptr [KERNELBASE.VirtualAlloc]

Throwing that sequence of bytes into ODA gives me this:

rex.W jmp QWORD PTR [rip+0x59b71] # 0x00059b78

Which I suspect is the culprit. OSDev's documentation for REX says that the prefix is valid but ignored for the JMP. I do not know why the Microsoft compiler emits this particular prefix, but it's there.

I tried (rather naively) commenting out the check at this line: https://github.com/martona/mhook/blob/master/disasm-lib/disasm_x86.c#L1745

Unfortunately this only makes the message go away, and doesn't result in a successful hook.

I've added a test case for you to try out: testcase.zip

gsuberland commented 7 years ago

I should note that this works perfectly on x86-32 releases from the same compiler.

gsuberland commented 7 years ago

Fixed in pull request #17.

gsuberland commented 1 year ago

Was this fixed by https://github.com/apriorit/mhook/pull/20 ?