darfink / detour-rs

A cross-platform detour library written in Rust
Other
406 stars 75 forks source link

How to go about using a custom calling convention #37

Open FrankvdStam opened 2 years ago

FrankvdStam commented 2 years ago

I'm dealing with a variation of thiscall where the this pointer is passed via EAX instead of ECX. Apparently this is some msvc-only optimization. While the detour is working with "thiscall" from https://github.com/darfink/detour-rs/pull/8, it is preserving ECX while I need EAX.

https://reverseengineering.stackexchange.com/questions/16919/what-compiler-uses-a-calling-convention-that-uses-eax-as-the-first-argument-the

Is there any way that I can tell the lib to use EAX or to control what registers go where? Would like to avoid writing my own asm to fix this.

Awesome library, thanks so much for the hard work!