Hpmason / retour-rs

A cross-platform detour library written in Rust
Other
99 stars 18 forks source link

Support for c_variadic #10

Open NotNite opened 1 year ago

NotNite commented 1 year ago

Using varargs in a detour causes a recursion crash: image

Hpmason commented 1 year ago

I wasn't aware of c_variadics were supported in Rust, but I found that's it's available under a nightly feature. I'll look into adding it. It may require a lot more than just updating the macro, but I'd have to look more into c variadics before I know for sure.

mgaertne commented 11 months ago

As a work-around, using a RawDetour with a C-variadic function works, though brings the drawback of transmuting back the trampoline function back whenever you call it. That said, I would appreciate C-variadic support as well for the two C-variadic detours I have in place in my project.

Hpmason commented 11 months ago

Sorry about that, I have a branch that I was working on adding support. Though, updating the static-detour macro has proved to be difficult since it's a nested macro, which is very hard to debug.

In the meantime, the branch has added trait implementations for c-variadics so it can be used with GenericDetour. You can update your current project to use the latest commit from the branch:

[patch.crates-io]
# a83e075 is the latest commit of the `c-variadics` branch
retour = {git = "https://github.com/Hpmason/retour-rs", rev = "a83e075"}
[dependencies]
# Add the new "c-variadic" feature to enable C variadics
retour = {version = "0.3.0", features = ["static-detour", "c-variadic"]}

I'll try to take another stab at the static-detour macro this week. If anyone else is interested in helping with that, I'm open to any help.

Edit: Updated rev since I forgot to impl Function for all variadics, not just fn(A, ...)s

Hpmason commented 11 months ago

So, I've hit the limitations of Rust's c-variadics. We can't properly implement retour's Function trait for variadic functions. Rust doesn't allow ... to be nested in another type, but we need to use it for Function::Arguments.

image

I'm not really sure if there is a way to properly support variadics atm. We sadly can't use Rust's Fn traits directly since they aren't implemented for unsafe functions, limiting a lot of use cases.