darfink / detour-rs

A cross-platform detour library written in Rust
Other
389 stars 71 forks source link

Compatibility issues with lazy_static #25

Closed joluch closed 2 years ago

joluch commented 2 years ago

When using detour-rs in conjunction with lazy_static in a loop something goes very wrong, to the point where nothing in the program works.

When using lazy_static in a loop, e.g. in this case:

use std::collections::HashMap;
use std::sync::Mutex;

lazy_static! {
    pub static ref HASHMAP: Mutex<HashMap<String, i32>> = Mutex::new(HashMap::new());
}

pub fn hashmap_test() {
    let mut hashmap = HASHMAP.lock().unwrap();
    for i in 0..5000 as i32 {
        hashmap.insert(format!("test{}", i).to_string(), i);
    }

    // Print out a random value from the hashmap to confirm it was successful
    println!("{:?}", hashmap.get("test4343"));
}

and then detouring a function, e.g. d3d9 EndScene:

use {
  detour::{Error, GenericDetour},
  std::mem,
  winapi::{shared::d3d9::LPDIRECT3DDEVICE9, um::winnt::HRESULT},
};

type EndScene = extern "stdcall" fn(LPDIRECT3DDEVICE9) -> HRESULT;

static mut END_SCENE_DETOUR: Result<GenericDetour<EndScene>, Error> = Err(Error::NotInitialized);

pub extern "stdcall" fn hk_end_scene(p_device: LPDIRECT3DDEVICE9) -> HRESULT {
  unsafe {
    return match &END_SCENE_DETOUR {
      Ok(f) => f.call(p_device),
      Err(_) => 80004005, //E_FAIL
    };
  }
}

pub fn hook_function(vtable: Vec<*const usize>) {
  unsafe {
    END_SCENE_DETOUR =
      GenericDetour::<EndScene>::new(mem::transmute(*vtable.get(42).unwrap()), hk_end_scene);
    println!("Created Endscene detour");
    match &END_SCENE_DETOUR {
      Ok(o) => {
        o.enable().unwrap();
        println!("Endscene detour enabled");
      }
      _ => {}
    }
  }
}

the whole program failes to execute, and dosen't even allocate a console, like it should:

#![feature(abi_thiscall)]
#![feature(core_intrinsics)]
#![cfg_attr(feature = "const", feature(const_fn, const_vec_new))]

mod d3d9_util;
mod hashmap_test;
mod hook;
mod process;

#[macro_use]
extern crate lazy_static;

use std::thread;

fn init() {
  unsafe {
    winapi::um::consoleapi::AllocConsole();

    hashmap_test::hashmap_test();

    let end_scene = d3d9_util::get_d3d9_vtable().unwrap();

    println!("d3d9Device[42]: {:p}", *end_scene.get(42).unwrap());
    hook::hook_function(end_scene);
  }
}

#[allow(non_snake_case)]
#[no_mangle]
pub unsafe extern "system" fn DllMain(
  _hinst_dll: winapi::shared::minwindef::HINSTANCE,
  reason: u32,
  _: *mut winapi::ctypes::c_void,
) {
  if reason == 1 {
    thread::spawn(init);
  }
}

Both functions function like expected when used seperately but when both are ran, it seems like it fails completely to execute. If the lazy_static HashMap is used outside a loop it functions completely normally.

darfink commented 2 years ago

Can you reproduce this behavior within any executable? I'll try to reproduce this behavior locally, and if possible, try to determine the cause. With that said, albeit unlikely (if it happens deterministically), threads not being synchronized may be the issue.

SK83RJOSH commented 2 years ago

Having made heavy use of detours and lazy statics I'm thinking this should be closed as user error.

Some advice if you're not starting the application in a suspended state then you should manually call suspend on all non-library threads before initialization and resume after. By the looks of this though you're not synchronizing at all since you're not resuming the main thread either... so yes you likely got caught by a synchronization issue here unfortunately.