hedge-dev / HMMCodes

Hedge Mod Manager community codes repository
14 stars 16 forks source link

Trying to implement initialization check to unmanaged function crashes game #132

Closed MegAmi24 closed 5 months ago

MegAmi24 commented 5 months ago

I've been working on updating the Bridge.SetGlobalVariables library in Sonic Origins, however I've been running into an issue with unmanaged functions. I've discovered that the code that my ASM hook is hooked into runs every frame, so I tried to implement an initialization check so that the unmanaged function only runs once per time an RSDK game is opened. However, no matter what I try, having a check that I'm 100% confident should work simply crashes the game when the unmanaged function runs with _isInitialised set to true. Even trying to wrap all of my function's code in an initialization check instead of using return results in a crash. I've done some testing and could not determine what the cause of the crash could be; removing the check for _isInitialised or replacing its return with a write to the console log seems to not crash, but of course, that's not what I'm trying to do with my code.

Here's what I currently have for the function. I've been testing this by directly replacing the code in Codes.hmm. Also included is a blank code for loading the library.

Library "Bridge.SetGlobalVariables" by "MegAmi"
{
    #include "Helpers" noemit

    #lib "RSDK"

    using System.Collections.Generic;

    private static bool _isInitialised = false;

    UNMANAGED_FUNCTION(void, SetV4GlobalVariables)
    {
        if (_isInitialised)
            return;

        Console.WriteLine(RSDK.GetCurrentGame() + " v4");
        _isInitialised = true;
        return;
    }

    UNMANAGED_FUNCTION(void, SetV3GlobalVariables)
    {
        if (_isInitialised)
            return;

        Console.WriteLine(RSDK.GetCurrentGame() + " v3");
        _isInitialised = true;
        return;
    }

    [LibraryInitializer]
    public void Init()
    {
        /* v2.0.2: 0x140100414 */
        long sig = ScanSignature
        (
            "\x48\x63\xC1\x48\x8D\x0D\x00\x00\x00\x00\x48\x8B\x8C\xC1\x00\x00\x00\x00\x83\xFA\x03\x0F\x84\x00\x00\x00\x00\x83\xFA\x04\x0F\x84\x00\x00\x00\x00\xB8\xFF\xFF\xFF\xFF\xC3",
            "xxxxxx????xxxx????xxxxx????xxxxx????xxxxxx"
        );

        WriteAsmHook
        (
            $@"
                ; Call the unmanaged function
                push rdi
                mov  rdi, {GET_UNMANAGED_FUNCTION_PTR(SetV3GlobalVariables)}
                call rdi
                pop  rdi
            ",

            (nint)(Memory.ReadEffectiveAddress(sig + 20)),

            HookBehavior.After
        );

        WriteAsmHook
        (
            $@"
                ; Call the unmanaged function
                push rdi
                mov  rdi, {GET_UNMANAGED_FUNCTION_PTR(SetV4GlobalVariables)}
                call rdi
                pop  rdi
            ",

            (nint)(Memory.ReadEffectiveAddress(sig + 29)),

            HookBehavior.After
        );
    }

    [LibraryUpdate]
    public void Update()
    {
        if (RSDK.GetEngineVersion() == 0) // If no game is running
            _isInitialised = false;
    }
}

Code "Load SetGlobalVariables" in by "MegAmi"
//
    #lib "Bridge.SetGlobalVariables"
//
{
    // Blank
}
hyperbx commented 5 months ago

Have you stepped through the code using a debugger like Cheat Engine?

Unmanaged functions are a bit of a hack, as .NET doesn't support the usual calling convention used by x64 (__fastcall), so you might have to preserve any registers before the function gets called yourself and restore them later.

You can check which registers are being affected by placing a breakpoint before and after the call to your unmanaged function and checking which registers have changed post-call. Whichever registers have changed, try pushing them pre-call and popping them afterwards. It's possible code after your call could be relying on the original data in these registers.

An alternative method to what you're trying to do would be to do this check in your ASM instead. For example;


fixed (bool* p_isInitialised = &_isInitialised)
{
    WriteAsmHook
    (
        @$"
            push rdi
            mov  rdi, {(long)p_isInitialised}
            cmp  byte ptr [rdi], 1
            je   exit
            mov  rdi, {GET_UNMANAGED_FUNCTION_PTR(SetV3GlobalVariables)}
            call rdi
        exit:
            pop  rdi
        ", 

        (nint)(Memory.ReadEffectiveAddress(sig + 20)),

        HookBehavior.After
    );
}
MegAmi24 commented 5 months ago

Did a check and there were indeed a bunch of registers getting changed. I'll try the push and pop thing next time I run into something like this, but the alternative method you gave works fine so I'll use that for this. Thanks for the response!