C# Harmony/MonoMod.RuntimeDetour library causes System.DllNotFoundException or System.NotImplementedException

[Frozenreflex]

Godot version

v4.1.1.stable.mono.official.bd6af8e0e

System information

Godot v4.1.1.stable.mono - Linux Mint 21.1 (Vera) - Vulkan (Forward+) - dedicated AMD Radeon RX 6600 (RADV NAVI23) () - AMD Ryzen 7 1800X Eight-Core Processor (16 Threads)

Issue description

Depending on the way that Harmony is set up, two different, but both undesirable, outcomes will occur. With both the latest Harmony pre-release, and MonoMod.RuntimeDetour referenced, as is set up in the reproduction project, any attempt to patch a method will result in an error that looks like this

System.DllNotFoundException: Unable to load shared library '/tmp/mm-exhelper.so.Jvmlua' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 
/tmp/mm-exhelper.so.Jvmlua: undefined symbol: _Unwind_RaiseException

(Jvmlua is normally a completely random string of characters, I somehow got lucky enough to get that) It will also not patch any methods, but will continue to run. I believe this is a bug with Godot because Harmony and MonoMod.RuntimeDetour work fine outside of Godot.

With only the Harmony pre-release referenced, and not MonoMod.RuntimeDetour, it instead throws this

System.NotImplementedException: The method or operation is not implemented.

and still does not patch any methods, and also continues running. This also does not occur outside of Godot.

Steps to reproduce

1. Install the latest Harmony pre-release (and optionally the latest MonoMod.RuntimeDetour) into the project through Nuget
2. Write a patch using Harmony that gets triggered somewhere in the scene
3. Run the player, I don't believe Harmony would work in the editor because of assembly reloading

Minimal reproduction project

GodotHarmonyBug.zip Remove this line in the .csproj to get the second case

    <PackageReference Include="MonoMod.RuntimeDetour" Version="25.0.2" />

[raulsntos]

• Sounds like a duplicate of https://github.com/godotengine/godot/issues/72428.

[Frozenreflex]

That issue does not mention throwing any errors, I do not believe these are the same.

[RedworkDE]

Does not reproduce on windows v4.1.1.stable.mono.official [bd6af8e0e] or v4.2.dev.mono.custom_build [d6d8cb1a1], with our without the direct MonoMod reference.

Please try to create a non-godot projects to check that harmony generally works on your system in the first place.

[nike4613]

This issue is fundamentally Linux-specific, as Windows does not require the exception helper in the first place. When this was first brought up on the MonoMod discord, I had @Frozenreflex try to use Harmony in a standalone application, and that had no issues. It was only after that we figured that this would be the right place for this.

[ErinTorno]

Confirming that this is still an issue as of 4.2.2 and 34b5e8f55cb7d09977074b1486bbdf00d5c16a01. I'm also on Linux Mint. Cherry-picking https://github.com/godotengine/godot/pull/72333 did not have any effect, so it's probably not related to https://github.com/godotengine/godot/issues/72428. Enabling Harmony.DEBUG = true; didn't log any more (useful) info.

Has anyone found a workaround or knows of where to start investigating? Searching for the error doesn't bring up any results other than this issue, so it does appear to be uniquely Godot-related.

[ErinTorno]

So the lib that was having issues was the linux/mac exception helper in MonoMod (see source here). That compile .so is embedded in the MonoMod.Core nuget package, and at runtime it will be copied to a temporary file and loaded using the C dlopen function.

I was able to replicate the issue from a single .c file calling that function. Same results when compiled with both Clang and GCC.

Removing references to _Unwind_RaiseException in that .asm just produced a build that complained about the next EXTERN symbol there. If you change

nasm -f elf64 -Ox exhelper_linux_x86_64.asm -o exhelper_linux_x86_64.o && ld -shared --eh-frame-hdr -z now -x -o exhelper_linux_x86_64.so exhelper_linux_x86_64.o

to

nasm -f elf64 -Ox exhelper_linux_x86_64.asm -o exhelper_linux_x86_64.o && ld -shared --eh-frame-hdr -z now -x -o exhelper_linux_x86_64.so exhelper_linux_x86_64.o -lc -lunwind

(add -lc -lunwind to ld), it fixes the issue (the missing EXTERN refs are all from libc and libunwind).

Then I had to build and publish my version of MonoMod to a local nuget source, and used that and Lib.Harmony.Thin for my Godot .csproj references.

    <PackageReference Include="MonoMod.Core" Version="1.1.2-alpha.dev" />
    <PackageReference Include="Lib.Harmony.Thin" Version="2.3.3" />

Right now I'm lost as to how the original still works when used in a non-Godot dotnet project, but fails in C/C++ projects. I'm assuming the dotnet app is doing some special linking behavior behind the scenes?

Unfortunately, this process will likely made the .so less portable I imagine. Diffing the ELF shows the following after the EXTERNAL symbols in the fix.

libc.so.6 \0 libunwind.so.1 \0 _edata \0 __bss_start \0 _end \0 GLIBC_2.34 \0 GLIBC_2.2.5

exhelper_repro.c.zip

[nike4613]

The issue with passing -lc -lunwind on the link line is that that makes it impossible to build on non-Linux systems. The current approach can create all targets on all OSs. libunwind is also named differently on certain platforms (I think it's part of libc on Musl? I remember something like that.)

By default, it seems that .NET loads libunwind with RTLD_GLOBAL, making its symbols globally available. (We did also actually test the exception helpers against a C++ binary, and it worked, though I may be thinking of macOS, not Linux.) My guess would be that Godot is loading libunwind with RTLD_LOCAL or equivalent instead.

[ErinTorno]

Fortunately exhelper_linux_x86_64.so is only compiled and used on Linux in MonoMod, so that likely wouldn't be an issue. There's a Mac one that shares much of the same code, but it doesn't seem like there are any reports of this being an issue on Mac. Windows doesn't use any exception helper.

Godot doesn't seem to load libunwind with dlopen. It's headers are referenced in some third party code though.

But it looks like if I include the following in a GDExtension register_types.cpp (using that RTLD_GLOBAL flag), Godot can patch successfully with the published HarmonyPatch/MonoMod. So it looks like that's a viable workaround.

#if defined(UNIX_ENABLED)
#include <dlfcn.h>
#include <unwind.h>
#endif
...
void initialize_this_gdextension_module(ModuleInitializationLevel p_level) {
    if (p_level != MODULE_INITIALIZATION_LEVEL_SCENE) {
        return;
    }
#if defined(UNIX_ENABLED)
    void *lib = dlopen("libunwind.so", RTLD_NOW | RTLD_GLOBAL);

    if (lib == nullptr) {
        UtilityFunctions::printerr(dlerror());
    }
#endif
}

[nike4613]

It might be possible for MonoMod to try to rebind libunwind as global when loading the exception helper as well.