Plugins loading unmanaged DLLs that depend on other unloaded unmanaged DLLs causes a load failure

[karashiiro]

Say that one ten times fast 🤖

When a plugin loads an unmanaged DLL via P/Invoke, that triggers the custom AssemblyLoadContext implementation, which redirects unmanaged DLL loads (1) to a shadow assembly (2) to enable hot-reloading. The implementation creates a copy of the assembly in a temporary directory (AppData/Local/Temp/... on Windows) and locks that copy for loading instead of the original copy. However, it does not copy any of that assembly's dependencies to the shadow location.

When the assembly attempts to call into a DLL it depends on, that bypasses the custom import logic (as expected, since it's an unmanaged import) and uses the standard LoadLibrary semantics, which can fail to find the required DLLs relative to the new shadow location. This is not an issue if the libraries in question are already installed system-wide, and may not be an issue if the libraries are already loaded as process modules, but if any other libraries are needed that don't fulfill these conditions, the load will fail, causing the plugin to throw an exception at the call site back in managed land.

My use case is bundling libav with a plugin to perform video decoding - I can't use C++/CLI to avoid the indirect load because plugins are loaded as collectible assemblies (more details), so my next best option is to wrap the functionality I need from libav in a simple C++ class, which I can then load directly with a small number of P/Invoke declarations. I would prefer not to directly import from libav, because I would need a dozen or more function declarations and a few dozen structs besides.

I can think of a few workarounds for this:

• Bite the bullet and write a bunch of interop boilerplate for libav to directly import it
• Statically-link against libav so I can directly import my wrapper DLL without indirectly loading the libav DLLs (I tried this and got a wall of linker errors, but I might come back to it)
• Load the indirect dependencies in a module initializer using NativeLibrary.Load

When I rebuild Dalamud with shadow assemblies disabled (just flipping this to false) everything loads correctly, so it would be ideal if either:

1. Shadow assembly loading were optional for specific assemblies, accepting that hot-reloading will break if you do that
2. Shadow assembly loading were changed to somehow detect dependencies and shadow-load them, too

[goaaats]

Can you attach a minimal repro?

[karashiiro]

Here: https://github.com/karashiiro/Dalamud1238 I vendored the native DLLs for convenience, but you can rebuild them, just copy them to the repository root afterwards so that they get copied to the plugin output directory.

The load order should be:

Dalamud1238 -> DynamicLibrary1 -> DynamicLibrary2

This fails with:

2023-06-04 08:05:27.223 -07:00 [ERR] [PLUGINW] Plugin installer threw an unexpected error
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.DllNotFoundException: Unable to load DLL 'C:\Users\karashiiro\AppData\Local\Temp\x0yif3l4.fza\DynamicLibrary1.dll' or one of its dependencies: The specified module could not be found. (0x8007007E)
   at System.Runtime.InteropServices.NativeLibrary.<LoadFromPath>g____PInvoke|1_0(UInt16* libraryName, Int32 throwOnError)
   at System.Runtime.InteropServices.NativeLibrary.Load(String libraryPath)
   at System.Runtime.Loader.AssemblyLoadContext.LoadUnmanagedDllFromPath(String unmanagedDllPath)
   at Dalamud.Plugin.Internal.Loader.ManagedLoadContext.LoadUnmanagedDllFromShadowCopy(String unmanagedDllPath) in C:\goatsoft\companysecrets\dalamud\Plugin\Internal\Loader\ManagedLoadContext.cs:line 361
   at Dalamud.Plugin.Internal.Loader.ManagedLoadContext.LoadUnmanagedDllFromResolvedPath(String unmanagedDllPath, Boolean normalizePath) in C:\goatsoft\companysecrets\dalamud\Plugin\Internal\Loader\ManagedLoadContext.cs:line 352
   at Dalamud.Plugin.Internal.Loader.ManagedLoadContext.LoadUnmanagedDll(String unmanagedDllName) in C:\goatsoft\companysecrets\dalamud\Plugin\Internal\Loader\ManagedLoadContext.cs:line 262
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUnmanagedDll(String unmanagedDllName, IntPtr gchManagedAssemblyLoadContext)
   at Dalamud1238.Dalamud1238.Product(Int32 a, Int32 b)
   at Dalamud1238.Dalamud1238..ctor() in D:\Users\luca\Documents\GitHub\Dalamud1238\Dalamud1238.cs:line 12
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   at System.Reflection.RuntimeConstructorInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at Dalamud.IoC.Internal.ServiceContainer.CreateAsync(Type objectType, Object[] scopedObjects, IServiceScope scope) in C:\goatsoft\companysecrets\dalamud\IoC\Internal\ServiceContainer.cs:line 119
   at Dalamud.Plugin.Internal.Types.LocalPlugin.LoadAsync(PluginLoadReason reason, Boolean reloading) in C:\goatsoft\companysecrets\dalamud\Plugin\Internal\Types\LocalPlugin.cs:line 444

[karashiiro]

At the moment, I've confirmed this works as a workaround:

[ModuleInitializer]
[SuppressMessage("Usage", "CA2255:The \'ModuleInitializer\' attribute should not be used in libraries")]
public static void Initialize()
{
    var assembly = Assembly.GetExecutingAssembly();
    NativeLibrary.Load(ResolvePath("avutil-57.dll"), assembly, DllImportSearchPath.SafeDirectories);
    NativeLibrary.Load(ResolvePath("swresample-4.dll"), assembly, DllImportSearchPath.SafeDirectories);
    NativeLibrary.Load(ResolvePath("swscale-6.dll"), assembly, DllImportSearchPath.SafeDirectories);
    NativeLibrary.Load(ResolvePath("avcodec-59.dll"), assembly, DllImportSearchPath.SafeDirectories);
    NativeLibrary.Load(ResolvePath("avformat-59.dll"), assembly, DllImportSearchPath.SafeDirectories);

    // add each to nativeLibraries

    var assemblyLoadContext = AssemblyLoadContext.GetLoadContext(Assembly.GetExecutingAssembly());
    if (assemblyLoadContext == null) return;
    assemblyLoadContext.Unloading += _ => { nativeLibraries.ForEach(NativeLibrary.Free); };
}

This triggers the custom load context logic, but since it forces all of the dependencies to be loaded first, it doesn't raise any exceptions.

[Minoost]

You can workaround this by inserting a custom dll resolver.

public sealed class Plugin : IDalamudPlugin {
    public Plugin() {
        var currentAlc = AssemblyLoadContext.GetLoadContext(Assembly.GetExecutingAssembly());
        if (currentAlc != null)
        {
            currentAlc.ResolvingUnmanagedDll += CustomAssemblyResolver.ResolveUnmanaged;
        }

        // do things...
    }

    public void Dispose()
    {
        var currentAlc = AssemblyLoadContext.GetLoadContext(Assembly.GetExecutingAssembly());
        if (currentAlc != null)
        {
            currentAlc.ResolvingUnmanagedDll -= CustomAssemblyResolver.ResolveUnmanaged;
        }
    }
}

internal static class CustomAssemblyResolver
{
    public static nint ResolveUnmanaged(Assembly assembly, string library)
    {
        var assemblyLocation = Path.GetDirectoryName(assembly.Location);
        if (assemblyLocation == null)
        {
            return nint.Zero;
        }

        var path = Path.Combine(assemblyLocation, library);
        Log.Information("ResolvePath: {Path}", path);

        if (NativeLibrary.TryLoad(path, out var handle))
        {
            return handle;
        }

        return nint.Zero;
    }
}