Multiple native dependencies are not copied to temporary storage when LoadInMemory=true

[nxrighthere]

Describe the bug NuGet packages such as Confluent.SchemaRegistry.Serdes.Avro that contains multiple native dependencies seems missing native libraries in temporary storage with enabled in-memory loading/unloading. Only the main library is being copied which cause DllNotFoundException in the application.

To Reproduce

1. Add Confluent.SchemaRegistry.Serdes.Avro to a library project
2. Publish the library with the following code executed from the entry point of the application:

   
   var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

using (var p = new ProducerBuilder<Null, string>(config).Build()) { try { var dr = await p.ProduceAsync("test-topic", new Message<Null, string> { Value="test" }); Console.WriteLine($"Delivered '{dr.Value}' to '{dr.TopicPartitionOffset}'"); }

catch (ProduceException<Null, string> e) {
    Console.WriteLine($"Delivery failed: {e.Error.Reason}");
}

}

3. Load the library with enabled in-memory loading/unloading:
`var loader = PluginLoader.CreateFromAssemblyFile(assembly, config => { config.DefaultContext = assembliesContext; config.IsUnloadable = true; config.LoadInMemory = true; });`

**Expected behavior**
All relevant native libraries from `*.deps.json` should persist in temporary storage and loaded along with the main library.

[natemcmaster]

What is the TargetFramework of your plugin?

[nxrighthere]

It's net5.0.

[natemcmaster]

I don't see any native dependencies in Confluent.SchemaRegistry.Serdes.Avro, only a netstandard2.0 library. Can you share more details? What is the full exception raised? Which specific dlls are you expecting but not seeing?

[natemcmaster]

Also, what is your csproj and how did you create the plugin folder (VS, dotnet command, etc. and which commands did you use?)

[nxrighthere]

I don't see any native dependencies in Confluent.SchemaRegistry.Serdes.Avro, only a netstandard2.0 library.

They stored in a separate NuGet package librdkafka.redist which is a dependency of Confluent.SchemaRegistry.Serdes.Avro (see dependencies section in Confluent.SchemaRegistry.Serdes.Avro.nuspec).

What is the full exception raised?

System.DllNotFoundException: Failed to load the librdkafka native library.
at Confluent.Kafka.Impl.Librdkafka.TrySetDelegates(List`1 nativeMethodCandidateTypes)
at Confluent.Kafka.Impl.Librdkafka.LoadNetStandardDelegates(String userSpecifiedPath)
at Confluent.Kafka.Impl.Librdkafka.Initialize(String userSpecifiedPath)
at Confluent.Kafka.Producer`2..ctor(ProducerBuilder`2 builder)
at Confluent.Kafka.ProducerBuilder`2.Build()

Which specific dlls are you expecting but not seeing?

librdkafkacpp.dll, libzstd.dll, msvcp120.dll, msvcr120.dll, zlib.dll are missed. Only librdkafka.dll is copied to temporary storage.

Also, what is your csproj and how did you create the plugin folder (VS, dotnet command, etc. and which commands did you use?)

It's a simple library project which includes Confluent.SchemaRegistry.Serdes.Avro, the plugin is published for win-x64 runtime using release configuration:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Library</OutputType>
    <TargetFrameworks>net5.0</TargetFrameworks>
    <Platforms>x64</Platforms>
  </PropertyGroup>
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Confluent.SchemaRegistry.Serdes.Avro" Version="1.7.0" />
  </ItemGroup>

</Project>

We use DotNetCorePlugins without any issues with NuGet packages where only one native library is used as a dependency.

[natemcmaster]

Thanks for the details. This looks to me like you've hit the edge of what we can reasonably support in the DotNetCorePlugins library and the "LoadInMemory" feature. I haven't set up a full repro to verify my hunch, but here it is: it seems this callback is not invoked https://github.com/natemcmaster/DotNetCorePlugins/blob/452f8d306ef17c84c8b02b948e93eb95ef182be3/src/Plugins/Loader/ManagedLoadContext.cs#L210.

This could be because there are native <> native calls happening within the unmanaged dlls, so the AssemblyLoadContext is not called to help load. Or, it could because copying the native assembly to a new place breaks something the .NET runtime runtime.

Either way, it looks like a legit bug. I'm going to mark as "help wanted" since at this point this project is in maintenance mode and am not planning to tackle this bug myself.

[github-actions[bot]]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Please comment if you believe this should remain open, otherwise it will be closed in 14 days. Thank you for your contributions to this project.

[github-actions[bot]]

Closing due to inactivity. If you are looking at this issue in the future and think it should be reopened, please make a commented here and mention natemcmaster so he sees the notification.