AssemblyLoadContext is unloading or was already unloaded.

Seabizkit commented 2 years ago

Description

My code fails to run when deployed, seem to work when running through VS

Details I have a plugin which im using AssemblyLoadContext and have marked as isCollectible: true It all seem to work, but does not work when deployed.

every project is on .net 6 to ensure no other variables when trying to debug this AssemblyLoadContext and usage.

-Web ---- Hosts - BackgroundProcessor -----------Kicks off Plugin code

Only code share with plugin is Component.Facade Lib which has no other references.(it works in VS).

Fails when trying to run plugin code. I deploy this with self container and targeting Arm32 Release mode, a PI4 with raspberry OS (32bit)

update: confirmed works in Debug but not Release in VS

Exception message: Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. An operation is not legal in the current state. (0x80131509)

Exception inner: Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. An operation is not legal in the current state. (0x80131509) AssemblyLoadContext is unloading or was already unloaded.

Plugin has:

my guess the above is this is the issue, but how are you meant to handle this?

the class which kicks off the plugin stuff is TaskComponentProcessor

inside there I have

 var pluginInfo = await _taskComponentHelper.EnsureLocalPluginAsync(item.BuilderComponentId.Value);
  var pluginLocation = Path.GetFullPath(Path.Combine(pluginInfo.Path, pluginInfo.Name));
  var loadContext = new PluginLoadContext(pluginLocation);
  var componentUI = loadContext.GetImplementations<BaseComponentUI>().Single();
  var plugin = loadContext.GetImplementations<IParadoxComponent>().Single();

i then call

 var res = await plugin.ExcuteAsync(executingContext, ct);

code for PluginLoadContext

public class PluginLoadContext : AssemblyLoadContext
  {
      private AssemblyDependencyResolver _resolver;
      private string _path;
      public PluginLoadContext() : base(isCollectible: true)
      { }

      public PluginLoadContext(string pluginPath) : base(isCollectible: true)
      {
          _path = pluginPath;
          _resolver = new AssemblyDependencyResolver(pluginPath);
      }

      protected override Assembly Load(AssemblyName assemblyName)
      {
          if (_resolver != null)
          {
              string assemblyPath = _resolver.ResolveAssemblyToPath(assemblyName);
              if (assemblyPath != null)
              {
                  return LoadFromAssemblyPath(assemblyPath);
              }
          }
          return null;
      }

      protected override IntPtr LoadUnmanagedDll(string unmanagedDllName)
      {
          string libraryPath = _resolver.ResolveUnmanagedDllToPath(unmanagedDllName);
          if (libraryPath != null)
          {
              return LoadUnmanagedDllFromPath(libraryPath);
          }

          return IntPtr.Zero;
      }

      public IEnumerable<T> GetImplementations<T>()
      {
          var type = typeof(T);
          var assName = AssemblyName.GetAssemblyName(_path);
          var assembly = LoadFromAssemblyName(assName);
          var types = assembly.GetTypes().ToList();
          var other = new List<Type>();
          if (type.IsAbstract && !type.IsInterface)
          {
              other = types.Where(m => m.IsClass && !m.IsAbstract
                                      && m.IsSubclassOf(type)
                              ).ToList();
          }
          else
          {
              other = types.Where(t => type.IsAssignableFrom(t)).ToList();
          }
          return other
              .Select(t => Activator.CreateInstance(t))
              .Cast<T>();

      }

Statcktrace:

 at Paradox.Component.FtpDownloadPlugin.DIBuilder.Start(ExecutingContext e, Action`1 configureDelegate) 
 at Paradox.Component.FtpDownloadPlugin.Plugin.SetupService(ExecutingContext e, IConfiguration configuration) 
in :\Projects\Paradox.Importer\src\ComponentContainer\ComponentFunctions\Paradox.Component.FtpDownload\Plugin.cs:line 31  at Paradox.Component.FtpDownloadPlugin.Plugin.ExcuteAsync(ExecutingContext ec, CancellationToken ct) 
in :\Projects\Paradox.Importer\src\ComponentContainer\ComponentFunctions\Paradox.Component.FtpDownload\Plugin.cs:line 58  at Paradox.Component.FtpDownloadPlugin.Plugin.ExcuteAsync(ExecutingContext ec, CancellationToken ct) 
in :\Projects\Paradox.Importer\src\ComponentContainer\ComponentFunctions\Paradox.Component.FtpDownload\Plugin.cs:line 93  at Paradox.ComponentCode.TaskComponentProcessor.PreformStepPluginAsync(Int32 actionRequestId, BuilderTaskRun taskRun, BuilderTaskRunItem item, ProgressDto progressDto, IArtifactManger storageManger, CancellationToken ct) 
in C:\Projects\Paradox.Importer\src\ComponentContainer\Paradox.ComponentCode\TaskComponentProcessor.cs:line 343 
at Paradox.ComponentCode.TaskComponentProcessor.PreformStepPluginAsync(Int32 actionRequestId, BuilderTaskRun taskRun, BuilderTaskRunItem item, ProgressDto progressDto, IArtifactManger storageManger, CancellationToken ct) 
in C:\Projects\Paradox.Importer\src\ComponentContainer\Paradox.ComponentCode\TaskComponentProcessor.cs:line 381 
 at Paradox.ComponentCode.TaskComponentProcessor.ExecuteAsync(Int32 actionRequestId, Int32 builderTaskRunId, CancellationToken ct) 
in C:\Projects\Paradox.Importer\src\ComponentContainer\Paradox.ComponentCode\TaskComponentProcessor.cs:line 164

Reproduction Steps

It work when running through VS so, not sure how to debug, Open to suggestions.

in my plugin i have which i used to return a IServiceProvider so i can register any service the plugin may need.

public class DIBuilder
    {
        private const string DefaultEnviroementName = "Development";
        public static IServiceProvider Start(ExecutingContext e, Action<IServiceCollection> configureDelegate)
        {

            var configuration = SetConfigation(e);
            var host = CreateHostBuilder(configuration, e, configureDelegate).Build();
            return host.Services;
        }

        public static IConfiguration SetConfigation(ExecutingContext e)
        {
            //this should be local to the plugin and not the app: Hack
            var directory = Directory.GetCurrentDirectory();

            //  e.Logger.LogDebugAsync(directory);

            var environment = Environment.GetEnvironmentVariable("ASPNET_ENVIROMENT") ??
                      Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ??
                      DefaultEnviroementName;

            //  e.Logger.LogDebugAsync(environment);

            var builder = new ConfigurationBuilder()
                .SetBasePath(directory)
                .AddJsonFile($"appsettings.json", true, true)
                .AddJsonFile($"appsettings.{environment}.json", optional: true, reloadOnChange: true);

            IConfiguration configuration = builder.Build();
            return configuration;
        }

        public static IHostBuilder CreateHostBuilder(IConfiguration configuration, ExecutingContext e,
            Action<IServiceCollection> configureDelegate, string[] args = null)
        {
            return Host.CreateDefaultBuilder(args)
                      .ConfigureServices(configureDelegate);
        }
    }

Expected behavior

Should not through exception, or at least help with how to fix it.

Actual behavior

fails when deployed, giving the exception

Exception message: Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. An operation is not legal in the current state. (0x80131509)

Exception inner: Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. An operation is not legal in the current state. (0x80131509) AssemblyLoadContext is unloading or was already unloaded.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost commented 2 years ago

Tagging subscribers to this area: @vitek-karas, @agocke, @vsadov See info in area-owners.md if you want to be subscribed.

Issue Details

### Description My code **fails** to run when **deployed,** seem to work when running through VS Details I have a plugin which im using AssemblyLoadContext and have marked as `isCollectible: true` It all seem to work, but does not work when deployed. **every project is on .net 6** to ensure no other variables when trying to debug this AssemblyLoadContext and usage. -Web ---- Hosts - BackgroundProcessor -----------Kicks off Plugin code Only code share with plugin is Component.Facade Lib which has no other references.(it works in VS). Fails when trying to run plugin code. I deploy this with **self container** and targeting **Arm32** **Release** mode, a PI4 with raspberry OS (32bit) **update: confirmed works in Debug but not Release in VS** **Exception message:** Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. An operation is not legal in the current state. (0x80131509) **Exception inner:** Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. **An operation is not legal in the current state. (0x80131509) AssemblyLoadContext is unloading or was already unloaded.** Plugin has: ![image](https://user-images.githubusercontent.com/1920090/188303993-b70335f6-3aea-48d0-971a-61db1d4e74e3.png) **my guess the above is this is the issue, but how are you mean to handle this?** the class which kicks off the plugin stuff is TaskComponentProcessor inside there I have ``` var pluginInfo = await _taskComponentHelper.EnsureLocalPluginAsync(item.BuilderComponentId.Value); var pluginLocation = Path.GetFullPath(Path.Combine(pluginInfo.Path, pluginInfo.Name)); var loadContext = new PluginLoadContext(pluginLocation); var componentUI = loadContext.GetImplementations().Single(); var plugin = loadContext.GetImplementations().Single(); ``` i then call ``` var res = await plugin.ExcuteAsync(executingContext, ct); ``` code for PluginLoadContext ``` public class PluginLoadContext : AssemblyLoadContext { private AssemblyDependencyResolver _resolver; private string _path; public PluginLoadContext() : base(isCollectible: true) { } public PluginLoadContext(string pluginPath) : base(isCollectible: true) { _path = pluginPath; _resolver = new AssemblyDependencyResolver(pluginPath); } protected override Assembly Load(AssemblyName assemblyName) { if (_resolver != null) { string assemblyPath = _resolver.ResolveAssemblyToPath(assemblyName); if (assemblyPath != null) { return LoadFromAssemblyPath(assemblyPath); } } return null; } protected override IntPtr LoadUnmanagedDll(string unmanagedDllName) { string libraryPath = _resolver.ResolveUnmanagedDllToPath(unmanagedDllName); if (libraryPath != null) { return LoadUnmanagedDllFromPath(libraryPath); } return IntPtr.Zero; } public IEnumerable GetImplementations() { var type = typeof(T); var assName = AssemblyName.GetAssemblyName(_path); var assembly = LoadFromAssemblyName(assName); var types = assembly.GetTypes().ToList(); var other = new List(); if (type.IsAbstract && !type.IsInterface) { other = types.Where(m => m.IsClass && !m.IsAbstract && m.IsSubclassOf(type) ).ToList(); } else { other = types.Where(t => type.IsAssignableFrom(t)).ToList(); } return other .Select(t => Activator.CreateInstance(t)) .Cast(); } ``` Statcktrace: ``` at Paradox.Component.FtpDownloadPlugin.DIBuilder.Start(ExecutingContext e, Action`1 configureDelegate) at Paradox.Component.FtpDownloadPlugin.Plugin.SetupService(ExecutingContext e, IConfiguration configuration) in :\Projects\Paradox.Importer\src\ComponentContainer\ComponentFunctions\Paradox.Component.FtpDownload\Plugin.cs:line 31 at Paradox.Component.FtpDownloadPlugin.Plugin.ExcuteAsync(ExecutingContext ec, CancellationToken ct) in :\Projects\Paradox.Importer\src\ComponentContainer\ComponentFunctions\Paradox.Component.FtpDownload\Plugin.cs:line 58 at Paradox.Component.FtpDownloadPlugin.Plugin.ExcuteAsync(ExecutingContext ec, CancellationToken ct) in :\Projects\Paradox.Importer\src\ComponentContainer\ComponentFunctions\Paradox.Component.FtpDownload\Plugin.cs:line 93 at Paradox.ComponentCode.TaskComponentProcessor.PreformStepPluginAsync(Int32 actionRequestId, BuilderTaskRun taskRun, BuilderTaskRunItem item, ProgressDto progressDto, IArtifactManger storageManger, CancellationToken ct) in C:\Projects\Paradox.Importer\src\ComponentContainer\Paradox.ComponentCode\TaskComponentProcessor.cs:line 343 at Paradox.ComponentCode.TaskComponentProcessor.PreformStepPluginAsync(Int32 actionRequestId, BuilderTaskRun taskRun, BuilderTaskRunItem item, ProgressDto progressDto, IArtifactManger storageManger, CancellationToken ct) in C:\Projects\Paradox.Importer\src\ComponentContainer\Paradox.ComponentCode\TaskComponentProcessor.cs:line 381 at Paradox.ComponentCode.TaskComponentProcessor.ExecuteAsync(Int32 actionRequestId, Int32 builderTaskRunId, CancellationToken ct) in C:\Projects\Paradox.Importer\src\ComponentContainer\Paradox.ComponentCode\TaskComponentProcessor.cs:line 164 ``` ### Reproduction Steps It work when running through VS so, not sure how to debug, Open to suggestions. in my plugin i have which i used to return a **IServiceProvider** so i can register any service the plugin may need. ``` public class DIBuilder { private const string DefaultEnviroementName = "Development"; public static IServiceProvider Start(ExecutingContext e, Action configureDelegate) { var configuration = SetConfigation(e); var host = CreateHostBuilder(configuration, e, configureDelegate).Build(); return host.Services; } public static IConfiguration SetConfigation(ExecutingContext e) { //this should be local to the plugin and not the app: Hack var directory = Directory.GetCurrentDirectory(); // e.Logger.LogDebugAsync(directory); var environment = Environment.GetEnvironmentVariable("ASPNET_ENVIROMENT") ?? Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? DefaultEnviroementName; // e.Logger.LogDebugAsync(environment); var builder = new ConfigurationBuilder() .SetBasePath(directory) .AddJsonFile($"appsettings.json", true, true) .AddJsonFile($"appsettings.{environment}.json", optional: true, reloadOnChange: true); IConfiguration configuration = builder.Build(); return configuration; } public static IHostBuilder CreateHostBuilder(IConfiguration configuration, ExecutingContext e, Action configureDelegate, string[] args = null) { return Host.CreateDefaultBuilder(args) .ConfigureServices(configureDelegate); } } ``` ### Expected behavior Should not through exception, or at least help with how to fix it. ### Actual behavior fails when deployed, giving the exception **Exception message:** Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. An operation is not legal in the current state. (0x80131509) **Exception inner:** Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. **An operation is not legal in the current state. (0x80131509) AssemblyLoadContext is unloading or was already unloaded.** ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration _No response_ ### Other information _No response_

Author:	Seabizkit
Assignees:	-
Labels:	`area-AssemblyLoader-coreclr`, `untriaged`
Milestone:	-

Seabizkit commented 2 years ago

update i added

   finally {
               loadContext.Unload();
            }

and it appears to be working... like surely not? what is going on here, are some how required to call unload in release..?

PS - before i was not calling Unload(); at all as i was like when it goes out the scope the GC will handle it

vitek-karas commented 2 years ago

It's hard to tell for sure without seeing everything, but based on your description I would guess that it has to do with lifetime of local variables. The runtime is free to unload the context if there are no references to it anywhere. If the only reference is in local variable I could see the JIT releasing the reference at different spots between Debug/Release. It would also explain why adding the finally solves this as that will make sure the local holds onto the context until the end. This is probably a bit more complicated due to the use of async...

You could try to gather a loader trace: https://docs.microsoft.com/en-us/dotnet/core/dependency-loading/collect-details - it might have enough detail to tell the order of things.

vitek-karas commented 2 years ago

Moving this to Future for now since for so far this doesn't look like a bug in the product to me (just complex behavior , which is probably by design for now).

Seabizkit commented 1 year ago

@vitek-karas i could you access to my private repo, further to this i have a memory leak, trying to diagnose but quite difficult.

I believe the memory leek is because unload is not actually unloading.

would you mind taking a look?

vitek-karas commented 1 year ago

You can try the debugging guide here: https://learn.microsoft.com/en-us/dotnet/standard/assembly/unloadability#debug-unloading-issues It describes some of the tools you can use to figure out what prevents the ALC to unload.

Seabizkit commented 1 year ago

Hi @vitek-karas

So been trying to simplify things so i can try see where things are going wrong.

I have test code ....,to try and test my plugin 'Paradox.Component.EmailPlugin'

[TestMethod]
public async Task TestMethod1Async()
{
    var directory = Directory.GetCurrentDirectory();
    var scrFolder = "src";
    var indexOfSrc = directory.IndexOf(scrFolder);
    var srcBaseDirectory = directory.Substring(0, indexOfSrc + scrFolder.Length);
    var pluginFolder = Path.Combine(srcBaseDirectory, "_Plugins");
    var plugins = Directory.GetDirectories(pluginFolder);
    foreach (var item in plugins) 
    {
        var folderName = Path.GetFileName(item);
        var fullpath = Path.Combine(item, folderName + ".dll");
        var loadContext = new PluginLoadContext(fullpath);
        var componentUI = loadContext.GetImplementations<BaseComponentUI>().Single();
        var plugin = loadContext.GetImplementations<IParadoxComponent>().Single();
        await plugin.ExcuteAsync();
    }
}

Paradox.Component.EmailPlugin has no other reference other than the interface project and that project reference no other projects.

the plugin code, I'm registering an IHost inside the plugin so I can get services via ServiceProvider

public class Plugin : IParadoxComponent
{
    public Guid IdentifferName => new Guid("0EFC02C1-E923-4183-B98F-1562237BA62B");
    public string Name => "Paradox email";
    public string ShortName => "Email";
    public string Author => "Paradox";
    public string Description => "Create and send basic email.";
    public string ImageIcon => "fa-envelope";
    public string ImageUrl => "";

    private Action<HostBuilderContext, IServiceCollection> SetupServiceNew()
    {
        Action<HostBuilderContext, IServiceCollection> configureDelegate = (hostBuilder, services) =>
        {
            var config = hostBuilder.Configuration;
            services.AddScoped<IEmailSender, EmailSender>();
            var provider = new FileExtensionContentTypeProvider();
            services.AddSingleton<IMimeMappingHelper>(new MimeMappingHelper(provider));
        };
        return configureDelegate;
    }
    public async Task ExcuteAsync()
    {
        using (var host = DIBuilder.Start(SetupServiceNew()))
        using (var scope = host.Services.CreateScope())
        {
            var services = scope.ServiceProvider;
            var emailSender = services.GetRequiredService<IEmailSender>();
            var mappingHelper = services.GetRequiredService<IMimeMappingHelper>();

            var instance = new ComponentProvider(emailSender, mappingHelper);

        }
    }
}

DIBuilder code

 public class DIBuilder
    {
        private const string DefaultEnviroementName = "Development";

        public static IHost Start(Action<HostBuilderContext, IServiceCollection> configureDelegate)
        {
            try
            {
                var host = CreateHostBuilder(configureDelegate).Build();
                return host;
            }
            catch (Exception ex)
            {
                throw;
            }

        }

        public static IHostBuilder CreateHostBuilder(Action<HostBuilderContext, IServiceCollection> configureDelegate, string[] args = null)
        {
            return Host.CreateDefaultBuilder(args)
                     .ConfigureHostConfiguration(configHost =>
                     {
                         //var global = Directory.GetCurrentDirectory();
                         string LocalPath = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);

                         var environment = Environment.GetEnvironmentVariable("ASPNET_ENVIROMENT") ??
                         Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ??
                         DefaultEnviroementName;

                         configHost
                             .SetBasePath(LocalPath)
                             .AddJsonFile($"appsettings.json", true, reloadOnChange: false)
                             .AddJsonFile($"appsettings.{environment}.json", optional: true, reloadOnChange: false);
                         // .AddDatabaseConfiguration();

                     })
                      .ConfigureServices(configureDelegate);
        }
    }

An exception is being thrown: Could not load file or assembly 'System.Diagnostics.EventLog, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

this make non sense to me, could you assit the files inside of the plugin folder

vitek-karas commented 1 year ago

It's hard to answer this without more details. I would:

Look at the callstack the exception came from - or even better catch it in the debugger to see exactly where it's coming from (which instance of the plugin and so on)
Use the technique described here: https://learn.microsoft.com/en-us/dotnet/core/dependency-loading/collect-details. This will tell you exactly what the runtime did to load the assembly and exactly where it failed.

dotnet / runtime