Orleans.Serialization.ReferencedAssemblyHelper.GetApplicationPartAssemblies() does not honor custom assembly load contexts

[MarkusGeigerDev]

I have (for reasons I cannot change) the following project structure:

• grain interfaces and shared models of the grain interfaces are in assembly "Shared" (package reference --> Orleans.Runtime and Orleans.Sdk)
• Grain implementation code is in assembly "Silo" (project reference --> "Shared")
• Service "Svc" hosts the silo and an ASP.NET MVC web api to access the grains via REST (project reference --> "Silo", package ref --> Orleans.Server)

So far so good.

Now, "Svc" dynamically loads a plug-in assembly ("Plugin") from a separate directory using a custom load context. "Plugin" also has a project reference to "Shared" because it uses some of its POCOs, but it is otherwise not Orleans-related and does neither provide nor use any grains. In pre-Orleans-7-times, this was not an issue.

Now with Orleans v7, I get System.IO.FileNotFoundException: 'Could not load file or assembly 'Plugin, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.' at startup.

I was able to track the problem down to Orleans.Serialization.ReferencedAssemblyHelper.GetApplicationPartAssemblies() which tries to load all assemblies listed in the [ApplicationPart] attributes - that works for all assemblies in the app domain except for "Plugin", because GetApplicationPartAssemblies() does not honor the fact that this assembly can only be loaded through it's custom load context.

As an aside: If I understand it correctly, there is no need to add the [ApplicationPart] attibute to the "Plugin" assembly at all, but I have not yet found a way to suppress it either. Some switch to keep the code generator from adding those attributes would also solve my problem, since Orleans.Serialization.ReferencedAssemblyHelper filters assemblies for them.

[ReubenBond]

Is your Svc project also referencing Plugin? If not, how does the code generator know that Plugin exists?

[MarkusGeigerDev]

No, Svc is not referencing Plugin. That's what puzzles me, too. There's only the transitive reference to the code generator through Shared, and the fact that both, Svc and Plugin are referencing Shared directly. Plugin, as I said, is using types from Shared for which serializers have been generated, but it is using them as POCOs, not in "the Orleans way". Is there a way to verbosely log the code generator's activities? Or to attach a debugger?

[MarkusGeigerDev]

Any ideas about this one yet? Is there anything I could try out, e.g. provide you with verbose logging of the code generator stage? If yes, how can I enable verbose logging?

[ReubenBond]

My apologies. If the Svc project doesn't reference Plugin at all, even transitively, then this is confusing to me. To clarify an earlier point you made, Plugin contains no types which Orleans needs to know about, is that correct? No serializable types, for example. Perhaps we need a property to explicitly enable/disable the code generator, eg <OrleansCodeGeneratorEnabled>false<OrleansCodeGeneratorEnabled>

[MarkusGeigerDev]

To clarify an earlier point you made, Plugin contains no types which Orleans needs to know about, is that correct? No serializable types, for example.

Correct. Plugin uses lots of types which happen to have the [GenerateSerializer] attribute set on them, but it doesn't define any. And Plugin does not perform any Orleans stuff at all, it is using the types as POCO arguments in public methods,

Perhaps we need a property to explicitly enable/disable the code generator, eg <OrleansCodeGeneratorEnabled>false<OrleansCodeGeneratorEnabled>

When browsing through the code, I hoped I had found such a "master switch" https://github.com/dotnet/orleans/blob/4778c2df79bfee932d5885949d63c810f22b7d8c/src/Orleans.CodeGenerator.MSBuild/build/Microsoft.Orleans.CodeGenerator.MSBuild.targets#L79 which depends on another build property "DesignTimeBuild" https://github.com/dotnet/orleans/blob/4778c2df79bfee932d5885949d63c810f22b7d8c/src/Orleans.CodeGenerator.MSBuild/build/Microsoft.Orleans.CodeGenerator.MSBuild.targets#L29 so I tried to set <DesignTimeBuild>false</DesignTimeBuild> but that didn't help either.

In the same file, there's this line https://github.com/dotnet/orleans/blob/4778c2df79bfee932d5885949d63c810f22b7d8c/src/Orleans.CodeGenerator.MSBuild/build/Microsoft.Orleans.CodeGenerator.MSBuild.targets#L85) so I tried this <OrleansCodeGenLogLevel>TRACE</OrleansCodeGenLogLevel> in the Plugin.csproj, but I couldn't find any log output.

[ReubenBond]

I'm still stuck on this:

If the Svc project doesn't reference Plugin at all, even transitively, then this is confusing to me.

How is Plugin being pulled into the build? That is the crux of the issue in my view.

[MarkusGeigerDev]

How is Plugin being pulled into the build? That is the crux of the issue in my view.

It is build as an independent project, then dotnet published to a file system path known to Svc and then loaded from there using Type.GetType() in a custom load context that is able to resolve assemblies from that well-known publishing path. Plugin is part of the build solution, but forms a build dependency tree entirely of its own

To prove the point that Plugin does not contain anything interesting, I dissembled the generated metadata provider. It is empty, whereas the corresponding type in Shared has more than a hundred lines

[assembly: global::Orleans.ApplicationPartAttribute("Plugin")]
[assembly: global::Orleans.ApplicationPartAttribute("Orleans.Core.Abstractions")]
[assembly: global::Orleans.ApplicationPartAttribute("Orleans.Serialization")]
[assembly: global::Orleans.ApplicationPartAttribute("Orleans.Core")]
[assembly: global::Orleans.ApplicationPartAttribute("Orleans.Multitenant")]
[assembly: global::Orleans.ApplicationPartAttribute("Orleans.Runtime")]
[assembly: global::Orleans.ApplicationPartAttribute("Orleans.Streaming")]
[assembly: global::Orleans.ApplicationPartAttribute("Shared")]
[assembly: global::Orleans.Serialization.Configuration.TypeManifestProviderAttribute(typeof(OrleansCodeGen.Plugin.Metadata_Plugin))]
namespace OrleansCodeGen.Plugin
{
    using global::Orleans.Serialization.Codecs;
    using global::Orleans.Serialization.GeneratedCodeHelpers;

    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("OrleansCodeGen", "7.0.0.0")]
    internal sealed class Metadata_Plugin : global::Orleans.Serialization.Configuration.ITypeManifestProvider
    {
        public void Configure(global::Orleans.Serialization.Configuration.TypeManifestOptions config)
        {
        }
    }
}

I

[ReubenBond]

It's not the plugin project which I'm concerned about, it's the way that the Svc project seems to be referencing it (possibly transitively).

Otherwise, I don't know how the code generator could be emitting a reference to it. That is the problem, from my perspective. Could you show the relevant csproj files?

[MarkusGeigerDev]

I have uploaded a demo reproduction of the issue to https://github.com/MarkusGeigerDev/OrleansCodeGenTest At the moment, this solution doesn't even contain Svc and yet the Plugin assembly has the following assembly level attributes and the generated metadata provider.

[assembly: CompilationRelaxations(8)]
[assembly: RuntimeCompatibility(WrapNonExceptionThrows = true)]
[assembly: Debuggable(DebuggableAttribute.DebuggingModes.Default | DebuggableAttribute.DebuggingModes.IgnoreSymbolStoreSequencePoints | DebuggableAttribute.DebuggingModes.EnableEditAndContinue | DebuggableAttribute.DebuggingModes.DisableOptimizations)]
[assembly: TargetFramework(".NETCoreApp,Version=v7.0", FrameworkDisplayName = ".NET 7.0")]
[assembly: AssemblyCompany("Plugin")]
[assembly: AssemblyConfiguration("Debug")]
[assembly: AssemblyFileVersion("1.0.0.0")]
[assembly: AssemblyInformationalVersion("1.0.0")]
[assembly: AssemblyProduct("Plugin")]
[assembly: AssemblyTitle("Plugin")]
[assembly: ApplicationPart("Plugin")]
[assembly: ApplicationPart("Orleans.Core.Abstractions")]
[assembly: ApplicationPart("Orleans.Serialization")]
[assembly: ApplicationPart("Orleans.Core")]
[assembly: ApplicationPart("Shared")]
[assembly: TypeManifestProvider(typeof (Metadata_Plugin))]
[assembly: AssemblyVersion("1.0.0.0")]

[module: RefSafetyRules(11)]

I will try to add the Svc project later to also demonstrate the assembly loading issue for which I opened this issue in the first place, however, that is not so easy to port to a simple demo without revealing too much of my customer's code.

[ReubenBond]

yet the Plugin assembly has the following assembly level attributes and the generated metadata provider.

This part is expected: the Plugin project references projects which contain generated code, so it contains attributed which point to those projects. That's by-design. The part I've been focusing on is where Svc tries to load Plugin even though it isn't present

[ugumba]

I think I'm hitting the same issue, in a similar context. My app A uses Assembly.LoadFrom() to load an extension assembly B, which references assembly C. B and C reside in an external folder. B has [ApplicationPart("C")].

GetRelevantAssemblies() finds B since it is already loaded, but fails to resolve C (with Assembly.Load(AssemblyName)).

I can hack it by explicitly loading all referenced assemblies before AddSerialization(). (I'd prefer not to!) Or I can ditch Orleans serialization codegen (and fall back on System.Text.Json).

.NET itself has no problem automatically resolving the referenced assemblies in external folders, so some logic seems to be missing from Orleans' assembly resolution logic.

[ReubenBond]

@ugumba I see, so if Orleans were to get the AssemblyLoadContext of B, then use that to load C, would that work for your case? I opened a PR (which may not be the right change). Are you able to test it against your case? https://github.com/dotnet/orleans/pull/8492 (branch: https://github.com/ReubenBond/orleans/tree/fix/use-assembly-load-context). You can build Orleans packages by running build.cmd from the repo root.