System.Text.Json 7.0.2 fails to load in a .net6 application.

[mjkkirschner]

Description

We are attempting to reference RestSharp 110.2 which references system.text.json 7.0.2. Our application has many integration scenarios (as a full wpf application, headless service, or as a desktop plugin) and targets net6. We do not currently use deps.json files as they caused our application to not find our own project dependencies during our migration to .net6.

When starting our application we try to use a RestSharp class, this attempts to resolve system.text.json which fails with an error that the assembly cannot be loaded. A reason is not given and the inner exception is null.

Using dotnet-trace I see the following:

HasStack="True" ThreadID="19,208" ProcessorNumber="0" ClrInstanceID="6" AssemblyName="System.Text.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" Stage="ApplicationAssemblies" AssemblyLoadContext="Default" Result="MismatchedAssemblyName" ResultAssemblyName="System.Text.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" ResultAssemblyPath="C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.23\System.Text.Json.dll" ErrorMessage="Requested version 7.0.0.0 is incompatible with found version 6.0.0.0" ActivityID="/#15712/1/225/" 

I have verified in the module window in visual studio that no version of system.text.json is loaded at this time.

In a small test project, we can recreate this issue by removing the deps.json file.

Reproduction Steps

please see: https://github.com/pinzart90/SytemTextDemo/tree/main

this test project sets GenerateDependencyFile to false at build time, setting it to true resolves the issue. But that causes other issues for our application and integration.

We'd like to understand if the behavior we are seeing is a bug or as designed and if there are any workarounds.

Expected behavior

We expect that STJ 7.0 will be loaded in net6 since we reference it and no other earlier version of STJ is loaded. We expect that it will be loaded even if a GenerateDependencyFile is false at build time and we do not have deps.json files.

Actual behavior

The assembly fails to load even when using an assembly resolver.

Regression?

unknown

Known Workarounds

Hoping you can provide some.

Configuration

6.0.415 windows 11 pro x64 no na

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Description We are attempting to reference RestSharp 110.2 which references system.text.json 7.0.2. Our application has many integration scenarios and targets net6. We do not currently use deps.json files as they caused our application to not find our own project dependencies during our migration to .net6. When starting our application we try to use a RestSharp class, this attempts to resolve system.text.json which fails with an error that the assembly cannot be loaded. Using dotnet-trace I see the following: ``` HasStack="True" ThreadID="19,208" ProcessorNumber="0" ClrInstanceID="6" AssemblyName="System.Text.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" Stage="ApplicationAssemblies" AssemblyLoadContext="Default" Result="MismatchedAssemblyName" ResultAssemblyName="System.Text.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" ResultAssemblyPath="C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.23\System.Text.Json.dll" ErrorMessage="Requested version 7.0.0.0 is incompatible with found version 6.0.0.0" ActivityID="/#15712/1/225/" ``` I have verified in the module window in visual studio that no version of system.text.json is loaded at this time. In a small test project, we can recreate this issue by removing the deps.json file. ### Reproduction Steps please see: https://github.com/pinzart90/SytemTextDemo/tree/main this test project sets `GenerateDependencyFile` to false at build time, setting it to true resolves the issue. But that causes other issues for our application and integration. We'd like to understand if the behavior we are seeing is a bug or as designed and if there are any workarounds. ### Expected behavior We expect that STJ 7.0 will be loaded in net6 since we reference it and no other earlier version of STJ is loaded. We expect that it will be loaded even if a `GenerateDependencyFile` is false at build time and we do not have deps.json files. ### Actual behavior The assembly fails to load even when using an assembly resolver. ### Regression? unknown ### Known Workarounds Hoping you can provide some. ### Configuration 6.0.415 windows 11 pro x64 no na ### Other information _No response_

Author:	mjkkirschner
Assignees:	-
Labels:	`area-System.Text.Json`, `untriaged`
Milestone:	-

[eiriktsarpalis]

My understanding is that *deps.json files are mandatory if your application has other dependencies, and generation should only be turned off if you or some other component in the build specifies a hand-rolled *.deps.json file.

I defer to @ericstj or @ViktorHofer who are probably more knowledgeable on the subject.

[mjkkirschner]

@ericstj @ViktorHofer - the docs say the following: MyApp.deps.json - A list of dependencies, compilation dependencies and version information used to address assembly conflicts. Not technically required, but required to use the servicing or package cache/shared package install features, and to assist during roll-forward scenarios to select the newest version of any assembly that exists more than once in the application and framework(s). If the file is not present, all assemblies in the current folder are used instead.

sounds like this behavior might be expected since we are not using deps.json files?

When we enable them, our exe cannot find any of our assemblies (built csproj dependencies), does using deps.json mean we also have to use dotnet publish?

[ViktorHofer]

I'm not the best person to answer your question but I think @ericstj or someone from the host team should know, cc @agocke

[ghost]

Tagging subscribers to this area: @vitek-karas, @agocke, @vsadov See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Description We are attempting to reference RestSharp 110.2 which references system.text.json 7.0.2. Our application has many integration scenarios (as a full wpf application, headless service, or as a desktop plugin) and targets net6. We do not currently use deps.json files as they caused our application to not find our own project dependencies during our migration to .net6. When starting our application we try to use a RestSharp class, this attempts to resolve system.text.json which fails with an error that the assembly cannot be loaded. A reason is not given and the inner exception is null. Using dotnet-trace I see the following: ``` HasStack="True" ThreadID="19,208" ProcessorNumber="0" ClrInstanceID="6" AssemblyName="System.Text.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" Stage="ApplicationAssemblies" AssemblyLoadContext="Default" Result="MismatchedAssemblyName" ResultAssemblyName="System.Text.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" ResultAssemblyPath="C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.23\System.Text.Json.dll" ErrorMessage="Requested version 7.0.0.0 is incompatible with found version 6.0.0.0" ActivityID="/#15712/1/225/" ``` I have verified in the module window in visual studio that no version of system.text.json is loaded at this time. In a small test project, we can recreate this issue by removing the deps.json file. ### Reproduction Steps please see: https://github.com/pinzart90/SytemTextDemo/tree/main this test project sets `GenerateDependencyFile` to false at build time, setting it to true resolves the issue. But that causes other issues for our application and integration. We'd like to understand if the behavior we are seeing is a bug or as designed and if there are any workarounds. ### Expected behavior We expect that STJ 7.0 will be loaded in net6 since we reference it and no other earlier version of STJ is loaded. We expect that it will be loaded even if a `GenerateDependencyFile` is false at build time and we do not have deps.json files. ### Actual behavior The assembly fails to load even when using an assembly resolver. ### Regression? unknown ### Known Workarounds Hoping you can provide some. ### Configuration 6.0.415 windows 11 pro x64 no na ### Other information _No response_

Author:	mjkkirschner
Assignees:	-
Labels:	`area-AssemblyLoader-coreclr`, `untriaged`
Milestone:	-

[ericstj]

this behavior might be expected since we are not using deps.json files?

I believe so - you end up getting the runtime's deps file (dotnet\shared\Microsoft.NETCore.App\6.0.23\Microsoft.NETCore.App.deps.json) which includes System.Text.Json and that will cause the runtime to prefer that for a bind - though it's a bit surprising it doesn't allow you to instead load your local copy with an AssemblyResolve handler -- perhaps you're not hooking the handler soon enough?

When we enable them, our exe cannot find any of our assemblies (built csproj dependencies)

That sounds unusual. How are you referencing those? Do you see them in the deps file? Do you see them in the application directory?

[mjkkirschner]

@ericstj thanks for your reply!

though it's a bit surprising it doesn't allow you to instead load your local copy with an AssemblyResolve handler -- perhaps you're not hooking the handler soon enough?

That's possible we are doing this: https://github.com/DynamoDS/Dynamo/blob/master/src/DynamoSandbox/Program.cs#L17

the very first thing in main, is we do is:

            AppDomain.CurrentDomain.AssemblyResolve += ResolveAssembly;

I should have clarified in my original post that the failure we see is actually inside our assembly resolve handler where we use Assembly.LoadFrom

That sounds unusual. How are you referencing those? Do you see them in the deps file? Do you see them in the application directory?

1. we reference them with project references.
2. we do see them in the deps file, but they are all version 1.0.0 in the deps file - we have a shared assembly generator that sets the version numbers of all projects to the same version. We have tried forcing all version to 1.0.0 and did not seem to help.
3. yes the deps.json files are in our bin folder (we build and copy everything to bin output folder including nuget package dependencies as in some situations we are loaded as a plugin in another application.

[ericstj]

the very first thing in main, is we do is:

If your main method itself uses types in an assembly, the JIT would load it before your Main() runs.

yes the deps.json files are in our bin folder

What about your project reference DLLs? The thing to examine would be why things are failing when you're using the deps file. See if you can get to the bottom of that and fix it.

[mjkkirschner]

If your main method itself uses types in an assembly, the JIT would load it before your Main() runs.

but we don't use those types in main directly (or anywhere besides the RestSharp dependency we are trying to update), whats strange is that at the time of the failure to load STJ 7 - STJ 6 is not loaded! (at least that is what the module window shows, maybe it's not illustrating everything that is loaded.)

What about your project reference DLLs? The thing to examine would be why things are failing when you're using the deps file. See if you can get to the bottom of that and fix it.

yes, in general our project reference dlls are in bin as well. We are looking into it, definitely not obvious so far.

[pinzart90]

Looks like the problem with using deps file was that we were using CopyLocal=False in most of our project dependencies. Switching it to CopyLocal=true allows our program to startup correctly with the deps files on disk. This also seems to allow us to force the version of System.Text.Json (vers 7) by adding it as a simple nuget package reference in one of our projects.

@ericstj our application is used as an addin (dynamically loaded as part of another host app). Our app also has around 50 assemblies. Do we need deps.json files for all of them. Or just the entrypoint assembly ? We want to make sure that a similar issue does not happen for other assemblies (other framework assemblies, or host app assemblies).

I am also not clear on the load order. Which deps file is used to resolve an assembly reference? Is it the "closest" deps file to the assembly that has the reference ? Something like AddingApp deps.json => HostApp deps.json => Dotnet Framework deps.json

[ericstj]

This is a better question for @agocke, @vitek-karas, @jkotas and others on the loader team.

You can check out the docs here: https://learn.microsoft.com/en-us/dotnet/core/tutorials/creating-app-with-plugin-support https://learn.microsoft.com/en-us/dotnet/core/dependency-loading/understanding-assemblyloadcontext

do we need deps.json files for all of them. Or just the entrypoint assembly

The deps file is needed for each "load context" - think of that as a cohesive graph of dependencies like a single application or single plugin.

When you have an application + plugins you probably want to have a deps file for the application and its dependencies, and allow for a deps file for plugins - so they are free to load their own dependencies without affecting the host or other plugins. The host will need to make sure that any exchange types between the host/plugin are unified. A plugin needs to agree with the host on all the assemblies that make up the host/plugin interface - and ideally those are minimal to allow for plugins to update components like System.Text.Json, Microsoft.Extensions.*, etc without breaking the host/plugin interface. I believe that's covered the docs linked above.

[pinzart90]

I will focus on part of the discussion:

Summary: A dotnet6 project with a nuget references to System.Text.Json 7.0.2 Microsoft.NETCore.App\6.0.23 - has a System.Text.Json vers 6.0.xx

Code example "TestApp":

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <OutputType>exe</OutputType>
    <SelfContained>false</SelfContained>
    <GenerateDependencyFile>false</GenerateDependencyFile>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
    <PackageReference Include="System.Text.Json" Version="7.0.2" />
  </ItemGroup>
</Project>

  public class Test
  {
    public static void Main(string[] args)
    {
      UseNewtonType();//Works just fine. Dependency is loaded from the app folder
      UseTextJsonType();//Could not load file or assembly 'System.Text.Json, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
    }

    public static void UseNewtonType()
    {
      var a = Newtonsoft.Json.Formatting.None;
    }

    public static void UseTextJsonType()
    {
      var a = new System.Text.Json.JsonDocumentOptions();
    }
  }

From the documentation:

https://github.com/dotnet/runtime/blob/main/docs/design/features/host-probing.md#probing-paths The list of probing paths ordered according to their priority.

1. Servicing paths
2. The application (or framework if we're resolving framework assets) directory
3. Framework directories If the app (or framework) has dependencies on frameworks
4. Shared store paths
5. Additional probing paths

https://learn.microsoft.com/en-us/dotnet/core/dependency-loading/default-probing#managed-assembly-default-probing When probing to locate a managed assembly, the AssemblyLoadContext.Default looks in order at: Files matching the AssemblyName.Name in TRUSTED_PLATFORM_ASSEMBLIES (after removing file extensions). Assembly files in APP_PATHS with common file extensions.

Printing System.AppContext.GetData("APP_CONTEXT_DEPS_FILES")

D:\\dev\\test projects\\c#\\TestApp\\bin\\Debug\\net6.0\\win-x64\\TestApp.deps.json;C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\6.0.23\\Microsoft.NETCore.App.deps.json

Printing System.AppContext.GetData("TRUSTED_PLATFORM_ASSEMBLIES")

All the assemblies under dotnet\shared\Microsoft.NETCore.App\6.0.23

My question is: Is this the expected behavior for an app that does not generate a deps.json file ?

Notes: AppDomain.CurrentDomain.AssemblyResolve + Assembly.LoadFrom does not help, the same exception is thrown. Why don't we generate deps.json files? Because the expectation was that (without a deps file) the dependencies would be loaded from the app folder (first).

[vitek-karas]

Let me start with: Please do not delete .deps.json files. The whole system is designed to work with them. The host supports running the app without it, but it's really only a backward compatibility behavior. Even if there are weird quirks in that behavior, we are very unlikely to try to fix those.

Specifically for the ability to carry the same assembly in the app as the one in the framework: The functionality is designed to let the app carry the minimum version it needs, but it is expected to upgrade to a newer version if the framework the app is running on has a newer version. For this, the host needs version information which it gets from the .deps.json. More specifically - it is not designed to "always use the one from the app", that would not work if the app is running on let's say .NET 8, because other parts of the framework would not be compatible with a 7.0 System.Text.Json.

I'll try to answer all the questions, let me know if I missed something. In no particular order:

If I delete .deps.json how come the assembly is resolved from the framework and not from the app's folder.

This is because the host reads version information from the .deps.json file - the host does not read version information from the assembly itself (it would be prohibitively expensive to do that). In the case the app has no .deps.json it has no version information about the file in the app, but it does have version information about the file in the framework (because the framework has .deps.json). In this case it prefers the one with the version information. This is the relevant log from the host when this happens:

Replacing deps entry [repro_app\bin\Debug\net6.0\System.Text.Json.dll, AssemblyVersion:, FileVersion:] with [C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.24\System.Text.Json.dll, AssemblyVersion:6.0.0.0, FileVersion:6.0.2423.51814]

Custom resolution logic as the first thing in Main

This should generally work, with the caveat mentioned by Eric above. Assembly resolution happens at JIT time, and JIT will process entire methods at once, before executing them. So, having a reference to the assembly JITed before the resolution handler is activated can happen. Please also note that JIT can inline methods, so just putting the code into a separate method may not solve this.

As for the code in your repro - I would advice against using Assembly.LoadFrom - honestly pretty much ever, but specifically in this case. Your custom resolution logic tries to help the runtime resolve assemblies as if it were resolved by the runtime itself. For that please use AssemblyLoadContext.Default.LoadFromAssemblyPath - that is as close as you can get to the behavior the runtime has when resolving the assembly on its own. Assembly.LoadFrom tries to implement the behavior ii had in .NET Framework which requires additional complexity, one of it being that there will be another handler for the AssemblyResolve event registered by the runtime, and the ordering might get "interesting".

Why using a custom resolution handler doesn't help with the System.Text.Json problem

The runtime prefers the TPA over other assemblies, specifically the version. If you try to load an assembly by hand which is also in the TPA, the runtime will actually compare the assembly version of the one you're trying to load to the one in the TPA (in your case the former is 7.0.0.0 and the latter is 6.0.0.0). The load will fail if they don't match.

Our app also has around 50 assemblies. Do we need deps.json files for all of them.

You should not even get .deps.json for anything but the main app. By default SDK doesn't copy .deps.json from dependency projects into the app - you should not need to do that either. The host will only consider the .deps.json for the app (so if the app is called MyApp.dll, then it will look at MyApp.deps.json) and then .deps.json files from the frameworks the app depends on (so Microsoft.NETCore.App.deps.json and potentially also ASP.NET or WindowsDesktop if your app uses those). Any other .deps.json files in the directory will be ignored.

How does this work for plugins

That depends on the application which hosts the plugins (to avoid confusion I'll call this the plugin-host here). If the plugin-host implements plugin isolation, it will typically do so via a new AssemblyLoadContext and in that case it may (and typically will) choose to use AssemblyDependencyResolver which will then try to read the .deps.json for the plugin. So if my main plugin assembly is MyPlugin.dll, then the AssemblyDependencyResolver will look for MyPlugin.deps.json. The processing there will be basically the same as if the plugin is an app - that is it will read the MyPlugin.deps.json and then also consider any frameworks. It will NOT consider the plugin-host's .deps.json - intentionally. It is up to the plugin-host to decide the resolution strategy for assemblies which are present in the plugin-host as well as the plugin. As Eric mentioned above, it will typically try to unify the ones used in the interface between the plugin-host and the plugin, and then let the rest be loaded per plugin's request (isolated). But plugin-hosts may choose a different strategy. The assembly resolution and what it needs from the plugin should be part of the plugin-host/plugin contract description to make it clear.

[pinzart90]

Thanks @vitek-karas . That pretty much answers all of the posted questions.

I still have a workflow that I would like to get clarity on. Our application (call it HybridApp) behaves as a host app and a plugin at the same time. HostApp loads HybridApp which loads a PluginApp. My question : How are unresolved references from the plugins going to bubble up to their host apps ? I would expect the unresolved reference to go through the HybridApp resolver first then to the HostApp resolver (depending on who manages to resolve that assembly reference). Or would I need to resolve everything in the Plugin resolver (by having all host resolver references stored here)?

[vitek-karas]

The load contexts (AssemblyLoadContext or ALC for short) do not form hierarchy. If you return null from AssemblyLoadContext.Load it falls back always to the Default ALC.

That said, it's your choice to do so. You can just as easily call the AssemblyLoadContext.LoadFromAssemblyName on the ALC in which the HybridApp is loaded (the ALC for the PluginApp is implemented in the HybridApp, so it can decide to forward that call to the ALC of itself - for example AssemblyLoadContext.GetLoadContext(typeof(PluginLoadContext).Assembly).LoadFromAssemblyName(asmName).

I should note that this will work regardless if the HybridApp is executed standalone or as a plugin.