Add documentation for the risks for version incompatibilities between msbuild and a plugin dependencies

[YuliiaKovalova]

Description:

We have encountered an issue, when MSBuild was using System.CodeDom net8.0 and the source project had System.CodeDom net6.0 package dependency that was copied in the output folder. During the runtime it caused the exception:

targets/Microsoft.NET.GenerateAssemblyInfo.targets: (199, 5): The "WriteCodeFragment" task failed unexpectedly. 
System.IO.FileLoadException: Could not load file or assembly 'System.CodeDom, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. Could not find or load a specific file. (0x80131621) 
File name: 'System.CodeDom, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' 
---> System.IO.FileLoadException: Could not load file or assembly 'System.CodeDom, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. 
  at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath) 
  at System.Reflection.Assembly.LoadFrom(String assemblyFile) 
  at Microsoft.Build.Locator.MSBuildLocator.<>c__DisplayClass15_0.<RegisterMSBuildPath>g__TryLoadAssembly|3(AssemblyName assemblyName) 
  at Microsoft.Build.Locator.MSBuildLocator.<>c__DisplayClass15_0.<RegisterMSBuildPath>b__2(AssemblyLoadContext _, AssemblyName assemblyName) 
  at System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(AssemblyName assemblyName) 
  at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName) 
  at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr gchManagedAssemblyLoadContext, AssemblyName assemblyName) 
  at Microsoft.Build.Tasks.WriteCodeFragment.GenerateCode(String& extension) 
  at Microsoft.Build.Tasks.WriteCodeFragment.Execute() 
  at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() 
  at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) 
======================================== 
  at MessagePack.Generator.MessagepackCompiler.OpenMSBuildProjectAsync(String projectPath, CancellationToken cancellationToken) in /home/vsts/work/1/s/src/MessagePack.Generator/MessagepackCompiler.cs:line 96 
  at MessagePack.Generator.MessagepackCompiler.RunAsync(String input, String output, String conditionalSymbol, String resolverName, String namespace, Boolean useMapMode, String multipleIfDirectiveOutputSymbols, String[] externalIgnoreTypeNames) in /home/vsts/work/1/s/src/MessagePack.Generator/MessagepackCompiler.cs:line 63 
  at ConsoleAppFramework.WithFilterInvoker.RunCore(ConsoleAppContext _) 
  at ConsoleAppFramework.WithFilterInvoker.InvokeAsync() 
  at ConsoleAppFramework.ConsoleAppEngine.RunCore(Type type, MethodInfo methodInfo, Object instance, String[] args, Int32 argsOffset) 
  at ConsoleAppFramework.ConsoleAppEngine.RunCore(Type type, MethodInfo methodInfo, Object instance, String[] args, Int32 argsOffset) 

Suggested action

Document the versions that MSBuild binds to for each feature-band release and publish it as a part of documentation. The ticket #9312 can be used as a basement for that.

[YuliiaKovalova]

@vitek-karas, this issue can be interesting for you. Before bumping to net8.0, there weren't any compatibility problems between used System.CodeDom net6.0 in the source project and net7.0 in MSBuild.

cc: @baronfel

[vitek-karas]

My guess would be that the problem is caused by:

 <PackageReference Include="System.CodeDom" ExcludeAssets="runtime" />

This brings in the 8.0 version of System.CodeDom but only as a compiler reference, it explicitly excludes it from the output. So when SDK builds this along with MSBuild - it will probably use the MSbuild's version (7.0) in the output. The end result is that you have a package/app which has code which references 8.0, but carries 7.0 as the implementation.

I don't know why the ExcludeAssets was added above.

That said there are other possible reasons, this involves custom ALCs and MSBuild's custom assembly resolution logic, so that might complicate things even more.

[YuliiaKovalova]

cc: @AArnott

[AArnott]

@vitek-karas I added ExcludeAssets above, and it worked, because I don't reference System.CodeDom 8.0. I reference the 6.0 version. That makes it work with MSBuild whether MSBuild provides 6.0, 7.0, or 8.0 of the System.CodeDom dll.

The original failure was because I shipped the 6.0 assembly while MSBuild wanted 8.0. Because I shipped it, .NET loaded the 6.0 version, and refused to load the 8.0 version that MSBuild itself required.

[vitek-karas]

@AArnott oh - sorry, my bad. Thanks for the explanation. I missed the package version part of your PR.

@YuliiaKovalova if you want I can look into this some more, but I would need to know what is the project you're building - is it the message pack executable, or something else?

/cc @elinor-fung

[AArnott]

@vitek-karas for the repo @YuliiaKovalova links to, which I maintain, there are two front-ends that matter:

1. An executable that uses MSBuildLocator.
2. An MSBuild Task.

Both of these front-end projects depend on MessagePack.Generator.Core, a library that depends on System.CodeDom.dll.

[vitek-karas]

I guess this problem then happens when the MSBuild task is loaded into some msbuild execution. It doesn't carry System.CodeDom with it, and instead relies on the one from msbuild. That would explain the behavior above.

I don't know what's the detailed design of allowing msbuild tasks to carry their own versions of some of the msbuild dependencies - like System.CodeDom.

[rainersigwald]

There are two relevant things here:

1. For applications that use the MSBuild API, we need to be able to load MSBuild from the SDK folder, so MSBuildLocator adds a resolver to find assemblies from there. This does not, however, override the runtime behavior that prefers to load assemblies "from next to the application" if they exist there.
2. For MSBuild task plugins, we are (usually) running in an application from the SDK folder. We use AssemblyDependencyResolver to respect a .deps.json if the task provides one which should allow using a different version of one of our dependencies in the task's ALC.

This started with the first situation: mpc.exe shipped CodeDOM v6.0.0.0 but an MSBuild assembly loaded from the SDK directory wanted to load CodeDOM v8.0.0.0. Removing it from the app directory via ExcludeAssets="runtime" cleared that up.

I'm not completely sure why removing the CodeDOM deployment from next to the task caused failures there; the SDK copy should be just as available in that context.