Cannot compile queries using nuget package Microsoft.CodeAnalysis 2.0.0

[wangzq]

To repro, create a new query and add nuget package Microsoft.CodeAnalysis which is 2.0.0, add following code:

var workspace = new AdhocWorkspace();

Add missing namespace imports and ensure it works in Linqpad.

Now try to generate an exe with lpless, it will fail to compile since it cannot correctly add the assemblies from the nuget package.

[atifaziz]

I prepared a LINQPad query as you described and named it LinqPadless-Issue-13.linq. Then I compiled it as follows:

lpless -v I:\LINQPad\Queries\LinqPadless-Issue-13.linq

The output from the compilation was:

I:\LINQPad\Queries\LinqPadless-Issue-13.linq
  <Query Kind="Expression">
    <NuGetReference>Microsoft.CodeAnalysis</NuGetReference>
  </Query>
  Packages referenced (1):
    Microsoft.CodeAnalysis
  Packages directory: I:\LINQPad\Queries\packages
  Packages target: .NETFramework,Version=v4.5.2
  Querying latest version of Microsoft.CodeAnalysis
    [INF]   GET https://api.nuget.org/v3/registration1-gz/microsoft.codeanalysis/index.json
    [INF]   GET https://api.nuget.org/v3/registration1-gz/microsoft.codeanalysis/index.json
    [INF]   GET https://www.myget.org/F/raboof/api/v2/FindPackagesById()?id='Microsoft.CodeAnalysis'
    [INF]   OK https://www.myget.org/F/raboof/api/v2/FindPackagesById()?id='Microsoft.CodeAnalysis' 65ms
    [INF]   OK https://api.nuget.org/v3/registration1-gz/microsoft.codeanalysis/index.json 636ms
    [INF]   OK https://api.nuget.org/v3/registration1-gz/microsoft.codeanalysis/index.json 639ms
    Using version 2.0.0
  Resolving references...
    Microsoft.CodeAnalysis 2.0.0
      Microsoft.CodeAnalysis.CSharp.Workspaces 2.0.0
        Microsoft.CodeAnalysis.CSharp 2.0.0
          Microsoft.CodeAnalysis.Common 2.0.0
        Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
      Microsoft.CodeAnalysis.VisualBasic.Workspaces 2.0.0
        Microsoft.CodeAnalysis.VisualBasic 2.0.0
          Microsoft.CodeAnalysis.Common 2.0.0
        Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
  Warning! Packages with no references for .NETFramework,Version=v4.5.2:
    Microsoft.CodeAnalysis.CSharp.Workspaces 2.0.0
    Microsoft.CodeAnalysis.CSharp 2.0.0
    Microsoft.CodeAnalysis.Common 2.0.0
    Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
    Microsoft.CodeAnalysis.VisualBasic.Workspaces 2.0.0
    Microsoft.CodeAnalysis.VisualBasic 2.0.0
    Microsoft.CodeAnalysis.Common 2.0.0
    Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
  "C:\Program Files (x86)\MSBuild\14.0\bin\csc.exe" I:\LINQPad\Queries\LinqPadless-Issue-13.cs /r:System.dll /r:Microsoft.CSharp.dll /r:System.Core.dll /r:System.Data.dll /r:System.Data.Entity.dll /r:System.Transactions.dll /r:System.Xml.dll /r:System.Xml.Linq.dll /r:System.Data.Linq.dll /r:System.Drawing.dll /r:System.Data.DataSetExtensions.dll
    Microsoft (R) Visual C# Compiler version 1.3.1.60616
    Copyright (C) Microsoft Corporation. All rights reserved.

    LinqPadless-Issue-13.cs(23,15): error CS0234: The type or namespace name 'CodeAnalysis' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
C# compiler finished with a non-zero exit code of 1.
lpless.exe Error: 0 : System.Exception: C# compiler finished with a non-zero exit code of 1.
   at LinqPadless.Program.GenerateExecutable(String cscPath, String queryFilePath, String packagesPath, QueryLanguage queryKind, String source, IEnumerable`1 imports, IEnumerable`1 references, IndentingLineWriter writer) in A:\LinqPadless\src\Program.cs:line 673
   at LinqPadless.Program.<>c__DisplayClass16_0.<GenerateExecutable>b__0(String queryFilePath, String packagesPath, QueryLanguage queryKind, String source, IEnumerable`1 imports, IEnumerable`1 references, IndentingLineWriter writer) in A:\LinqPadless\src\Program.cs:line 558
   at LinqPadless.Program.<>c__DisplayClass5_0.<Compiler>b__0(String queryFilePath) in A:\LinqPadless\src\Program.cs:line 280
   at LinqPadless.Program.Wain(IEnumerable`1 args) in A:\LinqPadless\src\Program.cs:line 185
   at LinqPadless.Program.Main(String[] args) in A:\LinqPadless\src\Program.Main.cs:line 33

The compilation was indeed unsuccessful because the C# compiler emitted the following error:

LinqPadless-Issue-13.cs(23,15): error CS0234: The type or namespace name 'CodeAnalysis' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)

I suspect that the following warning has something to do with (i.e. no references found):

  Warning! Packages with no references for .NETFramework,Version=v4.5.2:
    Microsoft.CodeAnalysis.CSharp.Workspaces 2.0.0
    Microsoft.CodeAnalysis.CSharp 2.0.0
    Microsoft.CodeAnalysis.Common 2.0.0
    Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
    Microsoft.CodeAnalysis.VisualBasic.Workspaces 2.0.0
    Microsoft.CodeAnalysis.VisualBasic 2.0.0
    Microsoft.CodeAnalysis.Common 2.0.0
    Microsoft.CodeAnalysis.Workspaces.Common 2.0.0

Since most of the libraries in those packages target .NET Standard 1.3, I also tried invoking lpless with the --fx=netstandard1.3 option but the end-result was the same.

[vcaraulean]

Indeed, without specifying the target, the default compilation target of linqpadless is used, so it's net452. As @atifaziz mentioned, libraries are targeting netstandard1.3 which is not covering 4.5.2. Closest target is 4.6.

Patch from #17 makes correct use of target framework ('-fx' key). Results of testing:

-fx=net462 or -fx=netstandard1.3

• compiles fine
• .cmd gives runtime error about missing types

Error details:

System.AggregateException: One or more errors occurred. ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Composition.TypedParts, Version=1.0.27.0, Cultu
re=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.                                                                  
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable`1 assemblies)                                                                                               
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.get_DefaultHost()                                                                                                              
   at Microsoft.CodeAnalysis.AdhocWorkspace..ctor()                                                                                                                                  
   at Submission#0.<<Initialize>>d__0.MoveNext()                                                                                                                                     
--- End of stack trace from previous location where exception was thrown ---                                                                                                         
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)                                                                                                      
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                 
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.<RunSubmissionsAsync>d__9`1.MoveNext()                                                                                   
--- End of stack trace from previous location where exception was thrown ---                                                                                                         
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)                                                                                                      
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                 
   at Microsoft.CodeAnalysis.Scripting.Script`1.<RunSubmissionsAsync>d__21.MoveNext()                                                                                                
   --- End of inner exception stack trace ---                                                                                                                                        
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)                                                                                                
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunScript(ScriptOptions options, String code, ErrorLogger errorLogger, CancellationToken cancellationToken)         
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunInteractiveCore(ErrorLogger errorLogger)                                                                         
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunInteractive()                                                                                                    
   at Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.Csi.Main(String[] args)                                                                                                        
---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'System.Composition.TypedParts, Version=1.0.27.0, Culture=neutral, PublicKeyToken=b03f5f7f
11d50a3a' or one of its dependencies. The system cannot find the file specified.                                                                                                     
File name: 'System.Composition.TypedParts, Version=1.0.27.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'                                                                       
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable`1 assemblies)                                                                                               
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.get_DefaultHost()                                                                                                              
   at Microsoft.CodeAnalysis.AdhocWorkspace..ctor()                                                                                                                                  
   at Submission#0.<<Initialize>>d__0.MoveNext()                                                                                                                                     
--- End of stack trace from previous location where exception was thrown ---                                                                                                         
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)                                                                                                      
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                 
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.<RunSubmissionsAsync>d__9`1.MoveNext()                                                                                   
--- End of stack trace from previous location where exception was thrown ---                                                                                                         
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)                                                                                                      
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                 
   at Microsoft.CodeAnalysis.Scripting.Script`1.<RunSubmissionsAsync>d__21.MoveNext()                                                                                                

WRN: Assembly binding logging is turned OFF.                                                                                                                                         
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.                                                             
Note: There is some performance penalty associated with assembly bind failure logging.                                                                                               
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].                                                                                      

[vcaraulean]

The type in case is residing in Microsoft.Composition package. But there's #20

[atifaziz]

With #16 and #20 resolved, the compilation now fails due to the following error from the C# compiler:

LinqPadless-Issue-13.cs(22,9): error CS0012: The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Runtime, Version=4.0.20.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

That happens when targeting .NET Framework 4.6. When targeting 4.6.2 instead, compilation passes but running the executable throws the following error:

Unhandled Exception: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeAssembly.get_DefinedTypes()
   at System.Composition.Hosting.ContainerConfiguration.<WithAssemblies>b__0(Assembly a)
   at System.Linq.Enumerable.<SelectManyIterator>d__16`2.MoveNext()
   at System.Composition.TypedParts.TypedPartExportDescriptorProvider..ctor(IEnumerable`1 types, AttributedModelProvider attributeContext)
   at System.Composition.Hosting.ContainerConfiguration.CreateContainer()
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable`1 assemblies)
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.get_DefaultHost()
   at Microsoft.CodeAnalysis.AdhocWorkspace..ctor()
   at UserQuery.Main()

The LoaderExceptions property had one FileLoadException entry whose Message property read:

Could not load file or assembly 'System.Collections.Immutable, Version=1.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

and FusionLog property read:

=== Pre-bind state information ===
LOG: DisplayName = System.Collections.Immutable, Version=1.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
 (Fully-specified)
LOG: Appbase = file:///B:/
LOG: Initial PrivatePath = NULL
Calling assembly : System.Reflection.Metadata, Version=1.4.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: B:\lpless-issue-13.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: The same bind was seen before, and was failed with hr = 0x80131040.

The System.Collections.Immutable package version installed during compilation was 1.3.1. If some assembly in the dependency chain has a reference to version 1.2.0 then the exception is understandable (and because lpless doesn't add any assembly binding redirects).

The actual version of System.Collections.Immutable.dll in the 1.3.1 package is 1.2.1. Adding the following redirection to the compiled executable's configuration file made the executable finally work as in LINQPad:

<dependentAssembly>
  <assemblyIdentity name="System.Collections.Immutable"
                    publicKeyToken="b03f5f7f11d50a3a"
                    culture="neutral" />
  <bindingRedirect  oldVersion="0.0.0.0-1.2.1.0"
                    newVersion="1.2.1.0" />
</dependentAssembly>

It seems then that lpless needs to add automatic binding redirections as NuGet would. How would this ever work for a C# script? 🤔

[vcaraulean]

I remember dealing with similar issues. If there's no API level compatibility issues (missing types, signatures, members) dynamic assembly resolution might work. A good example can be found in somewhat related issue for scriptsc. And implementation. Thoughts?

[atifaziz]

@vcaraulean Thanks for the scriptcs pointers, and yeah, I thought of dynamic assembly resolution too but at this stage things seem to be getting quite complicated and I'm wondering if the scenario of compiling to C# script versus an executable is buying anything extra. After all, it's not like the C# interpreter is readily downloadable from anywhere (even keeps moving locations and redistribution channels) so it's only good for environments where the build tools are installed.

With NuGet integration, compilation and more, I fear that LINQPadless might end up replicating the complexity of the build process and artifacts (developer packs, references per target framework, forwarded types, facades) & package installation. @wangzq took an interesting approach with lpmake where he compiles the LINQPad query to a project and then leverages the build tools to do all the heavy-lifting. With LINQPadless, the goal was having few files to manage and distribute (*.cmd + *.csx or *.cmd + *.exe with auto-restore at target if packages are missing) and least tooling (e.g. NuGet and MSBuild are not required). Just thinking out loud here for now. 💭

[wangzq]

This issue is really about how to get correct assembly references from a nuget package for a specific framework version. This used to be easier with just .NET 4.x, with NetStandard it is getting more complicated. In lpmake I am relying on project.json to figure out the correct references for me, but it also stopped working for recent packages such as 2.0 of Microsoft.CodeAnalysis. My plan is to either learn more about how Linqpad is able to do that, and/or check if the new msbuild is able to solve this problem.

[atifaziz]

After all, it's not like the C# interpreter is readily downloadable from anywhere (even keeps moving locations and redistribution channels) so it's only good for environments where the build tools are installed.

Looks like C# interpreter is available as part of Microsoft.Net.Compilers so doesn't require a Microsoft Build Tools 2015 installation.