Types from .NET 7 NuGet package are not found

oleg-shilo / cs-script

C# scripting platform

http://www.cs-script.net

MIT License

1.62k stars 235 forks source link

Types from .NET 7 NuGet package are not found #326

Closed UweKeim closed 1 year ago

UweKeim commented 1 year ago

Using latest .NET Core CS-Script and referencing a .NET 7 class library NuGet package with this .nuspec file:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>Foo.Bar</id>
    <version>1.0.2</version>
    <title>Foo Bar</title>
    <authors>uwekeim</authors>
    <icon>cloud_upload.png</icon>
    <readme>readme.md</readme>
    <projectUrl>https://example.org</projectUrl>
    <description>Foo bar.</description>
    <tags>foo bar</tags>
    <repository url="https://example.org" />
    <dependencies>
      <group targetFramework="net7.0">
        <dependency id="JetBrains.Annotations" version="2022.3.1" exclude="Build,Analyzers" />
        <dependency id="Newtonsoft.Json" version="13.0.2" exclude="Build,Analyzers" />
        <dependency id="RestSharp" version="108.0.3" exclude="Build,Analyzers" />
        <dependency id="SharpZipLib" version="1.4.2" exclude="Build,Analyzers" />
      </group>
    </dependencies>
  </metadata>
</package>

When adding this to my cs file with this line

//css_nuget -force:3600 Foo.Bar

running the script successfully prints that it has referenced the package:

...
info : Pakete für "C:\Users\ukeim\AppData\Local\Temp\csscript.core\.nuget\59176\59176.csproj" werden wiederhergestellt...
info : Das Paket "Foo.Bar" ist mit allen angegebenen Frameworks im Projekt "C:\Users\ukeim\AppData\Local\Temp\csscript.core\.nuget\59176\59176.csproj" kompatibel.
info : Die PackageReference für das Paket "Foo.Bar", Version 1.0.2, wurde der Datei "C:\Users\ukeim\AppData\Local\Temp\csscript.core\.nuget\59176\59176.csproj" hinzugefügt.
info : Die Assetdatei wird auf den Datenträger geschrieben. Pfad: C:\Users\ukeim\AppData\Local\Temp\csscript.core\.nuget\59176\obj\project.assets.json
log  : "C:\Users\ukeim\AppData\Local\Temp\csscript.core\.nuget\59176\59176.csproj" wiederhergestellt (in "102 ms").
...

Still, when trying to access a type of this package it says that the type cannot be found:

error CS0246: Der Typ- oder Namespacename "Foo" wurde nicht gefunden (möglicherweise fehlt eine using-Direktive oder ein Assemblyverweis).

I tried the same package within VS 2022 in a console test project and this compiled and ran successfully.

So I'm not sure what's the issue here.

Is it that the package is not coming from nuget.org but from a private repository?
Is it that my package is targeting .NET 7 only?

I've tried to take a look at the generated .csproj file from the log ("C:\Users\ukeim\AppData\Local\Temp\csscript.core.nuget\59176\59176.csproj") but unfortunately this file is not present anymore.

Is there a way to tell CS-Script to keep these temporary files for debugging purposes?

My primary question

Can you think of any way to debug/locate this behavior so that I am finally able to use the classes in my NuGet package?

Update 1

I found the compiled project for my script (named "do-deploy.cs") at "C:\Users\ukeim\AppData\Local\Temp\csscript.core\cache-361889454.build\do-deploy.cs\" and took a look at "do-deploy.csproj".

While I actually found my NuGet package downloaded at "C:\Users\ukeim.nuget\packages\foo.bar\1.0.2" I can confirm that "deploy.csproj" does not contain a <Reference Include="C:\Users\ukeim\.nuget\packages\foo.bar\1.0.2\lib\net7.0\Foo.Bar.dll">.

So there must be something that is preventing CS-Script to actually reference my NuGet library.

Update 2

After looking at the CS-Script source code here on GitHub, I tried the following:

//css_nuget -force:3600 -rt:net7.0 Foo.Bar

And to my surprise, this actually worked! 🎉

So explicitly specifying -rt:net7.0 is a workaround I could live with for now. Still it would be awesome if CS-Script would automatically detect this without specifying it. (I would have to adjust my code every year for every new .NET release)

oleg-shilo commented 1 year ago

Txs Uwe. Will have a look.

oleg-shilo commented 1 year ago

Is there a way to tell CS-Script to keep these temporary files for debugging purposes?

You can do it with this command: css -vs script.cs. It will create the same csproj and open it in VS.

You can also check how the references are resolved with this command css -proj script.cs:

============

Now I see your problem.

CS-Script is not a package manager like the nuget manager of VS. It cannot solve the complicated dependencies and does only a few things:

Asks CLI package manager (e.g. dotnet.exe) to download your package and all its dependencies
Tries to aggregate and reference all assemblies: your package and its dependencies. This is something that cannot be done by the nuget manager as it does not expose this functionality to third parties. This CS-Script does its best (e.g. relying on naming convention), but without promising complete success. That's why you need sometimes to specify the exact details of the package. For the cases when CS-SCript cannot resolve it for whatever reason (e.g. unusual naming convention).

UweKeim commented 1 year ago

Thanks, @oleg-shilo, this really surprises me. I would have thought that Microsoft provides all tools available as command line tools, too.

E.g. dotnet add package <PACKAGE_NAME> would install even my "strange" package, like outlined on MSDN. Too bad this does not work well.

oleg-shilo commented 1 year ago

The problem is not installing. dotnet restore imports all packages including dependencies. There is no problem here.

The problem is that no package manager has API that tells you the location of the library for a package specified by name.

.NET solves this problem easy:

dotnet downloads the packages and dependicies
dotnet copies the libraries in build folder and builds project assembly. This API is closed no one knows how dotnet does it.
dotnet publishes: copies the libraries and the project assembly in the publish output folder
publish output folder can be distributed as a product.

CS-Script works much harder:

dotnet downloads the packages and dependicies (the same as for .NET use-case)
cs-script converts the packages dependency tree in the list of referenced assemblies from the .nuget cache folder
cs-script complies script into assembly and executes it.
cs-script uses list of referenced assemblies from the .nuget cache folder to resolve assemblies at runtime when AppDomain cannot find packages assemblies in the working directory.

Step 2 is something that cs-script cannot "borrow" from any tool but has to implement by itself. And this implementation is rather limited. So sometimes the neser needs to help with extra context. Improving _package-to-assemblypath algorithm will require bringing dotnet scale of complexity to cs-script. Thus it is a matter of "value vs effort".

Though... After talking to you I got an interesting idea how the problem can possibly be solved but without extending cs-script. POC-ing it right now.

UweKeim commented 1 year ago

Yeah! I also experienced that my package has references to other NuGet packages like "RestSharp".

At first, my cs-script file did not run correctly because RestSharp was not automatically installed.

I had to manually add RestSharp as a dependency via //css_nuget RestSharp to my cs-script file.

No big deal and yet less comfortable than not having to do this (like in VS).

oleg-shilo commented 1 year ago

Yep, it looks like my idea works.

Below is the script that plays the role of the package manager that converts package names into the nuget cache assemblies.

using System;
using System.Diagnostics;
using System.IO;
using System.IO.Hashing;
using System.Linq;
using System.Reflection;

var packages = new[] { "Microsoft.CodeAnalysis.CSharp.Workspaces", "NLog" };

// ---------------------

var nugetRepo =
    Environment.OSVersion.Platform == PlatformID.Win32NT ?
        Environment.ExpandEnvironmentVariables(@"%userprofile%\.nuget\packages") :
        "~/.nuget/packages";

var projectDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "Temp", "csscript.core", "nuget", Guid.NewGuid().ToString());
Directory.CreateDirectory(projectDir);

try
{
    var projectFile = Path.Combine(projectDir, "nuget.ref.csproj");

    File.WriteAllText(projectFile, @"
<Project Sdk=""Microsoft.NET.Sdk"">
    <PropertyGroup>
        <OutputType>Library</OutputType>
        <TargetFramework>net7.0</TargetFramework>
    </PropertyGroup>
    <ItemGroup>
" + string.Join(Environment.NewLine, packages.Select(x => $"<PackageReference Include=\"{x}\" Version=\"*\"/>")) + @"
    </ItemGroup>
</Project>");

    var sw = Stopwatch.StartNew();
    Console.WriteLine("Restoring packages...");

    var p = new Process();
    p.StartInfo.FileName = "dotnet";
    p.StartInfo.Arguments = "publish -o ./publish";
    p.StartInfo.WorkingDirectory = projectDir;
    p.StartInfo.RedirectStandardOutput = true;
    p.Start();
    p.WaitForExit();

    var allRefAssemblies = Directory.GetFiles(Path.Combine(projectDir, "publish"), "*.dll")
                               .Concat(
                           Directory.GetFiles(Path.Combine(projectDir, "publish"), "*.exe"))
                               .Where(x => !x.EndsWith("nuget.ref.dll"));

    bool isSameData(byte[] a, byte[] b)
    {
        if (a.Length == b.Length)
        {
            for (int i = 0; i < a.Length; i++)
                if (a[i] != b[i])
                    return false;
            return true;
        }
        else
            return false;
    }

    Console.WriteLine("    " + sw.Elapsed.ToString());
    sw.Restart();
    Console.WriteLine("Mapping packages to assemblies...");
    Console.WriteLine("================");

    foreach (var x in allRefAssemblies)
    {
        var packageName = Path.GetFileNameWithoutExtension(x);

        var refAssemblyVersion = FileVersionInfo.GetVersionInfo(x).FileVersion;
        var refAssemblySize = new FileInfo(x).Length;
        byte[] refAssemblyBytes = File.ReadAllBytes(x);

        // Console.WriteLine($"package: {packageName} v{refAssemblyVersion}");

        var matchingAssemblies = Directory
                .GetDirectories(nugetRepo, $"{packageName}*")
                .SelectMany(d => Directory
                                     .GetFiles(d, Path.GetFileName(x), SearchOption.AllDirectories)
                                     .Where(f => refAssemblyVersion == FileVersionInfo.GetVersionInfo(f).FileVersion
                                            && refAssemblySize == new FileInfo(f).Length
                                            && isSameData(refAssemblyBytes, File.ReadAllBytes(f))
                                           ));

        Console.WriteLine($"    {matchingAssemblies.FirstOrDefault()} ");
    }

    Console.WriteLine("================");
    Console.WriteLine("    " + sw.Elapsed.ToString());
}
finally
{
    try { Directory.Delete(projectDir, true); }
    catch { }
}

The script above resolves two packages Microsoft.CodeAnalysis.CSharp.Workspaces and NLog into the list of assemblies to be reffed.

I used .NET publish and then looked up the aggregated assemblies in the cache based on file version, size and finally file content. The lookup is actually quite fast (300 msec on my PC). The package restoration is slow (~3 seconds). This is OK as we may even need to spend more time for downloading.

The script can be used to produce a script that only contains "//css_ref " for all assemblies discovered. And then this script can be simply included to the primary script with //css_include. The rest is as with any script, including caching.

Can you please test this script on your environment and see if it correctly discovers your Foo.Bar package? You will need to replace

var packages = new[] { "Microsoft.CodeAnalysis.CSharp.Workspaces", "NLog" };

with

var packages = new[] { "Foo.Bar" };

UweKeim commented 1 year ago

Thank you very much, Oleg.

I took your script and added this to the very top:

//css_nuget System.IO.Hashing

After adding this, the script compiled. Output is:

Restoring packages...
    00:00:02.5201611
Mapping packages to assemblies...
================

    C:\Users\ukeim\.nuget\packages\jetbrains.annotations\2022.3.1\lib\netstandard2.0\JetBrains.Annotations.dll
    C:\Users\ukeim\.nuget\packages\newtonsoft.json\13.0.3\lib\net6.0\Newtonsoft.Json.dll
    C:\Users\ukeim\.nuget\packages\restsharp\109.0.1\lib\net7.0\RestSharp.dll
    C:\Users\ukeim\.nuget\packages\foo.bar\1.0.8\lib\net7.0\Foo.Bar.dll
================
    00:00:03.6720715

So it seems to work correctly. Awesome 😊

Update 1

It seems that "SharpZipLib" was not resolved in the above output.

Interestingly enough, the generated "nuget.ref.deps.json" file does reference SharpZipLib:

...
"SharpZipLib/1.4.2": {
    "runtime": {
        "lib/net6.0/ICSharpCode.SharpZipLib.dll": {
            "assemblyVersion": "1.4.2.13",
            "fileVersion": "1.4.2.13"
        }
    }
},
...

oleg-shilo commented 1 year ago

//css_nuget System.IO.Hashing

My mistake, it was left over from the experiment I did try to use CRC to binary compare files. You can remove it in the using System.IO.Hashing;. Comparing byte-by-byte is actually faster (understandably).

"SharpZipLib" was not resolved

Interesting. Do you know where it is coming from? The mapping algorithm is very simple: Let dotnet to bring all package libraries into the "publish" folder and then try to find the location where dotnet copied them from. Currently, I am only searching for files in the default nuget packages folder. Is it possible that on our machine you have more than one source of hosted packages?

UweKeim commented 1 year ago

The initial reference to "SharpZipLib" was from my .nuspec file (see initial post here above):

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>Foo.Bar</id>
    <version>1.0.2</version>
    <title>Foo Bar</title>
    <authors>uwekeim</authors>
    <icon>cloud_upload.png</icon>
    <readme>readme.md</readme>
    <projectUrl>https://example.org</projectUrl>
    <description>Foo bar.</description>
    <tags>foo bar</tags>
    <repository url="https://example.org" />
    <dependencies>
      <group targetFramework="net7.0">
        <dependency id="JetBrains.Annotations" version="2022.3.1" exclude="Build,Analyzers" />
        <dependency id="Newtonsoft.Json" version="13.0.2" exclude="Build,Analyzers" />
        <dependency id="RestSharp" version="108.0.3" exclude="Build,Analyzers" />
        <dependency id="SharpZipLib" version="1.4.2" exclude="Build,Analyzers" />
      </group>
    </dependencies>
  </metadata>
</package>

I do have multiple package sources on my machine.

Calling

dotnet nuget list source

results in this output:

Registrierte Quellen:
  1.  nuget.org [Aktiviert]
      https://api.nuget.org/v3/index.json
  2.  DevExtreme [Aktiviert]
      https://nuget.devexpress.com/L565nFO6Jil59NpbDH7RU4KjrOBVaRwjphgz5kZer2urOBn6no/api
  3.  Microsoft Visual Studio Offline Packages [Aktiviert]
      C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\
  4.  DevExpress 21.2 Local [Aktiviert]
      C:\Program Files (x86)\DevExpress 21.2\Components\System\Components\Packages
  5.  GitLab (Someone) [Aktiviert]
      https://git.example.com/api/v4/groups/45/-/packages/nuget/index.json
  6.  DevExpress 22.1 Local [Aktiviert]
      C:\Program Files\DevExpress 22.1\Components\System\Components\Packages
  7.  VistaDB 6 [Deaktiviert]
      C:\Program Files (x86)\Gibraltar Software\VistaDB 6\Packages\
  8.  GitLab (KM) [Aktiviert]
      https://git.example.com/api/v4/groups/16/-/packages/nuget/index.json
  9.  GitLab (Otherone) [Aktiviert]
      https://git.example.com/api/v4/groups/26/-/packages/nuget/index.json
  10. DevExpress 22.2 Local [Aktiviert]
      C:\Program Files\DevExpress 22.2\Components\System\Components\Packages

("git.example.com" is a placeholder of my actual own local Git server)

oleg-shilo commented 1 year ago

Interesting... The remote sources do not really matter. The packages are downloaded from multiple sources but stored in the same local cache. My speculation was that maybe there is more than one local cache.

I was wrong and the answer is actually much simpler :)

The problem was caused by the package author deviating from the package naming convention. Thus SharpZipLib package contains the assembly that is con called <package_name>.dll but ICSharpCode.SharpZipLib.dll. Meaning that my probing algorithm cannot work in such cases as I assumed that the assembly name is always the same as the assembly name.

When I dropped this assumption and switched to brute force probing the algorithm started picking the required file. But brute-force probing can lead to a slower probing process. May I ask you to do a test to see the probing performance in a realistic scenario like yours? I want to understand if the brute force algorithm is still adequate or I need to go with an undesirable analysis of "nuget.ref.deps.json"

Can you please repeat the very same test but modify the algorithm like this:

// replace this
        var matchingAssemblies = Directory
                .GetDirectories(nugetRepo, $"{packageName}*")
                .SelectMany(d => Directory
// with this
        var matchingAssemblies = Directory
                .GetDirectories(nugetRepo, "*")
                .SelectMany(d => Directory

Run the test and let me know what was the performance of the test (the output contains stopwatch data).

UweKeim commented 1 year ago

First one (with $"{packageName}*"):

Restoring packages...
    00:00:06.0009183
Mapping packages to assemblies...
================

    C:\Users\ukeim\.nuget\packages\jetbrains.annotations\2022.3.1\lib\netstandard2.0\JetBrains.Annotations.dll
    C:\Users\ukeim\.nuget\packages\newtonsoft.json\13.0.3\lib\net6.0\Newtonsoft.Json.dll
    C:\Users\ukeim\.nuget\packages\restsharp\109.0.1\lib\net7.0\RestSharp.dll
    C:\Users\ukeim\.nuget\packages\foo.bar\1.0.9\lib\net7.0\Foo.Bar.dll
================
    00:00:00.9124856

Second one (with "*"):

Restoring packages...
    00:00:03.2517671
Mapping packages to assemblies...
================
    C:\Users\ukeim\.nuget\packages\sharpziplib\1.4.2\lib\net6.0\ICSharpCode.SharpZipLib.dll
    C:\Users\ukeim\.nuget\packages\jetbrains.annotations\2022.3.1\lib\netstandard2.0\JetBrains.Annotations.dll
    C:\Users\ukeim\.nuget\packages\newtonsoft.json\13.0.3\lib\net6.0\Newtonsoft.Json.dll
    C:\Users\ukeim\.nuget\packages\restsharp\109.0.1\lib\net7.0\RestSharp.dll
    C:\Users\ukeim\.nuget\packages\foo.bar\1.0.9\lib\net7.0\Foo.Bar.dll
================
    00:00:00.9851969

I've called dotnet nuget locals all --clear before each run.

(I wonder why there is a blank line after the ================ in the first run that is missing in the second run).

oleg-shilo commented 1 year ago

The empty line is because:

Console.WriteLine($"    {matchingAssemblies.FirstOrDefault()} ");

It means that the assembly was not found.

This would work even better.

Console.WriteLine($"    {(matchingAssemblies.FirstOrDefault() ?? $"{packageName} - not found")} ");

Anyway, your experiment just confirmed that the performance of the brute-force probing is quite acceptable. Great. Will see how best to integrate the new algorithm without breaking it.

oleg-shilo commented 1 year ago

Done. Have a look at https://github.com/oleg-shilo/cs-script/releases/tag/v4.6.5.2-rc2

oleg-shilo commented 1 year ago

Done. Release v4.7.0