search

anchore / syft

CLI tool and library for generating a Software Bill of Materials from container images and filesystems
Apache License 2.0
6.32k stars 580 forks source link

Incorrect package name and purl of dotnet nuget packages #2697

Open markusmuellerusi opened 9 months ago

markusmuellerusi commented 9 months ago

What happened: Package name and purl do not match expectet result. Found f#internalName :

      {
            "bom-ref": "pkg:nuget/f%23InternalName@15.00.0913.015?package-id=d8a74f75a594e230",
            "type": "library",
            "name": "f#InternalName",
            "version": "15.00.0913.015",
            "cpe": "cpe:2.3:a:f\\#InternalName:f\\#InternalName:15.00.0913.015:*:*:*:*:*:*:*",
            "purl": "pkg:nuget/f%23InternalName@15.00.0913.015",
            "properties": [{
                    "name": "syft:package:foundBy",
                    "value": "dotnet-portable-executable-cataloger"
                }, {
                    "name": "syft:package:language",
                    "value": "dotnet"
                }, {
                    "name": "syft:package:type",
                    "value": "dotnet"
                }, {
                    "name": "syft:package:metadataType",
                    "value": "dotnet-portable-executable-entry"
                }, {
                    "name": "syft:location:0:path",
                    "value": "\\packages\\Microsoft.Exchange.WebServices.2.2\\lib\\40\\Microsoft.Exchange.WebServices.dll"
                }
            ]
        }

What you expected to happen: The name should be Microsoft.Exchange.WebServices and same for purl

Steps to reproduce the issue: Create an SBoM for a dotnet project using this nuget package: Microsoft.Exchange.WebServices.2.2.nupkg packages\Microsoft.Exchange.WebServices.2.2\lib\40\Microsoft.Exchange.WebServices.dll

Anything else we need to know?:

Environment:

markusmuellerusi commented 9 months ago

Same for syft 1.0.1

markusmuellerusi commented 8 months ago

"\packages\Microsoft.Exchange.WebServices.2.2\lib\40\Microsoft.Exchange.WebServices.dll" Package name: Microsoft.Exchange.WebServices Version: 2.2

markusmuellerusi commented 8 months ago

Additional Information: This assembies are .Net Framework 4.x, not .NetCode or .Net6/7/8. The correct Versions you'll find in packages.config file.

tgerla commented 8 months ago

Thanks @markusmuellerusi for the report, we will take a look as soon as we can.

tgerla commented 8 months ago

Hey @markusmuellerusi, we've done a bunch of digging here and it looks as though the dll files inside that package have some bad metadata, probably from Microsoft's own build process. We are seeing f#InternalName in the FileDescription field for one package, and p(InternalName) in the same field on the other package. This is coming right from the nuget package.

We suspect that in this case, there is some failed template interpolation going on. We have found that Nuget packages don't really have a standard field name for the package name, unfortunately. We have a heuristic to choose which field to use for the package name, and for most Microsoft-built packages, FileDescription is the correct one.

Do you by chance have a support agreement with Microsoft at all? Have you seen this sort of behavior on any other Nuget packages? We'll be happy to try to work out a solution.

markusmuellerusi commented 8 months ago

I think it would be better for this kind of project to analyze the packages.config instead of the files (.dll). Here's a snippet created with CycloneDx-dotnet-tool:

{
  "type": "library",
  "bom-ref": "pkg:nuget/Microsoft.Exchange.WebServices@2.2",
  "author": "Microsoft",
  "name": "Microsoft.Exchange.WebServices",
  "version": "2.2",
  "description": "Exchange Web Services (EWS) Managed API",
  "scope": "required",
  "hashes": [
    {
      "alg": "SHA-512",
      "content": "1ABEED02B764FFA6A0C5DB2E96071E4FC85489DE91C6210AF8B236CD1824C8063F3DDDF6090156CDD499E0A66FBB49B84F4B0377F83759004DAF465ED49FC8C6"
    }
  ],
  "licenses": [
    {
      "license": {
        "name": "Unknown - See URL",
        "url": "https://github.com/OfficeDev/ews-managed-api/blob/master/license.txt"
      }
    }
  ],

It uses the packages.config file <?xml version="1.0" encoding="utf-8"?>

I can use the "syft:location:" property to search for matching package in that path. Yes we have a support contract with Microsoft and I can raise an issue there. But there are others too not only Microsoft. May be thinking about another way of finding packages, would be an option. Thanks a lot

markusmuellerusi commented 8 months ago

packages.config file <?xml version="1.0" encoding="utf-8"?>

markusmuellerusi commented 8 months ago

&lt ?xml version="1.0" encoding="utf-8"? &gt
&lt packages &gt
&lt package id="Common.Logging" version="3.3.1" targetFramework="net45" / &gt
&lt package id="Common.Logging.Core" version="3.3.1" targetFramework="net45" / &gt
&lt package id="Common.Logging.Log4Net.Universal" version="1.0.1" targetFramework="net45" / &gt
&lt package id="Iesi.Collections" version="4.0.0.4000" targetFramework="net45" / &gt
&lt package id="log4net" version="2.0.8" targetFramework="net45" / &gt
&lt package id="Microsoft.Exchange.WebServices" version="2.2" targetFramework="net45" / &gt
&lt package id="NHibernate" version="4.1.1.4000" targetFramework="net45" / &gt
&lt package id="NHibernate.Caches.SysCache" version="4.0.0.4000" targetFramework="net45" / &gt
&lt package id="Spring.Core" version="2.0.1" targetFramework="net45" / &gt
&lt package id="Spring.Web" version="2.0.1" targetFramework="net45" / &gt
&lt package id="System.Data.SQLite.Core" version="1.0.109.1" targetFramework="net45" requireReinstallation="true" / &gt
&lt /packages &gt

tgerla commented 8 months ago

Thanks for the hints, @markusmuellerusi, this is very helpful. Can you help me understand where packages.config fits in a bit better? I unzipped the microsoft.exchange.webservices.2.2.0.nupkg to poke around after replicating your problem but I don't see a packages.config file in there. Are they a part of Nuget packages at all, or something else? Thanks!

markusmuellerusi commented 8 months ago

For .Net4.x projects, the packages.config file may be located in the project directory, at the same level as the .csproj file itself. The packages directory may be located in the top-level solutions directory. .Net 6/7/8 projects do not have a separate configuration file, the package references are included in the .csproj file. But it all depends on what kind of setup for the package you want to use. (use within the project or for the solution or globally and use package reference). This task may not be easy to determine.

wagoodman commented 7 months ago

For folks looking to reproduce the issue:

$ dotnet new console
$ dotnet add package Microsoft.Exchange.WebServices --version 2.2.0
$ dotnet publish -c Release
$ syft . -o table -o json=sbom.json
 ✔ Indexed file system                                                                                                                                                                                       .
 ✔ Cataloged contents                                                                                                                         cdb4ee2aea69cc6a83331bbe96dc2caa9a299d21329efb0336fc02a82e1839a8
   ├── ✔ Packages                        [16 packages]
   └── ✔ Executables                     [16 executables]
NAME                            VERSION         TYPE
Microsoft.Exchange.WebServices  2.2.0           dotnet  (+1 duplicate)
f#InternalName                  15.00.0913.015  dotnet  (+1 duplicate)
p(InternalName                  15.00.0913.000  dotnet  (+1 duplicate)
...

cat sbom.json| jq '.artifacts[] | select(.name == "f#InternalName")'

{
  "id": "18551e301b570224",
  "name": "f#InternalName",
  "version": "15.00.0913.015",
  "type": "dotnet",
  "foundBy": "dotnet-portable-executable-cataloger",
  "locations": [
    {
      "path": "/bin/Release/net7.0/Microsoft.Exchange.WebServices.dll",
      "accessPath": "/bin/Release/net7.0/Microsoft.Exchange.WebServices.dll",
      "annotations": {
        "evidence": "primary"
      }
    }
  ],
  "licenses": [],
  "language": "dotnet",
  "cpes": [
    {
      "cpe": "cpe:2.3:a:f\\#InternalName:f\\#InternalName:15.00.0913.015:*:*:*:*:*:*:*",
      "source": "syft-generated"
    }
  ],
  "purl": "pkg:nuget/f%23InternalName@15.00.0913.015",
  "metadataType": "dotnet-portable-executable-entry",
  "metadata": {
    "assemblyVersion": "",
    "legalCopyright": "© 2014 Microsoft Corporation. All rights reserved.",
    "comments": "Service Pack 0",
    "internalName": "Microsoft.Exchange.WebServices.dll",
    "companyName": "Microsoft Corporation",
    "productName": "Microsoft® Exchange",
    "productVersion": "15.00.0913.015"
  }
}
{
  "id": "683c8944d460183a",
  "name": "f#InternalName",
  "version": "15.00.0913.015",
  "type": "dotnet",
  "foundBy": "dotnet-portable-executable-cataloger",
  "locations": [
    {
      "path": "/bin/Release/net7.0/publish/Microsoft.Exchange.WebServices.dll",
      "accessPath": "/bin/Release/net7.0/publish/Microsoft.Exchange.WebServices.dll",
      "annotations": {
        "evidence": "primary"
      }
    }
  ],
  "licenses": [],
  "language": "dotnet",
  "cpes": [
    {
      "cpe": "cpe:2.3:a:f\\#InternalName:f\\#InternalName:15.00.0913.015:*:*:*:*:*:*:*",
      "source": "syft-generated"
    }
  ],
  "purl": "pkg:nuget/f%23InternalName@15.00.0913.015",
  "metadataType": "dotnet-portable-executable-entry",
  "metadata": {
    "assemblyVersion": "",
    "legalCopyright": "© 2014 Microsoft Corporation. All rights reserved.",
    "comments": "Service Pack 0",
    "internalName": "Microsoft.Exchange.WebServices.dll",
    "companyName": "Microsoft Corporation",
    "productName": "Microsoft® Exchange",
    "productVersion": "15.00.0913.015"
  }
}

cat sbom.json| jq '.artifacts[] | select(.name == "p(InternalName")'

{
  "id": "7a319297af858a94",
  "name": "p(InternalName",
  "version": "15.00.0913.000",
  "type": "dotnet",
  "foundBy": "dotnet-portable-executable-cataloger",
  "locations": [
    {
      "path": "/bin/Release/net7.0/Microsoft.Exchange.WebServices.Auth.dll",
      "accessPath": "/bin/Release/net7.0/Microsoft.Exchange.WebServices.Auth.dll",
      "annotations": {
        "evidence": "primary"
      }
    }
  ],
  "licenses": [],
  "language": "dotnet",
  "cpes": [
    {
      "cpe": "cpe:2.3:a:p\\(InternalName:p\\(InternalName:15.00.0913.000:*:*:*:*:*:*:*",
      "source": "syft-generated"
    }
  ],
  "purl": "pkg:nuget/p(InternalName@15.00.0913.000",
  "metadataType": "dotnet-portable-executable-entry",
  "metadata": {
    "assemblyVersion": "",
    "legalCopyright": "© 2014 Microsoft Corporation. All rights reserved.",
    "comments": "Service Pack 0",
    "internalName": "Microsoft.Exchange.WebServices.Auth.dll",
    "companyName": "Microsoft Corporation",
    "productName": "Microsoft® Exchange",
    "productVersion": "15.00.0913.000"
  }
}
{
  "id": "5a0ac610c16eb8f8",
  "name": "p(InternalName",
  "version": "15.00.0913.000",
  "type": "dotnet",
  "foundBy": "dotnet-portable-executable-cataloger",
  "locations": [
    {
      "path": "/bin/Release/net7.0/publish/Microsoft.Exchange.WebServices.Auth.dll",
      "accessPath": "/bin/Release/net7.0/publish/Microsoft.Exchange.WebServices.Auth.dll",
      "annotations": {
        "evidence": "primary"
      }
    }
  ],
  "licenses": [],
  "language": "dotnet",
  "cpes": [
    {
      "cpe": "cpe:2.3:a:p\\(InternalName:p\\(InternalName:15.00.0913.000:*:*:*:*:*:*:*",
      "source": "syft-generated"
    }
  ],
  "purl": "pkg:nuget/p(InternalName@15.00.0913.000",
  "metadataType": "dotnet-portable-executable-entry",
  "metadata": {
    "assemblyVersion": "",
    "legalCopyright": "© 2014 Microsoft Corporation. All rights reserved.",
    "comments": "Service Pack 0",
    "internalName": "Microsoft.Exchange.WebServices.Auth.dll",
    "companyName": "Microsoft Corporation",
    "productName": "Microsoft® Exchange",
    "productVersion": "15.00.0913.000"
  }
}
wagoodman commented 7 months ago

We suspect that in this case, there is some failed template interpolation going on. We have found that Nuget packages don't really have a standard field name for the package name, unfortunately.

I think it's a pretty good guess.

I don't think having a ( or # is valid within a package name in this ecosystem. What we could do in the meantime is change syft to look for these prefixes and then use the referenced field to find the "real" value. This would be the fastest way to address this issue, however, the only problem is that I haven't been able to find any evidence of an existing convention for this p( and f# field indirection for these fields.

An initial look shows we can probably add this "fix" safely.

wagoodman commented 7 months ago

Though the downside with using this "indirect" value is that they appear to lead to incorrect versions relative to the package manager (nuget in this case): v15.00.0913.000 vs v2.2

I think it would be better for this kind of project to analyze the packages.config instead of the files (.dll)

In the context of the incorrect values described in this issue I think this makes sense, but there is more nuance there when it comes to parsing manifest-like files. This is a little more confusing since .NET core vs .NET have different ways to track project dependencies (package.config vs .csproj). There are also some trade offs.

Let's take a simple project as an example:

$ dotnet new console
$ dotnet add package Microsoft.Exchange.WebServices --version 2.2.0
$ dotnet add package System.Text.Json --version 9.0.0-preview.3.24172.9

Taking a look at the manifest, we will see only direct dependencies (cat syft-2697.csproj):

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <RootNamespace>syft_2697</RootNamespace>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Exchange.WebServices" Version="2.2.0" />
    <PackageReference Include="System.Text.Json" Version="8.0.3" />
  </ItemGroup>

</Project>

(I'm using dotnet v7)

This trade off is not the case when looking at the deps.json, which is output from the build itself (thus will show transitive dependencies):

deps.json contents ```json { "runtimeTarget": { "name": ".NETCoreApp,Version=v7.0", "signature": "" }, "compilationOptions": {}, "targets": { ".NETCoreApp,Version=v7.0": { "syft-2697/1.0.0": { "dependencies": { "Microsoft.Exchange.WebServices": "2.2.0", "System.Text.Json": "8.0.3" }, "runtime": { "syft-2697.dll": {} } }, "Microsoft.Exchange.WebServices/2.2.0": { "runtime": { "lib/40/Microsoft.Exchange.WebServices.Auth.dll": { "assemblyVersion": "15.0.0.0", "fileVersion": "15.0.913.0" }, "lib/40/Microsoft.Exchange.WebServices.dll": { "assemblyVersion": "15.0.0.0", "fileVersion": "15.0.913.15" } } }, "System.Text.Encodings.Web/8.0.0": { "runtime": { "lib/net7.0/System.Text.Encodings.Web.dll": { "assemblyVersion": "8.0.0.0", "fileVersion": "8.0.23.53103" } }, "runtimeTargets": { "runtimes/browser/lib/net7.0/System.Text.Encodings.Web.dll": { "rid": "browser", "assetType": "runtime", "assemblyVersion": "8.0.0.0", "fileVersion": "8.0.23.53103" } } }, "System.Text.Json/8.0.3": { "dependencies": { "System.Text.Encodings.Web": "8.0.0" }, "runtime": { "lib/net7.0/System.Text.Json.dll": { "assemblyVersion": "8.0.0.0", "fileVersion": "8.0.324.11423" } } } } }, "libraries": { "syft-2697/1.0.0": { "type": "project", "serviceable": false, "sha512": "" }, "Microsoft.Exchange.WebServices/2.2.0": { "type": "package", "serviceable": true, "sha512": "sha512-NlkaTD0uWtg9VbiWq0VDWMBPzHFknH3tmstrxt0acrT09LBGqATTjRpsGOODz6tUdX7/dehBinxsf75U5Uw/+A==", "path": "microsoft.exchange.webservices/2.2.0", "hashPath": "microsoft.exchange.webservices.2.2.0.nupkg.sha512" }, "System.Text.Encodings.Web/8.0.0": { "type": "package", "serviceable": true, "sha512": "sha512-yev/k9GHAEGx2Rg3/tU6MQh4HGBXJs70y7j1LaM1i/ER9po+6nnQ6RRqTJn1E7Xu0fbIFK80Nh5EoODxrbxwBQ==", "path": "system.text.encodings.web/8.0.0", "hashPath": "system.text.encodings.web.8.0.0.nupkg.sha512" }, "System.Text.Json/8.0.3": { "type": "package", "serviceable": true, "sha512": "sha512-hpagS9joOwv6efWfrMmV9MjQXpiXZH72PgN067Ysfr6AWMSD1/1hEcvh/U5mUpPLezEWsOJSuVrmqDIVD958iA==", "path": "system.text.json/8.0.3", "hashPath": "system.text.json.8.0.3.nupkg.sha512" } } } ```

(this tradeoff also does not exist with the binary version resources section, but the deps.json is easier to read)

In cases where there is only the binary and not the deps.json file, we also want to be accurate. I think there are changes that can be made to the PE cataloger to do that:

syft ./bin -o json | jq '.artifacts[] | select(.foundBy == "dotnet-portable-executable-cataloger") | select(.name == "System.Text.Json")'

{
  "id": "23eccff6e0f78092",
  "name": "System.Text.Json",
  "version": "8.0.324.11423",
  "metadata": {
    "productVersion": "8.0.3+9f4b1f5d664afdfc80e1508ab7ed099dff210fbd",
    "fileVersion": "8.0.324.11423",
    "fileDescription": "System.Text.Json",
  }
}
{
  "id": "0244a7a163f7c50e",
  "name": "System.Text.Json",
  "version": "8.0.324.11423",
  "metadataType": "dotnet-portable-executable-entry",
  "metadata": {
    "productVersion": "8.0.3+9f4b1f5d664afdfc80e1508ab7ed099dff210fbd",
    "fileVersion": "8.0.324.11423",
    "fileDescription": "System.Text.Json",
   }
}
full json ```json { "id": "23eccff6e0f78092", "name": "System.Text.Json", "version": "8.0.324.11423", "type": "dotnet", "foundBy": "dotnet-portable-executable-cataloger", "locations": [ { "path": "/Release/net7.0/System.Text.Json.dll", "accessPath": "/Release/net7.0/System.Text.Json.dll", "annotations": { "evidence": "primary" } } ], "licenses": [], "language": "dotnet", "cpes": [ { "cpe": "cpe:2.3:a:System.Text.Json:System.Text.Json:8.0.324.11423:*:*:*:*:*:*:*", "source": "syft-generated" } ], "purl": "pkg:nuget/System.Text.Json@8.0.324.11423", "metadataType": "dotnet-portable-executable-entry", "metadata": { "assemblyVersion": "8.0.0.0", "legalCopyright": "© Microsoft Corporation. All rights reserved.", "comments": "Provides high-performance and low-allocating types that serialize objects to JavaScript Object Notation (JSON) text and deserialize JSON text to objects, with UTF-8 support built-in. Also provides types to read and write JSON text encoded as UTF-8, and to create an in-memory document object model (DOM), that is read-only, for random access of the JSON elements within a structured view of the data.\r\n\r\nThe System.Text.Json library is built-in as part of the shared framework in .NET Runtime. The package can be installed when you need to use it in other target frameworks.", "internalName": "System.Text.Json.dll", "companyName": "Microsoft Corporation", "productName": "Microsoft® .NET", "productVersion": "8.0.3+9f4b1f5d664afdfc80e1508ab7ed099dff210fbd", "fileVersion": "8.0.324.11423", "fileDescription": "System.Text.Json", "originalFilename": "System.Text.Json.dll" } } { "id": "0244a7a163f7c50e", "name": "System.Text.Json", "version": "8.0.324.11423", "type": "dotnet", "foundBy": "dotnet-portable-executable-cataloger", "locations": [ { "path": "/Release/net7.0/publish/System.Text.Json.dll", "accessPath": "/Release/net7.0/publish/System.Text.Json.dll", "annotations": { "evidence": "primary" } } ], "licenses": [], "language": "dotnet", "cpes": [ { "cpe": "cpe:2.3:a:System.Text.Json:System.Text.Json:8.0.324.11423:*:*:*:*:*:*:*", "source": "syft-generated" } ], "purl": "pkg:nuget/System.Text.Json@8.0.324.11423", "metadataType": "dotnet-portable-executable-entry", "metadata": { "assemblyVersion": "8.0.0.0", "legalCopyright": "© Microsoft Corporation. All rights reserved.", "comments": "Provides high-performance and low-allocating types that serialize objects to JavaScript Object Notation (JSON) text and deserialize JSON text to objects, with UTF-8 support built-in. Also provides types to read and write JSON text encoded as UTF-8, and to create an in-memory document object model (DOM), that is read-only, for random access of the JSON elements within a structured view of the data.\r\n\r\nThe System.Text.Json library is built-in as part of the shared framework in .NET Runtime. The package can be installed when you need to use it in other target frameworks.", "internalName": "System.Text.Json.dll", "companyName": "Microsoft Corporation", "productName": "Microsoft® .NET", "productVersion": "8.0.3+9f4b1f5d664afdfc80e1508ab7ed099dff210fbd", "fileVersion": "8.0.324.11423", "fileDescription": "System.Text.Json", "originalFilename": "System.Text.Json.dll" } } ```

For these examples it looks like the better version to be using is 8.0.3+9f4b1f5d664afdfc80e1508ab7ed099dff210fbd or maybe a cleaned up value of 8.0.3 (dropping the semver optional metadata field).

Additionally, maybe the PE cataloger should be deduplicating (merging really) these packages so that only one is shown.