search

ossf / scorecard

OpenSSF Scorecard - Security health metrics for Open Source
https://scorecard.dev
Apache License 2.0
4.58k stars 496 forks source link

Add support for NuGet #1578

Open JonDouglas opened 2 years ago

JonDouglas commented 2 years ago

We've invested time into understanding this space. It seems that for NuGet to thrive with scorecards, we will be heavily dependent on #4177 for many Microsoft packages as they haven't quite moved to GH actions yet. This work however, will work just fine for other prominent packages that are already on GH actions and follow best practices outlined.

Additionally, it seems that no other package manager has implemented the basic pinned dependencies check for ensuring "pinned versions" are included.

Based on the recent SonaType survey, the following are emphasized across various ecosystems:

Image

Image

Therefore, this issue should track the following support (or lack thereof) of the most impactful/implemented today:

While some of these are repository & CI/CD specific, in the context of a package i.e. scorecard --nuget=System.Text.Json --show-details we have a lot of work to do.

OLD:

The scorecard project currently only supports npm, golang, and pip as far as I could tell. I'm a PM on the NuGet team at Microsoft and would love to help contribute adding support for NuGet in this tool or providing the right guidance to implement support for NuGet. This closely aligns with a proposal I had last year and would love to experiment with this scorecard in .NET:

https://github.com/dotnet/designs/pull/216

Please feel free to reach out to us over at NuGet/Home on GitHub or in this issue. Any steps on how to best contribute adding this support would be greatly appreciated!

naveensrinivasan commented 2 years ago

It would be great to add support for NuGet.

Here is an example of how we are looking for go https://github.com/ossf/scorecard/blob/696553be2dd358a1b177cf28a1c8e321ee9e779f/checks/shell_download_validate.go#L591

PR's are welcome!

laurentsimon commented 2 years ago

@JonDouglas that's great, thanks for reaching out!

@naveensrinivasan pointed you to the file where we parse commands and look for unpinned go, pip and npm commands. This is the place to add it.

Shameless plug: as part of the OSSF Best Practice WG, we have a work stream on package manager's best security practices. We've started with npm and pip. If you're interested in collaborating - or even lead! - a similar effort for NuGet, please let me know.

An idea we've been kicking around is to work with package managers and help them display scorecard information on their website. Would you imagine providing scorecard results on the NuGet's hub//website? I'd love to discuss this direction if it's of interest. You're more than welcome to attend our bi-weekly Thu scorecard meetings!

laurentsimon commented 2 years ago

an additional place to add support for NuGet is in the Packaging check https://github.com/ossf/scorecard/blob/main/checks/packaging.go, which tries to infer if the project published a package. We typically look for known GitHub actions and commands to detect this.

JonDouglas commented 2 years ago

@laurentsimon Let's discuss this further as it's very in-line with some areas I'm investigating and would love to join a scorecard meeting. Where can I get involved for those bi-weekly meetings?

azeemshaikh38 commented 2 years ago

Please see https://github.com/ossf/scorecard#connect-with-the-scorecards-community for details about joining the bi-weekly. Bi-weekly's happen Thursdays 1-2pm PST (next instance is on Feb 24th).

azeemshaikh38 commented 2 years ago

Also feel free to add an agenda item to the upcoming meet in this doc. You'll need to join ossf-scorecard-announce@googlegroups.com for being able to modify the doc.

laurentsimon commented 2 years ago

Looking forward to meeting you @JonDouglas! Thanks @azeemshaikh38 for the prompt info!

JonDouglas commented 2 years ago

Hi all,

I had originally planned to join the meeting later this afternoon, but unfortunately I have a conflict I cannot clear this month. I'll be looking to join the next one on the 10th instead.

laurentsimon commented 2 years ago

Thanks for letting us know. See you in 2 weeks then!

balteravishay commented 1 year ago

@laurentsimon I'm trying to plan the work needed to comply with your reply here:

an additional place to add support for NuGet is in the Packaging check https://github.com/ossf/scorecard/blob/main/checks/packaging.go, which tries to infer if the project published a package. We typically look for known GitHub actions and commands to detect this.

Looking at the packaging code and the IsPackagingWorkflow method it seems as if some support for Nuget is already accounted for when calculating the packaging score. Do you reckon some more development needs to be added or is that comment stale? cc. @JonDouglas

laurentsimon commented 1 year ago

Looking at the packaging code and the IsPackagingWorkflow method it seems as if some support for Nuget is already accounted for when calculating the packaging score. Do you reckon some more development needs to be added or is that comment stale? cc. @JonDouglas

Great, it seems it's already supported. No further comment from me on that front then

walbourn commented 1 year ago

Note in my case there's no GitHub action that is going to indicate my project has an associated NuGet package on nuget.org or a VC++ Package on the vcpkg.io.

I do have links in my Release comments that have those links though, so maybe you can scan those as well?

balteravishay commented 1 year ago

thanks for the feedback @walbourn!

Packaging scores in Scorecard, today, work by detecting github/gitlab packaging workflows only, for all ecosystems (correct me if I'm wrong here @laurentsimon). The aim of this check is to detect if the repo has an automatic, standardized packaging and delivery mechanism, rather than to check if "indicate my project has an associated NuGet package on nuget.org".

please review the test case here, noting how a GH workflows that match a run statement with "nuget push" commands is deemed as a packaging workflow. that also complies with the GitHub docs on pushing to nuget, here and with the nuget docs on pushing, here

Perhaps the scenario you're describinh can be better handled as its own packaging enhancement request, aiming all ecosystems, rather than under this issue that is specific to nuget?

Also, I suggest to separate dotnet/nuget from VC++ (both here and in your original issue #2535), as they would likely have rather different implementations, contribution paths and contributors.

walbourn commented 1 year ago

One option would be to use a file in the repo like is done with .onefuzz detection for Fuzzing. Maybe a .nugetorg file that just has text lines for the URL(s) to associated nuget packages?

The scorecard could then walk those URLs to see if the packages are signed.

You can also do validation where you extract the NuGet package metadata and verify that the <repository type="git" url="???" /> points back to the repo you are scoring to make sure it's consistent.

github-actions[bot] commented 8 months ago

This issue has been marked stale because it has been open for 60 days with no activity.

JonDouglas commented 1 month ago

Updated this issue with various comments and rationale after doing a small hackathon on this topic. We're investigating incremental improvements such as lock files in #4351 today and bringing more awareness to Microsoft/.NET teams on various other checks.