search

ossf / scorecard

OpenSSF Scorecard - Security health metrics for Open Source
https://scorecard.dev
Apache License 2.0
4.46k stars 489 forks source link

Improve Score Reporting: OSV coverage is limited #2162

Closed azeemshaikh38 closed 1 year ago

azeemshaikh38 commented 2 years ago

We only query the OSV database for vulns on a commitSHA so we are extremely limited on the vulns we can report through Scorecard. There is a high possibility that we are rating a commitSHA as 10 on Vulnerabilities check but it actually has reported Vulnerabilities elsewhere. This will quickly erode users trust in Scorecard, need to fix this one ASAP.

laurentsimon commented 2 years ago

@oliverchang can you comment on this?

oliverchang commented 2 years ago

I think this just boils down to the issue of not having a good repo -> package level mapping. Most vulnerability databases do not track commit level information, and when they do, it's sparse and incomplete.

Perhaps we should change the meaning of this Vulnerabilities check. Maybe we should instead check if a project has known vulnerabilities in its dependencies. This is much easier to check (because we just scan a project's manifest files) and provides a good signal about a project's hygiene when it comes to dealing with vulns.

We're building a tool in OSV that does exactly that (https://github.com/google/osv.dev/tree/master/tools/osv-scanner#scanning-a-lockfile), which could be integrated here.

scantist-admin commented 2 years ago

@azeemshaikh38 - can you share an example where this false-negative scenario plays out? We have been considering open-sourcing our vulnerability database and we do have significant repo-package and commit-version mapping data that may be able to address the issue.

oliverchang commented 1 year ago

With https://github.com/google/osv-scanner, we have a way of measuring how many dependencies are affected by known vulnerabilities for most repos. This could be a great way to augment the current vulnerabilities check and make it more useful.

CC @another-rex who owns osv-scanner.

laurentsimon commented 1 year ago

This is really nice, thanks for sharing. How would you envisage scorecard ingesting the results? Are you planning on exposing a Go API? For example, wrapping the CLI's options into an API (like we did for Scorecard https://github.com/ossf/scorecard/blob/main/main.go)?

oliverchang commented 1 year ago

Yes, a Go API should be pretty easily doable. @another-rex thoughts?