Get vulnerability score/severity with pip-audit

[matteoannotell]

Is your feature request related to a problem? Please describe.

We would like to be able to see the severity of vulnerabilities detected, and ideally filter them out based on being above/below a given threshold.

Describe the solution you'd like

Adding score/severity the information to the output would be a first step for enabling the user to ideally pipe the output to a json and have some case/if downstream based on that. Ideally one could go even further and perhaps pass a --threshold flag for muting vulnerabilities below a given score

Describe alternatives you've considered

I drafted a PR for addressing the first part of the problem, providing score/severity, https://github.com/pypa/pip-audit/pull/653. An alternative would be using pip-audit as it is, and calling the nvd api using pip-audit output as input. The main risk here is creating a dependency on a third-party api

Additional context

INFO:pip_audit._cli:This product uses the NVD API but is not endorsed or certified by the NVD.
Found 2 known vulnerabilities in 2 packages
Name       Version ID               Fix Versions Severity Score
---------- ------- ---------------- ------------ -------- -----
py         1.11.0  PYSEC-2022-42969              HIGH     7.5
setuptools 65.5.0  PYSEC-2022-43012 65.5.1       MEDIUM   5.9

[woodruffw]

Thank you for opening an issue, and for following our template!

Some previous discussion here: #207 (specifically, https://github.com/pypa/pip-audit/issues/207#issuecomment-1625111415 and below)

To summarize some of the concerns raised there:

1. We prefer PYSEC vulnerability IDs by default, and there's no formal guarantee that a PYSEC identifier has corresponding CVE, GHSA, etc. identifiers. Consequently, we need to be careful about introducing assumptions about relationships between these identifiers/assuming that the current alias-returning APIs will always return things that we can meaningfully link together.
2. Using a separate external service (the NVD API) for scoring metadata means that we'll need to merge reports/determine a policy around metadata timeliness, which is a problem I'd rather we didn't have to solve 🙂

TL;DR: If pip-audit is going to support this, we need vulnerability scores/severities to come directly from our upstream vulnerability feeds (i.e. PyPI and OSV), not from an external service. In addition to the problems mentioned above, depending on an additional external service means we inherit that service's rate limits, compatibility concerns, etc. -- I don't expect stability to be a huge issue for the NVD API, but every dependency adds additional error modes that I'd prefer be folded into an existing dependency.

I'm happy to give your changes in #653 a review if you'd like, but using the NVD API is likely to be a non-starter.

[matteoannotell]

Good feedback William, thanks! I understand the concerns: I will then close the PR, you can disregard it, but thanks a lot for your offer to review it 😄

At the best of your knowledge, do you know if there is any plans, or if there has been any discussions in either PyPI or OSV for adding some sort of vulnerability info?

Also, not fully familiar with how PYSEC/CVE, and GHSA relate to each other. Is it possible that a PYSEC ID has neither a GHSA nor a CVE counterpart?

[di]

I think the upstream issue in OSV is https://github.com/google/osv.dev/issues/783, PyPI would be able to add CVSS scores once OSV supported it.

[woodruffw]

Is it possible that a PYSEC ID has neither a GHSA nor a CVE counterpart?

It's currently unlikely, but it's technically possible: PYSEC IDs come from a separate database which is back-populated with relevant CVEs, but it's possible that future vulnerabilities are "unique" to PYSEC or may appear in PYSEC before appearing in other feeds (roughly analogously to RUSTSEC).

[matteoannotell]

I understand. Thank you both for the info!