Granular `Vulnerabilities` check

[laurentsimon]

We should make the Vulnerabilities check more granular by taking into account the severity of the vulnerabilities. Example: Critical vulnerabilities -> score -= 10 High -> score -= 7 Medium -> score -= 2 Low -> score -= 1

Suggestions welcome. We could also take the time the vulnerabilities have been present for. Bit I think this complicates too much and I'm not sure it helps.

[laurentsimon]

@oliverchang is this something you could do for q4?

[oliverchang]

Severity isn't actually a standard part of the schema, as it's hard to define a standardized value that makes sense for all ecosystems. There's also debate on if a severity value is generally that useful at all -- a real severity to a user depends on they they use it.

[laurentsimon]

agreed on the context being important from a dependent's point of view. For the repo owner, severity helps with prioritization and could have been useful. Any thoughts on how we could make the check more granular?

[github-actions[bot]]

Stale issue message

[laurentsimon]

another idea around improving the vulnerability check is the following. Since scorecard always works at HEAD for a repo, it's likely all repos have their OSV/CVE vulnerabilities fixed at HEAD on main. I think we need a way to assess existing vulnerabilities, and the time it take to be fixed. Some examples:

• retrieve vulnerabilities from CodeQl via API, their creation time, closing time, title
• retrieve existing vulnerabilities from the Omega project, which is essentially similar to the above
• retrieve data from lgtm.com or sonarcloud.io. Fyi, lgtm.com runs CodeQl for free on public projects

cc @scovetta

[laurentsimon]

There is also https://docs.github.com/en/rest/reference/code-scanning that we may use to retrieve existing vulnerabilities from scanners

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[Chealer]

[...] We could also take the time the vulnerabilities have been present for. Bit I think this complicates too much and I'm not sure it helps.

It would most certainly help. A critical vulnerability which is fixed before shipping should have less weight than a vulnerability a tenth as severe but for which a fix is only released a year after it's disclosed that it affects the current version (and it starts being exploited).

Thank you for reporting

[Chealer]

Severity isn't actually a standard part of the schema, as it's hard to define a standardized value that makes sense for all ecosystems. There's also debate on if a severity value is generally that useful at all -- a real severity to a user depends on they they use it.

Issue importance is very hard to model. Indeed, impact on individual A can be orders of magnitude the impact on individual B. Even if we could consider individual importance to each user, the model would still be imperfect until it would consider that each user's usage changes over time.

None of this is reason enough to ignore importance. If we refused to estimate importance, computing would be little more than a curiosity.