Wishlist / inquiry: "advisory impact" standard.

[nathan-at-least]

Advisory Impact Standard

Is there a standard for documenting how upstream dependency vulnerabilities impact a given crate, so that users of the crate know (a) the authors are aware and (b) they have some guidance on the impact for the dependent crate/binaries?

This is an idea that's been percolating for me and I'm posting it here because if there's already something like this (I haven't found it yet) it seems like this project would know, and if there is not, maybe this set of developers is best positioned to define it.

Motivation

Dependent Crate Developers

Suppose you develop a crate A that has a transitive dependency on B and B has an outstanding advisory. What should the A developer do? Assume they can't reasonably contribute to B's vulnerability resolution on a timeline that works for their needs for A.

• They should probably first try cargo update and see if that pulls in a resolution to the vulnerability. By far the best option.
• If that fails, they might find an alternative dependency subtree to switch to which does not have an advisory. This may be fairly invasive, depending on the alternative's API differences. This may in turn impact A's interface and complicate the decision making around semver semantics. However, sometimes, it may work out well enough.
• If both of the above do not work, they should investigate precisely how the vuln in B impacts A in any case any downstream user may exercise, then document that.

The idea here is to standardize that documentation for both users and automated tools.

Dependent Crate Consumers

Now suppose you are a consumer of crate A, and you regularly run cargo audit, like a good citizen, and you see that one of its transitive dependencies has an outstanding vulnerability disclosure. What should you do?

• First, you probably want to know if the A developers are even aware of the issue. If they are not, that's concerning, and implies the impacts on A haven't been investigated adequately. A good step here might be to file a ticket requesting that they document an impact analysis.
• Next, suppose the developers are aware and have documented how this impacts the use of A. This is the best case! Now you need to determine if your own use is safe (whether as a downstream crate dev, or a user of the binaries).

Ecosystem Safety Infrastructure

Finally, suppose you are providing infrastructure to improve safety across the ecosystem. You may want a way to scan many crates for impact statements on any particular advisory, or all advisories. Perhaps you want to add badges to repositories, doc sites, or package distribution sites.

Specification Brainstorm

I envision a specification that can help dependent crate developers and users track the impacts of upstream vulnerabilities, so here's a sketch:

• Define a standard path for documenting vulnerability impacts, such as ./doc/vulnerabilities.md.
• Define a human-readable format with well-defined embedded metadata. Maybe this is markdown with some unambiguously parseable way to associate impact analysis documentation to the advisories themselves.
• Create a feature of cargo-audit or a downstream tool (ex cargo-audit-impacts) which compares the audit report against the impact document, and warns/errors users where there are advisories with no associated impact section, or where there are impact sections that are no longer relevant.

In addition to those tool-centric bullets, the idea is a user who doesn't use cargo at all can simply browse the markdown to get a concise overview of advisory impacts, say by browsing github or another source mirroring website.

It may be useful to have a standardized set of impact assessment field values, such as UNKNOWN, NO IMPACT, CONDITIONAL IMPACTS, COMPREHENSIVE IMPACT. "Conditional" means that only some subsets of an API or binary usage under some conditions are vulnerable, in which case the details need to spelled out in natural language suitable for the crate/binary consumers. Meanwhile, "comprehensive" means many or most uses of the dependent crate are likely to be vulnerable to the issue. (Names are hard, I'm not wed at all to them.)

Thoughts?

Does this already exist?

Does this general approach seem like a useful thing to put effort into?

If so, I'd like to ensure the standard and cargo-audit interface well with each other, including backwards compatibility / future-proofing considerations.

[nathan-at-least]

Background: when I learned about cargo audit, I quickly started adding it as a CI / git commit-hook criteria (typically via cargo-checkmate). After some time, I found in a significant number of cases there are long-running / deep advisories present (such as RUSTSEC-2020-0071).

Some of those may not impact my crates, or some may, and if there were an impact standard like this, I could retool CI rules to allow merges so long as the impact is documented, or with other rules.

[tarcieri]

There are a few open issues along these lines:

• 21

• https://github.com/rustsec/advisory-db/issues/288

[ionut-arm]

I'm dealing with the same kind of issue - trying to figure out how to report to potential users about a vulnerability in our dependency tree. (My problem is compounded by the fact that the users of our service aren't expected to be Rust developers, so just using cargo-audit isn't necessarily a solution.)

Ultimately I'd be interested in seeing this treated as an ecosystem-wide issue, so for example having crates.io integrate with the Rustsec advisories to show which versions are vulnerable and, as you suggest, potentially include more detailed information written up by the maintainers about what is actually affected. If there are other standards or best-practice guides out there for other ecosystems, e.g., for Go packages, I'd be interested in reading about them.

[nathan-at-least]

Poking around the source I find most of what I want is already present, I just didn't catch it by looking at the output of cargo audit --help. Is this documented somewhere else?

First, there can be per-project audit.toml files as per this comment.

Next, I see rustsec itself ignores the two issues that crop up in cargo-checkmate which depends on cargo-audit [^1].

So what's missing?

First, I don't have any impact statement from rustsec devs on why those are ignored. I haven't yet taken the time to understand the vulnerabilities themselves, and instead just scanned issues to see the likelihood they would be resolved soon. So are they ignored in rustsec because (a) they have 0 impact in any circumstances, or (b) they have impacts in some circumstances, but it's considered not a big deal for various reasons?

So second, it takes pretty specialized knowledge to know to look for .cargo/audit.toml for ignored advisories.

Tying these together for this issue's wishlist item, what if there were a document like ./doc/vulnerabilities.md that spelled entries that look something like:

# RUSTSEC-2020-0159

    Crate:         chrono
    Version:       0.4.19
    Title:         Potential segfault in `localtime_r` invocations
    Date:          2020-11-10
    ID:            RUSTSEC-2020-0159
    URL:           https://rustsec.org/advisories/RUSTSEC-2020-0159
    Mitigation: IGNORED

This vulnerability is ignored in `rustsec` because we believe it never affects users because the `localtime_r` is never exercised by any `rustsec` code.

The thinking here is that users with less specialized knowledge would hopefully notice a file named ./doc/vulnerabilities.md and open it out of curiosity, then many technical users would understand the contents immediately without knowing anything about rust, cargo, cargo-audit, etc…

Meanwhile, the format could be machine-parsable, although the current audit.toml format already provides the same information. The motivation for machine-parsable documentation is to have a single source of truth for both humans and machines.

Finally, the doc format could hopefully be cross-language/framework to be of most use.

For now I'm going to use audit.toml files in my projects and keep pondering this idea of a standardized doc format for acknowledging vuln impacts in downstream projects. If I were to dig into that, I'd probably focus on extending cargo-audit to do what I want as a demo, then see if I could rally sufficient support in either the rust community or across languages. (I put low odds on having the bandwidth for that project, so if someone else is interested, please run with it.)

[^1]: The dependency of cargo-checkmate on cargo-audit is a bit wonky because all cargo-checkmate needs to do is execute the cargo-audit binary, but cargo has no concept of dependencies on binaries, a feature I've requested.

[ionut-arm]

Is there a standard for documenting how upstream dependency vulnerabilities impact a given crate, so that users of the crate know (a) the authors are aware and (b) they have some guidance on the impact for the dependent crate/binaries?

I've been made aware of VEX as a standard with this purpose in mind. More info here