Discussion: Triage workflow improvements

[terriko]

So, for #2422 I had to update our triage and I feel like we could improve the process but we might need to think about how we want to do that. I know @anthonyharrison and @raboof have talked about this (possibly mostly in PRs?) and we mention it in the gsoc ideas (#2354) but I wanted to pull it out into its own discussion thread.

@anthonyharrison suggests that our workflow should look something like this:

1. Scan product using cve-bin-tool
2. Create SBOM and VEX from the scan. The SBOM will need to be linked to the VEX I think
3. Triage the VEX and update the status of the vulnerabilities
4. Repeat

Problems:

1. We can do some of the triage side of this already, sort of, but there's a few gaps that make it not so smooth. The ouptut.json that we create doesn't quite match what we need for --triage-input-file, as I learned while doing #2422
2. We can't do the SBOM side yet but we're getting close there, and I expect we'll be working on this post 3.2 release
3. There's no well-defined triage format yet. VEX exists but is still evolving. We'll probably need to start with something or maybe even several things and pivot as needed
4. We may actually want a UI for this. I've avoided any serious UI on cve-bin-tool because it balloons our requirements and maintenance costs, but we could look at spinnoff tools maybe? Or integration with something like VSCode that already has good json editing tools?

Thoughts?

[raboof]

our workflow should look something like this

That general workflow seems really reasonable to me

The SBOM will need to be linked to the VEX I think

How strongly SBOM and VEX should be linked is an interesting question. From an 'upstream' perspective, I would really like to be able to provide VEX information about 'my' products, and then allow them to be combined with a cve-bin-tool-generated SBOM into a holistic report. There's lots of things we (as an industry) need to improve to get there, but that would be my 'dot on the horizon'.

We may actually want a UI for this

Personally I don't feel a strong need for a UI yet, but I agree it might be nice. I'm not aware of any nicely-scoped SBOM/VEX authoring tools, but that might indeed be an interesting spinoff.

[anthonyharrison]

The closest 'standard' for a VEX is CSAF and I have already got a sample implementaion available in #2401. The VEX format we are following from CycloneDX isn't a standard but in the absence of anything useful we have 'adopted' it.

Linking of SBOMs and VEXs is a hot topic with much ongoing debate. SPDX 2.3 provides facilities through the external reference facility to reference a VEX from a SBOM. As SBOMs are static (referring to a particular build or product release) and VEXes are dynamic being the latest report of the vulnerabilities (and their status), linking the two together requires some co-ordination (e.g. the latest VEX for a particular SBOM is always available at a fixed URL). I think cve-bin-tool does provide a useful summary report of vulnerabilities from the triage.

My take on the workflow from @terriko

• Scan product using cve-bin-tool and produce a SBOM of the products identified and a VEX.
• Triage the VEX and update the status to remove false reports etc
• Repeatedly scan the SBOM and the triaged VEX producing an updated VEX.

The last step needs a bit more thought as we need to try and get the information from the triaged VEX included in the updated VEX.

Having a simple UI to update the VEX is on my todo list (already sketched out a very simple UI). But it now needs to work with CSAF as well... :-). Most of the informaytion in a VEX doesn't change (it is mainly status and comments) so making all of the JSON available to an enduser might not be appropriate.

[anthonyharrison]

@terriko CSAF and VEX are a mess! Been spending a lot of time trying to understand what should be included in botth artefacts (there may also be a VDR (Vulnerability Dicslosure Report) to consider) and how aligned they are with SBOMs. Some vendors are producing CSAFs but I think it is best to will align CSAF production with SBOM production #1697 and to document our interpretation of the process which these artafacts are supported by cve-bin-tool.

[terriko]

Okay, I'm game to go with CSAF for now, with the understanding that things are likely to change. I suspect there's going to be more than a little bit of chaos and churn as folk try to comply with rules from multiple governments/corporations/etc. all at once.

[pdxjohnny]

• Similar work on similar formats
  • https://github.com/openvex/vexctl#3-vexing-a-results-set
  • https://github.com/seedwing-io/seedwing-policy#usage
  • https://github.com/ossf/wg-vulnerability-disclosures/discussions/127

[anthonyharrison]

@terriko @pdxjohnny I think we need an offline discussion about this. I am already following OpenVEX (and all of the other VEXes that are iut there!)

[rudrakshkarpe]

already sketched out a very simple UI

@anthonyharrison I was wondering if you could kindly share the UI concept you have in mind. As I am working on a GSoC'23 proposal for the idea of Integration of new formats into triage workflow

we could look at spinnoff tools maybe? Or integration with something like VSCode that already has good json editing tools?

@terriko @anthonyharrison @pdxjohnny I humbly believe that exploring the development of a VS Code extension that enables users to interact with the cve-bin-tool directly within the editor could be a potential approach worth considering. However, as I am not an expert in this area, I would appreciate your feedback on the feasibility and viability of this idea.

[anthonyharrison]

@rudrakshkarpe My thoughts are based on having a low barrier to entry so basing an initial solution on VSCode might not be the way to start (although this might be a suitable additional capability in the future). I was thinking of a web-based UI using python frameworks such as Flask or Django and maybe FastAPI. The triage process is a file based process so you need a way of creating, reading and manipulating files as well as developing a workflow.

[rudrakshkarpe]

@anthonyharrison, that is interesting. I am familiar with the flask and I am excited about creating a proposal that leverages it.

[anthonyharrison]

@rudrakshkarpe That's good. It is important to think about the bigger picture and how the UI fits in with other components. Defining an architecture will probably help you understand the scope and identify the areas you want to focus on. As this is an area subject to a lot of change, developing a modular approach which will probably help support this.

[rudrakshkarpe]

Exactly, I'm looking forward to defining an architecture within the proposal itself. I appreciate your suggestion to focus on a modular approach that can accommodate changes in the field of vulnerability reporting.

The VEX format we are following from CycloneDX isn't a standard but in the absence of anything useful, we have 'adopted' it.

Could you please navigate me to any documentation that explains the existing triage workflow of cve-bin-tool briefly or possibly you could elaborate it to me in short?

[terriko]

I'll add that if vscode gets us a prototype faster, I'm willing to give that a shot:

We don't have extensive user survey info, but most of our users right now I think fall into a few categories:

• open source devs running scans in a continuous integration system like Github Actions. They probably run at least daily scans, likely fix things themselves, and would likely provide their own triage to silence or modify warnings in their regular scans.
• devops or sysadmins who run scans across a wide range of projects (e.g. for a whole company or large business unit within a company). They probably run daily or fairly regular scans, but if anything's found they'd contact individual project teams to fix things or provide information that could be used in triage.
• people analyzing a number of projects or devices for security flaws. They likely don't fix things themselves, but report them to the "vendor" team that produced the code (which may be someone else in the same org, or may be an external company or open source project). I don't know how much they'd use triage: it might be a lot if they're stuck with these devices and need to keep track of mitigations; it might be not at all if they're deciding which device to buy or replace and they just want the data to make decisions.

I think that first group is pretty likely to be willing to use vscode for triage. Some might even already have it installed for other development purposes.

In the next two cases, I'm not sure our original "user" is likely to be the one doing the triage by themselves -- they'd either be asking someone else to do the analysis and fix, or potentially getting information form them to fill out the triage info. I'd guess that a lot of devops folk would probably prefer to use their text editor of choice (given our install patterns, probably that means vim or emacs), but they'd want us to generate the file so they could add their notes into an existing file rather than trying to make a new file from nothing.

[terriko]

Our current triage workflow is part of the "merged reports" workflow, described here in the docs:

https://cve-bin-tool.readthedocs.io/en/latest/how_to_guides/use_intermediate_reports.html?highlight=triage#adding-triage-information-to-a-merge-report

I think that we'd still potentially want a way to look at data across multiple reports and track trends, which is what the merged reports can do, but for our improved triage there's likely a few other use cases:

• combining triage from multiple sources (e.g. if you scanned a whole docker container and wanted to combine triage from multiple projects that are deployed in the same container together)
• sharing triage between projects (e.g. multiple components use the same library)
• following at least one of the proposed triage standards if we can (e.g. one of several versions of VEX probably. These are evolving rapidly but I think if we start with one, any one, we'll learn a lot and be able to translate to other formats.)
• making it easier for people to make and add triage (e.g. pre-made files with instructions in comments, a nice interface with syntax highlighting and help for editing these)

[terriko]

• Also, I've just merged #2817 so we now have SBOM generation in main. Hurray!

[terriko]

I think a lot of what we discussed here has been implemented/better documented, and any further work will likely be part of #3770 . So I'm closing this in favour of #3770