Improve empty states for Software > Vulnerabilities

[rachaelshaw]

Goal

User story
As a user on the vulnerabilities page,
I want to see a clearer explanation of why a CVE search returned no results
so that I can understand whether Fleet has no record of it, or whether I'm not affected.

The customer wants to be able to differentiate between the following scenarios:

• Fleet is not aware of CVE-123456
• Fleet is aware of CVE-123456, but no vulnerable assets detected
• Fleet can’t detect these types of vulnerabilities

Context

• Product designer: @noahtalerman

Changes

Product

• [x] UI changes: Figma
• [x] REST API changes: API design PR is here.

Engineering

• [x] Reference documentation changes: Add a "Source" section to the Vulnerability processing page to help the security engineer understands where Fleet gets vulnerabilities from (NVD, OVAL, etc.
• [ ] Database schema migrations: TODO
• [ ] Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Following this flow chart will help in QA: https://drive.google.com/file/d/1HHhJoiYdNR9xMmznd4GtQ6Sbim8aFvrP/view?usp=sharing

Confirmation

1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.

[noahtalerman]

Rachael: was discussed but I didn't file issue:

Adding more filtering/sorting throughout the UI — we have some issues for this in drafting to bring into the next sprint, so probably worth seeing what they think about those, and making specific requests for improvements going forward. (It's the kind of thing I see us iterating on a bit at a time; I agree it'd be nice to have better filters in a lot of places.)

[sharon-fdm]

EST: BE 5 FE 3

[zayhanlon]

@noahtalerman do you want to review the figma with customer-faltona or think we're good to go?

[noahtalerman]

Hey @zayhanlon, good catch. Do you think we can add someone from customer-faltona to a design review next week?

[zayhanlon]

@noahtalerman absolutely - let me see if they can make it

[zayhanlon]

@lukeheath for this first hackathon that's designed, estimated and ready to go - the customer has suggested 8/12-8/13 for the development days (next sprint). letting you know here for assignment of an engineer (let me know if we should follow a different process)

@sharon-fdm @noahtalerman FYI

[lukeheath]

@zayhanlon This will be the virtual hackathon format, correct? I recall discussing the format, but I'm not certain where we landed. Do you have a format in mind?

[zayhanlon]

correct @lukeheath ! so basically the engineers can do the work as they would at home, but the customer is available for an end of day check-in to review progress on the issue, and then @noahtalerman and I discussed if maybe we want to patch the feature out or depending what date the customer selects, we could also just ship in the upcoming release.

[lukeheath]

@zayhanlon Got it. So overall same process, but a check-in call at the beginning with the customer, and then an end of day check-in each of the two days?

@sharon-fdm Please work with your team to select one frontend and one backend engineer that can schedule two days to work on this story and join the virtual hackathon. Just let me know if you have any questions. Thanks!

[lukeheath]

@sharon-fdm To be clear, this will be happening next sprint.

[zayhanlon]

@lukeheath we actually did the beginning check in call already (they joined the final design review and gave a thumbs up) :)

thanks! @sharon-fdm keep me posted so i can get meeting invites out

[noahtalerman]

TODOs:

• ✅ New empty state when search query is not valid
  • Are there any CVEs that don’t start with a year? (CVE-YYYY-####). Assuming no: validate that the search query is in the format: <CVE (case-insensitive)>-<4-digits>-
  • Validation via regex on front-end
• Some kind of illustration for the new empty states to at a glance understand which scenario I'm in
  • Maybe shrug/error if it’s not known. We can try this in the copy
  • Thumbs up if it’s it’s known but doesn’t exist

More info: VINCE has the latest stuff: https://vuls.cert.org/confluence/display/VIN/VINCE+Documentation

[noahtalerman]

https://www.loom.com/share/8d288c9f30f54de9bddafc49586369f1?sid=9ca468bc-96d3-4d88-b46a-e0417f1fe922

[getvictor]

@noahtalerman

1. The CVE format is CVE-YYYY-<4 or more digits>, so CVE-2023-9 is not a valid format.
2. Will we always require the user to type in the CVE- prefix? Seems like we can make it optional.
3. Does known_vulnerability assume an exact match? For example, if the search for CVE-2024-1234 returned no results and is not known to Fleet, but CVE-2024-12345 is known to Fleet.

[noahtalerman]

The CVE format is CVE-YYYY-<4 or more digits>, so CVE-2023-9 is not a valid format.

@getvictor thanks! Makes sense to use that correct format for validation instead. I updated the copy in the Figma here.

Will we always require the user to type in the CVE- prefix? Seems like we can make it optional.

Hmm, good idea to make it optional to give the user more grace. I think let's keep the help text the same though.

Note that we want to do this validation on the frontend.

[noahtalerman]

Does known_vulnerability assume an exact match? For example, if the search for https://github.com/advisories/GHSA-44xr-qjp2-rmf4 (CVE-2024-1234) returned no results and is not known to Fleet, but CVE-2024-12345 is known to Fleet.

@getvictor I think we go with yes. As a user searching CVE-2024-1234 I would expect to see the "not a known CVE" state here.

In the meantime, I'll send a message to the customer in Slack to learn more.

[zayhanlon]

@getvictor @RachelElysia - this was a hackathon ask for customer-faltona. they are OOO this week, but can meet on monday or tuesday next week if you want to share what's been built and get some feedback (as part of the process) before closing it out :)

[getvictor]

@RachelElysia the backend change for this is on main

[RachelElysia]

@getvictor cc: @zayhanlon awesome!! I'll try to get started on the FE Monday! I was under the impression this was going to be a specific day so I hadn't started on this in favor of P2s, but probably can finish by EOD Tuesday

[noahtalerman]

Hey @RachelElysia I think let's plan on demoing this scenario to the customer when we meet w/ them.

It's hard to describe and get feedback over Slack...

cc @zayhanlon

[zayhanlon]

@RachelElysia @getvictor scheduling a call with you both and the customer for this week. stand by!

[RachelElysia]

@zayhanlon cc: @getvictor @mike-j-thomas frontend is merged to main, new graphics and all

[noahtalerman]

Hey @rachaelshaw and @getvictor I updated several dev notes and some copy in Figma following our call w/ the customer.

I recorded this Loom that includes changes we discussed during the call and changes I thought about after the call.

Please take a look at let me know if you have any questions or if it doesn't track with what y'all were thinking.

Thanks!

[getvictor]

@RachelElysia The latest backend change has been merged to main.

[lukeheath]

@sharon-fdm @getvictor @RachelElysia We're bringing this back to expedited drafting to make a small requirement change. Will update you later today.

[lukeheath]

@sharon-fdm I've added a new backend sub task to this story that we need to estimate: https://github.com/fleetdm/fleet/issues/21392. Would you please review with @getvictor, estimate, and bring onto the sprint board?

@RachelElysia has full context from our design review, but we decided not to enforce exact match searching on the frontend and instead do it on the backend so that this endpoint returns consistent results to the user between the UI and API. We're bringing this back through expedited drafting, but note this is not a priority ticket and I expect this change will mean we need to push to the next release.

[sharon-fdm]

Thanks, @lukeheath. @getvictor, do we need to roll back the changes you already merged?

[getvictor]

@lukeheath @sharon-fdm The current backend change that's on main is the following: If query is a legal CVE format (CVE-YYYY-<4+ digits>) then return a known_vulnerability field as part of the response.

I assume we still want the known_vulnerability field as part of https://github.com/fleetdm/fleet/issues/21392 In that case, we don't need to rip out the current code, just modify it.

[getvictor]

@marko-lisica @lukeheath @rachaelshaw

1. When we met with the customer, we agreed to only support the exact search for CVEs in the UI. Adding the double quotes requirements feels like a step in the wrong direction. For example, if the user searches for CVE-2023-1234 and they get back CVE-2023-12345, how do we indicate to them whether Fleet knows about CVE-2023-1234 or that the need to put double quotes around the query?

2. What if the customer misses a double quote, like "CVE-2023-1234 -- should frontend validate that?

3. We already have an endpoint for exact match which fronend can use:

GET /api/v1/fleet/vulnerabilities/{cve}

I propose that if Fleet knows about the vulnerability (but no hosts match it), that endpoint returns a 204 instead of 404

4. Since we are doing an exact match, why not display the vulnerability details on the Software->Vulnerabilities tab, instead of having the user click and go to another page to get the same info?

[marko-lisica]

When we met with the customer, we agreed to only support the exact search for CVEs in the UI. Adding the double quotes requirements feels like a step in the wrong direction. For example, if the user searches for CVE-2023-1234 and they get back CVE-2023-12345, how do we indicate to them whether Fleet knows about CVE-2023-1234 or that the need to put double quotes around the query?

@getvictor The idea is to wrap a string in double quotes to indicate it should return for exact match. So if user search for "CVE-2023-1234" and there's no results, Fleet will return one of the new empty states ("This is not a known CVE" or "This is a known vulnerability (CVE), but it wasn't detected on any hosts."). If they search without quotes it will be same fuzzy search we have today with same empty states. We could add a tooltip to indicate that users can search for exact match if they wrap CVE in quotes.

We already have an endpoint for exact match which fronend can use:

GET /api/v1/fleet/vulnerabilities/{cve}

Since we already have an endpoint for this, I think it makes sense to have logic on the frontend that checks if CVE is wrapped with quotes and based on that call this endpoint instead of GET /api/latest/fleet/vulnerabilities?query={search_string}

I propose that if Fleet knows about the vulnerability (but no hosts match it), that endpoint returns a 204 instead of 404

I agree, we can return 204 and probably 404 in case that CVE is not known?

[lukeheath]

Since we already have an endpoint for this, I think it makes sense to have logic on the frontend that checks if CVE is wrapped with quotes and based on that call this endpoint instead of GET /api/latest/fleet/vulnerabilities?query={search_string}

This makes sense to me. If the user indicates they want exact match (by wrapping the string in quotes), the frontend will route the request to the exact match endpoint. Otherwise, it will use the fuzzy search endpoint. Thoughts? @RachelElysia @marko-lisica

[marko-lisica]

This makes sense to me. If the user indicates they want exact match (by wrapping the string in quotes), the frontend will route the request to the exact match endpoint. Otherwise, it will use the fuzzy search endpoint. Thoughts?

@lukeheath @RachelElysia That's what I specified here in Figma. I'll bring this to design review so we take one more look at it before re-estimation.

[georgekarrv]

I would strongly urge to use a toggle (like show versions) and do this without quotes

Why?

• If I imagine myself as the customer maybe reading 5 new major vulns that they need to investigate they would copy the vuln from that report and then have to " paste " to search instead of having the toggle enabled and just paste
• Secondly this makes it more clear to the end-user who might not hover for the tooltip (we have seen this before where most users don't look for usage by tooltips as a first thought
• Thirdly if we ever want to back out and change this logic in the future, removing a toggle is much easier to convey to the current users than telling them 'how they have been using search with quotes is changing'

• Finally it will allow users who didn't have this workflow in mind have an easier ui element to key them into trying this.

  Even better IMO but more work would be to do this without any toggle or quotes and have the frontend detect if the exact search is present in the results to dynamically pop up a warning with this messaging as opposed to adding new elements (though in that case I can see the world where the average user that doesn't want this warning popping up when they do random fuzzy searches to be a bit annoying with no way to dismiss or disable it.)

  Anyways just my 2 cents.

[getvictor]

Remaining estimates:

FE:5 BE:3

@getvictor to add subtask if needed

[noahtalerman]

Hey @zayhanlon, heads up that this customer request (hackathon) was shipped! 🎉

[fleet-release]

Clearer explanations, CVE search results bloom, Understanding grows.

[Patagonia121]

@noahtalerman I received some interesting/actionable feedback from the customer about this feature that they have to put the CVE in quotes when searching for it in Fleet. General takeaway is that this feature is usable and solves the zero-day problem, but that it's a tad-bit annoying to have to put quotes around the CVE everytime they perform a search since they never want to do fuzzy searching for CVEs.

[noahtalerman]

Hey @Patagonia121 thanks!

Can you please open a fresh feature request w/ Gong snippet attached? This way, that request goes into the product "Inbox" where we intake new requests. With the gong recording we have the full context on the ask.