Design and implementation of the vulnerability-lookup

[adulau]

vulnerability-lookup project

vulnerability-lookup is a cve-search rewrite to support the following functionalities. This project will be a new software project under the cve-search organisation.

Functionalities

Core

• Use a key-value store (kvrocks) instead of the old MongoDB back-end to handle the fast lookup of cve.circl.lu (peak ~7000 req per second);
• Import the actual NVD json file with some meta-data but keep the raw JSON import in a key;
• Expose an API compatible with the original cve-search with minimal API such as the two following entry points: /api/cve/ and /browse/.
• Allow the import of other vulnerabilities without CVE-ID assigned;
• Import CPE from NVD or from additional sources;

Import

• Allow the import of CVE-ID state (not in the JSON) from https://cve.mitre.org/cve/search_cve_list.html;
• Allow the creation of vulnerability outside CVE-ID allocation;
• Allow the import of GSD source https://github.com/cloudsecurityalliance/gsd-database
• Add CISA known exploited vulnerabilities https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
• Add PyPi advisory https://github.com/pypa/advisory-database/tree/main/vulns

[Rafiot]

Work in progress on that (branch framework):

• Imports all CVEs individually straight into kvrocks (it takes 15 min and uses ~3G of ram to load the complete dataset)
• Supports updates for individual CVEs by erasing the old one
• API endpoint: /vulnerability/{vulnerability_id} - Reason: will support any ID, from NVD or not

[adulau]

Maybe starting with a new importer like the GSD source GSD-database would be a good example of a second ID and also how to map the existing CVE with the GSD source too.

[adulau]

New extensions

• Allow to have a web interface similar to the original version cve-search
• Allow to have an admin interface (also expose via the API) to easily create vulnerability with a minimal set of vulnerability fields
• Allow to have a basic public or embargo/non-public vulnerability publishing system
• Improving debug and monitoring on the import process in vulnerability-lookup (e.g. how to monitor if the NVD API is down or giving 500 errors)

[Rafiot]

Some of the UI is implemented (search/list recent entries). Now let's discuss the system to create a new vulnerability.

This is the form to report an advisory via github:

Should it be similar to that? What is the minimal viable set of settings we want in the form?

Then, how to we identify the advisory before it is assigned a CVE? And do we do that? If yes, one way is to do something similar to github with something like that: GHSA-c647-pxm2-c52w

[Rafiot]

As found by @adulau , we should use this interface for edit/submit: https://github.com/Vulnogram/Vulnogram

And push it to vulnerability-lookup instead of CVE for the ones created by our constituants

[Rafiot]

Open question regarding CVEList: it is more or less a duplicate of the NVD database, and it is not really possible to treat it as a new source. For now, it will be imported as a meta for the CVE entries.

[adulau]

Potential new source to add - https://www.variotdbs.pl/vulns/

[adulau]

@Rafiot https://github.com/cve-search/vulnerability-lookup/issues/42 that would be an interesting candidate but it's using the ADP extension of CVE. Not exactly sure the best way to do it.