Integrate with vulnerability databases

[AMDmi3]

Mark vulnerable package versions

The plan:

• [x] Implement harvesting CPE data from upstream repositories
  • :x: GUIX contains cpe_name (is useless without vendor)
  • :x: FreeBSD ports define CPE_VENDOR and CPE_PRODUCT, but these are not exposed in INDEX
  • [x] Gentoo contains usable CPE metadata
• [x] Implement database storage for project → CPE relations
• [x] Implement fetching and parsing CPE data (https://nvd.nist.gov/vuln/data-feeds#JSON_FEED)
• [x] Implement setting vulnerable flag for affected packages
  • ~Match incoming packages against vulnerable version ranges in the database~
  • ~Force project update on new CPE for it (by resetting its hash)~
  • [x] ~It turned out to be viable to bulk update vulnerable status on all packages~ no it didn't, as we can't update binding tables properly this way to be able to do filtering based on vulnerable property
• ~Implement stub for handling patched vulnerabilty information from repositories~ (discussed in #1045
• [x] Add vulnerable flag to binding tables to allow using it in project filtering
• [x] Integrate vulnerability updates into delta update process properly
  • [x] During update, only run on incoming packages
  • [x] When vulnerabilities are updated, queue affected projects in dedicated table, reset their hashes before pushing new packages
• [x] Add per-maintainer and per-repository vulnerable packages/projects counters
• [x] When all code is in place, force update for all projects with defined cpe_vendor/cpe_product
• [x] Add per-project vulnerability flag in order to be able to show "Vulns" project page conditionally
• [x] Add/ensure that vulnerability counts are saved in repository history, so we can plot graphs
• [ ] Implement vulnerability based history events

[davidak]

Maybe you can get inspiration how to implement this from https://github.com/flyingcircusio/vulnix#theory-of-operation?

[AMDmi3]

It has nothing to do with inspiration, there's just a pile of technical problems. Apart from just parsing nvd, we

• need a way to match nvd entries with repology metapackages, preferably without introducing another set of rules. The are cpe entries, may use them instead. For instance, FreeBSD provides these, but a special parser is needed
• we can't directly use nvd to mark stuff vulnerable, as it may be patched without updating to a new version, so we need to parse specific format of each distro which is used to publish info on discovered and patched vulnerabilities (I know at least debian, gentoo and freebsd)
• to make it even more harder, each distro exposes its own specifics when publishing this info (binary package names, custom version formats, epochs and revisions)

[AMDmi3]

• [x] :bulb:Ravenports has CPE_VENDOR/CPE_PRODUCT as well, usr it if it's available in the dump or request exposure otherwise

[AMDmi3]

For the record, I've spotted an incorrect CVE information which leads to false positive. Here's a CVE which makes latest OpenVPN version look vulnerable, while in fact it refers to OpenVPN Access Server and should have CPE cpe:2.3:a:openvpn:openvpn_access_server:*:*:*:*:*:*:*:* instead of cpe:2.3:a:openvpn:openvpn:*:*:*:*:*:*:*:*.

I've mailed to nvd@nist.gov and hope it'll get fixed soon - in fact I expect more corrections to follow as people spot them, and it's important to establish communication to NVD allowing to fix them quickly and make NVD itself more useful and reliable.

Update: it was fixed, but I've got no reply. Other similar problems will be listed in repology/repology-rules#367