Derek-Jones / ESEUR-book

Issue handling for Evidence-based Software Engineering: based on the publicly available data
http://www.knosof.co.uk/ESEUR/
278 stars 18 forks source link

NVD claim misleading, it reverses cause & effect #24

Open david-a-wheeler opened 3 years ago

david-a-wheeler commented 3 years ago

I think this NVD claim is misleading:

All mistakes have the potential to have costly consequences, but in practice most appear to be an annoyance. One study found that only 2.6% of the vulnerabilities listed in the NVD have been used, or rather their use has been detected, in viruses and network threat attacks on computers.

A key purpose of the National Vulnerability Database (NVD) is to enable "automation of vulnerability management, security measurement, and compliance". https://nvd.nist.gov/general ; its purposes is not just to report on vulnerabilities that are being actively attacked. In many cases, vulnerabilities are identified by security researchers to the supplier or by the supplier itself, the vulnerabiliity is fixed, and then the vulnerability is made public in the NVD when the fix is release. Thus, in many cases the NVD helps alert people to update software before the attackers exploit the vulnerability. Having a low percentage is not an indicator that they are mere annoyances.

An analogy: This is like saying that hurricanes cause few deaths, because few people die from the hurricanes reported by the weather service. This reverses cause & effect; the reason relatively few people die from hurricanes is because they're getting warnings from the weather service. In the same way, the NVD (like the weather service) feeds many warning systems that help people update their software (like escaping to a safe place) before they get hit by an attacker (like a hurricane).

Derek-Jones commented 3 years ago

There is how the NVD was intended to be used and how researchers seem to be actually using it.

In the early days use of the NVD appeared to be easy to follow (the cited paper is from 2012, during the 'sensible' period). Over time it became more difficult to understand the rules for including a vulnerability or not, and researchers seemed to be making all kinds of assumptions about what the NVD contents represented. For sometime now one use of the NVD (not intended, I'm sure) is validating a researcher's work (i.e., citing the number of discovered problems that have been accepted as NVD entries). These days I ignore any paper that analyses the NVD data, it is rarely worth the effort.

The confusion around the NVD must be one of the factors that drove the creation of other vulnerability databases.

I continue to be impressed by how quickly virus creators respond to newly discovered vulnerabilities and how effective they are at discovering new vulnerabilities.

Anyway, back to your points.

Re: your analogy: Few people die from hurricans because the weather service provides accurate and timely predictions. I once saw a paper on the impact of tornado predictions having low accuracy (cannot remember its title), warnings tended to be ignored and consequently people died.

Anyway. These days the NVD is an example of what appears to be many serious vulnerabilities, but in practice the difficulty of weaponising a vulnerability prevents many of them being encountered in the field.