Data does not validate anymore (2.1 vs 2.0)

[ostefano]

Something odd must have happened a few days ago: new records seem to have been updated to a new format.

See for example: https://github.com/fkie-cad/nvd-json-data-feeds/blob/main/CVE-2021/CVE-2021-30xx/CVE-2021-3007.json There is now a new cveTags field, which is not part of the NVD 2.0 data schema.

Even more interestingly, if you now download the 2.0 NVD schema from https://csrc.nist.gov/schema/nvd/api/2.0/cve_api_json_2.0.schema what you get back is actually version 2.1 cve_api_json_2.1.schema.json (for comparison here the old one cve_api_json_2.0.schema.json )

Format 2.1 does indeed include a new field, but it is named cveTag.

In other words there are three issues:

• New data has been silently upgraded to a new version
• NVD does not really know what schema is serving (2.0? 2.1?)
• New records do not validate either new or old format

[rhelmke]

Hey Stefano,

thanks for letting us know :-). Do you plan to contact the NVD regarding this issue?

We planned some things on our side that would also be affected by your observations on API response inconsistency:

• We want to add weekly re-syncs of the whole dataset since we found that incremental updates using lastModified timestamps lead to inconsistencies, as said value is not always updated. We talked to the NVD about this issue and identified some problems together. However, in case of the behavioral issues you are reporting, it would mean that we propagate them across all records in the cache. Well, at least we are then consistent with NVD API behavior 🤷‍♂️.
• We want to add a Github Action that validates the schema on push. This might help to identify such issues relatively quickly.

[rhelmke]

Hope you don't mind that I pinned this issue

[ostefano]

Hi @rhelmke ,

Not yet, especially because https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2021-3007 still returns 2.0 data (no cveTag(s)) and there is no API 2.1 endpoint. In other words I am not entirely sure how the problematic records percolated down and what APIs are affected (and if problematic records are still generated).

What APIs do your automations rely on?

[rhelmke]

Oh I see. We use the same 2.0 API as you mentioned. So it appears that the endpoint emitted this additional data field at some point in time. And our automations do not update this record since it gets deltas by modification timestamp. Affected CVEs obviously didn't receive a timestamp update as only the API response got fixed.

Yeah this basically means that our planned weekly full-syncs with the NVD would fix these records.

Then there's probably also little use in contacting the NVD as they already noticed and fixed the issue. The invalid data in our cache is basically a relic of this fault 🤔

[rhelmke]

FYI: 3390fbf introduces schema validation on push. Example Run

[ostefano]

Nice, I can see that it catches the records with the rogue cveTags field

[rhelmke]

Next week I'll have more time to implement the required functionality that fixes these records. 😃

[rhelmke]

Fixed, see #16 :-)