Missing vulnerable_configurations on some some CVEs

[maxime-huyghe]

Hello again, upon updating our cve-search instance from v4 to v5, and repopulating the database, we noticed that a few thousand CPEs had gone missing from the cve-search API's /cvefor/ route due to missing vulnerable_configurations on these CVEs.

Looking at core/database_maintenance/sources_process.py l.555-600, we found that the affected CVEs were those with a version range in their NVD criteria but no matching CPEs. One such example is CVE-1999-0176 (https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-1999-0176). This behavior is inconsistent with CVEs that have no version range (e.g. CVE-1999-0196), which still have a fallback when CPE-less in version 5, and thus still have their vulnerable_configuration and are still found by cve-search.

I've got a patch ready that restores the old behavior and adds the fallback to CVEs with versions ranges. This is not ideal but would fix this regression in the meantime.

[Tazmaniac]

One side note: CPE database is incomplete (known thing) and lag a lot and even more behind CVEs actually too. (https://nvd.nist.gov/general/news/nvd-program-transition-announcement)

As https://github.com/cve-search/CveXplore/pull/256 is not merged, when CPE concerning an already published CVE is published (CVE with or without configuration range) you need to reset and re-import the complete NIST CVE database.

With something like https://github.com/cve-search/CveXplore/pull/256, we could locally periodically re-run the same vulnerable configuration list building routine as on mogo CVE document creation time on the existing documents, without re-downloading the complete NIST database.

We could even add a new parameter for new slow requests that do it on the fly for CVEs, or new route (cvesearch) to do more advanced query like explained in https://github.com/cve-search/CveXplore/issues/238 or others like "is the CPE is in a configuration range of a CVE" even if the CPE is not in the database. This later one could be a reason to later remove the "populate vulnerable_configuration with the CPE part of the CVE criteria if the CPE stem is not in the CPE database or empty).

https://github.com/cve-search/CveXplore/issues/268 was found with Maxime when investigating on the present issue/regression.

[P-T-I]

Sorry Guys; #256 fell completely off my radar for some reason; will merge asap!

[P-T-I]

256 is merged; but to my understanding this does not solve this issue, does it?

[P-T-I]

'With something like #256, we could locally periodically re-run the same vulnerable configuration list building routine as on mogo CVE document creation time on the existing documents, without re-downloading the complete NIST database.' This would be something interesting to actually add to the current code base; @Tazmaniac is that something you would like to share?

[Tazmaniac]

256 is merged; but to my understanding this does not solve this issue, does it?

No it does not solve it but permit to mitigate the impact with a tool to rebuild the vulnerable_configuration without re-downloading the NIST database (for the case there is CPE updates).

[Tazmaniac]

'With something like #256, we could locally periodically re-run the same vulnerable configuration list building routine as on mogo CVE document creation time on the existing documents, without re-downloading the complete NIST database.' This would be something interesting to actually add to the current code base; @Tazmaniac is that something you would like to share?

Yes, we hope so. We could propose something.

[P-T-I]

related to #284