feat: Convert NVD API 2.0 to JSON files for mirroring

[terriko]

• Related to https://github.com/sec-data/mirror-sandbox/issues/27

Our GitHub mirror and the global mirrors on https://cveb.in/ currently mirror the following:

• All the files directly served by NVD on https://nvd.nist.gov/vuln/data-feeds
• Pre-parsed files generated by cve-bin-tool

The NVD intends to turn off their JSON data feeds on December 15, 2023 (extended from September). This is unfortunate as it's probably the easiest way for folk to get a full copy of the data: the mirrors deliver files very quickly, and NVD API 2.0 has limits that require repeated queries (and sometimes backoffs) to populate a full database.

We'd like to provide JSON files indefinitely for the mirrors because it's the easiest thing for the mirrors to serve. It's entirely possible that this shutoff will wind up extended or the NVD will pivot to other solutions over the next 3 months, but assuming things go as currently announced we probably want a script that grabs the data using NVD API 2.0 and converts it into JSON files. We want this script to convert everything from NVD (not only the parts we use, as our current mirroring stuff does) since the hope is for the global mirrors to be useful to everyone doing vulnerability scanning and research, not only the users of cve-bin-tool.

I think what we want here is

• A separate script that may use / re-use NVD API 2.0 code from cve-bin-tool (e.g. generate_nvd_json.py and NOT cve-bin-tool --generate-nvd-json)
• It should query NVD API 2.0 and generate appropriate JSON files per year
• It should keep existing JSON yearly files up to date (ideally without deleting them and grabbing all the data again, since that's not how NVD API 2.0 is intended to be used)
• We want to imagine this being run every 6-12 hours or so on a mirror server. So we should do our best to minimize the time the data people will download might be changing: probably you're going to want to update all the data into a temp folder, sign it, then copy it over the data in the mirror.
• We probably don't need a lot of history but you could keep the last 5 updates + logs or something for debugging issues. 5 is likely overkill, but you'll want to at least be able to do a rollback if the update files so definitely no deleting data before the job runs the way we do with cve-bin-tool -u now

If anyone wants to work on this, I've made a play "sandbox" repo available here: https://github.com/sec-data/nvd-json-sandbox (ping me to get added for access). This is similar to what we did with https://github.com/sec-data/mirror-sandbox in case anyone needs a place to play with some github actions and running the script once it's started. But the script itself can be stuck in the cve_bin_tool directory with our other stuff for now to facilitate code re-use.

[srividya-p]

@terriko could you please assign this issue to me?

[terriko]

@srividya-p done! Let us know if you get stuck. I think it should be mostly mechanical but batching API calls is surprisingly annoying.

[terriko]

The NVD News mailing list https://list.nist.gov/nvd-news says that the JSON will be kept on for a big longer. Cut/paste below:

Legacy Data Feed File Retirement Update: Due to feedback received from many different downstream data consumer groups after our previous reminder, we will again be extending the retirement date for the Legacy Data Feed files. However, we will still be retiring the 1.0 APIs. Going forward, we will be improving capabilities to allow for bulk download of the NVD dataset. The legacy data feeds will remain available until this effort is completed. The following data feed files will remain available until further notice: • https://nvd.nist.gov/vuln/data-feeds#JSON_FEED • https://nvd.nist.gov/vuln/data-feeds#cpeMatch • https://nvd.nist.gov/vuln/data-feeds#transxml • https://nvd.nist.gov/vuln/data-feeds#comments • https://nvd.nist.gov/products/cpe The following API endpoints will be retired on 12/18/2023 as previously communicated: • https://services.nvd.nist.gov/rest/json/cve/1.0/ • https://services.nvd.nist.gov/rest/json/cves/1.0/ • https://services.nvd.nist.gov/rest/json/cpes/1.0/ V/r, National Vulnerability Database Team National Institute of Standards and Technology (NIST)

This shouldn't change much for this issue -- we still want it done before they turn things off! -- but it does mean we're not desperate to have it done tomorrow. (phew!) I'll update the title to remove that deadline.

[mastersans]

Hey @srividya-p are you still working on this? If not I'll like to take this one. @terriko can I get assignment and repo access.

[terriko]

@mastersans No harm in having more than one person play with this so I've added you as assigned and granted you write access to https://github.com/sec-data/nvd-json-sandbox/ to play around. Don't wait for review on that repo (I don't check it daily like I do this one), you can just push directly for experiments. You can also just make your own repo if that's easier -- sec-data is just a free github org so it's not like you're getting anything particularly special out of the repo other than having our experiments in one spot.

[terriko]

pinging @mastersans to see how close this is to doable now.