NVD feed is stale due to NIST server overload

[iansltx]

💥  Actual behavior

NVD feed builds are failing (and have been for a few days) because NVD CPE DB pulls are failing. NVD CPE DB pulls are failing due to 503s/timeouts from NIST. 503s/timeouts are NIST are due to recent updates on their end causing everyone to re-pull feeds, creating much higher than usual load. See NIST status for more details.

🛠️ To fix

Pull NVD CPE data (with a bunch of retries) outside the NVD feed build process, serve the data at an alternate URL, then point the NVD build process to the alternate URL for long enough to get a successful NVD build.

We may need to prioritize pulling deltas on CPE data if NIST/NVD API unreliability persists. That fix is out of scope here.

[iansltx]

To pull data, I have a stack of commands of the form:

curl --retry 100 --retry-delay 10 "https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=70000" -H "apiKey: XYZ" -v > cpe-70000.json

Piping does result in 503 errors getting prepended to the valid JSON, so the above files will need to be cleaned up.

Will hack together a quick PHP script to map "startIndex" to the file on disk for serving and paste that here.

[iansltx]

This seems to do the trick for stripping the error messages when serving (php -S host:port index.php) without having to fiddle with the files themselvse (my sed-fu isn't great today):

<?php

$filename = 'cpe-' . ($_GET['startIndex'] ?? '0') . '.json';

if (!file_exists($filename)) {
    http_response_code(404);
    return;
}

$fp = fopen($filename, 'r');
while (fgetc($fp) !== '{') {
    continue;
}
echo '{';
fpassthru($fp);

[iansltx]

Got a run of this done, and am testing higher retry counts to see if I can get a successful run locally. Will PR if I do.

[fleet-release]

Stale feed now refreshed, NIST's storm calmed in the cloud. Secure data we fetch.