binkley / modern-java-practices

Modern Java/JVM Build Practices
The Unlicense
920 stars 69 forks source link

Red build: NVD API timing out #558

Closed binkley closed 19 hours ago

binkley commented 3 days ago

See GH action logs.

The initial build failure looked like JVM heap exhaustion. Further investigation shows this is an upstream problem with NVD API:

:   UpdateException: Error updating the NVD Data
Error:          caused by NvdApiRetryExceededException: NVD Update Failed: attempted to retrieve starting index 224000 from the NVD unsuccessfully five times.
Error:      NoDataException: No documents exist

But can manifest as heap exhaustion when the plugin sometimes retries infinite times.

binkley commented 2 days ago

It looks like upstream NVD API has changed a version number:

com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "cvssMetricV40" (class io.github.jeremylong.openvulnerability.client.nvd.Metrics), not marked as ignorable (3 known properties: "cvssMetricV30", "cvssMetricV31", "cvssMetricV2"])
binkley commented 2 days ago

Updating to DependencyCheck 10.0.0 at least now is not barfing on the JSON response from NVD's REST API, however, is timing out multiple times. And I needed to purge my local records cache before I could get this far:

[INFO] Checking for updates
[INFO] NVD API has 255,726 records in this update
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=2000 : 3 time
[INFO] Recoverable I/O exception (org.apache.hc.core5.http.RequestNotExecutedException) caught when processing request to {s}->https://services.nvd.nist.gov:443
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=6000 : 3 time
[INFO] Recoverable I/O exception (org.apache.hc.core5.http.ConnectionClosedException) caught when processing request to {s}->https://services.nvd.nist.gov:443
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=8000 : 3 time
[INFO] Recoverable I/O exception (org.apache.hc.core5.http.ConnectionClosedException) caught when processing request to {s}->https://services.nvd.nist.gov:443
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[INFO] Downloaded 10,000/255,726 (4%)

It's making progress ... eventually.

binkley commented 2 days ago

Reopen - also attach the commit for Gradle.