Unable to download NVD CVE data

[Jayaramvenkat]

I'm trying to use the DependencyCheck plugin for the first time, I'm getting the below error.

[DependencyCheck] OWASP Dependency-Check Plugin v3.3.4 [DependencyCheck] Executing Dependency-Check with the following options: [DependencyCheck] -name = Dependency Check [DependencyCheck] -scanPath = /data/jenkins_home/workspace/Dependency Check [DependencyCheck] -outputDirectory = /data/jenkins_home/workspace/Dependency Check [DependencyCheck] -dataDirectory = /data/jenkins_home/workspace/Dependency Check/dependency-check-data [DependencyCheck] -dataMirroringType = NIST CPE/CVE [DependencyCheck] -cveUrl12Modified = https://nvd.nist.gov/download/nvdcve-Modified.xml.gz [DependencyCheck] -cveUrl20Modified = https://nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-Modified.xml.gz [DependencyCheck] -cveUrl12Base = https://nvd.nist.gov/download/nvdcve-%d.xml.gz [DependencyCheck] -cveUrl20Base = https://nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-%d.xml.gz [DependencyCheck] -isQuickQueryTimestampEnabled = true [DependencyCheck] -jarAnalyzerEnabled = false [DependencyCheck] -nodePackageAnalyzerEnabled = true [DependencyCheck] -nodeAuditAnalyzerEnabled = true [DependencyCheck] -retireJsAnalyzerEnabled = false [DependencyCheck] -composerLockAnalyzerEnabled = false [DependencyCheck] -pythonDistributionAnalyzerEnabled = false [DependencyCheck] -pythonPackageAnalyzerEnabled = false [DependencyCheck] -rubyBundlerAuditAnalyzerEnabled = true [DependencyCheck] -rubyGemAnalyzerEnabled = true [DependencyCheck] -cocoaPodsAnalyzerEnabled = false [DependencyCheck] -swiftPackageManagerAnalyzerEnabled = false [DependencyCheck] -archiveAnalyzerEnabled = false [DependencyCheck] -assemblyAnalyzerEnabled = false [DependencyCheck] -msBuildProjectAnalyzerEnabled = false [DependencyCheck] -nuGetConfigAnalyzerEnabled = false [DependencyCheck] -nuspecAnalyzerEnabled = false [DependencyCheck] -centralAnalyzerEnabled = false [DependencyCheck] -nexusAnalyzerEnabled = false [DependencyCheck] -artifactoryAnalyzerEnabled = false [DependencyCheck] -autoconfAnalyzerEnabled = false [DependencyCheck] -cmakeAnalyzerEnabled = false [DependencyCheck] -opensslAnalyzerEnabled = false [DependencyCheck] -showEvidence = true [DependencyCheck] -formats = XML [DependencyCheck] -autoUpdate = true [DependencyCheck] -updateOnly = false [DependencyCheck] Data directory created [DependencyCheck] Scanning: /data/jenkins_home/workspace/Dependency Check [DependencyCheck] Analyzing Dependencies [DependencyCheck] One or more exceptions were thrown while executing Dependency-Check [DependencyCheck] Exception Caught: org.owasp.dependencycheck.data.update.exception.UpdateException [DependencyCheck] Message: The download was interrupted; unable to complete the update [DependencyCheck] org.owasp.dependencycheck.data.update.exception.UpdateException: The download was interrupted; unable to complete the update [DependencyCheck] at org.owasp.dependencycheck.data.update.NvdCveUpdater.performUpdate(NvdCveUpdater.java:274) [DependencyCheck] at org.owasp.dependencycheck.data.update.NvdCveUpdater.update(NvdCveUpdater.java:119) [DependencyCheck] at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:899) [DependencyCheck] at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:716) [DependencyCheck] at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:642) [DependencyCheck] at org.jenkinsci.plugins.DependencyCheck.DependencyCheckExecutor.executeDependencyCheck(DependencyCheckExecutor.java:172) [DependencyCheck] at org.jenkinsci.plugins.DependencyCheck.DependencyCheckExecutor.call(DependencyCheckExecutor.java:103) [DependencyCheck] at org.jenkinsci.plugins.DependencyCheck.DependencyCheckExecutor.call(DependencyCheckExecutor.java:46) [DependencyCheck] at hudson.remoting.LocalChannel.call(LocalChannel.java:45) [DependencyCheck] at org.jenkinsci.plugins.DependencyCheck.AbstractDependencyCheckBuilder.perform(AbstractDependencyCheckBuilder.java:85) [DependencyCheck] at org.jenkinsci.plugins.DependencyCheck.DependencyCheckBuilder.perform(DependencyCheckBuilder.java:206) [DependencyCheck] at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) [DependencyCheck] at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) [DependencyCheck] at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) [DependencyCheck] at hudson.model.Build$BuildExecution.build(Build.java:206) [DependencyCheck] at hudson.model.Build$BuildExecution.doRun(Build.java:163) [DependencyCheck] at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) [DependencyCheck] at hudson.model.Run.execute(Run.java:1794) [DependencyCheck] at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) [DependencyCheck] at hudson.model.ResourceController.execute(ResourceController.java:97) [DependencyCheck] at hudson.model.Executor.run(Executor.java:429) [DependencyCheck] Build step 'Invoke Dependency-Check analysis' changed build result to FAILURE [DependencyCheck] Collecting Dependency-Check analysis files... [DependencyCheck] Searching for all files in /data/jenkins_home/workspace/Dependency Check that match the pattern **/dependency-check-report.xml

[stevespringett]

If you turn on Jenkins debug log for org.owasp you'll likely get more info. But my first though is a firewall or proxy issue that is preventing access. Does your org have one? Does it require authentication? The Jenkins plugin uses the proxy settings defined in the Jenkins global config.

[tbartley]

I've started seeing this recently but can't know for sure it's the same issue as the op.

Turning on org.owasp logging shows that I'm getting 403 responses from e.g. nist.gov. For example:

org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://nvd.nist.gov/feeds/xml/cve/1.2/nvdcve-2019.xml.gz; received response code 403.
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:158)
Caused: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://nvd.nist.gov/feeds/xml/cve/1.2/nvdcve-2019.xml.gz; unable to connect.
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:177)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:93)
    at org.owasp.dependencycheck.data.update.nvd.DownloadTask.call(DownloadTask.java:152)
    at org.owasp.dependencycheck.data.update.nvd.DownloadTask.call(DownloadTask.java:43)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

It's not 100% failure e.g. this success from the log:

Apr 01, 2019 10:47:14 AM FINE org.owasp.dependencycheck.utils.Downloader fetchFile
Download of https://nvd.nist.gov/feeds/xml/cve/2.0/nvdcve-2.0-2005.xml.gz complete

I am able to run curl to perform the failed downloads from the same machine as we have Jenkins running on.

I wonder whether it's rate limiting - possibly due to the parallel downloads?

[pwhittlesea]

Having the same issue here.

I changed my local version of the plugin to have a single download thread and the download succeeded. The change was here.

My theory is that concurrent downloads from the NVD site get rate limited intermittently.

[stevespringett]

I've seen this with both NIST Data Mirror and Dependency-Track, so both of these projects do sequential downloading for this reason.

It's also interesting to note the change that @pwhittlesea highlighted. This may work as intended on bare-metal, but will only work with the newer versions of Java 8 in a containerized environment. Java has a long history of misidentifying the number of processors in Docker and stating the number allocated to the host rather than the container itself. So you could end up with a lot more processors than you think. Most of this has been fixed in Java 10 and has been backported to Java 8u191. But I still would not rely on this method, especially in server environments where a server (i.e. Jenkins) could have 32 cores or more.

[chrisbrook83]

We are seeing this problem too. Started a few weeks ago, and our build pipeline now fails regularly (but intermittently) as it cannot download the definitions:

[ERROR] The download was interrupted; unable to complete the update

[stevespringett]

Is there a reason why folks who are having this issue have not setup an NVD mirror? We've recommended this approach for years. https://github.com/stevespringett/nist-data-mirror is simple to setup and there are a few config things that would need to be done with Dependency-Check to get it to work. But even a cron job with curl could accomplish the same thing. By doing this, the org controls mirroring cadence and frequency and eliminates the possibility of the NVD failing builds.

[valentijnscholten]

Because it involves extra steps I guess. Ideally the dependency checker works out of the box to avoid any hurdles for people to start using it. I would say maybe make single threaded downloads the default and allow multithreaded via a command line param.

[jeremylong]

We have an update pending that should alleviate some of the issues.

[shark-horse]

@pwhittlesea I think your link to the code change is not correct? It points to a source file in this repo.

@stevespringett I understand how using a mirror can mitigate the issue, but what is different about it that it can eliminate the issue? Does it download sequentially? Currently, we cache the scan results from one build to the next so that we should not need a mirror. To be more precise, we have 3 CI VMs that each have their own Maven repo since, last I heard, you cannot safely share a Maven repo between concurrent processes. So a mirror would allow us only one third the downloads--not a big deal IMO. Two VMs have a populated cache, but the other keeps failing after a few downloads. Will the mirror download differently? (Of course I can manually copy data from one server to the other as a quick fix, but I'd rather not do that all the time.)

[stevespringett]

nist-data-mirror downloads sequentially.

[din-eshch]

I have the same issue. Im using the CLI.

@jeremylong - When is the update available? We are not behind any proxy and it fails irregularly (For example NVD CVE 2012, but failed for 2019). Not able to identify the root cause.

11:44:24 [INFO] Download Started for NVD CVE - 2017
11:44:24 [INFO] Processing Started for NVD CVE - 2006
11:44:26 [INFO] Download Complete for NVD CVE - 2012  (2502 ms)
11:44:26 [INFO] Download Started for NVD CVE - 2018
11:44:26 [INFO] Download Complete for NVD CVE - 2013  (2689 ms)
11:44:26 [INFO] Download Started for NVD CVE - 2019
11:44:26 [INFO] Download Complete for NVD CVE - 2010  (3383 ms)
11:44:26 [INFO] Download Started for NVD CVE - Modified
11:44:26 [INFO] Download Complete for NVD CVE - 2015  (2218 ms)
11:44:27 [WARN] Download Failed for NVD CVE - Modified
11:44:27 Some CVEs may not be reported.
11:44:27 [INFO] If you are behind a proxy you may need to configure dependency-check to use the proxy.
11:44:28 [WARN] Download Failed for NVD CVE - 2018
11:44:28 Some CVEs may not be reported.
11:44:28 [INFO] If you are behind a proxy you may need to configure dependency-check to use the proxy.
11:44:28 [WARN] Download Failed for NVD CVE - 2019
11:44:28 Some CVEs may not be reported.

[shark-horse]

Problem mitigated successfully for us by caching downloaded files between builds.

I imagine a clever individual could also package up the CVE files into a .jar to deploy to and download from their Maven repo (e.g. Nexus or Artifactory).

[jeremylong]

The 5.0.0-M3 milestone release reduces the download thread count to 2 and utilizes the META files from the NVD as opposed to using a HEAD request to determine if the file should be downloaded.

[tbartley]

How do I get to that release as a Jenkins plug-in - Latest available Jenkins plug-in according to Jenkins is still 4.0.2 - am I able to download a build and install directly rather than through Jenkins management?

Cheers,

Tim

[stevespringett]

For Jenkins, refer to https://groups.google.com/d/msg/dependency-check/2yMe6-Tg8tU/53SCfThcBQAJ

Snapshots are not being produced. I may be able to release v5 of Jenkins when v5 is released. If not, it’ll be shortly thereafter.

[jeremylong]

We don't recommend scanning with the Jenkins plugin if scanning maven or gradle (or sbt) based projects. You get better results using the respective plugins. In the 5.0.0-M3 I did, at the request of a friend, include a JUNIT report type so that one could simply publish the findings to any of the default plugins that can import JUNIT results. See the release notes for more information.

If there is another existing format that can be read by warnings-ng that would more correctly store the analysis data - we could easily add another report format.

[shark-horse]

I just upgraded from 4.01 to 5.0.0-M3, deleted the cached 'data' files, and did a test run. According to the console output, it is indeed only running 2 concurrent downloads at once and I was able to get all the files. :+1:

But, there are vulnerabilities now reported that were not previously reported and are not mentioned in our suppression file. Is that to be expected from a change in the dependency checker, or did I likely overlook a change I should make to accommodate the new version? (I can ask in a separate thread if required.)

[jeremylong]

Improvements were made in the matching - at least we hope they were improvements. Are the vulnerabilities identified valid?

[shark-horse]

They appear to be correct matches based on data from NVD. :+1: