search

jeremylong / DependencyCheck

OWASP dependency-check is a software composition analysis utility that detects publicly disclosed vulnerabilities in application dependencies.
https://owasp.org/www-project-dependency-check/
Apache License 2.0
6.47k stars 1.29k forks source link

NVD API request failures #7178

Open scumtydo opened 4 days ago

scumtydo commented 4 days ago

Describe the bug A clear and concise description of what the bug is.

When starting the plugin, there are numerous messages like this: [WARNING] NVD API request failures are occurring; retrying request for the 5 time

It does appear to download some updates, but there are many failures and it eventually crashes with the following stack trace.

[ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:392) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637) at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1967) at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1150) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103) at java.lang.reflect.Method.invoke (Method.java:580) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503 at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient._next (NvdCveClient.java:412) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:331) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:352) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637) at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1967) at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1150) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103) at java.lang.reflect.Method.invoke (Method.java:580) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

Version of dependency-check used

Maven plugin version: 11.1.0

Additional context

I noticed that NIST has recently changed their API. Perhaps this has broken the dependency checker?

ric777 commented 4 days ago

Found the same at version 10.0.2

2024-11-22T01:49:43.0836082Z [ERROR] Error updating the NVD Data 2024-11-22T01:49:43.0837437Z org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data 2024-11-22T01:49:43.0838424Z at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:389) 2024-11-22T01:49:43.0839445Z at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116) 2024-11-22T01:49:43.0840419Z at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906) 2024-11-22T01:49:43.0841472Z at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711) 2024-11-22T01:49:43.0842563Z at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637) 2024-11-22T01:49:43.0843796Z at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1960) 2024-11-22T01:49:43.0844868Z at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1143) 2024-11-22T01:49:43.0846986Z at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) 2024-11-22T01:49:43.0848054Z at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370) 2024-11-22T01:49:43.0850126Z at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351) 2024-11-22T01:49:43.0851169Z at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) 2024-11-22T01:49:43.0852172Z at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171) 2024-11-22T01:49:43.0853166Z at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163) 2024-11-22T01:49:43.0854487Z at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) 2024-11-22T01:49:43.0855587Z at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) 2024-11-22T01:49:43.0856676Z at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) 2024-11-22T01:49:43.0857908Z at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) 2024-11-22T01:49:43.0858961Z at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:299) 2024-11-22T01:49:43.0859973Z at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:193) 2024-11-22T01:49:43.0860964Z at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:106) 2024-11-22T01:49:43.0861949Z at org.apache.maven.cli.MavenCli.execute (MavenCli.java:963) 2024-11-22T01:49:43.0862926Z at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:296) 2024-11-22T01:49:43.0863896Z at org.apache.maven.cli.MavenCli.main (MavenCli.java:199) 2024-11-22T01:49:43.0864962Z at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103) 2024-11-22T01:49:43.0866018Z at java.lang.reflect.Method.invoke (Method.java:580) 2024-11-22T01:49:43.0867033Z at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) 2024-11-22T01:49:43.0868072Z at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) 2024-11-22T01:49:43.0869116Z at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) 2024-11-22T01:49:43.0870256Z at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) 2024-11-22T01:49:43.0871290Z Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503 2024-11-22T01:49:43.0872341Z at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:373) 2024-11-22T01:49:43.0889680Z at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:349) 2024-11-22T01:49:43.0891125Z at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116) 2024-11-22T01:49:43.0892244Z at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906) 2024-11-22T01:49:43.0893278Z at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711) 2024-11-22T01:49:43.0894950Z at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637) 2024-11-22T01:49:43.0896153Z at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1960) 2024-11-22T01:49:43.0897225Z at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1143) 2024-11-22T01:49:43.0898263Z at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) 2024-11-22T01:49:43.0899317Z at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370) 2024-11-22T01:49:43.0900317Z at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351) 2024-11-22T01:49:43.0901671Z at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) 2024-11-22T01:49:43.0903965Z at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171) 2024-11-22T01:49:43.0905281Z at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163) 2024-11-22T01:49:43.0906373Z at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) 2024-11-22T01:49:43.0907631Z at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) 2024-11-22T01:49:43.0908733Z at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) 2024-11-22T01:49:43.0937873Z at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) 2024-11-22T01:49:43.0939101Z at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:299) 2024-11-22T01:49:43.0940713Z at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:193) 2024-11-22T01:49:43.0941598Z at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:106) 2024-11-22T01:49:43.0942600Z at org.apache.maven.cli.MavenCli.execute (MavenCli.java:963) 2024-11-22T01:49:43.0943413Z at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:296) 2024-11-22T01:49:43.0944219Z at org.apache.maven.cli.MavenCli.main (MavenCli.java:199) 2024-11-22T01:49:43.0945099Z at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103) 2024-11-22T01:49:43.0945983Z at java.lang.reflect.Method.invoke (Method.java:580) 2024-11-22T01:49:43.0946902Z at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) 2024-11-22T01:49:43.0947799Z at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) 2024-11-22T01:49:43.0948921Z at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) 2024-11-22T01:49:43.0949822Z at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

ericdev1995 commented 4 days ago

I got the same issues with AZ pipelines.

chadlwilson commented 4 days ago

Yeah, seems they are having problems. As normal, probably nothing this project can do other than wait. Root cause are persistent 503s from the API:

Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503

mmonetha-sobis commented 4 days ago

+1 with Version 11.1.0

marcelstoer commented 3 days ago

Gee folks, can you please leave (the maintainers of) this project alone with such issues? HTTP 503 means "unavailable", literally nothing this project can do.

Besides, the top 2 results when searching for "nvd api 503" in your favorite search engine are previous issues with this project.

--> issue should be closed

(As for the root cause, it appears to me as if NIST had bulk-updated a ton of CVEs in their database. ODC reports that "NVD API has 245,142 records in this update" even though we update our local database very frequently. No wonder their API gets flooded with requests...)

OrangeDog commented 3 days ago

The actual issue is #6535, raised the last time this happened.

cstsw commented 3 days ago

Totally agree with @marcelstoer. There's no issue the Dependency-Check project could resolve in this situation. The software reacts correctly by retrying and reporting the 503 errors.

As the NVD API is unavailable every now and then I decoupled updating and analyzing dependencies in my builds some time ago. Documentation is available for Caching the database, especially useful for multiple jobs/pipelines running on the same build server, as well as Using a central database, which might be more appropriate if you build on different servers or e.g. on Kubernetes pods (via Gitlab CI/CD pipelines and the like). In my case of Jenkins servers with multiple jobs analyzing dependencies I simply added a job updating the Dependency-Check data twice a day, pointing all analyzing/aggregating tasks to the same data directory and disabling autoUpdate for them.

jeremylong commented 3 days ago

Easiest way to avoid this issue is just to use the open-vulnerability-data-mirror docker container. Then configure ODC to use the mirror.

pinkfloydx33 commented 3 days ago

But isn't the mirror only good for that point-in-time and thus would need to be regularly updated?

We currently have a worker that runs daily and rebuilds the data directory, with the CI jobs downloading the cache and running with --noupdate per the instructions here

If we are already doing that, I assume there is no benefit to mixing in the NVD data mirror since the job that builds the cache would likely be the same job building the mirror and they'd both hit any API limitations... or is there a meaningful way to combine these two options?

cbertoldi commented 3 days ago

@pinkfloydx33

https://github.com/jeremylong/DependencyCheck/issues/7178 Yes, but at least your build pipeline would not be blocked by an issue with the NVD API. Of course the scan would be based on an outdated database, but that's better than no scan at all, I suppose.

pinkfloydx33 commented 3 days ago

@cbertoldi I understand that. We are already using an outdated version by caching the whole data directory and using --noupdate during scans. My question was about whether there was a benefit to doing both things... for example:

I would assume since we are rebuilding the cache on a regular schedule that there's no benefit to using the mirror, since we'd have to rebuild that anyways.

lread commented 3 days ago

I run my vulnerability scans on GitHub Actions and use its caching mechanisms. I don't think a mirror fits into this usage.

A full db redownload is required when the local db is invalidated, for whatever reason:

When NIST feeds are struggling/erroring, it might make sense to bump the nvd max retry count up. More retries might allow the db to actually be fully downloaded and cached and would avoid the full db refetch attempt again on the next run. This would mean much less traffic to NIST for subsequent runs, benefiting everyone.

OrangeDog commented 3 days ago

More retries

less traffic

Unfortunately, that's not how it works. When the NVD is screwed like this you want fewer retries, so you don't sit there hammering their servers for hours until dependency-check finally gives up and fails anyway.

What is needed is the ability for it to continue analysis even if an update fails.

lread commented 3 days ago

@OrangeDog, thanks for responding. It is a trade-off, no?

With low retries:

  1. Run 1: fails after downloading 50% of db.
  2. Run 2: fails again after downloading 40% of db.
  3. Run 3: fails again after downloading 60% of db.
  4. ...

With elevated retries:

  1. Run 1: succeeds in downloading 100% of db.
  2. Run 2: succeeds after only downloading a very small db delta.
  3. Run 3: succeeds after only downloading a very small db delta.

Once we get that db downloaded, we are only hitting NVD for small deltas.

I agree that #6535 would be great, if practical, but we don't have that yet.

OrangeDog commented 2 days ago

Where are you getting these numbers from? Yesterday every run fails after downloading <1% of db, regardless of how many retries.

All elevated retires does is make it take longer for you, and make the problem worse for everyone else,.

pinkfloydx33 commented 2 days ago

FWIW it took us 2.5 hours, with retries set to 20 and the delay bumped to 6000 (using an api key) to finally update our cache due to the NVD bulk updates. Everything is smooth sailing now, updates are practically instantaneous again

lread commented 2 days ago

@OrangeDog when you say:

All elevated retires does is make it take longer for you, and make the problem worse for everyone else,.

I think you are right for the initial fetch of the db, but if elevated retries allow the db to be fully downloaded (which they did for me), then subsequent runs will only fetch the necessary very small delta, which is much less ongoing stress on the NIST servers. Maybe not so clear cut?

Yesterday every run fails after downloading <1% of db, regardless of how many retries.

I was getting much further in than that before failing. My default delay is 5s. Maybe that had something to do with it. Or maybe I just got "lucky". Or maybe the servers were a little healthier when I tried.

lread commented 2 days ago

I totally agree that erroring NIST servers is a NIST issue and not a DependencyCheck issue.

But given that NIST servers do sometimes get into a bad state, would it make sense to explore more aggressive delay algorithms? Perhaps some variant of an exponential delay algorithm? An exponential delay could reduce stress on NIST servers and maybe allow more successful NIST db downloads?

If the idea is interesting and not already been explored/rejected, I could open an issue to work out details.