search

jeremylong / DependencyCheck

OWASP dependency-check is a software composition analysis utility that detects publicly disclosed vulnerabilities in application dependencies.
https://owasp.org/www-project-dependency-check/
Apache License 2.0
6.32k stars 1.26k forks source link

NVD API returns transient 403 response with API key #6195

Closed aarongoldenthal closed 9 months ago

aarongoldenthal commented 9 months ago

Describe the bug When using an API key, the NVD API has started returning a transient 403 response. It occurs in the middle of a database update, so not a key configuration issue (and when retrying the key is used successfully). Some ODC database updates do complete, but this has occurred about half of the time since upgrading to v9.0.2.

This is related to https://github.com/jeremylong/DependencyCheck/issues/6180 and https://github.com/jeremylong/DependencyCheck/issues/6149, but is still occurring with the 9.0.2 CLI.

[INFO] Running: [/bin/sh -c /usr/share/dependency-check/bin/dependency-check.sh     --updateonly --nvdApiKey "$(cat /kaniko/NVD_API_KEY)"     --retireJsForceUpdate --hostedSuppressionsForceUpdate] 
[INFO] Checking for updates
[INFO] NVD API has 231,966 records in this update
[INFO] Downloaded 10,000/231,966 (4%)
[INFO] Downloaded 20,000/231,966 (9%)
[INFO] Downloaded 30,000/231,966 (13%)
[INFO] Downloaded 40,000/231,966 (17%)
[INFO] Downloaded 50,000/231,966 (22%)
[INFO] Downloaded 60,000/231,966 (26%)
[INFO] Downloaded 70,000/231,966 (30%)
[INFO] Downloaded 80,000/231,966 (34%)
[INFO] Downloaded 90,000/231,966 (39%)
[INFO] Downloaded 100,000/231,966 (43%)
[INFO] Downloaded 110,000/231,966 (47%)
[INFO] Downloaded 120,000/231,966 (52%)
[INFO] Downloaded 130,000/231,966 (56%)
[INFO] Downloaded 140,000/231,966 (60%)
[INFO] Downloaded 150,000/231,966 (65%)
[INFO] Downloaded 160,000/231,966 (69%)
[INFO] Downloaded 170,000/231,966 (73%)
[INFO] Downloaded 180,000/231,966 (78%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:340)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110)
    at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
    at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
    at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
    at org.owasp.dependencycheck.App.run(App.java:172)
    at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403
    at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:346)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:319)
    ... 6 common frames omitted
[ERROR] Failed to process CVE-2011-0074
org.owasp.dependencycheck.data.nvdcve.DatabaseException: Unable to retrieve id for new vulnerability for 'CVE-2011-0074'
    at org.owasp.dependencycheck.data.nvdcve.CveDB.updateOrInsertVulnerability(CveDB.java:1054)
    at org.owasp.dependencycheck.data.nvdcve.CveDB.updateVulnerability(CveDB.java:866)
    at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call(NvdApiProcessor.java:87)
    at org.owasp.dependencycheck.data.update.nvd.api.NvdApiProcessor.call(NvdApiProcessor.java:33)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "org.h2.mvstore.MVStoreException: Reading from file sun.nio.ch.FileChannelImpl@fec3929 failed at 43616274 (length -1), read 0, remaining 1024 [2.1.214/1]"; SQL statement:
DELETE FROM reference WHERE cveid = ? [50000-214]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:554)
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:477)
    at org.h2.message.DbException.get(DbException.java:212)
    at org.h2.message.DbException.convert(DbException.java:395)
    at org.h2.command.Command.executeUpdate(Command.java:264)
    at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:209)
    at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:169)
    at org.owasp.dependencycheck.data.nvdcve.H2Functions.updateVulnerability(H2Functions.java:223)
    at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.h2.schema.FunctionAlias$JavaMethod.execute(FunctionAlias.java:495)
    at org.h2.schema.FunctionAlias$JavaMethod.getTableValue(FunctionAlias.java:363)
    at org.h2.expression.function.table.JavaTableFunction.getValue(JavaTableFunction.java:34)
    at org.h2.table.FunctionTable.getResult(FunctionTable.java:51)
    at org.h2.index.VirtualConstructedTableIndex.find(VirtualConstructedTableIndex.java:38)
    at org.h2.index.IndexCursor.find(IndexCursor.java:161)
    at org.h2.table.TableFilter.next(TableFilter.java:394)
    at org.h2.command.query.Select$LazyResultQueryFlat.fetchNextRow(Select.java:1832)
    at org.h2.result.LazyResult.hasNext(LazyResult.java:78)
    at org.h2.result.FetchedResult.next(FetchedResult.java:34)
    at org.h2.command.query.Select.queryFlat(Select.java:728)
    at org.h2.command.query.Select.queryWithoutCache(Select.java:833)
    at org.h2.command.query.Query.queryWithoutCacheLazyCheck(Query.java:197)
    at org.h2.command.query.Query.query(Query.java:512)
    at org.h2.command.query.Query.query(Query.java:475)
    at org.h2.command.CommandContainer.query(CommandContainer.java:251)
    at org.h2.command.Command.executeQuery(Command.java:190)
    at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:128)
    at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:123)
    at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:123)
    at org.owasp.dependencycheck.data.nvdcve.CveDB.updateOrInsertVulnerability(CveDB.java:1049)
    ... 7 common frames omitted
Caused by: org.h2.mvstore.MVStoreException: Reading from file sun.nio.ch.FileChannelImpl@fec3929 failed at 43616274 (length -1), read 0, remaining 1024 [2.1.214/1]
    at org.h2.mvstore.DataUtils.newMVStoreException(DataUtils.java:1004)
    at org.h2.mvstore.DataUtils.readFully(DataUtils.java:470)
    at org.h2.mvstore.FileStore.readFully(FileStore.java:98)
    at org.h2.mvstore.Chunk.readBufferForPage(Chunk.java:422)
    at org.h2.mvstore.MVStore.readPage(MVStore.java:2569)
    at org.h2.mvstore.MVMap.readPage(MVMap.java:633)
    at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1125)
    at org.h2.mvstore.Page.get(Page.java:243)
    at org.h2.mvstore.MVMap.get(MVMap.java:436)
    at org.h2.mvstore.tx.TransactionMap.getFromSnapshot(TransactionMap.java:472)
    at org.h2.mvstore.tx.TransactionMap.getFromSnapshot(TransactionMap.java:467)
    at org.h2.mvstore.db.MVPrimaryIndex.getRow(MVPrimaryIndex.java:263)
    at org.h2.mvstore.db.MVTable.getRow(MVTable.java:331)
    at org.h2.mvstore.db.MVSecondaryIndex$MVStoreCursor.get(MVSecondaryIndex.java:421)
    at org.h2.index.IndexCursor.get(IndexCursor.java:270)
    at org.h2.table.TableFilter.get(TableFilter.java:515)
    at org.h2.command.dml.Delete.update(Delete.java:59)
    at org.h2.command.dml.DataChangeStatement.update(DataChangeStatement.java:74)
    at org.h2.command.CommandContainer.update(CommandContainer.java:169)
    at org.h2.command.Command.executeUpdate(Command.java:252)
    ... 34 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
    at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:159)
    at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:814)
    at org.h2.mvstore.DataUtils.readFully(DataUtils.java:456)
    ... 52 common frames omitted

Version of dependency-check used The problem occurs using version 9.0.2 of the cli (from the owasp/dependency-check image)

To Reproduce

Run /usr/share/dependency-check/bin/dependency-check.sh --updateonly, but since transient it's hard to reproduce reliably.

Expected behavior

Database update should occur without error.

michalszelagsonos commented 9 months ago

I can confirm the same issue with gradle. During initial NVD download, I get transient 403 errors. Here's the error:

07:27:40  Checking for updates and analyzing dependencies for vulnerabilities
07:27:41  Checking for updates
07:27:43  NVD API has 231,975 records in this update
07:27:47  Downloaded 10,000/231,975 (4%)
07:27:51  Downloaded 20,000/231,975 (9%)
07:27:55  Downloaded 30,000/231,975 (13%)
07:27:59  Downloaded 40,000/231,975 (17%)
07:28:04  Downloaded 50,000/231,975 (22%)
07:28:08  Downloaded 60,000/231,975 (26%)
07:28:12  Downloaded 70,000/231,975 (30%)
07:28:16  Downloaded 80,000/231,975 (34%)
07:28:19  Downloaded 90,000/231,975 (39%)
07:28:23  Downloaded 100,000/231,975 (43%)
07:28:28  
07:28:28  Recoverable I/O exception (org.apache.hc.core5.http.ConnectionClosedException) caught when processing request to {s}->[https://services.nvd.nist.gov:443](https://services.nvd.nist.gov/)
07:28:28  Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@34f9d5bc[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5464b49e[Wrapped task = org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$ScheduledRequestExecution@2c87e1ee]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@feb0e97[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
07:28:28  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@34f9d5bc[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5464b49e[Wrapped task = org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$ScheduledRequestExecution@2c87e1ee]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@feb0e97[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
07:28:28    at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
07:28:28    at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
07:28:28    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
07:28:28    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
07:28:28    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:705)
07:28:28    at java.base/java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:721)
07:28:28    at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.executeScheduled(InternalAbstractHttpAsyncClient.java:361)
07:28:28    at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec$1.failed(AsyncHttpRequestRetryExec.java:164)
07:28:28    at org.apache.hc.client5.http.impl.async.AsyncProtocolExec$1.failed(AsyncProtocolExec.java:295)
07:28:28    at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.failed(HttpAsyncMainClientExec.java:131)
07:28:28    at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.failed(ClientHttp1StreamHandler.java:285)
07:28:28    at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.disconnected(ClientHttp1StreamDuplexer.java:220)
07:28:28    at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onDisconnect(AbstractHttp1StreamDuplexer.java:409)
07:28:28    at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.disconnected(AbstractHttp1IOEventHandler.java:95)
07:28:28    at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.disconnected(ClientHttp1IOEventHandler.java:41)
07:28:28    at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.disconnected(SSLIOSession.java:247)
07:28:28    at org.apache.hc.core5.reactor.InternalDataChannel.disconnected(InternalDataChannel.java:204)
07:28:28    at org.apache.hc.core5.reactor.SingleCoreIOReactor.processClosedSessions(SingleCoreIOReactor.java:231)
07:28:28    at org.apache.hc.core5.reactor.SingleCoreIOReactor.doTerminate(SingleCoreIOReactor.java:106)
07:28:28    at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:93)
07:28:28    at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
07:28:28    at java.base/java.lang.Thread.run(Thread.java:833)
07:28:28  
07:28:28  > Task :sampleapp:dependencyCheckAnalyze
07:28:28  Error updating the NVD Data
07:28:28  org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
07:28:28    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:340)
07:28:28    at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110)
07:28:28    at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
07:28:28    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
07:28:28    at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
07:28:28    at org.owasp.dependencycheck.gradle.tasks.AbstractAnalyze.analyze(AbstractAnalyze.groovy:100)
07:28:28    at java.base@17.0.8.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
07:28:28    at java.base@17.0.8.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
07:28:28    at java.base@17.0.8.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
07:28:28    at java.base@17.0.8.1/java.lang.reflect.Method.invoke(Method.java:568)
07:28:28    at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
07:28:28    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
07:28:28    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
07:28:28    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
07:28:28    at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:248)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
07:28:28    at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:233)
07:28:28    at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:216)
07:28:28    at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:199)
07:28:28    at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)
07:28:28    at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:105)
07:28:28    at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)
07:28:28    at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:59)
07:28:28    at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:56)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
07:28:28    at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:56)
07:28:28    at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)
07:28:28    at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:67)
07:28:28    at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:37)
07:28:28    at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
07:28:28    at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
07:28:28    at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
07:28:28    at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)
07:28:28    at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:28)
07:28:28    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:100)
07:28:28    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:72)
07:28:28    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:50)
07:28:28    at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:40)
07:28:28    at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:29)
07:28:28    at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:179)
07:28:28    at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:70)
07:28:28    at org.gradle.internal.Either$Right.fold(Either.java:175)
07:28:28    at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
07:28:28    at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:68)
07:28:28    at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:46)
07:28:28    at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
07:28:28    at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
07:28:28    at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
07:28:28    at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
07:28:28    at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:91)
07:28:28    at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:55)
07:28:28    at java.base@17.0.8.1/java.util.Optional.orElseGet(Optional.java:364)
07:28:28    at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
07:28:28    at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:37)
07:28:28    at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:65)
07:28:28    at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:36)
07:28:28    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
07:28:28    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
07:28:28    at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:77)
07:28:28    at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:38)
07:28:28    at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:94)
07:28:28    at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:49)
07:28:28    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:71)
07:28:28    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:45)
07:28:28    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNonEmptySources(SkipEmptyWorkStep.java:177)
07:28:28    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:81)
07:28:28    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:53)
07:28:28    at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
07:28:28    at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
07:28:28    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
07:28:28    at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:36)
07:28:28    at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:23)
07:28:28    at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:75)
07:28:28    at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:41)
07:28:28    at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:32)
07:28:28    at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:293)
07:28:28    at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
07:28:28    at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:21)
07:28:28    at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
07:28:28    at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
07:28:28    at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:47)
07:28:28    at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:34)
07:28:28    at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:64)
07:28:28    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:146)
07:28:28    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:135)
07:28:28    at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
07:28:28    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
07:28:28    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
07:28:28    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)
07:28:28    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
07:28:28    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
07:28:28    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
07:28:28    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
07:28:28    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
07:28:28    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
07:28:28    at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
07:28:28    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)
07:28:28    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:324)
07:28:28    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:317)
07:28:28    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
07:28:28    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
07:28:28    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
07:28:28    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
07:28:28    at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
07:28:28    at java.base@17.0.8.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
07:28:28    at java.base@17.0.8.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
07:28:28    at java.base@17.0.8.1/java.lang.Thread.run(Thread.java:833)
07:28:28  Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403
07:28:28    at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:346)
07:28:28    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:319)
07:28:28    ... 127 more
michalszelagsonos commented 9 months ago

Update, I suspect this is related to API throttling. ODC has a default delay in between requests but I suspect that isn't enough to keep us out of API throttling limit entirely. After parsing this carefully: https://nvd.nist.gov/general/news/API-Key-Announcement it sounds like the limit is applied based on an endpoint mapped to the key. I think this bit is important:

If multiple employees are transmitting requests, the rate limits are for the user’s proxy server/firewall, not the individual user.

I read this as, depending on how your networking plumbing works and how that looks to NVD, they will aggregate all of your org's requests into a single quota. This means you could get throttled even though you're not sharing the key.

aarongoldenthal commented 9 months ago

I was thinking the same thing - this must one of NVD's methods to manage rate limiting (although 403 seems like the wrong initial response unless is was preceded by a 429 somewhere, but I'm not getting rate limiting messages from ODC).

I keep a container image that's preloaded with the latest vulnerability updates for multiple users/projects. Previously, I was rebuilding the full database every 4 hours - easy, and fast, so no driver to make more complicated. Since this I've moved to a build that takes the vulnerability data from the previous image build and updates ODC with that. The key is obviously getting one good build, which was a challenge with over 230,000 records to retrieve, but since then the OCD updates have typically less than 50 records to update.

SimonTaylor commented 9 months ago

I have run into this issue when running a build from a GitHub action, for example: https://github.com/spt-development/spt-development-cid-jms-spring/actions/runs/7071629550/job/19249535231

When running the action, because a new container is spun up each time the database is empty and therefore I get throttled as others have suggested. I know of no way of re-using containers from GitHub actions (I'm no expert) and therefore @aarongoldenthal 's solution wouldn't work for me. Depending on how NVD determines when to throttle, this problem could be exacerbated when running concurrent builds for other projects.

I have worked around this for now, by not passing in the API key and accepting that the build will be slower for example: https://github.com/spt-development/spt-development-cid-jms-spring/actions/runs/7072135068/job/19250625512

It would seem straight forward to fix this problem in the maven plugin, if the API returned a more appropriate error code and/or something to indicate throttling (if that is the issue) in the response body.

NOTE I have not run into this problem (yet) when running locally with a database that doesn't require completely re-populating.

gustafg commented 9 months ago

Setting -DnvdApiDelay=6000 when using Maven worked for me as per recommendations here https://nvd.nist.gov/general/news/API-Key-Announcement It is also recommended that users "sleep" their scripts for six seconds between requests.

(The default delay in the code is currently 2000ms, perhaps this should be changed).

SimonTaylor commented 9 months ago

Ah increasing the delay as suggested by @gustafg looks as though it should help.

I have also discovered this action to avoid rebuilding the database each time: https://github.com/marketplace/actions/dependency-check

jeremylong commented 9 months ago

The action is currently only 8.4.3 - it will be upgraded soon.

jeremylong commented 9 months ago

See https://github.com/jeremylong/DependencyCheck/pull/6204 - initial attempt at documenting strategies people use to cache the H2 database to improve execution time and avoid 403 errors.

jeremylong commented 9 months ago

Are people seeing this only in environments with multiple builds that may be sharing the same NVD API key?

gustafg commented 9 months ago

I had difficulty with an API key that was not shared.

jeremylong commented 9 months ago

@gustafg how recently? and with which version of ODC?

aarongoldenthal commented 9 months ago

@jeremylong I'm seeing as a single user, but I am running on gitlab.com shared CI runners. I last saw with ODC v9.0.2 on 12/01 19:10 CST. That's the last full build that I've done, since then starting from the cache and each build has no more than 150 NVD updates.

gfukushima commented 9 months ago

Having the similar issue, using ODC v9.0.2. Using an api key throws the error below org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data... Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 404 at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:346) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:319)

Not using a key seems to get better results. Get the update and check done

kle-pra commented 9 months ago

Same error with 9.0.2 plugin ... nvdApiDelay configuration property does not seem to help. 

 [ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
...
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403
    at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:346)

...
57947 [ERROR] Failed to execute goal org.owasp:dependency-check-maven:9.0.2:check (default) on project Utils: Fatal exception(s) analyzing ArnesUtils: One or more exceptions occurred during analysis:
57947 [ERROR]   UpdateException: Error updating the NVD Data
57947 [ERROR]       caused by NvdApiException: NVD Returned Status Code: 403
57947 [ERROR]   NoDataException: No documents exist
jeremylong commented 9 months ago

You can check to ensure you API Key is valid: https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

curl -H "Accept: application/json" -H "apiKey: ########-####-####-####-############" -v https://services.nvd.nist.gov/rest/json/cves/2.0\?cpeName\=cpe:2.3:o:microsoft:windows_10:1607:\*:\*:\*:\*:\*:\*:\*

If this does not return JSON you likely need to request a new api key.

david-pulkowski commented 9 months ago

I've been getting our projects up-to-date & using the latest version 9.0.2 Gradle seems to handle the nvd.delay setting better then the cli. Initially I never set the delay and the gradle task handled it fine & we would not see any 403 errors (testing locally)

Yet, in our jenkins instance (using the plugin), if I didnt set or add the delay flag we would always get a 403 error: (I tested values with 6000,5000,4000,3000) & landed on the default value. (all those values worked and we didnt get a 403 error anymore) Leave off the delay flag and it errors out

# add in cli-setting
--nvdApiDelay 2000

# or gradle
nvd.delay = 2000

So maybe gradle is respecting the default delay without setting it, and the cli isn't? Either setting the delay in both circumstances and not relying on the default value being passed in solved it thus far for us.

jeremylong commented 9 months ago

If you have multiple builds happening at the same time - using the same API key you could hit the NVD rate limiting threshold. Ideally, in an environment with multiple builds you would implement some sort of caching strategy. The documentation on this continues to evolve/improve: https://github.com/jeremylong/DependencyCheck/pull/6220

david-pulkowski commented 9 months ago

@jeremylong thank you! Got the daily catch strategy setup. Should have done this long ago. So much more efficient!

eissymont commented 9 months ago

I have the same issue with the version 9.0.7, with the maven plugin. the execution with the API-Key starts download but runs into a 403 error: [INFO] --- dependency-check:9.0.7:check (default-cli) @ spielprojekt --- [INFO] Checking for updates [INFO] NVD API has 233.797 records in this update [INFO] Downloaded 10.000/233.797 (4%) [INFO] Downloaded 20.000/233.797 (9%) [INFO] Downloaded 30.000/233.797 (13%) [INFO] Downloaded 40.000/233.797 (17%) [INFO] Downloaded 50.000/233.797 (21%) [INFO] Downloaded 60.000/233.797 (26%) [INFO] Downloaded 70.000/233.797 (30%) [INFO] Downloaded 80.000/233.797 (34%) [INFO] Downloaded 90.000/233.797 (38%) [INFO] Downloaded 100.000/233.797 (43%) [ERROR] Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@375cb85d[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@7dad6591[Wrapped tas k = org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$ScheduledRequestExecution@784c5717]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6ad823 9a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@375cb85d[Not completed, task = java.util.concurrent.Execu tors$RunnableAdapter@7dad6591[Wrapped task = org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$ScheduledRequestExecution@784c5717]] rejected from java.util.conc urrent.ScheduledThreadPoolExecutor@6ad8239a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution (ThreadPoolExecutor.java:2055) at java.util.concurrent.ThreadPoolExecutor.reject (ThreadPoolExecutor.java:825) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute (ScheduledThreadPoolExecutor.java:340) at java.util.concurrent.ScheduledThreadPoolExecutor.schedule (ScheduledThreadPoolExecutor.java:562) at java.util.concurrent.ScheduledThreadPoolExecutor.execute (ScheduledThreadPoolExecutor.java:705) at java.util.concurrent.Executors$DelegatedExecutorService.execute (Executors.java:687) at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.executeScheduled (InternalAbstractHttpAsyncClient.java:361) at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec$1.failed (AsyncHttpRequestRetryExec.java:164) at org.apache.hc.client5.http.impl.async.AsyncProtocolExec$1.failed (AsyncProtocolExec.java:295) at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.failed (HttpAsyncMainClientExec.java:131) at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.failed (ClientHttp1StreamHandler.java:285) at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.disconnected (ClientHttp1StreamDuplexer.java:220) at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onDisconnect (AbstractHttp1StreamDuplexer.java:409) at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.disconnected (AbstractHttp1IOEventHandler.java:95) ... [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:375) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:115) at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906) .. Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403 at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:357) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:348) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:115) at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637) ... [ERROR] Failed to execute goal org.owasp:dependency-check-maven:9.0.7:check (default-cli) on project spielprojekt: Fatal exception(s) analyzing Spielprojekt: One or more exceptions occurred during analysis: [ERROR] UpdateException: Error updating the NVD Data [ERROR] caused by NvdApiException: NVD Returned Status Code: 403 [ERROR] NoDataException: No documents exist [ERROR] -> [Help 1]

if i run the same check without the API-Key it succeeds.

eissymont commented 9 months ago

@jeremylong thank you - increasing the nvdApiDelay from 2000 to 3500 solved the problem for me, waiting for the new version :-)