croudet
commented
6 months ago Same for me.
Closed OrangeDog closed 6 months ago
croudet
commented
6 months ago Same for me.
OrangeDog
commented
6 months ago It looks like perhaps there is a retry mechanism, but for some reason the connection pool has been closed and it doesn't do anything about it.
[DEBUG] Shutdown connection pool GRACEFUL
[DEBUG] c-0000000006 Shutdown connection GRACEFUL
[DEBUG] Connection pool shut down
[DEBUG] Shutdown GRACEFUL
[DEBUG] c-0000000007[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][0] Enqueued ShutdownCommand with priority IMMEDIATE
[DEBUG] c-0000000007[ACTIVE][r:w][ACTIVE][r][NOT_HANDSHAKING][0][0][0] Event cleared [w]
[DEBUG] c-0000000007[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][0] Event set [w]
[DEBUG] ex-0000000008: consume response data, len 8192 bytes
[DEBUG] ex-0000000008: consume response data, len 8128 bytes
[DEBUG] c-0000000007[ACTIVE][r:w][ACTIVE][r][NOT_HANDSHAKING][0][0][0] Event cleared [w]
[DEBUG] ex-0000000008: execution failed: Connection is closed
[DEBUG] ex-0000000008 request failed: Connection is closed
[DEBUG] ep-0000000008 close IMMEDIATE
[DEBUG] ep-0000000008 endpoint closed
[DEBUG] ep-0000000008 discarding endpoint
[DEBUG] ep-0000000008 releasing endpoint
[DEBUG] ep-0000000008 connection released [route: {s}->https://services.nvd.nist.gov:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
[DEBUG] request failed
org.apache.hc.core5.http.ConnectionClosedException: Connection is closed
at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.disconnected (ClientHttp1StreamDuplexer.java:205)
at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onDisconnect (AbstractHttp1StreamDuplexer.java:409)
at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.disconnected (AbstractHttp1IOEventHandler.java:95)
at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.disconnected (ClientHttp1IOEventHandler.java:41)
at org.apache.hc.client5.http.impl.async.LoggingIOSession$1.disconnected (LoggingIOSession.java:258)
at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.disconnected (SSLIOSession.java:247)
at org.apache.hc.core5.reactor.InternalDataChannel.disconnected (InternalDataChannel.java:204)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.processClosedSessions (SingleCoreIOReactor.java:231)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute (SingleCoreIOReactor.java:133)
at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute (AbstractSingleCoreIOReactor.java:86)
at org.apache.hc.core5.reactor.IOReactorWorker.run (IOReactorWorker.java:44)
at java.lang.Thread.run (Thread.java:829)
[DEBUG] Ticket returned At: 17:11:13; count: 23
[DEBUG] Ticket taken At: 17:11:13; count: 24
[DEBUG] Requested At: 17:11:13; URI: /rest/json/cves/2.0?virtualMatchString=cpe%3A2.3%3Aa%3A&resultsPerPage=2000&startIndex=22000
[DEBUG] ex-0000000025 preparing request execution
[DEBUG] ex-0000000025 target auth state: UNCHALLENGED
[DEBUG] ex-0000000025 proxy auth state: UNCHALLENGED
[DEBUG] ex-0000000025 acquiring connection with route {s}->https://services.nvd.nist.gov:443
[DEBUG] ex-0000000025 acquiring endpoint (3 MINUTES)
[DEBUG] ex-0000000025 endpoint lease request (3 MINUTES) [route: {s}->https://services.nvd.nist.gov:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
[DEBUG] request failed
java.lang.IllegalStateException: Connection pool shut down
at org.apache.hc.core5.util.Asserts.check (Asserts.java:38)
at org.apache.hc.core5.pool.StrictConnPool.lease (StrictConnPool.java:176)
at org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager$3.<init> (PoolingAsyncClientConnectionManager.java:271)
at org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager.lease (PoolingAsyncClientConnectionManager.java:266)
at org.apache.hc.client5.http.impl.async.InternalHttpAsyncExecRuntime.acquireEndpoint (InternalHttpAsyncExecRuntime.java:105)
at org.apache.hc.client5.http.impl.async.AsyncConnectExec.execute (AsyncConnectExec.java:141)
at org.apache.hc.client5.http.impl.async.AsyncExecChainElement.execute (AsyncExecChainElement.java:54)
at org.apache.hc.client5.http.impl.async.AsyncProtocolExec.internalExecute (AsyncProtocolExec.java:207)
at org.apache.hc.client5.http.impl.async.AsyncProtocolExec.execute (AsyncProtocolExec.java:172)
at org.apache.hc.client5.http.impl.async.AsyncExecChainElement.execute (AsyncExecChainElement.java:54)
at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec.internalExecute (AsyncHttpRequestRetryExec.java:97)
at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec.execute (AsyncHttpRequestRetryExec.java:184)
at org.apache.hc.client5.http.impl.async.AsyncExecChainElement.execute (AsyncExecChainElement.java:54)
at org.apache.hc.client5.http.impl.async.AsyncRedirectExec.internalExecute (AsyncRedirectExec.java:112)
at org.apache.hc.client5.http.impl.async.AsyncRedirectExec.execute (AsyncRedirectExec.java:278)
at org.apache.hc.client5.http.impl.async.AsyncExecChainElement.execute (AsyncExecChainElement.java:54)
at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.executeImmediate (InternalAbstractHttpAsyncClient.java:347)
at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.lambda$doExecute$0 (InternalAbstractHttpAsyncClient.java:205)
at org.apache.hc.core5.http.nio.support.BasicRequestProducer.sendRequest (BasicRequestProducer.java:93)
at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.doExecute (InternalAbstractHttpAsyncClient.java:178)
at org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient.execute (CloseableHttpAsyncClient.java:97)
at org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient.execute (CloseableHttpAsyncClient.java:107)
at org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient.execute (CloseableHttpAsyncClient.java:124)
at org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient.execute (CloseableHttpAsyncClient.java:130)
at io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient.delayedExecute (RateLimitedClient.java:179)
at io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient.lambda$execute$0 (RateLimitedClient.java:152)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
[DEBUG] Ticket returned At: 17:11:13; count: 24
[DEBUG] Rate Limited API call - waiting for 2000ms
[DEBUG] Ticket taken At: 17:11:13; count: 25
[DEBUG] Requested At: 17:11:13; URI: /rest/json/cves/2.0?virtualMatchString=cpe%3A2.3%3Aa%3A&resultsPerPage=2000&startIndex=58000
[DEBUG] ex-0000000026 preparing request execution
[DEBUG] ex-0000000026 target auth state: UNCHALLENGED
[DEBUG] ex-0000000026 proxy auth state: UNCHALLENGED
[DEBUG] ex-0000000026 acquiring connection with route {s}->https://services.nvd.nist.gov:443
[DEBUG] ex-0000000026 acquiring endpoint (3 MINUTES)
[DEBUG] ex-0000000026 endpoint lease request (3 MINUTES) [route: {s}->https://services.nvd.nist.gov:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
[DEBUG] request failed
java.lang.IllegalStateException: Connection pool shut downThat repeats for a bit, with the ex-00000... number continuing to increment.
jeremylong
commented
6 months ago Can you try increasing the delay? For the CLI it would be --nvdApiDelay 16000?
jeremylong
commented
6 months ago By default it is attempting to use 8000 without an API key.
mastermc0
commented
6 months ago I’m using an API key and still getting a 503 after a while what should I try for the delay value?On Nov 22, 2023, at 19:07, Jeremy Long @.***> wrote: By default it is attempting to use 8000 without an API key.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>
croudet
commented
6 months ago I am using the gradle plugin with the following:
dependencyCheck {
nvd {
apiKey = '<my-api-key>'
delay = 16000
}
}Still result in 503:
$ ./gradlew dependencyCheckAnalyze
> Task :dependencyCheckAnalyze
Verifying dependencies for project access
Checking for updates and analyzing dependencies for vulnerabilities
Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:336)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:902)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:707)
at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:633)
at org.owasp.dependencycheck.gradle.tasks.AbstractAnalyze.analyze(AbstractAnalyze.groovy:100)
at java.base@17.0.9/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base@17.0.9/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base@17.0.9/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@17.0.9/java.lang.reflect.Method.invoke(Method.java:568)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:248)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:73)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:233)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:216)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:199)
at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)
at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:105)
at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:59)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:56)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:78)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:56)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:67)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:37)
at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:28)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:100)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:72)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:50)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:40)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:29)
at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:179)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:70)
at org.gradle.internal.Either$Right.fold(Either.java:175)
at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:68)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:46)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:91)
at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:55)
at java.base@17.0.9/java.util.Optional.orElseGet(Optional.java:364)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:37)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:65)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:36)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:77)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:38)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:108)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:55)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:71)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:45)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNonEmptySources(SkipEmptyWorkStep.java:177)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:81)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:53)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:36)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:23)
at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:75)
at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:41)
at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.lambda$execute$2(ExecuteWorkBuildOperationFiringStep.java:66)
at java.base@17.0.9/java.util.Optional.orElseGet(Optional.java:364)
at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:66)
at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:38)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:32)
at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:293)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:21)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:47)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:34)
at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:64)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:145)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:134)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:78)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:331)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:318)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.lambda$execute$0(DefaultTaskExecutionGraph.java:314)
at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:80)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:314)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
at java.base@17.0.9/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base@17.0.9/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base@17.0.9/java.lang.Thread.run(Thread.java:840)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:327)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:315)
... 133 more
Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
Unable to continue dependency-check analysis.
Region [NODEAUDIT] : Not alive and dispose was called, filename: NODEAUDIT
Region [CENTRAL] : Not alive and dispose was called, filename: CENTRAL
Region [POM] : Not alive and dispose was called, filename: POM
> Task :dependencyCheckAnalyze FAILEDyup... apparently the NVD API is under load... maybe they'll delay the retirement of the data feeds...
rkapoor028
commented
6 months ago same is happening for us.
nscuro
commented
6 months ago Yeah even with DT and API key I'm getting rate limited to no end.
@jeremylong maybe they'll delay the retirement of the data feeds...
If that is the outcome then it'd be a win for everyone 😆 DC adopting the NVD REST API must be one of the most brutal load tests you can get! 9.0 was only released today so I'm assuming the situation will not get better as folks proceed with upgrading.
jim-liu
commented
6 months ago Can you try increasing the delay? For the CLI it would be
--nvdApiDelay 16000?
Not sure if it's relevant. I've tried a 300000ms delay with vulnz (with or without API key), but still getting 503 after a couple requests.
chadlwilson
commented
6 months ago I'm getting 404s now (while using an API key) - wonder if it's related or something else bad in the API data or dep check's use of it.
Checking for updates
Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:336)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:902)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:707)
at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:633)
at org.owasp.dependencycheck.gradle.tasks.AbstractAnalyze.analyze(AbstractAnalyze.groovy:100)
... SNIP
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 404
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:327)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:315)
... 165 moreEdit: seems this was an invalid API key somehow, see https://github.com/jeremylong/DependencyCheck/issues/6107#issuecomment-1823967207
nico-arianto
commented
6 months ago Yes, I also experiencing 503 issue with or without the NVD API Key.
tornstam
commented
6 months ago I have the same problem with an api key
[INFO] Checking for updates
[INFO] NVD API has 171,457 records in this update
[INFO] Downloaded 20,000/171,457 (12%)
[INFO] Downloaded 40,000/171,457 (23%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
...
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503
JeremyJeanson
commented
6 months ago Same situation here with an API Key 😞
andrzej-talarek
commented
6 months ago Same here - despite 16k delay.
danshome
commented
6 months ago Same here...404
One or more exceptions occurred during analysis: UpdateException: Error updating the NVD Data caused by NvdApiException: NVD Returned Status Code: 404 NoDataException: No documents exist
I have the nvdApiKey set and
chadlwilson
commented
6 months ago Looks like the 404s are most likely be to do with invalid API Keys, according to https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-404-error-occurs (with some speculation about NVD giving out API keys that don't work here, as would seem likely in my case as I am pretty confident my key was correct).
Regenerated my key and am now onto the 503s like others :-)
danshome
commented
6 months ago @chadlwilson Yep. Ran the test curl -H "Accept: application/json" -H "apiKey: ########-####-####-####-############" -v https://services.nvd.nist.gov/rest/json/cves/2.0\?cpeName\=cpe:2.3:o:microsoft:windows_10:1607:*:*:*:*:*:*:*
The key was bad. Requested and new one, and the second one they sent me works. Looks like they have some issues to workout with this new API.
danshome
commented
6 months ago @chadlwilson How are you supposed to list the API key in nvdApiKey?
or
If I list it the first way I get a 503, if I list it the second way I get a 404, but the test using curl with the new API key it works fine.
chadlwilson
commented
6 months ago ########-####-####-####-############
Just the key on its own. The header is populated by the library: https://github.com/jeremylong/Open-Vulnerability-Project/blob/27c72f84943d28448a867f803c699867fcb864d3/open-vulnerability-clients/src/main/java/io/github/jeremylong/openvulnerability/client/nvd/NvdCveClient.java#L235-L237
danshome
commented
6 months ago The curl command is working fine, but from the plugin it's getting a 503...
One or more exceptions occurred during analysis: UpdateException: Error updating the NVD Data caused by NvdApiException: NVD Returned Status Code: 503 NoDataException: No documents exist
danshome
commented
6 months ago I had to add \<nvdApiDelay>16000\</nvdApiDelay> back, now it's working. There was about a 30 second delay before it finally started to download.
danshome
commented
6 months ago @chadlwilson Have you tested with maven parallel builds before? We build with -T1C and it's looking like it might be attempting to use multiple threads to connect to the API.
I'm running with mvn -X now and I can see it downloading, but after about a minute of downloading it got...
One or more exceptions occurred during analysis: UpdateException: Error updating the NVD Data caused by NvdApiException: NVD Returned Status Code: 503 NoDataException: No documents exist
I'm going to try again and double the nvdApiDelay
danshome
commented
6 months ago Looks like they have some issues to work on. I bumped up \<nvdApiDelay>32000\</nvdApiDelay>. It got further this time, but eventually failed with a 503. Here are the debug logs showing the failure.
212764934
commented
6 months ago Same issue also for me using the nvdApiKey and nvdApiDelay set at 16000
[INFO] Finished configuration in 104 ms.
[INFO] Checking for updates
Error: Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:336)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:902)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:707)
at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:633)
at org.owasp.dependencycheck.App.runScan(App.java:260)
at org.owasp.dependencycheck.App.run(App.java:192)
at org.owasp.dependencycheck.App.main(App.java:87)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:327)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:315)
... 7 common frames omitted
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[INFO] Begin database defrag
[INFO] End database defrag (70 ms)
[WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
Error: Unable to continue dependency-check analysis.
[INFO] Element event queue destroyed: org.apache.commons.jcs3.engine.control.event.ElementEventQueue@5cde6747
[INFO] In DISPOSE, [NODEAUDIT] fromRemote [false]
[INFO] In DISPOSE, [NODEAUDIT] auxiliary [NODEAUDIT]
[INFO] In DISPOSE, [NODEAUDIT] put 0 into auxiliary [NODEAUDIT]
[INFO] In dispose, destroying event queue.
[INFO] Cache event queue destroyed: CacheEventQueue [listenerId=-6824[39](https://github.com/DaVinciLab-bh/agreement-api/actions/runs/6969070955/job/18964233867#step:4:40)66, cacheName=NODEAUDIT]
[INFO] Region [NODEAUDIT] : Saving keys to: NODEAUDIT, key count: 0
[INFO] Region [NODEAUDIT] : Finished saving keys.
[INFO] Region [NODEAUDIT] : Shutdown complete.
[INFO] In DISPOSE, [NODEAUDIT] disposing of memory cache.
[INFO] Memory Cache dispose called.
[INFO] In DISPOSE, [CENTRAL] fromRemote [false]
[INFO] In DISPOSE, [CENTRAL] auxiliary [CENTRAL]
[INFO] In DISPOSE, [CENTRAL] put 0 into auxiliary [CENTRAL]
[INFO] In dispose, destroying event queue.
[INFO] Cache event queue destroyed: CacheEventQueue [listenerId=-68243966, cacheName=CENTRAL]
[INFO] Region [CENTRAL] : Saving keys to: CENTRAL, key count: 0
[INFO] Region [CENTRAL] : Finished saving keys.
[INFO] Region [CENTRAL] : Shutdown complete.
[INFO] In DISPOSE, [CENTRAL] disposing of memory cache.
[INFO] Memory Cache dispose called.
[INFO] In DISPOSE, [POM] fromRemote [false]
[INFO] In DISPOSE, [POM] auxiliary [POM]
[INFO] In DISPOSE, [POM] put 0 into auxiliary [POM]
[INFO] In dispose, destroying event queue.
[INFO] Cache event queue destroyed: CacheEventQueue [listenerId=-68243966, cacheName=POM]
[INFO] Region [POM] : Saving keys to: POM, key count: 0
[INFO] Region [POM] : Finished saving keys.
[INFO] Region [POM] : Shutdown complete.
[INFO] In DISPOSE, [POM] disposing of memory cache.
[INFO] Memory Cache dispose called.
[INFO] In dispose, destroying event queue.
Error: Region [NODEAUDIT] : Not alive and dispose was called, filename: NODEAUDIT
[INFO] In dispose, destroying event queue.
Error: Region [CENTRAL] : Not alive and dispose was called, filename: CENTRAL
[INFO] In dispose, destroying event queue.
Error: Region [POM] : Not alive and dispose was called, filename: POM
Error: One or more fatal errors occurred
Error: Error updating the NVD Data
Error: No documents exist
hooch1964
commented
6 months ago I am having the same issue with both the command line (v9.0.0) and Azure DevOps plug-in. I get the 503 errors when testing on the command line regardless of whether I have specified an API key or not.
Reverting to v8.4.0 completes an update check and allows the completion of a scan.
ferben
commented
6 months ago Same for me. I get 503 error with or without nvdApiKey. Can it depend on used JRE/JDK version? I'm using OpenJDK 17 (LTS).
nscuro
commented
6 months ago @ferben Can it depend on used JRE/JDK version? I'm using OpenJDK 17 (LTS).
It's the NVD REST API being unable to cope with the load it's exposed to right now. Not related to the Java version at all.
OrangeDog
commented
6 months ago maybe they'll delay the retirement of the data feeds...
More likely that they won't, and then this tool becomes useless as it's currently designed. It's far cheaper for them that way.
nscuro
commented
6 months ago It's far cheaper for them that way.
Somehow I doubt this is cost-driven. Serving static files can easily be done via CDN, way cheaper than any REST API will ever be. Being out of service constantly doesn't just make the clients useless, but the API itself as well.
ferben
commented
6 months ago @ferben Can it depend on used JRE/JDK version? I'm using OpenJDK 17 (LTS).
It's the NVD REST API being unable to cope with the load it's exposed to right now. Not related to the Java version at all.
Thank you. It also can be caused by malformed http request, typo in URL etc… Because it happed when upgrade to version 9.0. Version 8.4.3 was good, I'am trying to downgrade.
ferben
commented
6 months ago Confirmed! Version 8.4.3 (without support of nvdApiKey) works, so there must be a issue in implementation #5978.
rob-spoor
commented
6 months ago @ferben version 8.x doesn't use the NVD API but the NVD data feed. You can't really compare them, they probably don't even use the same servers.
ferben
commented
6 months ago @ferben version 8.x doesn't use the NVD API but the NVD data feed. You can't really compare them, they probably don't even use the same servers.
So in v9.0 should be any option --useNvdDatafeed for backward compatibility?
https://nvd.nist.gov/vuln/data-feeds : On December 15th, 2023, the NVD plans to retire all legacy data feeds while guiding any remaining data feed users to updated application-programming interfaces (APIs)
So option is useless.
OrangeDog
commented
6 months ago @ferben the point is that the NVD data feed is being removed on December 15th.
ferben
commented
6 months ago Can you please clarify this: When I use --nvdApiKey option I use automatically NVD API calls → Understand. When I not use --nvdApiKey i use NVD API too? If yes, why -> dependency-check can work until NVD switch off Data feeds (NVD says 15 Dec, but if API servers is not stable they should postpone this deadline) without problems and after Data feeds would go offline users can switch to NVD API adding one option into command line.
OK, so I' pray that NVD stabilize API servers availability ASAP.
lexek
commented
6 months ago I may be missing something, but wouldn't it be more efficient for most of use cases to fetch information for specific CPEs from API instead of syncing the whole NVD database?
This won't work for offline use cases of course, where complete database sync is probably the only option.
oliverlockwood
commented
6 months ago @jeremylong a couple of questions:
Given I'm using a shared (Postgres) database which was fully up-to-date before I switched from 8.4.3 to 9.0.0, how come Dependency-Check reports NVD API has 171,457 records in this update - surely what I actually need to update is much much smaller than that?
Might it be feasible for Dependency-Check to write to the DB in the case where it fails part way through consuming from the NVD API? That is, where a shared database is in use, why does run 2 still report the same number of records (171,457) as run 1, when (and I admit I'm guessing here) run 1 might have been able to download some updates before falling over in a heap due to the API unavailability?
jeremylong
commented
6 months ago See the note about breaking changes
aikebah
commented
6 months ago I may be missing something, but wouldn't it be more efficient for most of use cases to fetch information for specific CPEs from API instead of syncing the whole NVD database?
This won't work for offline use cases of course, where complete database sync is probably the only option.
The part you're missing is that the NVD data is not only the source for the CVEs, but also the source for 'which applicative CPEs do exist'. CPE is an identifier that is typically assigned the first time NIST receives a CVE for a product. DependencyCheck tries to textually match components of a CPE (product, vendor) to the evidences discovered in metadata of libraries and dependencies. It's not something that is readily available for a given library.
eimarfandino
commented
6 months ago I am using version 8.4.3 wit api key and i also get the 503 https error
oliverlockwood
commented
6 months ago See the note about breaking changes
Hmmm. Looking at https://github.com/jeremylong/DependencyCheck/blob/main/core/src/main/resources/data/initialize_postgres.sql, it hasn't changed since 10 months ago, and this is the schema that I installed when I created the database.
Is there somewhere else that I should be looking for how to manage the "schema will need to be updated" part of the breaking changes to which your comment refers?
jappaleinen
commented
6 months ago Adding myself to the list of 503-experiences here. So, the problem is obviously not an issue in dependency-check to begin with, but rather a capacity issue (?) on the nvd api server side. Or something causing the nvd server to misbehave at least. But maybe the symptoms could be mitigated in dependency-check after all, playing nice(er) with the server? Suggesting (don't know if supported out of the box in whatever client library is currently used) a retry + backoff strategy. This way, fire away as many requests as possible, as quickly as possible, with an optimistic approach, but as soon the server indicates abuse (returning 503:s, 420:s or other error codes indicating server side or non critical client side errors), backoff using a delay, and increase the delay for each retry until a valid response is returned, or with a sane/configurable max-delay / max-retry setting before giving up. Don't know how many requests a full sync would result in, but it is reasonable for everyone that a every sync attempt is eventually successful, although a bit slow, rather than restarting over and over just because one of all requests is rejected without retrying it. This would just add additional load to the nvd-server, further reducing the likeliness for anyone to succeed with a full sync. Don't know if at all possible , but if it is, it could be worth exploring? Or perhaps this is already the current behaviour. If so, disregard this suggestion :-)
raduking
commented
6 months ago I agree with @jappaleinen and @oliverlockwood since for me it downloaded more than half, and then 503. After that it started again from the beginning:
[INFO] Checking for updates
[INFO] NVD API has 171,457 records in this update
[INFO] Downloaded 20,000/171,457 (12%)
[INFO] Downloaded 40,000/171,457 (23%)
[INFO] Downloaded 60,000/171,457 (35%)
[INFO] Downloaded 80,000/171,457 (47%)
[INFO] Downloaded 100,000/171,457 (58%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:336)@jeremylong @chadlwilson
I've been looking through the code and I think the issue is this...
The NVD API documentation explicitly states that maximum limit for resultsPerPage is 1000.
"resultsPerPage optional{page limit}This parameter specifies the maximum number of source records to be returned in a single API response. For network considerations, the default value and maximum allowable limit is 1,000."
You have it hardcoded to 2000...
/**
are setting the value to be explicit. */ private static final int RESULTS_PER_PAGE = 2000;
This should probably be something that uses a property and is exposed as a configuration parameter in the plugin config.
I also agree with @jappaleinen that some retry logic probably needs to be added to guard against the instability issues they are having. My experience with the NVD 2.0 API so far is that it's extremely unstable; they have capacity issues for sure. I've also had about 4 API Keys that stopped working so far, and I've had two that never worked. I think it's not ready for production, and they have some work to do.
chadlwilson
commented
6 months ago @danshome I'm just another community member here 🙏
It's not clear to me that the page size is the reason for the 503s since such responses appear to be (and typically are) fast-fail circuit breaking responses from their API gateway that may happen regardless of page size before even hitting the underlying service. This is probably why increasing delay doesn't work for an individual if the API is under sustained load and dropping x% of requests in a time period.
If we want to make ODC more robust to the challenges within the NVD infra sounds most sensible to me to explore the ability to save the results-in progress so things don't have to restart all over again, however ability to do that typically depends on the API design when dealing with paging. I haven't looked at the NVD APIs enough to personally determine if the current design is unavoidable (where it appears to restart the whole process after it has given up on a 503).
Other than that, we can hope that NVD are working on getting improved caching (CDN, gateway or internally) of their API responses in place so the situation improves in the next few weeks before the current deadline.
ankurga
commented
6 months ago Even after using the key. I am still getting the same error.
[INFO] NVD API has 171,457 records in this update
[INFO] Downloaded 20,000/171,457 (12%)
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:336)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:110)
at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:902)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:707)
at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:633)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1936)
at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1119)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503
mmaeller
commented
6 months ago The NVD API documentation explicitly states that maximum limit for resultsPerPage is 1000.
"resultsPerPage optional{page limit}This parameter specifies the maximum number of source records to be returned in a single API response. For network considerations, the default value and maximum allowable limit is 1,000."
You have it hardcoded to 2000...
/* The number of results per page from the NVD API. The default is 2000; we are setting the value to be explicit. / private static final int RESULTS_PER_PAGE = 2000;
This should probably be something that uses a property and is exposed as a configuration parameter in the plugin config.
@danshome Unfortunately, it looks like there are either two documentation sources or your docs are outdated. Please see CVE API
This parameter specifies the maximum number of CVE records to be returned in a single API response. For network considerations, the default value and maximum allowable limit is 2,000.
It is recommended that users of the CVE API use the default resultsPerPage value. This value has been optimized to allow the greatest number of results over the fewest number of requests.
ZheSun88
commented
6 months ago I have the same issue with using the dependency-check-maven:9.0.0
mvn org.owasp:dependency-check-maven:9.0.0:check -DnvdApiKey=*** -DnvdApiDelay=16000
compare with others log, I dont even have the info for the progress part (Downloaded 20,000/171,457 (12%)), others are the same
[INFO] --- dependency-check:9.0.0:check (default-cli) @ test-project ---
[INFO] Checking for updates
[INFO] NVD API has 171▒457 records in this update
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:336)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:110)
PatrickGaissert
commented
6 months ago We also experience the issue.
Describe the bug The default configuration, with an API key, is either making requests too quickly, or not retrying enough, or both. It always eventually fails with a 503 error from NVD.
Version of dependency-check used Maven plugin 9.0.0
Log file
The debug log in this case is a nightmare as it logs every raw request and response, 16 bytes at a time, without synchronisation so they're all misordered. I'm not going through that to sanitise my API key.
To Reproduce
mvn dependency-check:checkExpected behavior Successful completion of the download.