Error Updating NVD Data in Azure DevOps Pipeline using OWASP Dependency Check Task

Was85 commented 1 year ago

Description

I am encountering an issue with the OWASP Dependency Check in my Azure DevOps pipeline. The primary error is related to updating the NVD (National Vulnerability Database) data, accompanied by warnings about not using an NVD API key and other cache-related errors.

Environment

OWASP Dependency Check Version: latest Pipeline Type (YAML) ubuntu-latest

Logs

Here are the relevant parts of the log: 2023-11-25T23:41:16.3053219Z [WARN] An NVD API Key was not provided - it is highly recommended to use an NVD API key as the update can take a VERY long time without an API Key 2023-11-25T23:42:57.5728332Z [INFO] NVD API has 171,546 records in this update 2023-11-25T23:45:17.3944764Z [ERROR] Error updating the NVD Data 2023-11-25T23:45:17.3945785Z org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data 2023-11-25T23:45:17.3946843Z at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:336) 2023-11-25T23:45:17.3947901Z at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110) 2023-11-25T23:45:17.3948761Z at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:902) 2023-11-25T23:45:17.3949536Z at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:707) 2023-11-25T23:45:17.3950367Z at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:633) 2023-11-25T23:45:17.3951046Z at org.owasp.dependencycheck.App.runScan(App.java:260) 2023-11-25T23:45:17.3951596Z at org.owasp.dependencycheck.App.run(App.java:192) 2023-11-25T23:45:17.3952128Z at org.owasp.dependencycheck.App.main(App.java:87) 2023-11-25T23:45:17.3952956Z Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503 2023-11-25T23:45:17.3954037Z at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:327) 2023-11-25T23:45:17.3955077Z at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:315) 2023-11-25T23:45:17.3955809Z ... 7 common frames omitted 2023-11-25T23:45:17.6460369Z [INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json 2023-11-25T23:45:18.0123461Z [INFO] Begin database defrag 2023-11-25T23:45:18.2239840Z [INFO] End database defrag (214 ms) 2023-11-25T23:45:18.2262528Z [WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities. 2023-11-25T23:45:18.2295723Z [ERROR] Unable to continue dependency-check analysis. 2023-11-25T23:45:18.2296908Z [INFO] Element event queue destroyed: org.apache.commons.jcs3.engine.control.event.ElementEventQueue@2f48b3d2 2023-11-25T23:45:18.2305219Z [INFO] In DISPOSE, [NODEAUDIT] fromRemote [false] 2023-11-25T23:45:18.2306007Z [INFO] In DISPOSE, [NODEAUDIT] auxiliary [NODEAUDIT] 2023-11-25T23:45:18.2307796Z [INFO] In DISPOSE, [NODEAUDIT] put 0 into auxiliary [NODEAUDIT] 2023-11-25T23:45:18.2308316Z [INFO] In dispose, destroying event queue. 2023-11-25T23:45:18.2308924Z [INFO] Cache event queue destroyed: CacheEventQueue [listenerId=148583792, cacheName=NODEAUDIT] 2023-11-25T23:45:18.2318185Z [INFO] Region [NODEAUDIT] : Saving keys to: NODEAUDIT, key count: 0 2023-11-25T23:45:18.2322405Z [INFO] Region [NODEAUDIT] : Finished saving keys. 2023-11-25T23:45:18.2326375Z [INFO] Region [NODEAUDIT] : Shutdown complete. 2023-11-25T23:45:18.2336290Z [INFO] In DISPOSE, [NODEAUDIT] disposing of memory cache. 2023-11-25T23:45:18.2337114Z [INFO] Memory Cache dispose called. 2023-11-25T23:45:18.2337504Z [INFO] In DISPOSE, [CENTRAL] fromRemote [false] 2023-11-25T23:45:18.2337891Z [INFO] In DISPOSE, [CENTRAL] auxiliary [CENTRAL] 2023-11-25T23:45:18.2338356Z [INFO] In DISPOSE, [CENTRAL] put 0 into auxiliary [CENTRAL] 2023-11-25T23:45:18.2338775Z [INFO] In dispose, destroying event queue. 2023-11-25T23:45:18.2339363Z [INFO] Cache event queue destroyed: CacheEventQueue [listenerId=148583792, cacheName=CENTRAL] 2023-11-25T23:45:18.2350734Z [INFO] Region [CENTRAL] : Saving keys to: CENTRAL, key count: 0 2023-11-25T23:45:18.2363475Z [INFO] Region [CENTRAL] : Finished saving keys. 2023-11-25T23:45:18.2363895Z [INFO] Region [CENTRAL] : Shutdown complete. 2023-11-25T23:45:18.2364315Z [INFO] In DISPOSE, [CENTRAL] disposing of memory cache. 2023-11-25T23:45:18.2364706Z [INFO] Memory Cache dispose called. 2023-11-25T23:45:18.2365017Z [INFO] In DISPOSE, [POM] fromRemote [false] 2023-11-25T23:45:18.2365355Z [INFO] In DISPOSE, [POM] auxiliary [POM] 2023-11-25T23:45:18.2365715Z [INFO] In DISPOSE, [POM] put 0 into auxiliary [POM] 2023-11-25T23:45:18.2366094Z [INFO] In dispose, destroying event queue. 2023-11-25T23:45:18.2366664Z [INFO] Cache event queue destroyed: CacheEventQueue [listenerId=148583792, cacheName=POM] 2023-11-25T23:45:18.2380904Z [INFO] Region [POM] : Saving keys to: POM, key count: 0 2023-11-25T23:45:18.2386200Z [INFO] Region [POM] : Finished saving keys. 2023-11-25T23:45:18.2415948Z [INFO] Region [POM] : Shutdown complete. 2023-11-25T23:45:18.2423623Z [INFO] In DISPOSE, [POM] disposing of memory cache. 2023-11-25T23:45:18.2424022Z [INFO] Memory Cache dispose called. 2023-11-25T23:45:18.2424337Z [INFO] In dispose, destroying event queue. 2023-11-25T23:45:18.2461712Z [ERROR] Region [NODEAUDIT] : Not alive and dispose was called, filename: NODEAUDIT 2023-11-25T23:45:18.2462294Z [INFO] In dispose, destroying event queue. 2023-11-25T23:45:18.2467250Z [ERROR] Region [CENTRAL] : Not alive and dispose was called, filename: CENTRAL 2023-11-25T23:45:18.2473270Z [INFO] In dispose, destroying event queue. 2023-11-25T23:45:18.2477990Z [ERROR] Region [POM] : Not alive and dispose was called, filename: POM 2023-11-25T23:45:18.2490433Z [ERROR] One or more fatal errors occurred 2023-11-25T23:45:18.2491007Z [ERROR] Error updating the NVD Data 2023-11-25T23:45:18.2491492Z [ERROR] No documents exist 2023-11-25T23:45:18.2722574Z 2023-11-25T23:45:18.2729540Z Dependency Check completed with exit code 13.

Steps to Reproduce

Run the OWASP Dependency Check task in the Azure DevOps pipeline. Observe the errors in the pipeline logs.

Expected Behavior

The OWASP Dependency Check should be completed without errors.

Actual Behavior

The task fails with errors related to updating the NVD Data, specifically mentioning a 503 Service Unavailable error from the NVD API.

Additional Context

The issue seems to be related to the NVD API returning a 503 error code. There is also a warning about the absence of an NVD API key, which might contribute to the issue.

shehanWeer commented 1 year ago

Came across the same issue. The workaround was to set the dependencyCheckVersion: 8.4.3 instead of latest as mentioned here

Was85 commented 1 year ago

@shehanWeer What's the point? It's a vulnerability scan and has to use the latest.

Wes-Love commented 12 months ago

@shehanWeer Thanks for the workaround, at least I can get my builds working again. Is anyone actually trying to fix this issue? It looks like Jeremy has resolved/hid the issue here: https://github.com/jeremylong/DependencyCheck/pull/6153

ariguererro commented 12 months ago

I am using same plugin in Azure pipeline and I am getting this error as of today, any update? [WARN] NVD API request failures are occurring; retrying request for the 7 time [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:338) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:110) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:904) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:709) at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:635) at org.owasp.dependencycheck.App.runScan(App.java:261) at org.owasp.dependencycheck.App.run(App.java:193) at org.owasp.dependencycheck.App.main(App.java:88) Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 403 at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:346) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:317) ... 7 common frames omitted [INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json [INFO] Begin database defrag [INFO] End database defrag (283 ms) [WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities. [ERROR] Unable to continue dependency-check analysis. [ERROR] One or more fatal errors occurred [ERROR] Error updating the NVD Data [ERROR] No documents exist [INFO] Shutdown hook activated. Shutdown was not called. Shutting down JCS.

bm-fez commented 12 months ago

@shehanWeer What's the point? It's a vulnerability scan and has to use the latest.

If you use this workaround you are still using the newest dependency scan data as this is download as part of the scan process. It is just you are using version 8.4.3 of the tool to run the scan as opposed to 9.0.1

ruben0909 commented 12 months ago

It work`s but be careful with this workaround because only works until December 15th, 2023 as you can see [here](here https://github.com/dependency-check/azuredevops/issues/141#issuecomment-1824244347) and here
"December 15th, 2023, the NVD will retire its legacy data feeds while working to guide any remaining data feed users to updated application-programming interfaces"

bm-fez commented 12 months ago

Good new, I have just retested this issue now that 9.0.2 has been released of the Dependancy Checker CLI has been release.

I used this YAML - note I pass an API key as is recommended with 9.0.0 onwards

- task: dependency-check-build-task@6
  displayName: 'Vunerability Scan'
  inputs:
    projectName: '$(Projectname)'
    scanPath: '$(Build.SourcesDirectory)/**'
    format: HTML, XML
    reportsDirectory: '$(Build.ArtifactStagingDirectory)/vunerabilityscan'
    reportFilename: 'vunerabilityscan.html'
    additionalArguments: '--nvdApiKey $(nvdapikey)'

And it worked, 9.0.2 was automatically downloaded and ran.

The only point to note is that the task took around 15 minutes to run (downloading the vulnerability content via the API to the DB). Presumably future DB updates will be quicker

Update: On some of our build agents the task did fail to merge the vulnerability DB changes into the local cache. It was suggested in the 9.0.0 notes this could be an issues and some commands could be run to resolve the issues.

I took the quicker approach, on any of our agents with a DB cache merge issue I just deleted the task folder and hence the local db cache i.e. the c:\agent\_work\_task\dependency-check-build-task_47ea1f4a-57ba-414a-b12e-c44f42765e72 and reran my build

pippolino commented 12 months ago

The issue has been resolved with version 9.0.2 of DependencyCheck. The task automatically use the latest version of the DependencyCheck tool.

As suggested by @bm-fez you need to use --nvdApiKey to indicate the NVD API Key, see https://nvd.nist.gov/developers/request-an-api-key

damu9618 commented 8 months ago

Hi @pippolino I am using the owasp dependency as below My Dependency-Check Core version 9.0.9

task: dependency-check-build-task@6 displayName: Run OWASP dependency check inputs: projectName: test scanPath: path failOnCVSS: 7 format: HTML, JSON, JUNIT suppressionPath: path reportsDirectory: path reportFilename: dependency-check-report additionalArguments: '--nvdApiKey $(OWASPNVDAPIKey) --nvdApiDelay 8000' and I tried without --nvdApiDelay 8000 However it is still taking around 20minutes to finish this task. by any chance do we have any workaround/fix to reduce this time

dependency-check / azuredevops