https://services.nvd.nist.gov/ endopint is giving 503 Service Unavailable

hpriya19 commented 5 months ago

OWASP Client version: 9.0.8

We are seeing the services endpoint timeout from last couple of days

RUN /tmp/dependency-check/bin/dependency-check.sh --updateonly  --nvdApiKey ${NVD_API_KEY} --nvdApiDelay 3000
Picked up JAVA_TOOL_OPTIONS: -Dhttp.proxyHost=*** -Dhttp.proxyPort=80 -Dhttps.proxyHost=*** -Dhttps.proxyPort=80
[INFO] Checking for updates
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 7 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 7 time

We have verified the nvdApiKey is valid. The curl response to the endpoint gives the response below

curl -H "Accept: application/json" -H "apiKey:******" -v https://services.nvd.nist.gov/rest/json/cves/2.0\?cpeName\=cpe:2.3:o:microsoft:windows_10:1607:\*:\*:\*:\*:\*:\*:\*
* Uses proxy env variable https_proxy ***
*   Trying 10.68.69.53...
* TCP_NODELAY set
* Connected to <proxy> (10.68.69.53) port 80 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to services.nvd.nist.gov:443
> CONNECT services.nvd.nist.gov:443 HTTP/1.1
> Host: services.nvd.nist.gov:443
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.0 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.nvd.nist.gov
*  start date: May 31 01:13:03 2024 GMT
*  expire date: Aug 29 01:13:02 2024 GMT
*  subjectAltName: host "services.nvd.nist.gov" matched cert's "*.nvd.nist.gov"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /rest/json/cves/2.0?cpeName=cpe:2.3:o:microsoft:windows_10:1607:*:*:*:*:*:*:* HTTP/1.1
> Host: services.nvd.nist.gov
> User-Agent: curl/7.61.1
> Accept: application/json
> apiKey:****
> 
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 503 Service Unavailable
< content-length: 107
< cache-control: no-cache
< content-type: text/html
< 
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
* Connection #0 to host *** left intact

Is there a change in Service endpoint? Can we get some assistance with this.

hpriya19 commented 5 months ago

From the Web; The endpoint intermittently shows

And also TBD with initial public draft

richardzaat commented 5 months ago

We are facing the same issue, cannot update the OWASP DC database. Might it be related to NVD moving to CVSS 4.0 on June 27th ? https://nvd.nist.gov/general/news/cvss-v4-0-official-support

fc-skaviedes commented 5 months ago

We also have myriads of jobs/pipelines failing due to this issue. Though this should teach us a lesson not to rely on its availability.

himanshukumar4642 commented 5 months ago

@jeremylong Can you please help us with this issue ?

jruhe-adesso commented 5 months ago

Same here 9.2.0

kheyang commented 5 months ago

The NVD API (https://nvd.nist.gov/developers/vulnerabilities) seems on and off at the moment

1 attempt shows 200 response with expected results

[root@blabla ~]# curl -H "Accept: application/json" -v https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2019-1010218
* About to connect() to services.nvd.nist.gov port 443 (#0)
*   Trying 18.235.227.114...
* Connected to services.nvd.nist.gov (18.235.227.114) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.nvd.nist.gov
*       start date: May 31 01:13:03 2024 GMT
*       expire date: Aug 29 01:13:02 2024 GMT
*       common name: *.nvd.nist.gov
*       issuer: CN=R3,O=Let's Encrypt,C=US
> GET /rest/json/cves/2.0?cveId=CVE-2019-1010218 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: services.nvd.nist.gov
> Accept: application/json
> 
< HTTP/1.1 200 OK
< content-type: application/json
< x-frame-options: SAMEORIGIN
< access-control-allow-origin: *
< access-control-allow-headers: accept, apiKey, content-type, origin, x-requested-with
< access-control-allow-methods: GET, HEAD, OPTIONS
< access-control-allow-credentials: false
< date: Mon, 01 Jul 2024 09:13:29 GMT
< content-length: 2639
< apikey: No
< strict-transport-security: max-age=31536000
< 
{"resultsPerPage":1,"startIndex":0,"totalResults":1,"format":"NVD_CVE","version":"2.0","timestamp":"2024-07-01T09:13:30.227","vulnerabilities":[{"cve":{"id":"CVE-2019-1010218","sourceIdentifier":"josh@bress.net","published":"2019-07-22T18:15:10.917","lastModified":"2020-09-30T13:40:18.163","vulnStatus":"Analyzed","cveTags":[],"descriptions":[{"lang":"en","value":"Cherokee Webserver Latest Cherokee Web server Upto Version 1.2.103 (Current stable) is affected by: Buffer Overflow - CWE-120. The impact is: Crash. The component is: Main cherokee command. The attack vector is: Overwrite argv[0] to an insane length with execl. The fixed version is: There's no fix yet."},{"lang":"es","value":"El servidor web de Cherokee más reciente de Cherokee Webserver Hasta Versión 1.2.103 (estable actual) está afectado por: Desbordamiento de Búfer - CWE-120. El impacto es: Bloqueo. El componente es: Comando cherokee principal. El vector de ataque es: Sobrescribir argv[0] en una longitud no sana con execl. La versión corregida es: no hay ninguna solución aún."}],"metrics":{"cvssMetricV31":[{"source":"nvd@nist.gov","type":"Primary","cvssData":{"version":"3.1","vectorString":"CVSS:3.1\/AV:N\/AC:L\/PR:N\/UI:N\/S:U\/C:N\/I:N\/A:H","attackVector":"NETWORK","attackComplexity":"LOW","privilegesRequired":"NONE","userInteraction":"NONE","scope":"UNCHANGED","confidentialityImpact":"NONE","integrityImpact":"NONE","availabilityImpact":"HIGH","baseScore":7.5,"baseSeverity":"HIGH"},"exploitabilityScore":3.9,"impactScore":3.6}],"cvssMetricV2":[{"source":"nvd@nist.gov","type":"Primary","cvssData":{"version":"2.0","vectorString":"AV:N\/AC:L\/Au:N\/C:N\/I:N\/A:P","accessVector":"NETWORK","accessComplexity":"LOW","authentication":"NONE","confidentialityImpact":"NONE","integrityImpact":"NONE","availabilityImpact":"PARTIAL","baseScore":5.0},"baseSeverity":"MEDIUM","exploitabilityScore":10.0,"impactScore":2.9,"acInsufInfo":false,"obtainAllPrivilege":false,"obtainUserPrivilege":false,"obtainOtherPrivilege":false,"userInteractionRequired":false}]},"we* Connection #0 to host services.nvd.nist.gov left intact
aknesses":[{"source":"nvd@nist.gov","type":"Primary","description":[{"lang":"en","value":"CWE-787"}]},{"source":"josh@bress.net","type":"Secondary","description":[{"lang":"en","value":"CWE-120"}]}],"configurations":[{"nodes":[{"operator":"OR","negate":false,"cpeMatch":[{"vulnerable":true,"criteria":"cpe:2.3:a:cherokee-project:cherokee_web_server:*:*:*:*:*:*:*:*","versionEndIncluding":"1.2.103","matchCriteriaId":"DCE1E311-F9E5-4752-9F51-D5DA78B7BBFA"}]}]}],"references":[{"url":"https:\/\/i.imgur.com\/PWCCyir.png","source":"josh@bress.net","tags":["Exploit","Third Party Advisory"]}]}}]}

On another attempt right after shows 503 response:

[root@blabla ~]# curl -H "Accept: application/json" -v https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2019-1010218
* About to connect() to services.nvd.nist.gov port 443 (#0)
*   Trying 18.235.227.114...
* Connected to services.nvd.nist.gov (18.235.227.114) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.nvd.nist.gov
*       start date: May 31 01:13:03 2024 GMT
*       expire date: Aug 29 01:13:02 2024 GMT
*       common name: *.nvd.nist.gov
*       issuer: CN=R3,O=Let's Encrypt,C=US
> GET /rest/json/cves/2.0?cveId=CVE-2019-1010218 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: services.nvd.nist.gov
> Accept: application/json
> 
< HTTP/1.1 503 Service Unavailable
< content-length: 107
< cache-control: no-cache
< content-type: text/html
< 
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
* Connection #0 to host services.nvd.nist.gov left intact

No luck on the builds though, all of them are failing.

schilli91 commented 5 months ago

Also affected with Dependency-Check Core version 9.0.9. Is there perhaps an official page by nvd that indicates availability? I couldn't find anything.

jruhe-adesso commented 5 months ago

Workaround is autoUpdate=false

ftiercelin commented 5 months ago

another workaround:

download meta and json.gz files from https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
put them on a local http server
configure this location as data feed: <nvdDatafeedUrl>http://internal-mirror.mycorp.com/nvdcve-{0}.json.gz</nvdDatafeedUrl>

richardzaat commented 5 months ago

Workaround in our case: We have private build servers on which we have OWASP DC installed. Each team running their build runs with --noupdate, and every 4 hours we run a separate job to update all OWASP DC databases. In this case at least the builds from the teams keep running, and they never have to wait for updates. In case updating fails, and it has failed before, only we, the system team, are impacted.

mebigfatguy commented 5 months ago

probably not much this project can do anything about, altho it would be nice if it would fail fast. rather than hanging for hours.

chadlwilson commented 5 months ago

IMHO, it's probably at root due to https://github.com/jeremylong/DependencyCheck/issues/6746 and the NVD APIs getting a DDoS from many thousands of ODC clients constantly retrying requests that will never succeed (until fixed within ODC, and every user around the world upgrading from v9 to v10).

himanshukumar4642 commented 5 months ago

Any updates on this issue ? looks like 10.0.0 version has some bug while updating CVEs.

LukaszRacon commented 5 months ago

Try with setting api key --nvdApiKey that resolved issues for us with fetching data.

https://nvd.nist.gov/general/news/API-Key-Announcement https://nvd.nist.gov/developers/request-an-api-key

chadlwilson commented 5 months ago

Needing to use an API key to avoid certain errors/rate limiting has been true since v9 of ODC (especially if starting without an existing cache/database), but if you are recently upgrading from v8 -> v9 or v8 -> v10 you may have this as a root cause for your issue.

But there also appear to be some other load problems, at least yesterday.

As for bugs and compatibility problems with v10 and certain databases/setups, you probably need to follow https://github.com/jeremylong/DependencyCheck/issues/6746 . Right now I don't think there is any working version of ODC v9 or v10 that is fine across all setups. Might be OK with an H2 Database but definitely some issues with Postgres and MySQL? Not sure about v8.

jeremylong commented 5 months ago

@ftiercelin the datafeeds are no longer supported. To use them you would have to be using an unsupported version of ODC - prior to 9.0.0.

jeremylong commented 5 months ago

Yes, the NVD API is having capacity issues right now. I have seen successful calls into the API - generally using an API key. There is not much the project (ODC) can do about this.

I am releasing 10.0.1 - which contains a few minor fixes if you are using postgres or mssql as a centralized database. The release also removes a bit of debug logging that makes it look like errors occurred when they did not.

zg2pro commented 5 months ago

Hi @jeremylong Where can you follow status of that capacity issue? Do they have a page with a news feed? I've also had issues for 2 days now and being not able to update our CVE database is becoming critical.

chadlwilson commented 5 months ago

@zg2pro No, they don't.

They have widespread and well documented funding and governance problems which you can google and read about. They have had persistent periodic technical problems since the launch of the API especially.

In any case, if you are relying upon OSS and a free service from the US government's NVD for your critical workflows rather than a commercial vendor, it's all best efforts.

Personally (just a hunch), I have a feeling that the NVD capacity issues will abate as people update to ODC 10.0.1+ and gradually get their caches back in sync - or another bunch of folks get forced by the issues to relook at their caching strategy.

In any case, my H2-database ODC cache updated slowly but successfully on 10.0.1 after 3 days of nothing flowing through, so it does seem generally to be working, especially if you are not trying to populate a database from scratch.

ftiercelin commented 5 months ago

Hi @jeremylong , if the datafeed option is not relevant anymore, you might want to update the example page which is showcasing this option: http://jeremylong.github.io/DependencyCheck/dependency-check-maven/index.html#Example_5: (and btw thanks for the fantastic job, this project is really neat!)

seancorfield commented 4 months ago

In any case, my H2-database ODC cache updated slowly but successfully on 10.0.1 after 3 days of nothing flowing through, so it does seem generally to be working, especially if you are not trying to populate a database from scratch.

Sounds like I just picked a bad time to blow away my local database (it seemed to have gotten corrupted) but this was also a good opportunity to update to 10.0.1 from 9.x!

Update: it took a while, with lots of retry messages in the output, but it did succeed in rebuilding/updating the database using 10.0.1.

jeremylong commented 4 months ago

The NVD is aware of the availability issues and they are working on it; see https://groups.google.com/a/list.nist.gov/g/nvd-news/c/sJmF-2XIA80

juanmanuelromeraferrio commented 4 months ago

Hi @jeremylong, I'm currently using Dependency Check v9.0.9 with a Postgres database and have been encountering 503 errors for the NVD services since last Friday. I noticed that there are several related issues reported.

My doubt is, given my setup (v9.0.9 of Dependency Check with a Postgres database), is upgrading to v10.0.1 the only solution, or are there alternative fixes available?

Thanks!

jeremylong commented 4 months ago

Version 9 will not be able to parse the current results from the NVD. 10+ is the only currently supported version.

RobSHK commented 4 months ago

Does the 10+ version also support Oracle databases? Our NVD mirroring is hosted on Oracle DB.

jeremylong commented 4 months ago

yes

henrykuijpers commented 4 months ago

@jeremylong On a fresh build (without local database available), we still get the following warnings:

[INFO] Checking for updates

[INFO] NVD API has 255,810 records in this update

[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=2000 : 3 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 9 time

[WARNING] NVD API request failures are occurring; retrying request for the 9 time

[WARNING] NVD API request failures are occurring; retrying request for the 9 time

[WARNING] NVD API request failures are occurring; retrying request for the 9 time

[WARNING] NVD API request failures are occurring; retrying request for the 10 time

[WARNING] NVD API request failures are occurring; retrying request for the 10 time

[WARNING] NVD API request failures are occurring; retrying request for the 10 time

[WARNING] NVD API request failures are occurring; retrying request for the 10 time

[INFO] Downloaded 10,000/255,810 (4%)

[WARNING] NVD API request failures are occurring; retrying request for the 11 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=20000 : 3 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=22000 : 3 time

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[INFO] Downloaded 20,000/255,810 (8%)

[WARNING] NVD API request failures are occurring; retrying request for the 8 time

[WARNING] NVD API request failures are occurring; retrying request for the 7 time

[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=28000 : 3 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[WARNING] NVD API request failures are occurring; retrying request for the 5 time

[INFO] Downloaded 30,000/255,810 (12%)

[WARNING] NVD API request failures are occurring; retrying request for the 6 time

It does seem that it is downloading something? But, also, a lot of warnings. Not sure if we will end up with a complete database. It is taking a very long time, also.

somera commented 4 months ago

@henrykuijpers after update to 10.0.0 I deleted my DB and the initial download needen some time and I had some WARNINGs. It's normal. Cause NVD API is the problem.

henrykuijpers commented 4 months ago

But, it is still going wrong, when using an empty repo (without any DB files locally cached yet):

[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:389)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
    at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
    at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1960)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1143)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:904)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:281)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:204)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503
    at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next (NvdCveClient.java:360)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi (NvdApiDataSource.java:349)
    at org.owasp.dependencycheck.data.update.NvdApiDataSource.update (NvdApiDataSource.java:116)
    at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:906)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:711)
    at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:637)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1960)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1143)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:904)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:281)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:204)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[INFO] Begin database defrag
[INFO] End database defrag (11566 ms)
[WARNING] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.

NVD returns 503 statuses etc.

adriaanse-deltares commented 4 months ago

Here (in NL) I got it working again on my laptop, using the 10.1 update from ant, I am surprised to see the data folder is still named data\9.0 ?

I want to put the data directory here on the network so all our teamcity agents can share this update rather than each of the agents downloading their own, is that a good idea or not ? No time and resources to setup a central database for this now...

chadlwilson commented 4 months ago

Here (in NL) I got it working again on my laptop, using the 10.1 update from ant, I am surprised to see the data folder is still named data\9.0 ?

Yes it’s expected. The data directory is versioned independently, it’s just a coincidence that they matched in the last version, and there is much more than H2 DBs potentially in there.

I want to put the data directory here on the network so all our teamcity agents can share this update rather than each of the agents downloading their own, is that a good idea or not ? No time and resources to setup a central database for this now...

Yes, you can cache and share the whole data directory, but IMHO might not be a good idea to have multiple agents potentially writing to the same network folder. I’ve not seen documented a locking mechanism that would make that safe across all files in the dir. read the docs in any case: https://jeremylong.github.io/DependencyCheck/data/cacheh2.html

adriaanse-deltares commented 4 months ago

Thanks Chad, there is a lock file appearing there during the update so for now I'll just see what happens on the next update, I suppose I can disable the auto-update on all dependency check builds except for the one in trunk running most frequently...

juanmanuelromeraferrio commented 4 months ago

Version 9 will not be able to parse the current results from the NVD. 10+ is the only currently supported version.

Thanks for the response. Regarding v9, are there any plans to release an update that, while it won't support CVSS4.0, ensures compatibility without breaking?

chadlwilson commented 4 months ago

Version 9 will not be able to parse the current results from the NVD. 10+ is the only currently supported version.

Thanks for the response. Regarding v9, are there any plans to release an update that, while it won't support CVSS4.0, ensures compatibility without breaking?

@juanmanuelromeraferrio you can read some discussion about this at https://github.com/jeremylong/DependencyCheck/pull/6756 but it seems unlikely at this stage. However there shouldn't be any remaining bugs preventing you from using v10 to my knowledge and it's not really a breaking change except for folks with certain DB configurations. It will also make no difference to whether the NVD API responds with timeout related errors to have a new V9.

As to whether this needed to be done with a v10, I think it would have been possible to first release a v9 with my original compatibility patch, but then use a different approach to deal with Jeremy's concerns about lost CVSSv4 data (release v10 later after more time for testing, with CVSSv4 support and a data validity epoch "snap back* that forces re-retrieval of all CVEs since the CVSSv4 NVD go-live date when v10 is installed, which would override local last updated records as a one-off), but it's all work and like all OSS needs time and effort, sometimes more than is available for free 😅

marcelstoer commented 4 months ago

\

IMHO might not be a good idea to have multiple agents potentially writing to the same network folder.

there is a lock file appearing there during the update

We've had this setup for years with potentially ~10 agents going against the same data directory simultaneously. Haven't had any issues so far. If an agent sees there's a lock file in place (i.e. another agent running an update currently), it waits until the lock file disappears. It prints a message to that effect to the console. Of course, should the agent running the update die, the lock file won't be removed automatically thereby causing issues for the waiting agents. \

chadlwilson commented 4 months ago

We've had this setup for years with potentially ~10 agents going against the same data directory simultaneously. Haven't had any issues so far. If an agent sees there's a lock file in place (i.e. another agent running an update currently), it waits until the lock file disappears. It prints a message to that effect to the console. Of course, should the agent running the update die, the lock file won't be removed automatically thereby causing issues for the waiting agents.

Thanks @marcelstoer ! TIL!

Might be a good idea for us to update the documentation to note this as a valid configuration for the multiple-writers option. Maybe I'll have a go at that (and try and fix the formatting at the same time!)

ankurga commented 4 months ago

@chadlwilson I can also vouch for the setup that @marcelstoer is having because we are also using a similar config for more than 2 years and likewise, we also get the warning if an update is already in place and another agent tries to update the same data.

jeremylong commented 4 months ago

@ftiercelin I missed your comment about the data feed option - it is used by many organizations that create a datafeed using the vulnz-cli.

chadlwilson commented 4 months ago

@marcelstoer @ankurga If you'd like to review my suggested update to the docs at https://github.com/jeremylong/DependencyCheck/pull/6804/files?short_path=2fc6754#diff-2fc67544c882e9fcef577dbcac2c1d6285dbb9bc8d890a9637fa35d09f88e72c it may be useful :-)

ankurga commented 4 months ago

@chadlwilson Looks good. Thank you !! :)

adriaanse-deltares commented 4 months ago

Right, thanks again Chad, this setup has been running fine now with dozens of teamcity agents running on the same data directory, significantly reducing our load on the NVD backend,

njefsky commented 4 months ago

Still same issue after upgrading to 10.0.1. ` ========================== Starting Command Output =========================== "C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\agent_work_temp\e3476919-ec8c-430a-ac54-f3bd7695d183.ps1'" 09:19:52,511 |-INFO in ch.qos.logback.classic.LoggerContext[dependency-check] - Could NOT find resource [logback-test.xml] 09:19:52,511 |-INFO in ch.qos.logback.classic.LoggerContext[dependency-check] - Found resource [logback.xml] at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml] 09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs multiple times on the classpath. 09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml] 09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-9.0.9.jar!/logback.xml] 09:19:52,527 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@4d3167f4 - URL [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml] is not of type file 09:19:52,667 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 09:19:52,667 |-INFO in ch.qos.logback.classic.joran.action.ContextNameAction - Setting logger context name as [dependency-check] 09:19:52,667 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] 09:19:52,683 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [console] 09:19:52,683 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.commons.jcs] to ERROR 09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.hc] to ERROR 09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO 09:19:52,699 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [console] to Logger[ROOT] 09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 09:19:52,699 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@ed9d034 - Registering current configuration as safe fallback point

[INFO] Checking for updates [WARN] NVD API request failures are occurring; retrying request for the 1 time [WARN] NVD API request failures are occurring; retrying request for the 1 time [WARN] NVD API request failures are occurring; retrying request for the 2 time [WARN] NVD API request failures are occurring; retrying request for the 3 time [WARN] NVD API request failures are occurring; retrying request for the 1 time [WARN] NVD API request failures are occurring; retrying request for the 1 time [WARN] NVD API request failures are occurring; retrying request for the 2 time [WARN] NVD API request failures are occurring; retrying request for the 3 time [WARN] NVD API request failures are occurring; retrying request for the 4 time [ERROR] Error updating the NVD Data org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906) at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711) at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637) at org.owasp.dependencycheck.App.runScan(App.java:262) at org.owasp.dependencycheck.App.run(App.java:194) at org.owasp.dependencycheck.App.main(App.java:89) Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503 at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:357) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341) at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349) ... 7 common frames omitted [INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours. [WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities. [ERROR] Unable to continue dependency-check analysis. [ERROR] One or more fatal errors occurred [ERROR] Error updating the NVD Data [ERROR] No documents exist

[error]PowerShell exited with code '1'.

`

Chris-Turner-NIST commented 4 months ago

Looks like this issue is referenced in a lot of places. Adding this here for awareness.

See Mandatory Upgrade Notice (10.0.2)

See Reason for Upgrade

AmyChihHi commented 4 months ago

Workaround in our case: We have private build servers on which we have OWASP DC installed. Each team running their build runs with --noupdate, and every 4 hours we run a separate job to update all OWASP DC databases. In this case at least the builds from the teams keep running, and they never have to wait for updates. In case updating fails, and it has failed before, only we, the system team, are impacted.

@richardzaat can you show that about how do you run a separate job to update all OWASP DC databases?

jeremylong commented 4 months ago

@AmyChihHi see the documentation here; it sounds like they are using option 3 and caching the data directory.

chadlwilson commented 4 months ago

To add on further, probably with the single node database updater option if --noupdate is involved as described by @richardzaat earlier. To summarise what is documented, you typically either have shared (readonly) storage for the database, or you have all the readers download-unpack or rsync a copy of the data dir from somewhere central prior to running.

The specific methodology for doing so is highly dependent on your environment and what you have available to you.

AmyChihHi commented 4 months ago

@AmyChihHi see the documentation here; it sounds like they are using option 3 and caching the data directory.

I tried it but got the following error message:

[INFO] Checking for updates [ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error

Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

If your NVD API Key is valid try increasing the NVD API Delay.

If this is ocurring in a CI environment org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data; the NVD returned a 403 or 404 error

Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

If your NVD API Key is valid try increasing the NVD API Delay.

If this is ocurring in a CI environment at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:387) at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906) at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878) at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427) at org.owasp.dependencycheck.App.run(App.java:172) at org.owasp.dependencycheck.App.main(App.java:89)

chadlwilson commented 4 months ago

@AmyChihHi see https://github.com/jeremylong/DependencyCheck/issues/6816 and #6817 - unless you are getting 503s, you have a different problem than this one.

ftiercelin commented 4 months ago

@AmyChihHi in my case the error ... was actually a SSL handshake error with services.nvd.nist.gov when debugging, the error was printed, but otherwise, I was just getting [ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error

I used https://github.com/jeremylong/InstallCert to install the new cert for services.nvd.nist.gov. In my case I am behind a corporate proxy (Zscaler) which probably explains why I had to add the certs for this new url

@jeremylong : it seems the "Error updating the NVD Data; the NVD returned a 403 or 404 error" error message is not always accurate. It seems to be also triggered by caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target '

chadlwilson commented 4 months ago

@ftiercelin it'd probably be helpful if you filed any misleading error messages as a separate issue with full stack trace, as opportunities for improvement are otherwise going to get lost here (just as they did with 9.x when the switch to this domain was first done - the use of this NVD domain is only new for folks who haven't been using 9.x earlier than this change).

Also, we are now conflating 503 errors with 403 errors in one ticket which is even more confusing for folks trying to figure out whether their problem is transient or permanent (without user action).

jeremylong / DependencyCheck

https://services.nvd.nist.gov/ endopint is giving 503 Service Unavailable #6758

[error]PowerShell exited with code '1'.