search

jeremylong / DependencyCheck

OWASP dependency-check is a software composition analysis utility that detects publicly disclosed vulnerabilities in application dependencies.
https://owasp.org/www-project-dependency-check/
Apache License 2.0
6.35k stars 1.27k forks source link

Is NVD api host in throubles? #7004

Open limkinZero opened 6 days ago

limkinZero commented 6 days ago

Our ci jobs are not finishing because they exceed the timeout of one hour. The task of updating the NVD database (for the first time) does not finish. In one hour, only 20%. Do you know if there is a problem?

INFO] NVD API has 264,240 records in this update
[INFO] Downloaded 10,000/264,240 (4%)
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=10000 : 2 time
[INFO] Downloaded 20,000/264,240 (8%)
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=18000 : 2 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=32000 : 2 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=36000 : 2 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=18000 : 3 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=32000 : 3 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=38000 : 2 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=36000 : 3 time
[INFO] Downloaded 30,000/264,240 (11%)
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=32000 : 4 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=38000 : 3 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=36000 : 4 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=38000 : 4 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=34000 : 2 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=34000 : 3 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 10 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 10 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=34000 : 4 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 10 time
[WARNING] NVD API request failures are occurring; retrying request for the 11 time
aikebah commented 5 days ago

No issues seen on my side. We have a job that runs (with API key) every 4 hours to take in NVD API updates that are stored in the cveDB on the data folder on a persistent share on our CI environment.

Your logs indicate that you don't have such a persistent share, as it has the full 260k+ entries to download. You should cache the data folder of dependency-check across runs in order to play nicely with NVD rather than abusing their service to reload full dataset on each scan.

See also what NVD themselves have to say about using their API responsibly at https://nvd.nist.gov/general/news/API-Key-Announcement