search

vehemont / nvdlib

A simple wrapper for the National Vulnerability CVE/CPE API
https://nvdlib.com
MIT License
81 stars 27 forks source link

nvd endpoint down? #45

Open mhdawson opened 3 days ago

mhdawson commented 3 days ago

The automation we have in the Node.js project has been reporting errors for the last week or so. I updated to the latest version of nvdlib but still see the issue.

From the errors it looks like an issue on the server side:

Traceback (most recent call last):
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 261, in <module>
    exit(main())
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 236, in main
    nvd_vulnerabilities: list[Vulnerability] = query_nvd(
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line [15](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/9803358356/job/27069412872#step:6:16)5, in query_nvd
    for cve in searchCVE(
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/nvdlib/cve.py", line [16](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/9803358356/job/27069412872#step:6:17)1, in searchCVE
    raw = __get('cve', headers, parameters, limit, verbose, delay)
  File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/nvdlib/get.py", line 27, in __get
    raw.raise_for_status()
  File "/opt/hostedtoolcache/Python/3.9.[19](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/9803358356/job/27069412872#step:6:20)/x64/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://services.nvd.nist.gov/rest/json/cves/2.0?virtualMatchString=cpe%3A2.3%3Aa%3Aada-url%3Aada%3A2.7.8%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A
Error: Process completed with exit code 1.

Are other people having the same issue?

johnlabuyfoy1024 commented 3 days ago

I received this from @.**@.> on July 2, 2024:

"...We are aware of availability issues with the NVD API Endpoints and are working to resolve them..."

From: Michael Dawson @.> Sent: Friday, July 5, 2024 9:38 AM To: vehemont/nvdlib @.> Cc: Subscribed @.***> Subject: [vehemont/nvdlib] nvd endpoint down? (Issue #45)

The automation we have in the Node.js project has been reporting errors for the last week or so. I updated to the latest version of nvdlib but still see the issue.

From the errors it looks like an issue on the server side:

Traceback (most recent call last):

File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 261, in 

exit(main())

File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 236, in main

nvd_vulnerabilities: list[Vulnerability] = query_nvd(

File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 155, in query_nvd

for cve in searchCVE(

File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/nvdlib/cve.py", line 161, in searchCVE

raw = __get('cve', headers, parameters, limit, verbose, delay)

File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/nvdlib/get.py", line 27, in __get

raw.raise_for_status()

File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status

raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://services.nvd.nist.gov/rest/json/cves/2.0?virtualMatchString=cpe%3A2.3%3Aa%3Aada-url%3Aada%3A2.7.8%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A%3A%2A

Error: Process completed with exit code 1.

Are other people having the same issue?

- Reply to this email directly, view it on GitHubhttps://github.com/vehemont/nvdlib/issues/45, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACCWEOIN5OXRYPYC7NXOAUTZK2OSZAVCNFSM6AAAAABKNHIDLGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4TENRSHEZTMNY. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

vehemont commented 2 days ago

Yeah it looks like the NVD API is having issues at the moment. They pushed some additional features on July 2nd to the API which may have broke some stuff. We can only sit tight and wait until NVD resolves the issue. If you are unsure of the status of the issue, you can email the NVD at nvd@nist.gov. I will leave this issue open so users are aware of the on-going problem, and will close it once we can confirm it is resolved.

I got a timeout on a simple search, so it seems to be a 503 error, or timeout. Not sure what other results we could be observing. 


>>> r = nvdlib.searchCVE(keywordSearch="Microsoft")
Traceback (most recent call last):
  File "C:\Users\***\AppData\Roaming\Python\Python312\site-packages\urllib3\connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "C:\Users\***\AppData\Roaming\Python\Python312\site-packages\urllib3\connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\http\client.py", line 1411, in getresponse
    response.begin()
  File "C:\Program Files\Python312\Lib\http\client.py", line 324, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\http\client.py", line 285, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\socket.py", line 707, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\ssl.py", line 1249, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\ssl.py", line 1105, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out```
johnlabuyfoy1024 commented 6 hours ago

Now I'm consistently getting a JSON decode error - the error occurs at a different location in the download from session to session.

Here is an example:

https://services.nvd.nist.gov/rest/json/cves/2.0?cvssV3Severity=MEDIUM&pubStartDate=2023-07-13T14:11:36.755588&pubEndDate=2023-11-10T14:11:36.755588&resultsPerPage=2000&startIndex=2000 Traceback (most recent call last): File "/home/johnf/.local/lib/python3.10/site-packages/requests/models.py", line 974, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3/dist-packages/simplejson/init.py", line 525, in loads return _default_decoder.decode(s) File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode obj, end = self.raw_decode(s) File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/johnf/Scripts/./acquire-reform.py", line 613, in datclass.get_dat(datStore, kev_df, epss_df) # Call the get_dat() class function passing the datStore List as argument File "/home/johnf/Scripts/./acquire-reform.py", line 102, in get_dat r = nvdlib.searchCVE(pubStartDate=dat_list[i], pubEndDate=dat_list[i+1], cvssV3Severity = dat_list[i+2], key='a3340ee7-6c27-4f56-a7b6-55cbafa0cf1e',verbose=True,timeout=120) File "/home/johnf/.local/lib/python3.10/site-packages/nvdlib/cve.py", line 158, in searchCVE raw = get('cve', headers, parameters, limit, verbose, delay, timeout) File "/home/johnf/.local/lib/python3.10/site-packages/nvdlib/get.py", line 71, in get getData = getReq.json()[path] File "/home/johnf/.local/lib/python3.10/site-packages/requests/models.py", line 978, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Nothing has changed on my end. Prior to July, I was able to successfully download data without errors. I've reached out to nvd.nist.gov