nvdlib.searchCVE occasionally times out

[johnlabuyfoy1024]

I'm using nvdlib-0.7.4 and occasionally I get a time out error from urllib3 when executing: r = nvdlib.searchCVE(cveId=s2,key='xyz',verbose=True,delay=6) Timeout is currently set to 30 Is there a way I can increase the timeout? Thanks

[vehemont]

Hi,

Can you please provide the entire error output when the timeout occurs?

Thanks

Edit: https://github.com/vehemont/nvdlib/issues/26#issuecomment-1645806289 NVD API is having issues at the moment.

[SSarka69]

Hi,

I am also facing similar timeout issue occasionally. Please find the attached picture for reference and is there any solution to this error?

Thanks

[johnlabuyfoy1024]

Code:

Get the details of the CVE from NVD Database using nvdlib

           r = []                       #Initialize r
           r = nvdlib.searchCVE(cveId=s2,key='xyzxyzxyzxyz',verbose=True)

Error Message:

Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22954 Traceback (most recent call last): File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request httplib_response = conn.getresponse() File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 1374, in getresponse response.begin() File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/home/johnf/anaconda3/lib/python3.10/socket.py", line 705, in readinto return self._sock.recv_into(b) File "/home/johnf/anaconda3/lib/python3.10/ssl.py", line 1274, in recv_into return self.read(nbytes, buffer) File "/home/johnf/anaconda3/lib/python3.10/ssl.py", line 1130, in read return self._sslobj.read(len, buffer) TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/johnf/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send resp = conn.urlopen( File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen retries = retries.increment( File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise raise value File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 451, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/johnf/cve_manager_VS/./first-api-epss.py", line 169, in r = nvdlib.searchCVE(cveId=s2,key='a3340ee7-6c27-4f56-a7b6-55cbafa0cf1e',verbose=True) File "/home/johnf/anaconda3/lib/python3.10/site-packages/nvdlib/cve.py", line 156, in searchCVE raw = get('cve', headers, parameters, limit, verbose, delay) File "/home/johnf/anaconda3/lib/python3.10/site-packages/nvdlib/get.py", line 23, in get raw = requests.get(link, params=stringParams, headers=headers, timeout=30) File "/home/johnf/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, kwargs) File "/home/johnf/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, kwargs) File "/home/johnf/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, send_kwargs) File "/home/johnf/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, kwargs) File "/home/johnf/.local/lib/python3.10/site-packages/requests/adapters.py", line 578, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

[vehemont]

Thank you for the output. Could you please see if you still experience the issue after adding delay=12 to nvdlib.searchCVE?

Example: r = nvdlib.searchCVE(cveId=s2,key='xyzxyzxyzxyz',verbose=True, delay=12)

This will add a 12 second delay to the search. NVD has firewall rules in place that could be dropping the connection (or putting up a 404) because it believes your IP address to be going over the rate limit. NVDLib has built-in rate limiting, but it can be bypassed unintentionally by stopping and starting code repeatedly that makes searches.

[johnlabuyfoy1024]

Still getting timeout error - Code:

Get the details of the CVE from NVD Database using nvdlib

           r = []                       #Initialize r
           r = nvdlib.searchCVE(cveId=s2,key='xyzxyzxyzxyz',verbose=True, delay=12)

/Error message:

Now running first-api-epss.py url = https://api.first.org/data/v1/epss?order=!epss&limit=1500 Wrote intermediate output-epss.txt Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-30525 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22947 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22965 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22954 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-26134 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-1388 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22963 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-37061 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-35405 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-29464 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-26352 Filter: https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-37042 Traceback (most recent call last): File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request httplib_response = conn.getresponse() File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 1374, in getresponse response.begin() File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/home/johnf/anaconda3/lib/python3.10/socket.py", line 705, in readinto return self._sock.recv_into(b) File "/home/johnf/anaconda3/lib/python3.10/ssl.py", line 1274, in recv_into return self.read(nbytes, buffer) File "/home/johnf/anaconda3/lib/python3.10/ssl.py", line 1130, in read return self._sslobj.read(len, buffer) TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/johnf/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send resp = conn.urlopen( File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen retries = retries.increment( File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise raise value File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 451, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/johnf/cve_manager_VS/./first-api-epss.py", line 169, in r = nvdlib.searchCVE(cveId=s2,key='a3340ee7-6c27-4f56-a7b6-55cbafa0cf1e',verbose=True, delay=12) File "/home/johnf/anaconda3/lib/python3.10/site-packages/nvdlib/cve.py", line 156, in searchCVE raw = get('cve', headers, parameters, limit, verbose, delay) File "/home/johnf/anaconda3/lib/python3.10/site-packages/nvdlib/get.py", line 23, in get raw = requests.get(link, params=stringParams, headers=headers, timeout=30) File "/home/johnf/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, kwargs) File "/home/johnf/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, kwargs) File "/home/johnf/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, send_kwargs) File "/home/johnf/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, kwargs) File "/home/johnf/.local/lib/python3.10/site-packages/requests/adapters.py", line 578, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

[jacobocasado]

I get the same error. Could be a srv issue?

[vehemont]

nvdlib-0.7.5-py3-none-any.zip

Attached is a compressed wheel file that contains an update to allow a custom timeout parameter. By default it set to 30 seconds, but you can enter a custom value. I tested it when setting the value to 60 and it seems to work for me, but please try it and see if it resolves your issue. If it works, then I will release this version.

Example: r = nvdlib.searchCVE(cveId=s2,key='xyzxyzxyzxyz',verbose=True, timeout=60)

To install the wheel, you must first right click and extract the contents, then navigate to the same directory as the wheel file and run the following. You might need to first uninstall the current nvdlib by running pip uninstall nvdlib then running the below: python install nvdlib-0.7.5-py3-none-any.whl

I believe this is caused by the NVD server taking too long to reply, which isn't an issue as long as we an define an appropriate timeout.

[jacobocasado]

I'll try it today, thx!

[johnlabuyfoy1024]

I tried nvdlib 0.7.5 and with timeout = 60 and some other settings as well and still get read timeout errors and HTTP 503 errors. However, I emailed the nvd today and got this response which seems to confirm that the problem is on the nvd side:

"Thank you for notifying the NVD. We are aware of sporadic errors being provided to users of the APIs and are investigating the root cause for resolution. We apologize for the inconvenience during this time."

[vehemont]

Thanks for reaching out to them. I will keep this issue open in the mean time so people can find updates.

[mhdawson]

We've been getting persistent timeouts like this

Run (
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connection.py", line 454, in getresponse
    httplib_response = super().getresponse()
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/ssl.py", line 1242, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/ssl.py", line 1100, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 538, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 261, in <module>
    exit(main())
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 236, in main
    nvd_vulnerabilities: list[Vulnerability] = query_nvd(
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line [15](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:16)5, in query_nvd
    for cve in searchCVE(
  File "/opt/hostedtoolcache/Python/3.9.[17](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:18)/x64/lib/python3.9/site-packages/nvdlib/cve.py", line 270, in searchCVE
    raw = __get('cve', headers, parameters, limit, verbose, delay)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/nvdlib/get.py", line [22](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:23), in __get
    raw = requests.get(link, params=stringParams, headers=headers, timeout=[30](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:31))
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/adapters.py", line 5[32](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:33), in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='services.nvd.nist.gov', port=4[43](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:44)): Read timed out. (read timeout=30)
Error: Process completed with exit code 1.

Since Wednesday July 5th.

I upgraded to 0.7.4 today but that did not change the behaviour.

I then found this discussion and see the issue is likely on the nvd side. Do those on the thread think it is worth adjusting the timeout (which is 6) etc. or just waiting until we hear the the problem on the nvd side is resolved.

[vehemont]

Do those on the thread think it is worth adjusting the timeout (which is 6) etc. or just waiting until we hear the the problem on the nvd side is resolved.

From the evidence in this thread, changing the timeout does not make any difference. It will be best to wait for an update from NVD.

[mhdawson]

@vehemont thanks for confirming.

[Fares-Harri]

using nvdlib 0.7.4, for around 10 days it is not working for me and now I tested this https://services.nvd.nist.gov/rest/json/cves/2.0?keywordSearch=django I get the response after around 1:40 minutes and sometimes I just get 503/504 server error response is it the same issue?

[vehemont]

using nvdlib 0.7.4, for around 10 days it is not working for me and now I tested this https://services.nvd.nist.gov/rest/json/cves/2.0?keywordSearch=django I get the response after around 1:40 minutes and sometimes I just get 503/504 server error response is it the same issue?

Yes. That is the issue we are experiencing at this time. Feel free to reach out to the NVD and inform them the issue is still occurring.

[johnlabuyfoy1024]

Update from NVD on timeout / 503 errors on 7-17-2023:

"We are aware of the root cause for the issue and are working to find appropriate resolutions that will have minimal impact to the userbase while improving reliability of the site and search. We do not have a timeline for this currently and appreciate your patience during this period."

[vehemont]

I will push out an update disabling the keyword and keyword exact match options. I will re-enable them once the issue is resolved.

Update from the NVD:

Keyword and Keyword Exact Match Searches Temporarily Disabled The NVD has been experiencing issues with website and API availability. We have identified the root cause, however, due to the particular complexities and other operational needs, a larger scale solution must be put into place. This will take time to implement and resolve. In the interim, to ensure continuity of services that are not impacted, we will be disabling both the keyword and keyword exact match capabilities of the vulnerability search page and APIs. We are aware that this will impact the daily efforts of many that make use of our data and request understanding and patience while we move towards a viable solution. For questions and concerns you can contact nvd@nist.gov . Please refrain from requesting timelines on resolution, we will notify all users through the various channels available when we have information to share on the topic. V/r, The National Vulnerability Database Team

[mhdawson]

@vehemont I assume it 0.7.5 that I should wait for in terms of the above?

[vehemont]

NVD released news regarding this issue. NVDLib should be back in functioning order. I don't think any changes will need to be made to NVDLib but I'll do some testing and find out.

Keyword and keyword exact match searches have been re-enabled. Clarifications on how keyword search operates can be found in the documentation for keyword parameters. For questions and concerns you can contact nvd@nist.gov .

[Fares-Harri]

Yes it is working now again

[mhdawson]

Seems to be working for the nodejs automation, thanks for this issue and the updates.