blackducksoftware / hub-rest-api-python

HUB REST API Python bindings
Apache License 2.0
89 stars 104 forks source link

get_vulnerabilities call can give a stacktrace: HTTPException('got more than 100 headers') #176

Closed miketj5 closed 4 months ago

miketj5 commented 3 years ago

Trying to retrieve vulnerability information for CVE-2020-27216 (and potentially others I suppose) produces a stacktrace (or 3). The following code snippet highlights the issue (noting that line 3 runs fine, but fails on line 4):

from blackduck.HubRestApi import HubInstance
hub = HubInstance()
hub.get_vulnerabilities('CVE-2021-26117')
hub.get_vulnerabilities('CVE-2020-27216')

and the stacktrace produced is below:

Traceback (most recent call last):
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  ...
  File "C:\Users\me\.virtualenvs\blackduck\python\lib\http\client.py", line 217, in parse_headers
    raise HTTPException("got more than %d headers" % _MAXHEADERS)
http.client.HTTPException: got more than 100 headers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  ...
  File "C:\Users\me\.virtualenvs\...\python\lib\http\client.py", line 217, in parse_headers
    raise HTTPException("got more than %d headers" % _MAXHEADERS)
urllib3.exceptions.ProtocolError: ('Connection aborted.', HTTPException('got more than 100 headers'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/me/work/git/utilcode/blackduck-scripts/headers_bug_poc.py", line 4, in <module>
    hub.get_vulnerabilities('CVE-2020-27216')
  File "c:\users\me\.virtualenvs\blackduck\src\blackduck\blackduck\Vulnerabilities.py", line 15, in get_vulnerabilities
    response = self.execute_get(url, custom_headers=headers)
  File "c:\users\me\.virtualenvs\blackduck\src\blackduck\blackduck\Core.py", line 138, in execute_get
    response = requests.get(url, headers=headers, verify = not self.config['insecure'])
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', HTTPException('got more than 100 headers'))
skiyooka commented 3 years ago

We have seen this issue before for some CVEs. Please try adding the following to the script to work-around the issue:

import http.client http.client._MAXHEADERS = 1000