Open raffaem opened 2 years ago
Hi @raffaem,
Can you try using the HTTPProxyDigestAuth
class with Requests 2.25.1? There is currently a bug with Requests 2.26.0 with proxy authorization headers. I'm not certain, but I believe this may be related to what you're seeing. We're intending to release Requests 2.27.0 tomorrow which will address this issue.
Hi @raffaem,
Can you try using the
HTTPProxyDigestAuth
class with Requests 2.25.1? There is currently a bug with Requests 2.26.0 with proxy authorization headers. I'm not certain, but I believe this may be related to what you're seeing. We're intending to release Requests 2.27.0 tomorrow which will address this issue.
Nada, it fails even with requests 2.25.1
Here is the code:
#!/usr/bin/env python3
import requests
from requests_toolbelt.auth.http_proxy_digest import HTTPProxyDigestAuth
print("Requests version="+str(requests.__version__))
proxies = {
"http": "http://proxy.polimi.it:8080",
"https": "http://proxy.polimi.it:8080",
}
auth = HTTPProxyDigestAuth("USERNAME", "PASSWORD")
url = "http://verify.proxy.polimi.it"
s = requests.get(url, proxies=proxies, auth=auth)
print(s.status_code)
url = "https://dev.elsevier.com"
s = requests.get(url, proxies=proxies, auth=auth)
print(s.status_code)
Here is the output:
$ ./toolbelt_test.py
Requests version=2.25.1
200
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 366, in connect
self._tunnel()
File "/usr/lib64/python3.10/http/client.py", line 924, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 407 Proxy Authentication Required
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/raffaele/.local/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/usr/lib/python3.10/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dev.elsevier.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/run/media/raffaele/55ab61c4-83cf-4d9f-a5cd-7fcfdc14b4fb/data/progetti_miei/pybliometrics_test/./toolbelt_test.py", line 20, in <module>
s = requests.get(url, proxies=proxies, auth=auth)
File "/home/raffaele/.local/lib/python3.10/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/home/raffaele/.local/lib/python3.10/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/raffaele/.local/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/raffaele/.local/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/raffaele/.local/lib/python3.10/site-packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='dev.elsevier.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))
Works fine with the implementation of Tey' from StackOverflow
Thanks for checking @raffaem. Looking at the StackOverflow post, there's some pretty extensive monkey-patching for both Requests, urllib3 and httplib to make this possible. This isn't something we're likely to ever add directly to Requests and I'm doubtful it would even make it into the Requests toolbelt. Unfortunately, I don't see a clean way for us to do this with what's currently exposed from urllib3/httplib.
Uhm can we work with urllib3/httplib
for them to expose what we need?
HTTPProxyDigestAuth
is less useful if you can't connect to HTTPS websites, as most websites today are HTTPS
You can read all of the patching changes made here, with several of them overriding behavior from the standard library. That would require changes to CPython itself, meaning the earliest this behavior could be available is in 3.11 or 3.12. We'd need someone to drive those changes to expose APIs for urllib3 to do this, or potentially rework how urllib3 interacts with tunnel
calls. I'm not sure if the latter is even possible at first glance though.
That's all going to be an extremely time intensive change though that likely won't be widely available for at least a couple years. The short term solution may be the proposed patch in the StackOverflow question. It's not something we can support or include directly in Requests though.
http.client
in Python3.12 has made some modifications to the HTTPConnection._tunnel()
as seen here. It still raises OSError but preserves header information which can be accessed via self.get_proxy_response_headers()
.
Do these changes make it a little bit easier to extract the challenge and re-use it to send the correct header using requests or requests-toolbelt?
I understand it would require urllib3 to be updated but the change would be much smaller now that headers are accessible to urllib3 when calling _tunnel()
and failing.
Related request to httpx: https://github.com/encode/httpx/issues/2033
Requests seems not to have support for proxies which requires DigestAuth, only BasicAuth.
In this SO post there is an authorization module that implements support for proxies that require DigestAuth, and also allow to connect to HTTPS websites.
requests-toolbet (I don't know whether it is an official project of requests) has a HTTPProxyDigestAuth class, but doesn't allow to connect to HTTPS websites.