Proxy Issue - 407 Proxy Authentication Required

[Kenith]

Hello Requests Team,

We found that the proxy setting is not working. We tried every possible proxy auth, only one success: using cntlm + username + password, like: http://username:password@127.0.0.1:3128

System Info: Python 2.7 Requests 2.21.0

How to reproduce:

1. Make sure you are behind a proxy
2. Run the following code to reproduce issue

   
   import requests
   from requests.auth import HTTPProxyAuth
   import base64

url = 'https://alexgaynor.net/'

user = 'username' password = 'password'

proxy1 = 'http://{0}:{1}@proxyserver.com:80'.format(user, password) # not working proxy2 = 'http://proxyserver.com:80' # not working proxy3 = 'http://{0}:{1}@127.0.0.1:3128'.format(user, password) # this working by using cntlm + username + password proxy4 = 'http://127.0.0.1:3128' # not working

import os

os.environ["HTTPS_PROXY"] = proxy3

os.environ["HTTP_PROXY"] = proxy3

proxies = {'http': proxy1, 'https': proxy1} proxy_auth_1 = 'Basic ' + base64.b64encode("{0}:{1}".format(user, password)) proxy_auth_2 = HTTPProxyAuth(user, password) headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36', 'Proxy-Authorization': proxy_auth_1}

session = requests.Session() session.headers = headers session.verify = False session.proxies = proxies session.auth = proxy_auth_2

session.trust_env = False

res = session.get(url, verify=False, headers=headers, proxies=proxies, auth=proxy_auth_2)

status_code = res.status_code

print status_code

3. You could get the following error:

/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/ken/PycharmProjects/Debug/Debug4.py Traceback (most recent call last): File "/Users/ken/PycharmProjects/Debug/Debug4.py", line 34, in res = session.get(url, verify=False, headers=headers, proxies=proxies, auth=proxy_auth_2) File "/Library/Python/2.7/site-packages/requests/sessions.py", line 546, in get return self.request('GET', url, kwargs) File "/Library/Python/2.7/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/Library/Python/2.7/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/Library/Python/2.7/site-packages/requests/adapters.py", line 510, in send raise ProxyError(e, request=request) requests.exceptions.ProxyError: HTTPSConnectionPool(host='alexgaynor.net', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',)))

Process finished with exit code 1

[Kenith]

Found the root cause, for some certain url, our company security mechanism will redirect the url to the following url, that's the reason, 407 error happens.

Why the proxy auth does not follow to the redirected url?

https://safe.menlosecurity.com/safeview-auth-server/proxy_auth?url=aHR0cHM6Ly9hbGV4Z2F5bm9yLm5ldC8=&ip=MnwxOjB8MTA6MTU1MzE0NjAwM3wyOmlwfDIwOk1qQXpMakV4TUM0eU16VXVNak09fDU0ODEyNzk0ZDk5NzU4YjA3NWNkYTM4ZGJkYWVlYWNlN2UzMjViMzM4YjBhMTEzOTk3MGU2MzgyYzM4MTZlNWQ=&pnr=2&saml=true

[gabrielnoguerales]

Prob fixed with pr#5030

[blresells]

how was this fixed, i have the same issue

[mirandadam]

@blresells @gabrielnoguerales it seems #5030 was not merged. I did not understand the patch and I am unsure whether it should have been merged or not.

I have the same issue - my corporate proxy relies on NTLM, I am running cntlm locally and python requests does not send headers on connect. Firefox works fine.

I have run wireshark to tell the difference between python requests and firefox and it seems python requests does not send the "User-Agent", "Host", "Proxy-Connection: keep-alive" and "Connection: keep-alive" headers upon CONNECTing. I have tried to force python requests to send those headers (especially the "Host" one) with no success.

I have tried the cntlm + username + password method sugested by the OP also with no success.

I believe this issue concerns the headers that python requests sends with CONNECT.

[mirandadam]

I am using cntlm and this works (@Kenith, @blresells, this may be a workaround):

import urllib3
p = urllib3.ProxyManager(
    "http://127.0.0.1:3128/",
    proxy_headers={'Proxy-Connection': 'Keep-Alive'})
r = p.request('GET', 'https://www.google.com')

This does not, and reads exactly like the python requests CONNECT in wireshark:

import urllib3
p = urllib3.ProxyManager(
    "http://127.0.0.1:3128/",
    proxy_headers={})
r = p.request('GET', 'https://www.google.com')

The output in the second case is:

Traceback (most recent call last):
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/connection.py", line 366, in connect
    self._tunnel()
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/http/client.py", line 907, in _tunnel
    raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 407 Proxy Authentication Required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/a/Desktop/git/teste-api-maxar/teste.py", line 35, in <module>
    r = p.request('GET', 'https://www.google.com')
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/poolmanager.py", line 532, in urlopen
    return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/srv/miniconda/envs/apimaxar/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))

So, the real issue is that: 1) Python requests does not send the 'Proxy-Connection': 'keep-alive' header by default, like Firefox and curl do (also does not send the Host and the User-Agent, but for my setup that is not important). 2) As far as I know, there is no way for the user to add a proxy_headers to a python requests session.

There is a simple workaround, though, which is to revert to urllib3 which exposes lower level funcionality.

[prosper86]

@blresells @gabrielnoguerales it seems #5030 was not merged. I did not understand the patch and I am unsure whether it should have been merged or not.

I have the same issue - my corporate proxy relies on NTLM, I am running cntlm locally and python requests does not send headers on connect. Firefox works fine.

I have run wireshark to tell the difference between python requests and firefox and it seems python requests does not send the "User-Agent", "Host", "Proxy-Connection: keep-alive" and "Connection: keep-alive" headers upon CONNECTing. I have tried to force python requests to send those headers (especially the "Host" one) with no success.

I have tried the cntlm + username + password method sugested by the OP also with no success.

I believe this issue concerns the headers that python requests sends with CONNECT.

HI I'm getting the same problem with my proxy it doesn't give mee the credentials I'm getting 407 Proxy-Authenticate how can I bypass the 407 Proxy-Authenticate?

[prosper86]

Found the root cause, for some certain url, our company security mechanism will redirect the url to the following url, that's the reason, 407 error happens.

Why the proxy auth does not follow to the redirected url?

https://safe.menlosecurity.com/safeview-auth-server/proxy_auth?url=aHR0cHM6Ly9hbGV4Z2F5bm9yLm5ldC8=&ip=MnwxOjB8MTA6MTU1MzE0NjAwM3wyOmlwfDIwOk1qQXpMakV4TUM0eU16VXVNak09fDU0ODEyNzk0ZDk5NzU4YjA3NWNkYTM4ZGJkYWVlYWNlN2UzMjViMzM4YjBhMTEzOTk3MGU2MzgyYzM4MTZlNWQ=&pnr=2&saml=true

HI what type of a browser you using? I have same proxy with my proxy

[sethmlarson]

We can't help debug your personal proxy for you as they are all different and sometimes very precise with what they want you to provide. If you can show this is a defect in Requests instead of a misconfiguration of your proxy then we can reopen.