Proxy support not working with squid proxy

[GoogleCodeExporter]

I am not able to use proxy support while httplib and urrlib2 works fine.

I'm using the newest version of httplib2.

Here's the code that I use:
    proxy = {
             'host':'http://144.16.192.245',
             'port':8000}
    httplib2.debuglevel=4
    conn = httplib2.Http(proxy_info =
httplib2.ProxyInfo(myhttp.socks.PROXY_TYPE_HTTP, proxy['host'], proxy['port']))
    # ...snip...
    response,content = conn.request(url, headers = headers1)    
    return response, content

And this is the output I see:
INFO:404trainer:Retrieving http://www.example.org/
Traceback (most recent call last):

  File "/home/anirudhs/workspace/iviz/src/test.py", line 71, in <module>
    response, content = http_request("http://www.example.org/")
  File "/home/anirudhs/workspace/iviz/src/test.py", line 32, in http_request
    response,content = conn.request(url, headers = headers1)    
  File "/home/anirudhs/workspace/iviz/src/httplib2/__init__.py", line 1071,
in request
    (response, content) = self._request(conn, authority, uri, request_uri,
method, body, headers, redirections, cachekey)
  File "/home/anirudhs/workspace/iviz/src/httplib2/__init__.py", line 863,
in _request
    (response, content) = self._conn_request(conn, request_uri, method,
body, headers)
  File "/home/anirudhs/workspace/iviz/src/httplib2/__init__.py", line 836,
in _conn_request
    raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
httplib2.ServerNotFoundError: Unable to find the server at www.example.org

Please delete this issue if it's not a bug. I just need proxy support as I
do not have a direct connection.

Original issue reported on code.google.com by prototyp...@gmail.com on 26 Oct 2008 at 3:02

[GoogleCodeExporter]

Try '144.16.192.245', instead of 'http://144.16.192.245'.

BTW, it looks like SocksiPy only supports HTTP proxies that support tunneling 
via 
CONNECT method. ( http://socksipy.sourceforge.net/readme.txt )

If your HTTP proxy does not support CONNECT method, you'll get 403 error.
( happened to me )

Original comment by RyuSun...@gmail.com on 5 Nov 2008 at 1:28

[GoogleCodeExporter]

urllib2 works fine for me.
with httplib2 I get the same error as  RyuSungho above.

Traceback (most recent call last):
  File "<pyshell#22>", line 1, in <module>
    response,content = conn.request("http://google.com")
  File "C:\Python25\lib\site-packages\httplib2\__init__.py", line 1050, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method,
body, headers, redirections, cachekey)
  File "C:\Python25\lib\site-packages\httplib2\__init__.py", line 854, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "C:\Python25\lib\site-packages\httplib2\__init__.py", line 823, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "C:\Python25\Lib\httplib.py", line 866, in request
    self._send_request(method, url, body, headers)
  File "C:\Python25\Lib\httplib.py", line 889, in _send_request
    self.endheaders()
  File "C:\Python25\Lib\httplib.py", line 860, in endheaders
    self._send_output()
  File "C:\Python25\Lib\httplib.py", line 732, in _send_output
    self.send(msg)
  File "C:\Python25\Lib\httplib.py", line 699, in send
    self.connect()
  File "C:\Python25\lib\site-packages\httplib2\__init__.py", line 705, in connect
    self.sock.connect(sa)
  File "C:\Python25\lib\site-packages\socks.py", line 383, in connect
    self.__negotiatehttp(destpair[0],destpair[1])
  File "C:\Python25\lib\site-packages\socks.py", line 349, in __negotiatehttp
    raise HTTPError((statuscode,statusline[2]))
HTTPError: (403, 'Forbidden')

Original comment by Almaten....@gmail.com on 4 Apr 2010 at 3:22

[GoogleCodeExporter]

I am also having this problem ... still no workaround?

Original comment by jon.k...@gmail.com on 9 Aug 2010 at 7:14

[GoogleCodeExporter]

Here is a patch for SocksiPy that allows us to use proxies which do not allow 
tunneling for standard HTTP traffic. It adds a "PROXY_TYPE_HTTP_NO_TUNNEL" type 
and fails back to "PROXY_TYPE_HTTP" for SSL requests.

Original comment by jon.k...@gmail.com on 11 Aug 2010 at 4:25

Attachments:

• socks.py.patch

[GoogleCodeExporter]

I also found a fork of SocksiPy which is continuing development / maintenance: 
http://code.google.com/p/socksipy-branch/

It works well for me.
I submitted a patch against their trunk as well:
http://code.google.com/p/socksipy-branch/issues/detail?id=1

Original comment by jon.k...@gmail.com on 11 Aug 2010 at 4:56

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I'm not sure if this is the same as the original issue report, but I finding 
the following when trying to make http and https requests via a squid proxy:

if the request is to an http url, then the squid proxy does not support 
tunnelling via CONNECT, and so the proxy request needs to be made without 
tunnelling. this sounds very much like what the patches submitted by jon.k 
above are trying to fix, but niether the patch posted here nor the patch posted 
on the socksipy-branch project would apply cleanly. 

if the request is to an https url, then the squid proxy expects a "User-Agent" 
header with the initial CONNECT request, and will respond with a misleading 407 
if the User-Agent header is absent. socksipy does not provide a user-agent 
header, and there is no way to pass one in.

if it helps anyone, I diagnosed these issues by running tcpwatch-httpproxy in 
forwarding mode, forwarding from localhost:1234 to the URL of the squid proxy, 
then used localhost:1234 as the proxy url in the httplib2 config, which is one 
way to get eyes on the tcp traffic. i compared httplib2 to curl and wget, 
niether of which have any problems if you export http_proxy and https_proxy 
environment variables.

Original comment by alimanfoo@googlemail.com on 9 Feb 2011 at 2:03

[GoogleCodeExporter]

The patch above is jon seems to have an error. it has a self.__httptunnel 
defaulting to true but later for no tunnelling changes a a self.httptunnel 
(missing underscores)

Original comment by exob...@gmail.com on 26 Feb 2011 at 9:41

[GoogleCodeExporter]

For those that need it - I have a modified socks.py here

http://code.google.com/p/xbmc-iplayerv2/source/browse/trunk/plugin.video.iplayer
/lib/httplib2/socks.py

that includes jon's patches with the fix i mentioned (and perhaps other minor 
tweaks), as well as http authentication support (for http proxy and tunnelled 
http proxy). I tested the authentication against mod_proxy on apache and 
mod_proxy_connect and it worked for me.

Original comment by exob...@gmail.com on 27 Feb 2011 at 12:33

[GoogleCodeExporter]

Issue 100 has been merged into this issue.

Original comment by joe.gregorio@gmail.com on 13 Jun 2011 at 4:51

[GoogleCodeExporter]

socks.py has been updated to latest.

Original comment by joe.gregorio@gmail.com on 13 Jun 2011 at 4:52

• Changed state: Fixed

[GoogleCodeExporter]

Why has this been marked as fixed? The patch / fix referred to here isn't 
upstream.

Original comment by exob...@gmail.com on 26 Jul 2011 at 3:35

[GoogleCodeExporter]

in fact upstream socksipy seems to be dead. with patches not being applied, and 
no response from the author. 

Original comment by exob...@gmail.com on 26 Jul 2011 at 3:36

[GoogleCodeExporter]

I've patched this in https://bitbucket.org/jaraco/httplib2 and a packaged 
version is available at 
https://bitbucket.org/jaraco/httplib2/downloads/httplib2-0.8.0dev2.zip

Original comment by jaraco%jaraco.com@gtempaccount.com on 9 Aug 2011 at 4:15

[GoogleCodeExporter]

I ran into this as well and I think this bug needs to be reopened. Given that 
socksipy upstream seems to be not responding I think its best to fork the 
socks.py inside httplib2 and put this patch in.

Original comment by michael....@gmail.com on 4 Oct 2011 at 7:52

[GoogleCodeExporter]

This issue was closed by revision f6701a80444e.

Original comment by joe.gregorio@gmail.com on 6 Oct 2011 at 6:54