mechanize silently fails to validate SSL certificates over https:

[andersk]

mechanize does not throw an error when connecting to a website using an invalid SSL certificate. This means that mechanize users are vulnerable to man-in-the-middle attacks even when they think they are protected by SSL.

Here’s a test case: mechanize.Browser().open('https://scripts-vhosts.mit.edu/').read()

[jjlee]

This is documented urllib2 behaviour, so I'm calling this a feature request.

Personally it doesn't matter to me that it doesn't check certificates. Seems few other people care much either, since nobody has fixed it in urllib2.

Would you like to implement this? Best to do it in urllib2 first.

[andersk]

It was fixed in Python 3.2: http://bugs.python.org/issue9003 http://docs.python.org/py3k/library/urllib.request.html#urllib.request.urlopen

I recognize that it may be some time before mechanize gets ported to Python 3. But given that mechanize forks several other parts of urllib2, maybe it would be reasonable to also include a patched or wrapped HTTPSConnection with a backported fix.

Until then, there should at least be warnings in the mechanize documentation, like there are in the urllib2 documentation.

[jjlee]

The mechanize documentation includes the urllib2 documentation "by reference".

Re backporting: feel free to do it. Again, I think it would be best to do this initially as a patch against Python (even if it doesn't get applied there) rather than mechanize.

[andersk]

I’m not sure I understand. Like I said, there already is a patch against Python, which I linked to in my previous comment, and it was applied in Python 3.2.

I’m willing to try to write the mechanize patch; just trying to make sure we agree on how it should be done.

[jjlee]

So I'm saying that if you want to backport that patch to Python 2, go ahead. I have no idea what is involved in that since I know nothing about Python 3.

[ab]

You might find this gist useful if you want to implement cert verification: https://gist.github.com/1347055

[hurzl]

Is the new behaviour to fail always on https like urllib2 does?

In urllib2 I can define and use an ssl context with .verify_mode= ssl.CERT_NONE. Can I do this with mechanize?

  File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 541, in submit
    return self.open(self.click(*args, **kwds))
  File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 230, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_opener.py", line 204, in open
    response = meth(req, response)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 457, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_opener.py", line 221, in error
    result = apply(self._call_chain, args)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 571, in http_error_302
    return self.parent.open(new)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 230, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 344, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 1170, in https_open
    return self.do_open(conn_factory, req)
  File "/usr/local/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 1118, in do_open
    raise URLError(err)
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>