SSL error

[Roh-codeur]

Hi

I get the below error while sending mail using djrill.

app_1 | Traceback (most recent call last): app_1 | File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response app_1 | response = wrapped_callback(request, _callback_args, _callback_kwargs) app_1 | File "/code/invitations/views.py", line 102, in request_invite app_1 | msg.send() app_1 | File "/usr/local/lib/python2.7/site-packages/django/core/mail/message.py", line 303, in send app_1 | return self.get_connection(fail_silently).send_messages([self]) app_1 | File "/usr/local/lib/python2.7/site-packages/djrill/mail/backends/djrill.py", line 81, in send_messages app_1 | sent = self._send(message) app_1 | File "/usr/local/lib/python2.7/site-packages/djrill/mail/backends/djrill.py", line 132, in _send app_1 | response = requests.post(api_url, data=api_data) app_1 | File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 109, in post app_1 | return request('post', url, data=data, json=json, _kwargs) app_1 | File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 50, in request app_1 | response = session.request(method=method, url=url, _kwargs) app_1 | File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request app_1 | resp = self.send(prep, _send_kwargs) app_1 | File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send app_1 | r = adapter.send(request, *_kwargs) app_1 | File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 431, in send app_1 | raise SSLError(e, request=request) app_1 | SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

Any ideas? I appreciate your help in this

Cheers Roh

[medmunds]

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

This means pretty much exactly what it sounds like: python was unable to verify the validity of the SSL certificate when it tried to contact the Mandrill API server over https.

Assuming Mandrill hasn't accidentally let their SSL certificate expire (and in fact, they have not) [EDIT: but Mandrill is sending an incorrect root cert -- see below], the problem is something in your environment:

• Something is MITM-ing your connection to Mandrill (inserting itself into the communication and pretending to be Mandrill). Are you running an internet proxy? Does your organization use some sort of firewall or security system that might be intercepting traffic to Mandrill?
• Your version of python or the requests package is somehow incompatible with the SSL protocol options used by Mandrill's servers.

To narrow it down, try running this command in a shell:

curl --verbose 'https://mandrillapp.com/api/1.0/users/ping.json'

You should see something like this:

* Connected to mandrillapp.com (x.x.x.x) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: mandrillapp.com
* Server certificate: thawte SSL CA - G2
* Server certificate: thawte Primary Root CA
...

(and then an error response from Mandrill that you didn't specify a key value).

If curl doesn't report a successful "TLS connection", then the problem is most likely something in your network (or upstream of your network). If curl does work, then the problem is probably in your python or installed packages.

To help any more, we'd need to know what OS version you're on, what python version, what version of requests and djrill you have installed, etc. (And where you're hosted, if it's a MITM issue.)

[donspaulding]

I've tracked this down to djrill using requests, which depends on the certifi project, which bundles up the Mozilla-approved Certificate Authorities. Running pip install certifi==2015.04.28 "fixes" the issue for me, though I suspect the real issue is that there's either a misconfigured cert at Mandrill or their CA has recently been dropped from the Mozilla default CA bundle.

To reproduce this behavior, make sure you're using a more-recent certifi version (2015.9.6 and up, I think) pip install certifi==2015.9.6.2 suffices to trigger it for me.

[donspaulding]

Errr, sorry. I just edited my last comment to include the right version that triggers it for me.

[Roh-codeur]

Hi Donspaulding

You are right! I tried the steps you suggested and can replicate. Also, pip install certifi==2015.04.28 fixes the issue

many thanks Cheers Roh

[medmunds]

Thanks @donspaulding -- that's some seriously-helpful detective work!

It does look like Mandrill is (unnecessarily) including a "thawte Primary Root CA" cert in their cert chain (scroll down to additional certificates # 3). And that root cert Mandrill includes has a different fingerprint than any of the thawte Primary Root CAs currently listed on Thawte's site (which are the same as the ones in the current Mozilla Included CA List used by certifi).

I'm guessing requests balks on the outdated/incorrect root cert provided by mandrillapp.com. (I believe curl and browsers just ignore any root CA provided by the server, whether or not it's valid, because they're only going to trust their own roots anyway.) Still, seems like Mandrill shouldn't be sending it.

I'll leave this issue open until Mandrill fixes the problem, to help anyone else who runs into it.

[medmunds]

This also affects Mandrill's own python API -- see the discussion in the certifi project: https://github.com/certifi/python-certifi/issues/26#issuecomment-140172969.

For Djrill users who don't want to pin their certifi to an older version (and it's probably not a good idea to depend on outdated security packages), there's a workaround available in newer certifi packages: set env.REQUESTS_CA_BUNDLE to certifi.old_where() (see https://github.com/certifi/python-certifi/issues/26#issuecomment-138336586).

(If anyone from Mandrill stops by here, your Zendesk case 140013 is about this issue.)

[medmunds]

I'm concluding this has either been fixed by Mandrill switching to a different SSL root cert, or is an issue with certifi more aggressively curating its CA roots than the browser vendors. [Disclaimer: I'm not a security professional, and it's entirely possible I'm misunderstanding something.]

Mandrill is including an old Thawte SHA-1 root cert to address compatibility issues in a bunch of customer environments.

The browsers are apparently continuing to include the existing SHA-1 roots (which is why your browser is OK with https://mandrillapp.com despite the old CA root). From Google: "SHA-1-based signatures for trusted root certificates are not a problem because TLS clients trust them by their identity, rather than by the signature of their hash." From Mozilla: "Root certificates are trusted by virtue of their inclusion in Firefox, so it does not matter how they are signed." And Qualys confirms: "Don't worry if the root certificate uses SHA1; signatures on roots are not used."

As of today, the Thawte root cert in mandrillapp.com's bundle has the fingerprint 91:C6:D6:EE:3E:8A:C8:63:84:E5:48:C2:99:29:5C:75:6C:81:7B:81, which is in Mozilla's included CA list. Since that list is the basis for certifi's trusted roots, it seems like certifi should accept it.

Certifi's readme now covers 1024-bit roots. Their workaround of setting env.REQUESTS_CA_BUNDLE to certifi.old_where() is probably the best option for Djrill users who are still running into SSL errors with the current version of certifi.