Script has started failing with 'certificate verify failed'

[silverl]

Started getting this error just recently.

$ /home/zenoss/bin/slack_zenoss.py --message='Test message' --summary='Test Summary' --device='null-device' --component='null-component' --severity='null-severity' --detail_url='http://null.net' --ack_url='' --close_url='' --dev_events_url=''
Traceback (most recent call last):
  File "/home/zenoss/bin/slack_zenoss.py", line 131, in <module>
    main(hookurl)
  File "/home/zenoss/bin/slack_zenoss.py", line 128, in main
    (resp, content) = h.request(hookurl, "POST", body=payload, headers={'content-type':'application/json'} )
  File "/opt/zenoss/lib/python/httplib2/__init__.py", line 1436, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/opt/zenoss/lib/python/httplib2/__init__.py", line 1188, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/opt/zenoss/lib/python/httplib2/__init__.py", line 1123, in _conn_request
    conn.connect()
  File "/opt/zenoss/lib/python/httplib2/__init__.py", line 911, in connect
    raise SSLHandshakeError(e)
httplib2.SSLHandshakeError: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

[ssplatt]

This just started for me, also. I'm guessing something changed at Slack. If I hear back from them I'll update here.

[silverl]

It appears the httplib2 is failing to verify the wildcard certificate the Slack is using. Perhaps they started using it only recently? Don't know.

The following hack makes the code less secure, but it does work. It tells the httplib2 library to NOT verify the TLS certificate (which is generally a no-no).

Change line 127:

h = httplib2.Http(disable_ssl_certificate_validation=True)

Not sure what the "right" fix is yet.

[silverl]

Yuck. Seems like one solution is to download and bundle the signing root certificate, and tell the httplib2 where to find the PEM file.

https://pwnetics.wordpress.com/2012/02/06/ssl-certificate-verification-and-httplib2/

Is there a better http library available that is smarter about validating the certificate chain authority?

Bundling publicly available root certs is fragile, as Slack may choose a different certificate authority in the future and the whole mess breaks again.

[ssplatt]

It's my understanding the httplib2 is the prefered library for these types of transactions but I'll research a little more this afternoon.

[silverl]

Ok, here's the fix for me, on CentOS 6:

h = httplib2.Http(ca_certs='/etc/pki/tls/certs/ca-bundle.crt')

You need to instantiate httplib2 and let it know where your system-maintained bundle of CA certs resides. On CentOS, it's at that path. This could very well vary by OS and distro. It should likely be a config specified at the top or in a separate custom config file.

I'd recommend making a separate config file that can be added to .gitignore and included in your code. This will enable other users to pull the latest code without conflicts or other manual intervention required.

[ssplatt]

from Slack:

We applied OpenSSL security updates released yesterday, and this connection problem may be related (https://www.openssl.org/news/secadv_20150319.txt)

Applying available OpenSSL and/or Python updates to your server may help.

We may be able to help diagnose a little further if you can please execute openssl s_client -connect api.slack.com:443 on your server and provide the results back to us.

Thanks!

But there doesn't seem to be any new updates for CentOS 6.6 to catch up with this, so I may be expediting my migration to CentOS 7 and Zenoss 5 now.

[silverl]

I'm sure the openssl update is coming soon to CentOS 6. I was expecting it today. My CentOS 7 machine doesn't have it either. Both OSes are still supported.

[silverl]

FYI, openssl updates were available in the repos today for CentOS 6 and 7 when I checked. They released a backported fix into the 1.0.1e release.

You can check to see the CVEs were addressed via: rpm -q --changelog openssl | head

The fix I came up with to specify the ca_certs location is still required. I tried removing it and got the same old error.