search

googleapis / google-auth-library-python

Google Auth Python Library
https://googleapis.dev/python/google-auth/latest/
Apache License 2.0
778 stars 308 forks source link

SSL Certificate verification issue #192

Closed lukesneeringer closed 7 years ago

lukesneeringer commented 7 years ago

From @Spikey123 on August 23, 2017 13:5

Hello.

I've been using tswast's method to query BigQuery results and it's been purring along fine for a while now. Over the last several days, I've been getting an SSL certificate verification issue. Below is the full error message and below that is the code I use.

I'll note that I updated the python client, bigquery library, cloud storage library, and gcloud components. pip install --upgrade google-api-python-client pip install --upgrade google-cloud-bigquery pip install --upgrade google-cloud-storage gcloud components update

Service account permissions should be fine image

Update: I'm getting the same exact error on other scripts I have running - like this one that deletes blobs in Google Cloud Storage:

Traceback (most recent call last):
  File "sfyExport/sfyExport.py", line 334, in <module>
    compression="GZIP" if args.gzip else "NONE")
  File "sfyExport/sfyExport.py", line 203, in main
    didDelete = delete_blobs_matching("mwg-bigqueryexport", deleteString)
  File "sfyExport/sfyExport.py", line 50, in delete_blobs_matching
    blobs = list_blobs(bucket_name)
  File "sfyExport/sfyExport.py", line 38, in list_blobs
    bucket = storage_client.get_bucket(bucket_name)
  File "/usr/lib/python2.7/site-packages/google/cloud/storage/client.py", line 173, in get_bucket
    bucket.reload(client=self)
  File "/usr/lib/python2.7/site-packages/google/cloud/storage/_helpers.py", line 99, in reload
    _target_object=self)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/lib/python2.7/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)'),))

I checked the certs on accounts.google.com and they appeared to be fine. So I'm not sure why the SSL error would be thrown.

Thanks!

Traceback (most recent call last):
  File "bqTests.py", line 295, in <module>
    args.num_retries)
  File "bqTests.py", line 254, in main
    failCount = async_query("SELECT Count(*) from [Tests.{0}]".format(bqTest["tempTable"]))
  File "bqTests.py", line 104, in async_query
    query_job.begin()
  File "/usr/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 380, in begin
    method='POST', path=path, data=self._build_resource())
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python2.7/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/lib/python2.7/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/lib/python2.7/site-packages/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/usr/lib/python2.7/site-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/usr/lib/python2.7/site-packages/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)'),))

The code is:

87- # [START wait_for_job]
88- def wait_for_job(job):
89-     while True:
90-        job.reload()  # Refreshes the state via a GET request.
91-        if job.state == 'DONE':
92-            if job.error_result:
93-                raise RuntimeError(job.errors)
94-            return
95-        time.sleep(1)
96- # [END wait_for_job]
99- # [START async_query]
100- def async_query(query):
101-    client = bigquery.Client()
102-    query_job = client.run_async_query(str(uuid.uuid4()), query)
103-    query_job.use_legacy_sql = True
104-    query_job.begin()
105-
106-    wait_for_job(query_job)
107-
108-    rows = query_job.results().fetch_data(max_results=10)
109-    
110-    for row in rows:
111-        return row[0]
112-    
113- # [END async_query]

Copied from original issue: GoogleCloudPlatform/google-cloud-python#3857

lukesneeringer commented 7 years ago

Hi @Spikey123, Thanks for reporting. Even though it will not change the audience for your issue much, I am going to move this over to google-auth-library-python, which is the actual source of the error you are getting.

theacodes commented 7 years ago

There are a lot of things that can lead to this and none of them are the fault of this library. I would check, in no particular order:

  1. Your system clock.
  2. Your Python version. Older versions of 2.7 lack SNI support.
  3. Your version of openssl, older versions can lack SNI support.
  4. Your version of requests and certifi.

You can try installing requests[security] to see if it automatically fixes it.

greg-finley commented 7 years ago

We are also affected by this issue within the same timeframe.

Spikey123 commented 7 years ago

To reply back about what I did: 1.) I checked all the things @jonparrott mentioned and all those things were configured correctly for me 2.) I asked for a VM reload of the day prior the issue surfaced

Therefore I have no idea what the issue was actually caused by and fortunately had a systems team that kept good VM backups for us :)

The fact that it worked after the VM refresh confirms that it's not an issue with the libraries used. It must be a VM configuration issue of some sort.

boosh commented 7 years ago

Seeing this too after upgrading google-cloud to 0.27.0. Was working fine at 0.24.0. Working in a docker container, so minimal scope for other dependencies to be out of whack. Already at the latest versions of requests and certifi.

Here's my trace:

Traceback (most recent call last):
  ...
  File "/app/content/lib.py", line 50, in <listcomp>
    table_dates = reversed(sorted([t.name.replace(table, '')
  File "/usr/lib/python3.5/site-packages/google_cloud_core-0.26.0-py3.5.egg/google/cloud/iterator.py", line 218, in _items_iter
    for page in self._page_iter(increment=False):
  File "/usr/lib/python3.5/site-packages/google_cloud_core-0.26.0-py3.5.egg/google/cloud/iterator.py", line 248, in _page_iter
    page = self._next_page()
  File "/usr/lib/python3.5/site-packages/google_cloud_core-0.26.0-py3.5.egg/google/cloud/iterator.py", line 348, in _next_page
    response = self._get_next_page_response()
  File "/usr/lib/python3.5/site-packages/google_cloud_core-0.26.0-py3.5.egg/google/cloud/iterator.py", line 399, in _get_next_page_response
    query_params=params)
  File "/usr/lib/python3.5/site-packages/google_cloud_core-0.26.0-py3.5.egg/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/usr/lib/python3.5/site-packages/google_cloud_core-0.26.0-py3.5.egg/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/lib/python3.5/site-packages/google_cloud_core-0.26.0-py3.5.egg/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/usr/lib/python3.5/site-packages/google_auth-1.0.2-py3.5.egg/google/auth/transport/requests.py", line 179, in request
    method, url, data=data, headers=request_headers, **kwargs)
  File "/usr/lib/python3.5/site-packages/requests-2.18.4-py3.5.egg/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.5/site-packages/requests-2.18.4-py3.5.egg/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.5/site-packages/requests-2.18.4-py3.5.egg/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/my-proj/datasets/pixel/tables (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
boosh commented 7 years ago

Issue goes away downgrading to google-cloud 0.24.0.

thomasw commented 7 years ago

https://github.com/requests/requests/issues/3859#issuecomment-311275977 is likely the solution to this issue if the following fails for you.

import requests
requests.get('https://google.com/')

Upgrading certifi and requests alone isn't going to solve the issue because it's being caused by a buggy openssl installation.

ldm314 commented 7 years ago

What @thomasw mentioned work for me when I ran into this issue.

For CentOS 7 all I had to do was 'yum update openssl' and the problem was resolved.

saurav955 commented 6 years ago

thanks @jonparrott, I was using python 2.7.5 and moved to 2.7.11 it solved it for me. SNI was the culprit.

theacodes commented 6 years ago

@saurav955 good to hear. SSL on older pythons is a huge pain.

iethan commented 6 years ago

I've installed 2.7.11, downgraded google-cloud to 0.24.0, installed the latest version of openssl via brew and requests[security] and it still hasn't changed the error.

Marigold commented 6 years ago

Same thing is happenings to us TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:661)'),)). Updating SSL didn't help. Nothing did.

theacodes commented 6 years ago

EOF is a completely different kind of error, please file a separate issue for that.

iethan commented 6 years ago

The fix that worked for me is installing requests-toolbelt via pip. It appears GAE uses requests in connecting to Datastore. Surprised it doesn't come standard in GAE.

erickertz commented 6 years ago

Same thing is happenings to us TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:661)'),)). Updating SSL didn't help. Nothing did.

did you ever figure out the SSLEOFError issue?

Marigold commented 6 years ago

@erickertz unfortunately not. We ended up coding our own wrapper around PubSub REST API with auth. We haven't tried newer versions though.

miaojun1023 commented 4 years ago

downgraded google-cloud-core to 0.24.1 works for me, but don't know why

RakibCSE commented 4 years ago

Issue solved after downgrading to google-cloud 0.30.0

anilmathew21 commented 4 years ago

I had same issue. Resolved by installing "pip install requests-toolbelt"

Issue : image

kmeeraj commented 4 years ago

I did this step and it worked for me https://stackoverflow.com/questions/45927259/google-cloud-python-sdk-installation-error-ssl-certification-error/50524168#50524168