search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

SSL certificate verify failed (was: Token Problem?) #61

Closed chriscardillo closed 4 years ago

chriscardillo commented 4 years ago

Currently on Python 3.6.5.

Attempted to login with coiled login --token my-token and received the following error.

Traceback (most recent call last):
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 804, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 830, in _create_connection_transport
    yield from waiter
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/HomeBase/Desktop/coiled_demo/venv/bin/coiled", line 11, in <module>
    sys.exit(cli())
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/coiled/cli/login.py", line 17, in login
    handle_credentials(server=server, token=token, save=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/coiled/utils.py", line 104, in handle_credentials
    response = await session.request("GET", server + "/api/v1/users/me/")
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/client.py", line 483, in _request
    timeout=real_timeout
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
    req, traces, timeout)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
    req=req, client_error=client_error)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host beta.coiled.io:443 ssl:default [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]
(venv) ChrisComputer:coiled_demo HomeBase$ 
(venv) ChrisComputer:coiled_demo HomeBase$ python --version
Python 3.6.5
(venv) ChrisComputer:coiled_demo HomeBase$ coiled login --token 2d01b77afe50811f4ed0e7b56c1a4822b1e4b695
Traceback (most recent call last):
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 804, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 830, in _create_connection_transport
    yield from waiter
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/HomeBase/Desktop/coiled_demo/venv/bin/coiled", line 11, in <module>
    sys.exit(cli())
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/coiled/cli/login.py", line 17, in login
    handle_credentials(server=server, token=token, save=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/coiled/utils.py", line 104, in handle_credentials
    response = await session.request("GET", server + "/api/v1/users/me/")
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/client.py", line 483, in _request
    timeout=real_timeout
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 859, in _create_connection
    req, traces, timeout)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 986, in _create_direct_connection
    req=req, client_error=client_error)
  File "/Users/HomeBase/Desktop/coiled_demo/venv/lib/python3.6/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host beta.coiled.io:443 ssl:default [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]

I am on the coiled slack, as well! Happy to talk there.

chriscardillo commented 4 years ago

This SO article solved the problem: https://stackoverflow.com/questions/42098126/mac-osx-python-ssl-sslerror-ssl-certificate-verify-failed-certificate-verify

"Just browse to Applications/Python 3.6 and double-click Install Certificates.command"

mrocklin commented 4 years ago

Thank you for raising the issue and posting the response Chris. Hopefully it helps others who come across this issue in the future.

On Thu, Sep 3, 2020 at 7:16 PM Rami Chowdhury notifications@github.com wrote:

Closed #61 https://github.com/coiled/coiled-issues/issues/61.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/coiled/coiled-issues/issues/61#event-3727538220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTBOJFGOBLIL562DHELSEBEZDANCNFSM4QWXEPRQ .

hugobowne commented 4 years ago

I've reopened this as I'm wondering if it's something we can flag in Coiled, in error messaging, or solve for?

My most naive solution is to include under "common problems" at some point, if it ends up being common (and it sounds like it will happen for anybody downloading python from python.org?), but I'm sure there are other solutions.

If we think this is an edge case that will happen rarely, then we could do nothing also for the time being.

necaris commented 4 years ago

From what I can see, this is a combination of macOS + Python from python.org. The 2019 Jetbrains survey https://www.jetbrains.com/lp/python-developers-survey-2019/ indicates 31% of those surveyed installed from Python.org (but doesn't give a hint as to what platform they were on), so it may be more common than I've been assuming.

I've renamed the issue to capture the SSL error, and suggest we leave it open until we either get more reports of it (in which case we put it in the FAQ), or go a month without hearing about it (in which case we close it and do nothing).

marin123 commented 4 years ago

Hi, I'm getting a similar problem, but on conda on macOS

Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 946, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "/opt/anaconda3/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/opt/anaconda3/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/opt/anaconda3/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '3.131.1.205'. (_ssl.c:1123)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/anaconda3/bin/coiled", line 8, in <module>
    sys.exit(cli())
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/anaconda3/lib/python3.8/site-packages/coiled/cli/login.py", line 15, in login
    asyncio.get_event_loop().run_until_complete(
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/opt/anaconda3/lib/python3.8/site-packages/coiled/utils.py", line 123, in handle_credentials
    response = await session.request("GET", server + "/api/v1/users/me/")
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/client.py", line 490, in _request
    conn = await self._connector.connect(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 528, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 868, in _create_connection
    _, proto = await self._create_direct_connection(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 1023, in _create_direct_connection
    raise last_exc
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 999, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 948, in _wrap_create_connection
    raise ClientConnectorCertificateError(
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host cloud.coiled.io:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '3.131.1.205'. (_ssl.c:1123)")]

I'm on Python 3.8. Any idea what the issue could be? My google-fu didn't find anything useful

necaris commented 4 years ago

@marin123 thanks for raising this and for the traceback! Can you tell us more about your setup -- conda version, exact Python version, version of coiled installed, etc?

(Our SSL certificate doesn't declare an IP address, although that is the IP of our load balancer, so I'm just trying to understand more of what's going on here.)

marin123 commented 4 years ago

@necaris Thanks for help, here is part of my conda info 

          conda version : 4.9.0
    conda-build version : 3.20.4
         python version : 3.8.5.final.0
       virtual packages : __osx=10.15.7=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
               platform : osx-64
             user-agent : conda/4.9.0 requests/2.24.0 CPython/3.8.5 Darwin/19.6.0 OSX/10.15.7
                UID:GID : 501:20
             netrc file : None
           offline mode : False

and I'm using the version of coiled

~ coiled --version
0.0.27
necaris commented 4 years ago

@marin123 could you please run a couple of other quick checks to see what your system (outside of Python) is doing with our IP / SSL certificate?

nslookup cloud.coiled.io

and

curl -i --verbose https://cloud.coiled.io

Thanks!

marin123 commented 4 years ago

This is what I'm getting 

 nslookup cloud.coiled.io
Server:     192.168.178.1
Address:    192.168.178.1#53

Non-authoritative answer:
cloud.coiled.io canonical name = cloud-dev-1376754671.us-east-2.elb.amazonaws.com.
Name:   cloud-dev-1376754671.us-east-2.elb.amazonaws.com
Address: 3.138.59.179
Name:   cloud-dev-1376754671.us-east-2.elb.amazonaws.com
Address: 3.135.111.100
Name:   cloud-dev-1376754671.us-east-2.elb.amazonaws.com
Address: 3.131.1.205

and

curl -i --verbose https://cloud.coiled.io
*   Trying 3.138.59.179:443...
* Connected to cloud.coiled.io (3.138.59.179) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /opt/anaconda3/ssl/cacert.pem
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=cloud.coiled.io
*  start date: Sep 22 00:00:00 2020 GMT
*  expire date: Oct 22 12:00:00 2021 GMT
*  subjectAltName: host "cloud.coiled.io" matched cert's "cloud.coiled.io"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: cloud.coiled.io
> User-Agent: curl/7.71.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Sat, 24 Oct 2020 14:49:03 GMT
Date: Sat, 24 Oct 2020 14:49:03 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Content-Length: 1581
Content-Length: 1581
< Connection: keep-alive
Connection: keep-alive
< X-Frame-Options: DENY
X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
Referrer-Policy: same-origin
< Vary: Origin
Vary: Origin

< 
* Connection #0 to host cloud.coiled.io left intact
<!DOCTYPE html><html><head><title>Coiled</title><link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Roboto:ital,wght@0,300;0,400;0,500;0,700;1,300;1,400;1,500;1,700&display=swap"><link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons"><link rel="icon" sizes="144x144" href="https://cloud.coiled.io/static/favicon-144.png"><link rel="icon" sizes="114x114" href="https://cloud.coiled.io/static/favicon-114.png"><link rel="icon" sizes="32x32" href="https://cloud.coiled.io/static/favicon-32.png"><link rel="apple-touch-icon-precomposed" sizes="167x167" href="https://cloud.coiled.io/static/favicon-167.png"><link rel="apple-touch-icon-precomposed" sizes="152x152" href="https://cloud.coiled.io/static/favicon-152.png"><link rel="apple-touch-icon-precomposed" sizes="144x144" href="https://cloud.coiled.io/static/favicon-144.png"><link rel="apple-touch-icon-precomposed" sizes="114x114" href="https://cloud.coiled.io/static/favicon-144.png"><link rel="apple-touch-icon-precomposed" sizes="72x72" href="https://cloud.coiled.io/static/favicon-72.png"><meta name="viewport" content="minimum-scale=1, initial-scale=1, width=device-width"><meta charset="utf-8"><meta property="og:image" content="https://cloud.coiled.io/static/og-image.png"><meta property="og:title" content="Coiled Computing"><meta property="og:description" content="Scaling python simply. Founded by creators of Dask, Coiled helps you run at maximum speed and minimum cost."></head><body> <div id="root"></div> <script src="/static/app.7f0f0fc5.js"></script> </body></html>%
necaris commented 4 years ago

Moved this to #85 since this IP address mismatch issue doesn't seem related to the original missing certificates. Closing this one, too, since it hasn't seen activity around the original problem in >1 month. Please reopen if you experience this!