search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

SSL connection failure #85

Closed necaris closed 1 year ago

necaris commented 4 years ago

See thread starting at https://github.com/coiled/coiled-issues/issues/61#issuecomment-715905260 -- created a new issue because this is distinct from needing to run Install Certificates on macOS.

/cc @marin123

Pasting original post below:

Hi, I'm getting a similar problem, but on conda on macOS

Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 946, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "/opt/anaconda3/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/opt/anaconda3/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/opt/anaconda3/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '3.131.1.205'. (_ssl.c:1123)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/anaconda3/bin/coiled", line 8, in <module>
    sys.exit(cli())
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/anaconda3/lib/python3.8/site-packages/coiled/cli/login.py", line 15, in login
    asyncio.get_event_loop().run_until_complete(
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/opt/anaconda3/lib/python3.8/site-packages/coiled/utils.py", line 123, in handle_credentials
    response = await session.request("GET", server + "/api/v1/users/me/")
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/client.py", line 490, in _request
    conn = await self._connector.connect(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 528, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 868, in _create_connection
    _, proto = await self._create_direct_connection(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 1023, in _create_direct_connection
    raise last_exc
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 999, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 948, in _wrap_create_connection
    raise ClientConnectorCertificateError(
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host cloud.coiled.io:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '3.131.1.205'. (_ssl.c:1123)")]

I'm on Python 3.8. Any idea what the issue could be? My google-fu didn't find anything useful

necaris commented 4 years ago

@marin123 moved this discussion here since the cause seems to be different. From your posting of the curl results it seems clear the rest of the SSL libraries in your system don't see this IP address mismatch, and as I'm not able to reproduce it locally I'm a little stumped. Can we see if the standard Python library is affected? Could you run the below with your conda environment activated?

python -c 'import urllib.request; print(urllib.request.urlopen("https://cloud.coiled.io/login").headers)'
marin123 commented 4 years ago

@necaris thanks, I tried running the above command:

python -c 'import urllib.request; print(urllib.request.urlopen("https://cloud.coiled.io/login").headers)'
Date: Mon, 26 Oct 2020 15:21:32 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1581
Connection: close
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Vary: Origin
necaris commented 4 years ago

Thanks @marin123 ! One more, to check the aiohttp/ asyncio stack, please:

python -c 'import aiohttp, asyncio
async def _():
  async with aiohttp.ClientSession() as s:
    async with s.get("https://cloud.coiled.io/login") as r:
      print("Status:", r.status)
      print("Headers:", r.headers)
      print("Body:", await r.text())
asyncio.get_event_loop().run_until_complete(_())'

Also, could you try rerunning coiled login? If the above works, I'll be completely stumped as to why that doesn't work.

marin123 commented 4 years ago

@necaris now we finally got to the point where it's failing! This is now the exact bug I'm getting

➜  ~ python -c 'import aiohttp, asyncio
async def _():
  async with aiohttp.ClientSession() as s:
    async with s.get("https://cloud.coiled.io/login") as r:
      print("Status:", r.status)
      print("Headers:", r.headers)
      print("Body:", await r.text())
asyncio.get_event_loop().run_until_complete(_())'
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 946, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "/opt/anaconda3/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/opt/anaconda3/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/opt/anaconda3/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '3.131.1.205'. (_ssl.c:1123)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 8, in <module>
  File "/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "<string>", line 4, in _
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/client.py", line 1083, in __aenter__
    self._resp = await self._coro
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/client.py", line 490, in _request
    conn = await self._connector.connect(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 528, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 868, in _create_connection
    _, proto = await self._create_direct_connection(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 1023, in _create_direct_connection
    raise last_exc
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 999, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/opt/anaconda3/lib/python3.8/site-packages/aiohttp/connector.py", line 948, in _wrap_create_connection
    raise ClientConnectorCertificateError(
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host cloud.coiled.io:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '3.131.1.205'. (_ssl.c:1123)")]
necaris commented 4 years ago

@marin123 one more piece of information that might help -- unfortunately I'm still not able to reproduce this, but I'm on Linux. Could you run this:

python -c 'import asyncio.sslproto; print(asyncio.sslproto.ssl.OPENSSL_VERSION)'
marin123 commented 4 years ago

@necaris I get

python -c 'import asyncio.sslproto; print(asyncio.sslproto.ssl.OPENSSL_VERSION)'
OpenSSL 1.1.1h  22 Sep 2020
jonashaag commented 3 years ago

I also have this problem

coiled login --token ...
Traceback (most recent call last):
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/home/jo/miniconda3/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/home/jo/miniconda3/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "/home/jo/miniconda3/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/home/jo/miniconda3/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/home/jo/miniconda3/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jo/.venvs/au/bin/coiled", line 8, in <module>
    sys.exit(cli())
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/coiled/cli/login.py", line 21, in login
    asyncio.get_event_loop().run_until_complete(
  File "/home/jo/miniconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/coiled/utils.py", line 159, in handle_credentials
    response = await session.request("GET", server + "/api/v1/users/me/")
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/client.py", line 520, in _request
    conn = await self._connector.connect(
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 535, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 892, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection
    raise last_exc
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 1020, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 971, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host cloud.coiled.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')]

cc @fjetter

necaris commented 3 years ago

Any chance you could do us a favor and dig into this a bit more? Could you tell us more about your OS, version, etc? I have no been able to reproduce this on Linux, and reports have been rare, so would love more details if you have them available.

On Fri, Apr 23, 2021 at 9:28 AM Jonas Haag @.***> wrote:

I also have this problem

coiled login --token ...

Traceback (most recent call last): File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa File "/home/jo/miniconda3/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection transport, protocol = await self._create_connection_transport( File "/home/jo/miniconda3/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport await waiter File "/home/jo/miniconda3/lib/python3.8/asyncio/sslproto.py", line 529, in data_received ssldata, appdata = self._sslpipe.feed_ssldata(data) File "/home/jo/miniconda3/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata self._sslobj.do_handshake() File "/home/jo/miniconda3/lib/python3.8/ssl.py", line 944, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/jo/.venvs/au/bin/coiled", line 8, in sys.exit(cli()) File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/jo/.venvs/au/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/home/jo/.venvs/au/lib/python3.8/site-packages/coiled/cli/login.py", line 21, in login asyncio.get_event_loop().run_until_complete( File "/home/jo/miniconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/home/jo/.venvs/au/lib/python3.8/site-packages/coiled/utils.py", line 159, in handle_credentials response = await session.request("GET", server + "/api/v1/users/me/") File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/client.py", line 520, in _request conn = await self._connector.connect( File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 535, in connect proto = await self._create_connection(req, traces, timeout) File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 892, in _createconnection , proto = await self._create_direct_connection(req, traces, timeout) File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection raise last_exc File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 1020, in _create_direct_connection transp, proto = await self._wrap_create_connection( File "/home/jo/.venvs/au/lib/python3.8/site-packages/aiohttp/connector.py", line 971, in _wrap_create_connection raise ClientConnectorCertificateError(req.connection_key, exc) from exc aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host cloud.coiled.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')]

cc @fjetter https://github.com/fjetter

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/85#issuecomment-825658485, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADQPYOWTDEPVSYBB5QIFH3TKFYY7ANCNFSM4S7PJHHQ .

-- Rami Chowdhury coiled.io "A mind all logic is like a knife all blade -- it makes the hand bleed that uses it." -- Rabindranath Tagore

jonashaag commented 3 years ago

Tried it with: 5.10.19-1-MANJARO, Python 3.8.5, virtualenv

Some of the packages installed:

local/ca-certificates 20181109-4
    Common CA certificates (default providers)
local/ca-certificates-mozilla 3.62-1
    Mozilla's set of trusted CA certificates
local/ca-certificates-utils 20181109-4
    Common CA certificates (utilities)
local/libksba 1.4.0-2
    Library for working with X.509 certificates, CMS data and related objects
local/lib32-openssl 1:1.1.1.j-1
    The Open Source toolkit for Secure Sockets Layer and Transport Layer Security (32-bit)
local/openssl 1.1.1.j-1
    The Open Source toolkit for Secure Sockets Layer and Transport Layer Security
local/openssl-1.0 1.0.2.u-1
    The Open Source toolkit for Secure Sockets Layer and Transport Layer Security
local/pkcs11-helper 1.27.0-1
    A library that simplifies the interaction with PKCS11 providers for end-user applications using a simple API
    and optional OpenSSL engine
shughes-uk commented 1 year ago

Haven't seen anyone run into this in a long time. Hopefully we're in the clear. We did change some of our SSL policies.