`coiled login` failure SSLCertVerificationError on cloud.coiled.io:443

[fkaleo]

Using python 3.11 on Linux with coiled 1.1.11 installed from pypi and doing coiled login:

Traceback (most recent call last):
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 1112, in create_connection
    transport, protocol = await self._create_connection_transport(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 1145, in _create_connection_transport
    await waiter
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/sslproto.py", line 575, in _on_handshake_complete
    raise handshake_exc
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/sslproto.py", line 557, in _do_handshake
    self._sslobj.do_handshake()
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/ssl.py", line 979, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/bin/coiled", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/coiled/cli/login.py", line 23, in login
    asyncio.run(
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/coiled/utils.py", line 279, in login_if_required
    await handle_credentials(server=server, token=token, account=account, save=save, retry=retry, browser=browser)
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/coiled/utils.py", line 360, in handle_credentials
    result = await client_token_grant_flow(server, browser, account=account)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/coiled/auth.py", line 33, in client_token_grant_flow
    token_data = await make_unattached_token(server, label=get_local_user())
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/coiled/auth.py", line 26, in make_unattached_token
    async with session.post(url, data={"label": label}) as resp:
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1167, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 562, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
    raise last_exc
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 982, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host cloud.coiled.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)')]

[dchudz]

Hi! Looks like something is wrong with SSL in your local Python environment. A few steps that would help us help you debug it:

Does this work, or give an error?

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://google.com') as resp:
            print(resp.status)
            print(await resp.text())

asyncio.run(main())

How about this?:

curl -vvI https://google.com

Or this?:

curl -vvI https://cloud.coiled.io/

[fkaleo]

curl -vvI https://cloud.coiled.io/
*   Trying 13.32.27.122:443...
* Connected to cloud.coiled.io (13.32.27.122) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=cloud.coiled.io
*  start date: Feb 21 00:00:00 2023 GMT
*  expire date: Jan 12 23:59:59 2024 GMT
*  subjectAltName: host "cloud.coiled.io" matched cert's "cloud.coiled.io"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x56328d3dae90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> HEAD / HTTP/2
> Host: cloud.coiled.io
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
HTTP/2 200 
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 1605
content-length: 1605
< vary: Accept-Encoding
vary: Accept-Encoding
< date: Mon, 13 Nov 2023 19:18:16 GMT
date: Mon, 13 Nov 2023 19:18:16 GMT
< server: uvicorn
server: uvicorn
< expires: Mon, 13 Nov 2023 19:33:16 GMT
expires: Mon, 13 Nov 2023 19:33:16 GMT
< cache-control: max-age=900
cache-control: max-age=900
< x-frame-options: DENY
x-frame-options: DENY
< strict-transport-security: max-age=15780000
strict-transport-security: max-age=15780000
< x-content-type-options: nosniff
x-content-type-options: nosniff
< referrer-policy: same-origin
referrer-policy: same-origin
< cross-origin-opener-policy: same-origin
cross-origin-opener-policy: same-origin
< vary: origin
vary: origin
< x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
< via: 1.1 b25bc331cb2e5e7e25d9488f5ecdc940.cloudfront.net (CloudFront)
via: 1.1 b25bc331cb2e5e7e25d9488f5ecdc940.cloudfront.net (CloudFront)
< x-amz-cf-pop: FRA56-C2
x-amz-cf-pop: FRA56-C2
< x-amz-cf-id: MAOv-iQ5x8W74JbFaGXnUDpFqMJHs5snE_1NNxaN18E9eUXcOnvZzQ==
x-amz-cf-id: MAOv-iQ5x8W74JbFaGXnUDpFqMJHs5snE_1NNxaN18E9eUXcOnvZzQ==

< 
* Connection #0 to host cloud.coiled.io left intact

[fkaleo]

curl -vvI https://google.com
*   Trying 2a00:1450:4007:810::200e:443...
* Connected to google.com (2a00:1450:4007:810::200e) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.google.com
*  start date: Oct 16 08:02:35 2023 GMT
*  expire date: Jan  8 08:02:34 2024 GMT
*  subjectAltName: host "google.com" matched cert's "google.com"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x558a04d89e90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> HEAD / HTTP/2
> Host: google.com
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
< HTTP/2 301 
HTTP/2 301 
< location: https://www.google.com/
location: https://www.google.com/
< content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8
< content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-a4nK22oPpColdTKEW6jfXg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-a4nK22oPpColdTKEW6jfXg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< date: Mon, 13 Nov 2023 19:18:57 GMT
date: Mon, 13 Nov 2023 19:18:57 GMT
< expires: Mon, 13 Nov 2023 19:18:57 GMT
expires: Mon, 13 Nov 2023 19:18:57 GMT
< cache-control: private, max-age=2592000
cache-control: private, max-age=2592000
< server: gws
server: gws
< content-length: 220
content-length: 220
< x-xss-protection: 0
x-xss-protection: 0
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< set-cookie: CONSENT=PENDING+001; expires=Wed, 12-Nov-2025 19:18:57 GMT; path=/; domain=.google.com; Secure
set-cookie: CONSENT=PENDING+001; expires=Wed, 12-Nov-2025 19:18:57 GMT; path=/; domain=.google.com; Secure
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

< 
* Connection #0 to host google.com left intact

[fkaleo]

>>> asyncio.run(main())
Traceback (most recent call last):
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 1112, in create_connection
    transport, protocol = await self._create_connection_transport(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 1145, in _create_connection_transport
    await waiter
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/sslproto.py", line 575, in _on_handshake_complete
    raise handshake_exc
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/sslproto.py", line 557, in _do_handshake
    self._sslobj.do_handshake()
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/ssl.py", line 979, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "<stdin>", line 3, in main
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1167, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 562, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
    raise last_exc
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kaleo/shared_home/code/nlp_experiments/tools/universal/.venv/lib/python3.11/site-packages/aiohttp/connector.py", line 982, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host google.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)')]

[fkaleo]

I have aiohttp 3.8.6

[dchudz]

Hmm, looks like it's not Coiled-specific-- SSL is broken generally in this Python environment.

If you list your other package versions (especially certifi) that might help us.

Is there any custom/special networking going on? (A corporate VPN, etc?)

[fkaleo]

no vpn, no special networking.

name = "certifi"
version = "2023.7.22"

[fkaleo]

Thank you so much for taking the time!

[dchudz]

We might be stuck for now, but will continue pondering. Sorry!

As mentioned, whatever this is, I don't think it's Coiled-specific. If you can get HTTPS to work with aiohttp at all, then it should work for Coiled too. (I realize this might not be very helpful, though.)

[dchudz]

@fkaleo Okay, one more question for now:

How are you creating the environment? (System python with virtualenv/pip? Conda Etc?) I think I'd suggest making a new Python environment and seeing if you fare any better with that. Maybe an environment using miniconda or mamba.

[AdarshNamala]

I was facing the same issue - I was able to ping using curl, but asyncio and aiohttp were failing with the SSL error.

Setting this env resolved the coiled login issue - "export SSL_CERT_FILE=/path/to/your/ca-certificates.crt"

[ntabris]

If anyone with the problem wants to try something, I'd be curious what this shows:

import ssl

ca_path = ssl.get_default_verify_paths().openssl_cafile
print(ca_path)

with open(ca_path) as f:
  ca_certs = f.read()

# check for fingerprint of ISRG Root X1
print('96:bc:ec:06:26:49:76:f3:74' in ca_certs)