certificate verify failed

[darose]

I have jupyterhub-ssh installed (via helm chart), but the SSH command fails, apparently due to SSL issues:

drosen2@drosen2:~$ ssh drosen2@172.16.181.5
Password:
Connection closed by 172.16.181.5 port 22

When I look in the jupyterhub-ssh log, I see this:

[asyncssh] [conn=2] Accepted SSH client connection
[asyncssh] [conn=2]   Local address: 172.18.4.100, port 8022
[asyncssh] [conn=2]   Peer address: 172.18.4.96, port 53214
[asyncssh] [conn=2] Sending version SSH-2.0-AsyncSSH_2.7.0
[asyncssh] [conn=2] Received version SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
[asyncssh] [conn=2] Requesting key exchange
[asyncssh] [conn=2] Received key exchange request
[asyncssh] [conn=2] Beginning key exchange
[asyncssh] [conn=2] Completed key exchange
[asyncssh] [conn=2] Beginning auth for user drosen2
[asyncssh] [conn=2] Trying keyboard-interactive auth
[asyncssh] [conn=2] Uncaught exception
Traceback (most recent call last):
  File "/home/jovyan/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

It's apparently tossing a cert verify failed error when trying to verify the cert at my hub address:

aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host <my hub-address>:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')]

Any idea what the issue might be / how to fix/workaround?

[consideRatio]

I'm not sure. What version of the Helm chart is installed? Only the latest versions of the helm chart automatically generate some relevant keys for you.

[darose]

0.0.1-n142.h402a3d6 (The most recent, I think?)

[darose]

Hypothesis: I think our organization might require us to supply our custom cert chain for the SSL verify to work. Is there any hook for me to provide that?

[darose]

Or to disable cert verification?

[aurany]

Thanks for a great repo!

I am struggling with the same error message, see below. Probably this is because of bad configuration but still i need some help. For the host_key_path I put in the .key file used by JupyterHub TLS (i have one .cer and one .key file) and for the hub_url I use https://:8000.

I run jupyterhub, user-notebooks and jupyterhub-ssh in separate containers. jupyterhub uses port 8000 on the host and jupyterhub-ssh uses 8022. I am trying to connect to jupyterhub-ssl via putty. All containers run on the same machine (Linux x86).

Startup is looking ok

[asyncssh] Creating SSH listener on port 8022
[asyncssh] [conn=0] Accepted SSH client connection
[asyncssh] [conn=0]   Local address: xx.xxx.xxx.161, port 8022
[asyncssh] [conn=0]   Peer address: xx.xxx.x.104, port 49220
[asyncssh] [conn=0] Sending version SSH-2.0-AsyncSSH_2.8.1
[asyncssh] [conn=0] Received version SSH-2.0-PuTTY_Release_0.73
[asyncssh] [conn=0] Requesting key exchange
[asyncssh] [conn=0] Received key exchange request
[asyncssh] [conn=0] Beginning key exchange
[asyncssh] [conn=0] Completed key exchange
[asyncssh] [conn=0] Beginning auth for user USERID
[asyncssh] [conn=0] Trying keyboard-interactive auth

But when I try to login I see this message

[asyncssh] [conn=0] Uncaught exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

EDIT It is working if TLS is disabled in JupyterHub but i'm not allowed to do that. <<<

[stubclan]

@darose I had the same issue, the SSL error happens when trying to connect to jupyterhub host url. If you include certs in the jupyterhub-ssh image it seems to be working fine.

If you have ssl enabled to jupyterhub, the same certs should work

[5uperpalo]

@stubclan @darose I am stuck on the same issue as you were 1.5year ago .. I have a company certificate generated by Sectigo, I used it in the values.yaml file for jupyterhub .. how can I use it in the jupyterhub-ssh? I did not get it how can I include it in the jupyterhub-ssh image? .. please, could you help me?

[stubclan]

@5uperpalo I added the certificate in the base image of jupyterhub-ssh manually. Basically create another Dockerfile with the the image and move your cert to appropriate location and install it.

sudo cp mycert.crt /usr/local/share/ca-certificates/ sudo update-ca-certificates

On a side note, this implementation web based ssh is very limiting and its probably better to write your own ssh proxy. I wrote one with Go, its a bit hacky to work with jhub but much better experience.

[5uperpalo]

@stubclan thank you for your response, I was trying to avoid digging deeper into this and making my own customized implementation but without a root user, I can't even add the SSL certificate to trusted certificates in the pod, so I think I will follow your advise and adjust the docker.... thank you for an idea...

[5uperpalo]

@stubclan could you by any chance share your implementation of ssh proxy? I am still struggling with certs .. for some reason they are not working in async mode, I tried also disabling ssl in iohttp, but I am always getting new and different bugs

[stubclan]

@5uperpalo Sorry for very late response. I've completely diverted from tornado implementation and using go crypto/ssh proxy in combination with other things for authentication and authorization. Hopefully you found your solution