TLJH upgrade causes SSLV3_ALERT_HANDSHAKE_FAILURE until ciphers explicitly configured

fredcy commented 3 weeks ago

Bug description

Upgrading TLJH breaks LDAP authentication with this error:

ldap3.core.exceptions.LDAPSocketOpenError: ("('socket ssl wrapping error: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)',)",)

If I explicitly set the list of TLS ciphers in the TLJH config as below, then authentication starts working again. So maybe this is not a bug-report per se, but report of a workaround that took me a long time to discover.

    tls_strategy: "on_connect"
    tls_kwargs: { "ciphers": "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:RSA-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-GCM-SHA384:RSA-PSK-CHACHA20-POLY1305:DHE-PSK-CHACHA20-POLY1305:ECDHE-PSK-CHACHA20-POLY1305:AES256-GCM-SHA384:PSK-AES256-GCM-SHA384:PSK-CHACHA20-POLY1305:RSA-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-GCM-SHA256:AES128-GCM-SHA256:PSK-AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:RSA-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA384:RSA-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA:AES256-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:RSA-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA256:RSA-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA:AES128-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA" }

How to reproduce

Start with a year-old TLJH instance that uses ldapauthenticator configured with use_ssl: True.

Update TLJH to the latest version, and thus ldapauthenticator also.

Try to login at the TLJH web home page:

Expected behaviour

I expected to be able to login as I did before the upgrade.

Actual behaviour

It fails with HTTP 500 error, and the internal logs show the above SSLV3_ALERT_HANDSHAKE_FAILURE error.

Your personal set up

I'm using the TLJH on Ubuntu 22.04.5 LTS.

We originally installed TLJH a year ago.

Our OpenLDAP server expects to connect with TLS 1.2 using TLS-over-SSL on port 636.

Version(s):

Python 3.10.12

Full environment

``` aiohttp==3.8.5 aiosignal==1.3.1 alembic==1.12.0 annotated-types==0.7.0 arrow==1.3.0 async-generator==1.10 async-timeout==4.0.3 attrs==23.1.0 backoff==2.2.1 bcrypt==4.0.1 certifi==2023.7.22 certipy==0.1.3 cffi==1.15.1 charset-normalizer==3.2.0 cryptography==41.0.3 escapism==1.0.1 filelock==3.16.1 fqdn==1.5.1 frozenlist==1.4.0 greenlet==2.0.2 idna==3.4 isoduration==20.11.0 Jinja2==3.1.2 jsonpointer==3.0.0 jsonschema==4.19.0 jsonschema-specifications==2023.7.1 jupyter-events==0.10.0 jupyter-telemetry==0.1.0 jupyterhub==5.2.1 jupyterhub-firstuseauthenticator==1.1.0 jupyterhub-idle-culler==1.4.0 jupyterhub-ldapauthenticator==2.0.1 jupyterhub-nativeauthenticator==1.3.0 jupyterhub-systemdspawner==1.0.2 jupyterhub-tmpauthenticator==1.0.0 jupyterhub-traefik-proxy==2.0.0 ldap3==2.9.1 Mako==1.2.4 MarkupSafe==2.1.3 multidict==6.0.4 oauthenticator==17.1.0 oauthlib==3.2.2 onetimepass==1.0.1 packaging==23.1 pamela==1.1.0 passlib==1.7.4 pluggy==1.3.0 prometheus-client==0.17.1 pyasn1==0.5.0 pycparser==2.21 pycurl==7.45.3 pydantic==2.9.2 pydantic_core==2.23.4 PyJWT==2.9.0 pyOpenSSL==23.2.0 python-dateutil==2.8.2 python-json-logger==2.0.7 PyYAML==6.0.2 referencing==0.30.2 requests==2.31.0 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rpds-py==0.10.2 ruamel.yaml==0.18.6 ruamel.yaml.clib==0.2.7 six==1.16.0 SQLAlchemy==2.0.20 the-littlest-jupyterhub @ git+https://github.com/jupyterhub/the-littlest-jupyterhub.git@a47a171850afc22d46f3b5b5bb0d09f2fae0937f toml==0.10.2 tornado==6.3.3 traitlets==5.9.0 types-python-dateutil==2.9.0.20241003 typing_extensions==4.7.1 uri-template==1.3.0 urllib3==2.0.4 webcolors==24.8.0 yarl==1.9.2 ```

fredcy commented 3 weeks ago

I got the cipher list from the output of openssl ciphers -tls1_2

consideRatio commented 2 weeks ago

I think you configured the following ciphers

['AES128-GCM-SHA256',
 'AES128-SHA',
 'AES128-SHA256',
 'AES256-GCM-SHA384',
 'AES256-SHA',
 'AES256-SHA256',
 'DHE-PSK-AES128-CBC-SHA',
 'DHE-PSK-AES128-CBC-SHA256',
 'DHE-PSK-AES128-GCM-SHA256',
 'DHE-PSK-AES256-CBC-SHA',
 'DHE-PSK-AES256-CBC-SHA384',
 'DHE-PSK-AES256-GCM-SHA384',
 'DHE-PSK-CHACHA20-POLY1305',
 'DHE-RSA-AES128-GCM-SHA256',
 'DHE-RSA-AES128-SHA',
 'DHE-RSA-AES128-SHA256',
 'DHE-RSA-AES256-GCM-SHA384',
 'DHE-RSA-AES256-SHA',
 'DHE-RSA-AES256-SHA256',
 'DHE-RSA-CHACHA20-POLY1305',
 'ECDHE-ECDSA-AES128-GCM-SHA256',
 'ECDHE-ECDSA-AES128-SHA',
 'ECDHE-ECDSA-AES128-SHA256',
 'ECDHE-ECDSA-AES256-GCM-SHA384',
 'ECDHE-ECDSA-AES256-SHA',
 'ECDHE-ECDSA-AES256-SHA384',
 'ECDHE-ECDSA-CHACHA20-POLY1305',
 'ECDHE-PSK-AES128-CBC-SHA',
 'ECDHE-PSK-AES128-CBC-SHA256',
 'ECDHE-PSK-AES256-CBC-SHA',
 'ECDHE-PSK-AES256-CBC-SHA384',
 'ECDHE-PSK-CHACHA20-POLY1305',
 'ECDHE-RSA-AES128-GCM-SHA256',
 'ECDHE-RSA-AES128-SHA',
 'ECDHE-RSA-AES128-SHA256',
 'ECDHE-RSA-AES256-GCM-SHA384',
 'ECDHE-RSA-AES256-SHA',
 'ECDHE-RSA-AES256-SHA384',
 'ECDHE-RSA-CHACHA20-POLY1305',
 'PSK-AES128-CBC-SHA',
 'PSK-AES128-CBC-SHA256',
 'PSK-AES128-GCM-SHA256',
 'PSK-AES256-CBC-SHA',
 'PSK-AES256-CBC-SHA384',
 'PSK-AES256-GCM-SHA384',
 'PSK-CHACHA20-POLY1305',
 'RSA-PSK-AES128-CBC-SHA',
 'RSA-PSK-AES128-CBC-SHA256',
 'RSA-PSK-AES128-GCM-SHA256',
 'RSA-PSK-AES256-CBC-SHA',
 'RSA-PSK-AES256-CBC-SHA384',
 'RSA-PSK-AES256-GCM-SHA384',
 'RSA-PSK-CHACHA20-POLY1305',
 'SRP-AES-128-CBC-SHA',
 'SRP-AES-256-CBC-SHA',
 'SRP-RSA-AES-128-CBC-SHA',
 'SRP-RSA-AES-256-CBC-SHA',
 'TLS_AES_128_GCM_SHA256',
 'TLS_AES_256_GCM_SHA384',
 'TLS_CHACHA20_POLY1305_SHA256']

where the default ciphers may have been:

['DHE-RSA-AES128-GCM-SHA256',
 'DHE-RSA-AES128-SHA256',
 'DHE-RSA-AES256-GCM-SHA384',
 'DHE-RSA-AES256-SHA256',
 'ECDHE-ECDSA-AES128-GCM-SHA256',
 'ECDHE-ECDSA-AES128-SHA256',
 'ECDHE-ECDSA-AES256-GCM-SHA384',
 'ECDHE-ECDSA-AES256-SHA384',
 'ECDHE-ECDSA-CHACHA20-POLY1305',
 'ECDHE-RSA-AES128-GCM-SHA256',
 'ECDHE-RSA-AES128-SHA256',
 'ECDHE-RSA-AES256-GCM-SHA384',
 'ECDHE-RSA-AES256-SHA384',
 'ECDHE-RSA-CHACHA20-POLY1305',
 'TLS_AES_128_GCM_SHA256',
 'TLS_AES_256_GCM_SHA384',
 'TLS_CHACHA20_POLY1305_SHA256']

Hmmm...

I wonder what cipher ended up being needed. Do you know what ciphers was supported via OpenLDAP? I observe that the default ciphers was all part of the larger list you specified.

consideRatio commented 2 weeks ago

In Python < 3.10, the following ciphers were part of the default ssl context:

['SSLv3__AES128-SHA',
 'SSLv3__AES256-SHA',
 'SSLv3__DHE-RSA-AES128-SHA',
 'SSLv3__DHE-RSA-AES256-SHA',
 'TLSv1.0__ECDHE-ECDSA-AES128-SHA',
 'TLSv1.0__ECDHE-ECDSA-AES256-SHA',
 'TLSv1.0__ECDHE-RSA-AES128-SHA',
 'TLSv1.0__ECDHE-RSA-AES256-SHA',
 'TLSv1.2__AES128-GCM-SHA256',
 'TLSv1.2__AES128-SHA256',
 'TLSv1.2__AES256-GCM-SHA384',
 'TLSv1.2__AES256-SHA256',
 'TLSv1.2__DHE-RSA-AES128-GCM-SHA256',
 'TLSv1.2__DHE-RSA-AES128-SHA256',
 'TLSv1.2__DHE-RSA-AES256-GCM-SHA384',
 'TLSv1.2__DHE-RSA-AES256-SHA256',
 'TLSv1.2__DHE-RSA-CHACHA20-POLY1305',
 'TLSv1.2__ECDHE-ECDSA-AES128-GCM-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES128-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES256-GCM-SHA384',
 'TLSv1.2__ECDHE-ECDSA-AES256-SHA384',
 'TLSv1.2__ECDHE-ECDSA-CHACHA20-POLY1305',
 'TLSv1.2__ECDHE-RSA-AES128-GCM-SHA256',
 'TLSv1.2__ECDHE-RSA-AES128-SHA256',
 'TLSv1.2__ECDHE-RSA-AES256-GCM-SHA384',
 'TLSv1.2__ECDHE-RSA-AES256-SHA384',
 'TLSv1.2__ECDHE-RSA-CHACHA20-POLY1305',
 'TLSv1.3__TLS_AES_128_GCM_SHA256',
 'TLSv1.3__TLS_AES_256_GCM_SHA384',
 'TLSv1.3__TLS_CHACHA20_POLY1305_SHA256']

In Python 3.10+, that became strictly narrower list with no new ciphers added:

['TLSv1.2__DHE-RSA-AES128-GCM-SHA256',
 'TLSv1.2__DHE-RSA-AES128-SHA256',
 'TLSv1.2__DHE-RSA-AES256-GCM-SHA384',
 'TLSv1.2__DHE-RSA-AES256-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES128-GCM-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES128-SHA256',
 'TLSv1.2__ECDHE-ECDSA-AES256-GCM-SHA384',
 'TLSv1.2__ECDHE-ECDSA-AES256-SHA384',
 'TLSv1.2__ECDHE-ECDSA-CHACHA20-POLY1305',
 'TLSv1.2__ECDHE-RSA-AES128-GCM-SHA256',
 'TLSv1.2__ECDHE-RSA-AES128-SHA256',
 'TLSv1.2__ECDHE-RSA-AES256-GCM-SHA384',
 'TLSv1.2__ECDHE-RSA-AES256-SHA384',
 'TLSv1.2__ECDHE-RSA-CHACHA20-POLY1305',
 'TLSv1.3__TLS_AES_128_GCM_SHA256',
 'TLSv1.3__TLS_AES_256_GCM_SHA384',
 'TLSv1.3__TLS_CHACHA20_POLY1305_SHA256']

consideRatio commented 2 weeks ago

I suspect upgrading to using a more modern version of Python led to the list of ciphers was too narrow for an agreement with the server, which doesn't sound bad security wise --- could it be that your LDAP server isn't allowing any cipher part of the python 3.10 list?

I figure its probably not suitable to expand the list of ciphers by default within this project.

fredcy commented 2 weeks ago

Using tcpdump and wireshark, I found that the TLJH setup that I've been using over the last year would settle on the TLS_RSA_WITH_AES_256_GCM_SHA384 cipher when connecting successfully to our LDAP server. (That's the name as displayed by wireshark, apparently encoded as 0x009d). When I got the connection to work again on the latest TLJH update and with the ciphers explicitly listed as above, the same cipher was chosen.

I don't have access to our LDAP server but it's running RHEL 7 and has correspondingly older versions of the openssl libs.

fredcy commented 2 weeks ago

In lieu of extending the list of default ciphers, perhaps the docs for ldapauthenticator could mention the symptoms and workaround for the case of cipher mismatch.

consideRatio commented 2 weeks ago

I think TLS_RSA_WITH_AES_256_GCM_SHA384 is what openssl calls AES256-GCM-SHA384 (TLS 1.2) which was indeed used with Python < 3.10, but not after Python >= 3.10.

Amazing debugging into this @fredcy, I fully agree on docs about this - possibly also a log message if this error ocurr to point users in the right direction.

consideRatio commented 2 weeks ago

@fredcy I opened https://github.com/jupyterhub/ldapauthenticator/pull/297 about this, thank you for digging in deep to this and reporting this -- I feel confident this will help others as well!

jupyterhub / ldapauthenticator