search

jupyterhub / the-littlest-jupyterhub

Simple JupyterHub distribution for 1-100 users on a single server
https://tljh.jupyter.org
BSD 3-Clause "New" or "Revised" License
1.01k stars 341 forks source link

`pycurl` update causing "500 : Internal Server Error" after logging into newly created TLJHs using Google Authenticator #964

Closed jrdnbradford closed 4 months ago

jrdnbradford commented 4 months ago

Bug description

Installing TLJH and configuring the Google Authenticator results in 500 : Internal Server Error after login.

Bug resolution

I resolved this by comparing the hub environment on a working TLJH with the hub environment on a newly created TLJH and downgrading pycurl from 7.45.3 to 7.45.2:

sudo su
source /opt/tljh/hub/bin/activate
pip install --force-reinstall pycurl==7.45.2
tljh-config reload

I see requirements-hub-env.txt is setup with pycurl>=7.45.2,<8 . Not sure if it should be pinned to 7.45.2 or some other resolution should be implemented.

How to reproduce

  1. Install TLJH and configure Google Authenticator: 
    
from oauthenticator.google import GoogleOAuthenticator
c.JupyterHub.authenticator_class = GoogleOAuthenticator

c.GoogleOAuthenticator.google_service_account_keys = {DOMAIN: GROUP_ADMIN_SECRET_PATH} c.GoogleOAuthenticator.gsuite_administrator = {DOMAIN: GSUITE_ADMINISTRATOR} c.GoogleOAuthenticator.allowed_google_groups = {DOMAIN: ALLOWED_GOOGLE_GROUPS} c.GoogleOAuthenticator.admin_google_groups = {DOMAIN: ADMIN_GOOGLE_GROUPS} c.GoogleOAuthenticator.client_id = OAUTH_CLIENT_ID c.GoogleOAuthenticator.client_secret = OAUTH_CLIENT_SECRET c.GoogleOAuthenticator.hosted_domain = DOMAIN c.GoogleOAuthenticator.login_service = DOMAIN_NAME c.GoogleOAuthenticator.oauth_callback_url = 'https://DNS_NAME/hub/oauth_callback'

2. Attempt to sign-in with an allowed Google account

#### Expected behaviour

User should be able to login.

#### Actual behaviour

 `500 : Internal Server Error`  occurs after user attempts to login with allowed account.

Logs reveal a `tornado.curl_httpclient.CurlError`. See `Logs` section.

### Your personal set up
Ubuntu 22.04 
Littlest JupyterHub 1.0.0

<details><summary>Full environment</summary>
Here's the offending environment:

source /opt/tljh/hub/bin/activate pip list Package Version

aiohttp 3.9.3 aiosignal 1.3.1 alembic 1.13.1 async-generator 1.10 async-timeout 4.0.3 attrs 23.2.0 backoff 2.2.1 bcrypt 4.1.2 cachetools 5.3.2 certifi 2024.2.2 certipy 0.1.3 cffi 1.16.0 charset-normalizer 3.3.2 cryptography 42.0.3 escapism 1.0.1 frozenlist 1.4.1 google-api-core 2.17.1 google-api-python-client 2.118.0 google-auth 2.28.0 google-auth-httplib2 0.2.0 google-auth-oauthlib 1.2.0 googleapis-common-protos 1.62.0 greenlet 3.0.3 httplib2 0.22.0 idna 3.6 Jinja2 3.1.3 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter-telemetry 0.1.0 jupyterhub 4.0.2 jupyterhub-firstuseauthenticator 1.0.0 jupyterhub-idle-culler 1.2.1 jupyterhub-ldapauthenticator 1.3.2 jupyterhub-nativeauthenticator 1.2.0 jupyterhub-systemdspawner 1.0.1 jupyterhub-tmpauthenticator 1.0.0 jupyterhub-traefik-proxy 1.1.0 ldap3 2.9.1 Mako 1.3.2 MarkupSafe 2.1.5 multidict 6.0.5 oauthenticator 16.2.1 oauthlib 3.2.2 onetimepass 1.0.1 packaging 23.2 pamela 1.1.0 passlib 1.7.4 pip 24.0 pluggy 1.4.0 prometheus_client 0.20.0 protobuf 4.25.3 pyasn1 0.5.1 pyasn1-modules 0.3.0 pycparser 2.21 pycurl 7.45.3 pyOpenSSL 24.0.0 pyparsing 3.1.1 python-dateutil 2.8.2 python-json-logger 2.0.7 referencing 0.33.0 requests 2.31.0 requests-oauthlib 1.3.1 rpds-py 0.18.0 rsa 4.9 ruamel.yaml 0.17.40 ruamel.yaml.clib 0.2.8 setuptools 59.6.0 six 1.16.0 SQLAlchemy 2.0.27 the-littlest-jupyterhub 1.0.0 toml 0.10.2 tornado 6.4 traitlets 5.14.1 typing_extensions 4.9.0 uritemplate 4.1.1 urllib3 2.2.1 yarl 1.9.4

</details>

<details><summary>Logs</summary>
Error logs:

Feb 19 03:02:55 $VM_NAME python3[9285]: [E 2024-02-19 03:02:55.548 JupyterHub oauth2:626] Error fetching 599 POST https://www.googleapis.com/oauth2/v4/token: HTTP 599: error setting certificate verify locations: Feb 19 03:02:55 $VM_NAME python3[9285]: CAfile: /etc/pki/tls/certs/ca-bundle.crt Feb 19 03:02:55 $VM_NAME python3[9285]: CApath: none ... ... Feb 19 03:02:55 $VM_NAME python3[9285]: Traceback (most recent call last): Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/tornado/web.py", line 1790, in _execute Feb 19 03:02:55 $VM_NAME python3[9285]: result = await result Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 208, in get Feb 19 03:02:55 $VM_NAME python3[9285]: user = await self.login_user() Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 828, in login_user Feb 19 03:02:55 $VM_NAME python3[9285]: authenticated = await self.authenticate(data) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/auth.py", line 492, in get_authenticated_user Feb 19 03:02:55 $VM_NAME python3[9285]: authenticated = await maybe_future(self.authenticate(handler, data)) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 969, in authenticate Feb 19 03:02:55 $VM_NAME python3[9285]: token_info = await self.get_token_info(handler, access_token_params) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 843, in get_token_info Feb 19 03:02:55 $VM_NAME python3[9285]: token_info = await self.httpfetch( Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 661, in httpfetch Feb 19 03:02:55 $VM_NAME python3[9285]: return await self.fetch( Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 627, in fetch Feb 19 03:02:55 $VM_NAME python3[9285]: raise e Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 606, in fetch Feb 19 03:02:55 $VM_NAME python3[9285]: resp = await self.http_client.fetch(req, **kwargs) Feb 19 03:02:55 $VM_NAME python3[9285]: tornado.curl_httpclient.CurlError: HTTP 599: error setting certificate verify locations: Feb 19 03:02:55 $VM_NAME python3[9285]: CAfile: /etc/pki/tls/certs/ca-bundle.crt Feb 19 03:02:55 $VM_NAME python3[9285]: CApath: none Feb 19 03:02:55 $VM_NAME python3[9285]:
... ... Feb 19 03:02:55 $VM_NAME python3[9285]: [E 2024-02-19 03:02:55.554 JupyterHub log:191] 500 GET /hub/oauth_callback?state=[secret]&code=[secret]&scope=email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=[secret]&hd=bc.edu&prompt=none (@$IP_ADDRESS) 15.03ms


</details>
consideRatio commented 4 months ago

Should resolve with https://github.com/pycurl/pycurl/issues/834

AashitaK commented 3 months ago

I have a TLJH instance running for more than a year on a bare metal server. Recently upgraded the TLJH instance to the latest version and since then I was getting the same 500 : Internal Server Error as @jrdnbradford after trying to login. My TLJH instance is using Github Authenticator but the logs looked very similar. I could fix the error by downgrading pycurl from 7.45.3 to 7.45.2