search

jupyterhub / oauthenticator

OAuth + JupyterHub Authenticator = OAuthenticator
https://oauthenticator.readthedocs.io
BSD 3-Clause "New" or "Revised" License
408 stars 362 forks source link

`pycurl` update causing "500 : Internal Server Error" after logging into newly created TLJHs using Google Authenticator #731

Closed jrdnbradford closed 6 months ago

jrdnbradford commented 6 months ago

I originally posted this issue at https://github.com/jupyterhub/the-littlest-jupyterhub/issues/964 but it may belong here instead.

Bug description

Installing TLJH and configuring the Google Authenticator results in 500 : Internal Server Error after login.

Bug resolution

I resolved this by comparing the hub environment on a working TLJH with the hub environment on a newly created TLJH and downgrading pycurl from 7.45.3 to 7.45.2:

sudo su
source /opt/tljh/hub/bin/activate
pip install --force-reinstall pycurl==7.45.2
tljh-config reload

I see requirements-hub-env.txt is setup with pycurl>=7.45.2,<8 . Not sure if it should be pinned to 7.45.2 in that repo or if it should be pinned here.

How to reproduce

  1. Install TLJH and configure Google Authenticator: 
    
from oauthenticator.google import GoogleOAuthenticator
c.JupyterHub.authenticator_class = GoogleOAuthenticator

c.GoogleOAuthenticator.google_service_account_keys = {DOMAIN: GROUP_ADMIN_SECRET_PATH} c.GoogleOAuthenticator.gsuite_administrator = {DOMAIN: GSUITE_ADMINISTRATOR} c.GoogleOAuthenticator.allowed_google_groups = {DOMAIN: ALLOWED_GOOGLE_GROUPS} c.GoogleOAuthenticator.admin_google_groups = {DOMAIN: ADMIN_GOOGLE_GROUPS} c.GoogleOAuthenticator.client_id = OAUTH_CLIENT_ID c.GoogleOAuthenticator.client_secret = OAUTH_CLIENT_SECRET c.GoogleOAuthenticator.hosted_domain = DOMAIN c.GoogleOAuthenticator.login_service = DOMAIN_NAME c.GoogleOAuthenticator.oauth_callback_url = 'https://DNS_NAME/hub/oauth_callback'

2. Attempt to sign-in with an allowed Google account

#### Expected behaviour

User should be able to login.

#### Actual behaviour

 `500 : Internal Server Error`  occurs after user attempts to login with allowed account.

Logs reveal a `tornado.curl_httpclient.CurlError`. See `Logs` section.

### Your personal set up
Ubuntu 22.04 
Littlest JupyterHub 1.0.0

<details><summary>Full environment</summary>
Here's the offending environment:

source /opt/tljh/hub/bin/activate pip list Package Version

aiohttp 3.9.3 aiosignal 1.3.1 alembic 1.13.1 async-generator 1.10 async-timeout 4.0.3 attrs 23.2.0 backoff 2.2.1 bcrypt 4.1.2 cachetools 5.3.2 certifi 2024.2.2 certipy 0.1.3 cffi 1.16.0 charset-normalizer 3.3.2 cryptography 42.0.3 escapism 1.0.1 frozenlist 1.4.1 google-api-core 2.17.1 google-api-python-client 2.118.0 google-auth 2.28.0 google-auth-httplib2 0.2.0 google-auth-oauthlib 1.2.0 googleapis-common-protos 1.62.0 greenlet 3.0.3 httplib2 0.22.0 idna 3.6 Jinja2 3.1.3 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter-telemetry 0.1.0 jupyterhub 4.0.2 jupyterhub-firstuseauthenticator 1.0.0 jupyterhub-idle-culler 1.2.1 jupyterhub-ldapauthenticator 1.3.2 jupyterhub-nativeauthenticator 1.2.0 jupyterhub-systemdspawner 1.0.1 jupyterhub-tmpauthenticator 1.0.0 jupyterhub-traefik-proxy 1.1.0 ldap3 2.9.1 Mako 1.3.2 MarkupSafe 2.1.5 multidict 6.0.5 oauthenticator 16.2.1 oauthlib 3.2.2 onetimepass 1.0.1 packaging 23.2 pamela 1.1.0 passlib 1.7.4 pip 24.0 pluggy 1.4.0 prometheus_client 0.20.0 protobuf 4.25.3 pyasn1 0.5.1 pyasn1-modules 0.3.0 pycparser 2.21 pycurl 7.45.3 pyOpenSSL 24.0.0 pyparsing 3.1.1 python-dateutil 2.8.2 python-json-logger 2.0.7 referencing 0.33.0 requests 2.31.0 requests-oauthlib 1.3.1 rpds-py 0.18.0 rsa 4.9 ruamel.yaml 0.17.40 ruamel.yaml.clib 0.2.8 setuptools 59.6.0 six 1.16.0 SQLAlchemy 2.0.27 the-littlest-jupyterhub 1.0.0 toml 0.10.2 tornado 6.4 traitlets 5.14.1 typing_extensions 4.9.0 uritemplate 4.1.1 urllib3 2.2.1 yarl 1.9.4

</details>

<details><summary>Logs</summary>
Error logs:

Feb 19 03:02:55 $VM_NAME python3[9285]: [E 2024-02-19 03:02:55.548 JupyterHub oauth2:626] Error fetching 599 POST https://www.googleapis.com/oauth2/v4/token: HTTP 599: error setting certificate verify locations: Feb 19 03:02:55 $VM_NAME python3[9285]: CAfile: /etc/pki/tls/certs/ca-bundle.crt Feb 19 03:02:55 $VM_NAME python3[9285]: CApath: none ... ... Feb 19 03:02:55 $VM_NAME python3[9285]: Traceback (most recent call last): Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/tornado/web.py", line 1790, in _execute Feb 19 03:02:55 $VM_NAME python3[9285]: result = await result Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 208, in get Feb 19 03:02:55 $VM_NAME python3[9285]: user = await self.login_user() Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 828, in login_user Feb 19 03:02:55 $VM_NAME python3[9285]: authenticated = await self.authenticate(data) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/auth.py", line 492, in get_authenticated_user Feb 19 03:02:55 $VM_NAME python3[9285]: authenticated = await maybe_future(self.authenticate(handler, data)) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 969, in authenticate Feb 19 03:02:55 $VM_NAME python3[9285]: token_info = await self.get_token_info(handler, access_token_params) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 843, in get_token_info Feb 19 03:02:55 $VM_NAME python3[9285]: token_info = await self.httpfetch( Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 661, in httpfetch Feb 19 03:02:55 $VM_NAME python3[9285]: return await self.fetch( Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 627, in fetch Feb 19 03:02:55 $VM_NAME python3[9285]: raise e Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 606, in fetch Feb 19 03:02:55 $VM_NAME python3[9285]: resp = await self.http_client.fetch(req, **kwargs) Feb 19 03:02:55 $VM_NAME python3[9285]: tornado.curl_httpclient.CurlError: HTTP 599: error setting certificate verify locations: Feb 19 03:02:55 $VM_NAME python3[9285]: CAfile: /etc/pki/tls/certs/ca-bundle.crt Feb 19 03:02:55 $VM_NAME python3[9285]: CApath: none Feb 19 03:02:55 $VM_NAME python3[9285]:
... ... Feb 19 03:02:55 $VM_NAME python3[9285]: [E 2024-02-19 03:02:55.554 JupyterHub log:191] 500 GET /hub/oauth_callback?state=[secret]&code=[secret]&scope=email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=[secret]&hd=bc.edu&prompt=none (@$IP_ADDRESS) 15.03ms


</details>
welcome[bot] commented 6 months ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

pastram-i commented 6 months ago

Confirming this is unrelated to TLJH/Google Auth - I had this error with Z2JH+Azure.

I would also like to confirm the resolution of pycurl downgrading from 7.45.3 -> 7.45.2 resolved this issue for me as well. I changed in my requirements.txt for jupyterhub to include pycurl==7.45.2.

Pycurl appears to be an indirect dependency (only used in certain cases/not installed by default) of jupyterhub itself, so the issue may be worth mentioning there for a dependency update to resolve.

I haven't been able to dig into what changed/actually broke on pycurl, but if someone can point to it it may be worth reporting in the pycurl github.

consideRatio commented 6 months ago

Tracked by https://github.com/pycurl/pycurl/issues/834

consideRatio commented 6 months ago

Thank you for an amazing issue writeup @jrdnbradford!! And thanl you @pastram-i for connecting the dots on what this is about! Nice work!!

jrdnbradford commented 6 months ago

Awesome, I somehow missed that filed issue over at https://github.com/pycurl/pycurl/issues/834.

Thanks for the assist!