Closed jrdnbradford closed 6 months ago
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:
Confirming this is unrelated to TLJH/Google Auth - I had this error with Z2JH+Azure.
I would also like to confirm the resolution of pycurl downgrading from 7.45.3
-> 7.45.2
resolved this issue for me as well. I changed in my requirements.txt for jupyterhub to include pycurl==7.45.2
.
Pycurl appears to be an indirect dependency (only used in certain cases/not installed by default) of jupyterhub itself, so the issue may be worth mentioning there for a dependency update to resolve.
I haven't been able to dig into what changed/actually broke on pycurl, but if someone can point to it it may be worth reporting in the pycurl github.
Tracked by https://github.com/pycurl/pycurl/issues/834
Thank you for an amazing issue writeup @jrdnbradford!! And thanl you @pastram-i for connecting the dots on what this is about! Nice work!!
Awesome, I somehow missed that filed issue over at https://github.com/pycurl/pycurl/issues/834.
Thanks for the assist!
I originally posted this issue at https://github.com/jupyterhub/the-littlest-jupyterhub/issues/964 but it may belong here instead.
Bug description
Installing TLJH and configuring the Google Authenticator results in
500 : Internal Server Error
after login.Bug resolution
I resolved this by comparing the hub environment on a working TLJH with the hub environment on a newly created TLJH and downgrading
pycurl
from7.45.3
to7.45.2
:I see requirements-hub-env.txt is setup with
pycurl>=7.45.2,<8
. Not sure if it should be pinned to7.45.2
in that repo or if it should be pinned here.How to reproduce
c.GoogleOAuthenticator.google_service_account_keys = {DOMAIN: GROUP_ADMIN_SECRET_PATH} c.GoogleOAuthenticator.gsuite_administrator = {DOMAIN: GSUITE_ADMINISTRATOR} c.GoogleOAuthenticator.allowed_google_groups = {DOMAIN: ALLOWED_GOOGLE_GROUPS} c.GoogleOAuthenticator.admin_google_groups = {DOMAIN: ADMIN_GOOGLE_GROUPS} c.GoogleOAuthenticator.client_id = OAUTH_CLIENT_ID c.GoogleOAuthenticator.client_secret = OAUTH_CLIENT_SECRET c.GoogleOAuthenticator.hosted_domain = DOMAIN c.GoogleOAuthenticator.login_service = DOMAIN_NAME c.GoogleOAuthenticator.oauth_callback_url = 'https://DNS_NAME/hub/oauth_callback'
source /opt/tljh/hub/bin/activate pip list Package Version
aiohttp 3.9.3 aiosignal 1.3.1 alembic 1.13.1 async-generator 1.10 async-timeout 4.0.3 attrs 23.2.0 backoff 2.2.1 bcrypt 4.1.2 cachetools 5.3.2 certifi 2024.2.2 certipy 0.1.3 cffi 1.16.0 charset-normalizer 3.3.2 cryptography 42.0.3 escapism 1.0.1 frozenlist 1.4.1 google-api-core 2.17.1 google-api-python-client 2.118.0 google-auth 2.28.0 google-auth-httplib2 0.2.0 google-auth-oauthlib 1.2.0 googleapis-common-protos 1.62.0 greenlet 3.0.3 httplib2 0.22.0 idna 3.6 Jinja2 3.1.3 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter-telemetry 0.1.0 jupyterhub 4.0.2 jupyterhub-firstuseauthenticator 1.0.0 jupyterhub-idle-culler 1.2.1 jupyterhub-ldapauthenticator 1.3.2 jupyterhub-nativeauthenticator 1.2.0 jupyterhub-systemdspawner 1.0.1 jupyterhub-tmpauthenticator 1.0.0 jupyterhub-traefik-proxy 1.1.0 ldap3 2.9.1 Mako 1.3.2 MarkupSafe 2.1.5 multidict 6.0.5 oauthenticator 16.2.1 oauthlib 3.2.2 onetimepass 1.0.1 packaging 23.2 pamela 1.1.0 passlib 1.7.4 pip 24.0 pluggy 1.4.0 prometheus_client 0.20.0 protobuf 4.25.3 pyasn1 0.5.1 pyasn1-modules 0.3.0 pycparser 2.21 pycurl 7.45.3 pyOpenSSL 24.0.0 pyparsing 3.1.1 python-dateutil 2.8.2 python-json-logger 2.0.7 referencing 0.33.0 requests 2.31.0 requests-oauthlib 1.3.1 rpds-py 0.18.0 rsa 4.9 ruamel.yaml 0.17.40 ruamel.yaml.clib 0.2.8 setuptools 59.6.0 six 1.16.0 SQLAlchemy 2.0.27 the-littlest-jupyterhub 1.0.0 toml 0.10.2 tornado 6.4 traitlets 5.14.1 typing_extensions 4.9.0 uritemplate 4.1.1 urllib3 2.2.1 yarl 1.9.4
Feb 19 03:02:55 $VM_NAME python3[9285]: [E 2024-02-19 03:02:55.548 JupyterHub oauth2:626] Error fetching 599 POST https://www.googleapis.com/oauth2/v4/token: HTTP 599: error setting certificate verify locations: Feb 19 03:02:55 $VM_NAME python3[9285]: CAfile: /etc/pki/tls/certs/ca-bundle.crt Feb 19 03:02:55 $VM_NAME python3[9285]: CApath: none ... ... Feb 19 03:02:55 $VM_NAME python3[9285]: Traceback (most recent call last): Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/tornado/web.py", line 1790, in _execute Feb 19 03:02:55 $VM_NAME python3[9285]: result = await result Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 208, in get Feb 19 03:02:55 $VM_NAME python3[9285]: user = await self.login_user() Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 828, in login_user Feb 19 03:02:55 $VM_NAME python3[9285]: authenticated = await self.authenticate(data) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/auth.py", line 492, in get_authenticated_user Feb 19 03:02:55 $VM_NAME python3[9285]: authenticated = await maybe_future(self.authenticate(handler, data)) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 969, in authenticate Feb 19 03:02:55 $VM_NAME python3[9285]: token_info = await self.get_token_info(handler, access_token_params) Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 843, in get_token_info Feb 19 03:02:55 $VM_NAME python3[9285]: token_info = await self.httpfetch( Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 661, in httpfetch Feb 19 03:02:55 $VM_NAME python3[9285]: return await self.fetch( Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 627, in fetch Feb 19 03:02:55 $VM_NAME python3[9285]: raise e Feb 19 03:02:55 $VM_NAME python3[9285]: File "/opt/tljh/hub/lib/python3.10/site-packages/oauthenticator/oauth2.py", line 606, in fetch Feb 19 03:02:55 $VM_NAME python3[9285]: resp = await self.http_client.fetch(req, **kwargs) Feb 19 03:02:55 $VM_NAME python3[9285]: tornado.curl_httpclient.CurlError: HTTP 599: error setting certificate verify locations: Feb 19 03:02:55 $VM_NAME python3[9285]: CAfile: /etc/pki/tls/certs/ca-bundle.crt Feb 19 03:02:55 $VM_NAME python3[9285]: CApath: none Feb 19 03:02:55 $VM_NAME python3[9285]:
... ... Feb 19 03:02:55 $VM_NAME python3[9285]: [E 2024-02-19 03:02:55.554 JupyterHub log:191] 500 GET /hub/oauth_callback?state=[secret]&code=[secret]&scope=email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=[secret]&hd=bc.edu&prompt=none (@$IP_ADDRESS) 15.03ms