[openshift] oauthentication error after upgrading to k8s-hub:3.1.0

[do-it-tim]

Bug description

After upgrading from k8s-hub:2.0.0 to 3.1.0, we are facing an error with oauthenticator (provider is openshift). The same config works fine with prior versions.

How to reproduce

1. Start Jupyterhub
2. Click on 'Sign in with OpenShift'

Expected behaviour

Redirect to OpenShift Login

Actual behaviour

We encounter a "500 : Internal Server Error"

Your personal set up

• OS:
  • Kubernetes - Openshift

• Version(s):

  • Kubernetes: v1.25.11
  • jupyterhub: 4.0.2
  • Python 3.11.5

Full environment

``` aiohttp==3.8.5 aiosignal==1.3.1 alembic==1.12.0 async-generator==1.10 async-timeout==4.0.3 attrs==23.1.0 bcrypt==4.0.1 certifi==2023.7.22 certipy==0.1.3 cffi==1.15.1 charset-normalizer==3.2.0 cryptography==41.0.4 escapism==1.0.1 frozenlist==1.4.0 greenlet==2.0.2 idna==3.4 Jinja2==3.1.2 jsonschema==4.19.1 jsonschema-specifications==2023.7.1 jupyter-telemetry==0.1.0 jupyterhub==4.0.2 jupyterhub-firstuseauthenticator==1.0.0 jupyterhub-hmacauthenticator==1.0 jupyterhub-idle-culler==1.2.1 jupyterhub-kubespawner==6.1.0 jupyterhub-ldapauthenticator==1.3.2 jupyterhub-ltiauthenticator==1.6.1 jupyterhub-nativeauthenticator==1.2.0 jupyterhub-tmpauthenticator==1.0.0 kubernetes-asyncio==26.9.0 ldap3==2.9.1 Mako==1.2.4 MarkupSafe==2.1.3 multidict==6.0.4 mwoauth==0.3.8 nullauthenticator==1.0.0 oauthenticator==16.1.0 oauthlib==3.2.2 onetimepass==1.0.1 packaging==23.1 pamela==1.1.0 prometheus-client==0.17.1 psycopg2==2.9.8 py-spy==0.3.14 pyasn1==0.5.0 pycparser==2.21 pycurl==7.45.2 PyJWT==2.8.0 PyMySQL==1.1.0 pyOpenSSL==23.2.0 python-dateutil==2.8.2 python-json-logger==2.0.7 python-slugify==8.0.1 PyYAML==6.0.1 referencing==0.30.2 requests==2.31.0 requests-oauthlib==1.3.1 rpds-py==0.10.3 ruamel.yaml==0.17.33 ruamel.yaml.clib==0.2.7 six==1.16.0 SQLAlchemy==2.0.21 sqlalchemy-cockroachdb==2.0.1 statsd==4.0.1 text-unidecode==1.3 tornado==6.3.3 traitlets==5.10.1 typing_extensions==4.8.0 urllib3==2.0.5 yarl==1.9.2 ```

Configuration

``` App: JupyterHub, Cfg: {'admin_access': True, 'authenticator_class': 'openshift'} App: OpenShiftOAuthenticator, Cfg: { 'admin_groups': ['scm000000043773_scm000000302611_sd_jupyterhub_dev_admin'], 'allowed_groups': ['scm000000043773_scm000000302611_sd_jupyterhub_dev_allowed'], 'client_id': 'system:serviceaccount:smartdata-ng-etu:jupyterhub-oauth', 'oauth_callback_url': 'https://jupyterhub-smartdata-ng-etu.apps.xxxxxxx..xxxxx.de/hub/oauth_callback', 'scope': ['user:full']} ```

Logs

``` [E 2023-10-12 14:21:10.789 JupyterHub web:1871] Uncaught exception GET /hub/oauth_login?next=%2Fhub%2F (::ffff:100.64.3.1) HTTPServerRequest(protocol='http', host='jupyterhub-smartdata-ng-etu.apps.xxxx.xx.xxxxxx.de', method='GET', uri='/hub/oauth_login?next=%2Fhub%2F', version='HTTP/1.1', remote_ip='::ffff:100.64.3.1') Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/tornado/web.py", line 1784, in _execute result = method(*self.path_args, **self.path_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 108, in get self.authorize_redirect( File "/usr/local/lib/python3.11/site-packages/tornado/auth.py", line 585, in authorize_redirect url = self._OAUTH_AUTHORIZE_URL # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 63, in _OAUTH_AUTHORIZE_URL return self.authenticator.authorize_url ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/traitlets/traitlets.py", line 716, in __get__ return t.cast(G, self.get(obj, cls)) # the G should encode the Optional ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/traitlets/traitlets.py", line 626, in get default = obj.trait_defaults(self.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/traitlets/traitlets.py", line 1902, in trait_defaults return self._get_trait_default_generator(names[0])(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/traitlets/traitlets.py", line 1255, in __call__ return self.func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/openshift.py", line 96, in _authorize_url_default return f"{self.openshift_auth_api_url}/oauth/authorize" ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/traitlets/traitlets.py", line 716, in __get__ return t.cast(G, self.get(obj, cls)) # the G should encode the Optional ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/traitlets/traitlets.py", line 626, in get default = obj.trait_defaults(self.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/traitlets/traitlets.py", line 1902, in trait_defaults return self._get_trait_default_generator(names[0])(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/traitlets/traitlets.py", line 1255, in __call__ return self.func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/openshift.py", line 87, in _openshift_auth_api_url_default client = HTTPClient() ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/tornado/httpclient.py", line 109, in __init__ self._async_client = self._io_loop.run_sync(make_client) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/tornado/ioloop.py", line 521, in run_sync self.start() File "/usr/local/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 195, in start self.asyncio_loop.run_forever() File "/usr/local/lib/python3.11/asyncio/base_events.py", line 596, in run_forever self._check_running() File "/usr/local/lib/python3.11/asyncio/base_events.py", line 590, in _check_running raise RuntimeError( RuntimeError: Cannot run the event loop while another loop is running ```

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

[manics]

requests.get was replaced by tornado.httpclient.HTTPClient in https://github.com/jupyterhub/oauthenticator/commit/b0367d72bce73cf3d35627aa8b8b862dfe7524d8

The code is used to set the default value of openshift_auth_api_url: https://github.com/jupyterhub/oauthenticator/blob/2fa1c4818f7ba3444f3e2e44f8ddb38127553ffd/oauthenticator/openshift.py#L80-L92 so until the bug is fixed a workaround should be to configure c.OpenShiftOAuthenticator.openshift_auth_api_url

[do-it-tim]

Thanks for the quick response and fix. Still struggling to find the correct auth_api_url - I encounter a 403 from OpenShift. Will post a solution once I found one.

forbidden: User "system:anonymous" cannot get path "/oauth/authorize"

[consideRatio]

Visit {self.openshift_url}/.well-known/oauth-authorization-server and look for the value of issuer. openshift_url's default value is https://openshift.default.svc.cluster.local unless you have configured it yourself.

you could perhaps access https://openshift.default.svc.cluster.local/.well-known/oauth-authorization-server from a pod inside the k8s cluster. So if you for example do kubectl exec -it deploy/hub -- /bin/bash or something like that, and then you can use curl https://openshift.default.svc.cluster.local/.well-known/oauth-authorization-server and look for the value of issuer in some JSON blob of data in the response.

[do-it-tim]

Thats actually what I did with no success. The auth_api_url was fine.

For me it looks like the client_id (configured with our serviceAccount) is not passed or read correctly

[do-it-tim]

Found the error. It was a misconfiguration of the client_id in my configuration that happend during debugging. The login is working now. Thanks a lot for your help!

[consideRatio]

@do-it-tim did you need to pass client_id etc to make the request to https://openshift.default.svc.cluster.local/.well-known/oauth-authorization-server?

Also since I don't have an openshift environment, can you examplify the format of the value of issuer as returned by the URL?

[do-it-tim]

The value of issuer was: { "issuer": "https://oauth-openshift.apps.xxx.xxxxxxx.de", ... ..... }