search

opendatahub-io-contrib / jupyterhub-odh

Example JupyterHub deployment using OpenShift OAuth authenticator.
16 stars 31 forks source link

Cannot login when OAuth endpoint is using a customCA #75

Open X-dark opened 3 years ago

X-dark commented 3 years ago

Describe the bug I get a 500 error about certificate validation after authenticating on the oauth endpoint. I have tried to inject the CA bundle using [1] but the system CA seems to be ignored.

I suspect this is related to this: https://github.com/jupyterhub/oauthenticator/issues/411 but I would be happy to get feedback on my investigation.

[1] https://docs.openshift.com/container-platform/4.6/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki

vpavlin commented 3 years ago

Based on the linked issue it seems there might be 2 issues

  1. The self-signed certificates might be handled incorrectly by default tornado backend, which should be fixed by https://github.com/jupyterhub/oauthenticator/pull/410 when merged and updated in this repo
  2. IIUC your issue the problem is that we use the serviceaccount/ca.cert by default, which causes the injected custom CA to be ignored. Correct?
X-dark commented 3 years ago
  1. From what I understood, tornado is using CA bundle from certifi. I have tried to overwrite that bundle (/opt/app-root/lib/python3.6/site-packages/certifi/cacert.pem with no impact). I am not sure to understand the impact of the PR you linked on my topic but I will be happy to try when merged.
  2. Yes. That file contains the kube-apiserver and the ingress certificate (used by the oauth endpoint) but not the CA chain of the ingress certificate
wseaton commented 3 years ago

@X-dark For context, I added a conditional import to that PR that defaults to the pycurl powered tornado backend if available, which should bypass certifi entirely as under the hood it uses libcurl. At that point it should be much more straightforward to debug cert issues, and for me that fixed issues with the self-signed certs I was seeing (where they 'just worked' with standard curl). In regards to a custom PKI bundle I'm not sure what the impact would be for this change.

There's still some work that happens to get pycurl compiled for RHEL's UBI into the hub images, but I think @vpavlin and team are working on that.

X-dark commented 3 years ago

Yes I think this could be great. pycurl should use default system CA certificates and using the certificate injection mechanism it is easy to inject custom CA bundle at that level.

akchinSTC commented 3 years ago

Hi All, I just installed the latest operator (v1.0) on openshift (v4.5) on IBM Cloud with the latest tagged release of the odh-manifests (v1.0.3). On startup of jupyterhub, after accepting the permissions, Im greeted with the following: image instead of the spawner page, which I assume is the issue at large here, or hopefully related? At this time, is there a recommended workaround for this problem? Thanks in advance!

nobusue commented 3 years ago

I got into same situation with @akchinSTC OpenShift 4.7.2 on AWS ODH operator 1.0.0 odh-manifests v1.0.0

X-dark commented 3 years ago

@wseaton so I see your PR has been merged 3 months ago but the latest odh-jupyterhub still uses a commit from Feb 10 of oauthenticator. Is there anything specific blocking this?

wseaton commented 3 years ago

@X-dark are you still seeing this issue in the latest Z-stream version of odh-manifests? odh-jupyterhub was configured to vendorize the fix in a fork, so the issue should be resolved AFAIK.

(cc @vpavlin)

X-dark commented 3 years ago

I am using 0.2.8 image of odh-jupyterhub from quay with manifests from latest HEAD from https://github.com/opendatahub-io/odh-manifests. So I guess I should be up to date. I am still seeing HTTP 599: Peer's Certificate issuer is not recognized from tornado.curl_httpclient.CurlError with CA injected into /etc/pki/ca-trust/extracted/pem.

lucianolacurcia commented 2 years ago

I am having the same problem, any workaround? I am using image 0.3.5, getting: " tornado.curl_httpclient.CurlError: HTTP 599: Peer's Certificate issuer is not recognized."

shalberd commented 1 year ago

https://github.com/opendatahub-io/jupyterhub-odh/issues/137 might offer some hints. There, the same type of configmap injection mechanism was used by me, then the configmap data section referenced in a volume and added to a spawned container volume mount programmatically. Unsure how this works here with Tornado, but still, might be worth a look in general.