squat / jupyter-operator

A Kubernetes operator for Jupyter Notebooks
Apache License 2.0
5 stars 1 forks source link

Having trouble accessing the notebook #1

Open ScottBerman opened 5 years ago

ScottBerman commented 5 years ago

Which kubernetes IP address am I supposed to resolve as example-notebook.example.com? My notebook (custom resource) doesn't get assigned an IP address, so I tried the IP of the operator but that got me no where.

squat commented 5 years ago

For every notebook resource you create, the operator will create a pod, a service pointing to the pod, and an ingress pointing to the service. The ingress resource will use the host you specified in the custom resource. From there on, this setup works just like any other ingress resource.

To access your notebook via the specified host, you must ensure the host resolves to an IP address for your cluster that can serve ingress, e.g. a public IP for a worker node. Note that this setup depends on an ingress controller to configure ingress.

The ingress controller accepts all HTTP requests to the cluster and provides them to the correct layer 7 service based on host.

ScottBerman commented 5 years ago

hmm maybe I messed something up? Because the operator didn't deploy a service or an ingress along with my notebook... I actually didn't even see the custom resource definition for the notebook in your repository.

Minor side note, your README might be out of date, most of your yaml definitions are in manifests/ not examples/

Otherwise this is a really cool project. I'll play around with it this weekend and see if I can get it working. Thanks for responding Lucas!

squat commented 5 years ago

Hi Scott, That’s right, the manifests directory contains everything you need to install the operator. The operator itself will create the custom resource definition. Once that is all in place, you must create a notebook custom resource, like the one in the examples directory.

So:

ScottBerman commented 5 years ago

Okay I follow you. So I got the operator up and running and tried to deploy the notebook but nothing came up (no service, no ingress, no pod, no notebook). Not sure how I should try to debug, the logs from the operator aren't terribly helpful:


time="2019-06-10T13:55:30Z" level=info msg="initializing CRD" pkg=controller 
time="2019-06-10T13:55:31Z" level=info msg="CRD already exists" pkg=controller 
time="2019-06-10T13:55:31Z" level=info msg="CRD is ready" pkg=controller 
time="2019-06-10T13:55:32Z" level=info msg="successfully synced all caches" pkg=controller ```
squat commented 5 years ago

Hmm I see, I think this is an issue with the examples and/or documentation. The deployment manifest of the operator only watches the jupyter namespace for notebook resources [0] (and in fact it is only privileged for that namespace [1]), while the example creates a notebook in the default namespace [2]. There are no interesting logs because the operator has does see any events in the namespace it is watching.

Can you please try deploying the example notebook instead to the jupyter namespace? I can update the example to make this clearer from the get-go.

[0] https://github.com/squat/jupyter-operator/blob/master/manifests/deployment.yaml#L22 [1] https://github.com/squat/jupyter-operator/blob/master/manifests/role.yaml#L5 [2] https://github.com/squat/jupyter-operator/blob/master/examples/notebook.yaml#L5

ScottBerman commented 5 years ago

Yeah sure, I actually thought it might have to do with the namespace, I should have noticed that in notebook.yaml myself! While you're doing that may I just add the suggestion to include owner: value in your example notebook.yaml under spec. It seems to be a required key that is expected. Let me try deploying in the Jupyter namespace.

ScottBerman commented 5 years ago

Hey so I got everything working, the notebook pod is up, and there's a connected service as well as an ingress. I haven't worked with Ingresses before so I was wondering if you'd be able to explain how I connect to my notebook, which I'm still unsure of how to do. I included host: example.com in my notebook.yaml, my service has cluster IP:10.104.35.182 and it's internal endpoint is listed as jupyter-notebook-example-notebook.jupyter:8888 TCP.

One last thing to note is that I am running this on docker for desktop so I don't know if that changes anything in terms of connecting to the cluster (when developing locally I have been using NodePort).

ScottBerman commented 5 years ago

Okay sorry to keep bothering you. I'm pretty sure the issue is with using ingress locally. So I tried to use port-forwarding to forward the pod endpoint directly to local host with kubectl port-forward jupyter-notebook-example-notebook-0 8888:8888. That seemed to allow a connection as at least it was showing up in the logs, but I still wasn't able to connect. This is what was logged: [W 19:38:09.992 NotebookApp] SSL Error on 9 ('127.0.0.1', 44010): [SSL: HTTP_REQUEST] http request (_ssl.c:1056)