Attempt to use Istio to secure connections between components of jupyterhub

[summerswallow-whi]

I opened this issue as a place to discuss an attempt to implement istio's side car injection to supply encryption in transit for jupyterhub on a kubernetes cluster.

So far I've followed Istio's quick start at https://istio.io/docs/setup/kubernetes/quick-start.html#aws-wkops for Kubernetes with KOps and automatic sidecar injection at https://istio.io/docs/setup/kubernetes/sidecar-injection.html. However, when I do this a helm install of jupyter hub fails.

I realize this is uncharted territory and I will continue to update this as I learn more.

[consideRatio]

@summerswallow-whi excellent! I'm learning about Envoy and Istio as well. Very new to this.

Related from KubeCon 2018 Europe - Envoy, Istio...

• [x] Envoy Project Intro
• [x] Envoy Deep Dive
  • Consul, keeps track of all discovered services?
  • Gimbal, a heptio project to function as a multi-cluster ingress load balancer
  • Contour, ingress controller based on envoy
  • SPIRE, a SPIFFE runtime environment... TLS for envoy communication?
• [x] Envoy Internals Deep Dive
  • TLS Thread Local Storage
  • RCU Read Copy Update
  • xDS Envoy discovers its various dynamic resources via the filesystem or by querying one or more management servers.
  • C10k - Supports massive amounts of connections by handling multiple connections per thread (one worker thread per core) with an event loop.
  • Hot Restart of Envoy supported (shared memory region, scale up Envoy to 2 -> scale down to 1)
• [ ] From Kubelet to Istio: Kubernetes Network Security Demystified
• [ ] Introduction to Istio Configuration
• [ ] Extending Istio Service Mesh w/Envoy v2 APIs
• [ ] Building Hybrid Clouds with Istio
• [ ] GitOps for Istio - Manage Istio Config like Code
• [ ] Replacing NGINX with Envoy in a Traffic Control System
• [ ] Canary Deploys with Kubernetes, Istio and Envoy
• [ ] Accelerating Envoy with the Linux Kernel
• [ ] Practical and Useful Latency Analysis using Istio and OpenCenus
• [ ] Observing and Troubleshooting your Microservices with Istio
• [ ] Istio Tells me my Service has Slow Response Time, Now What?
• [ ] Istio - The Weather Company's Journey
• [ ] From Kubelet to Istio: Kubernetes Network Security Demystified
• [ ] Istio Multi-Cluster Mesh Expansion BOF

[summerswallow-whi]

Have you made any progress getting it work with jupyterhub.

Right now I am finding I'm injecting the sidecars but nothing can communicate with each other, but I have tagged the entire kubernetes jupyterhub namespace for automatic sidecar injection. I don't think automatic injection will work as it is stated that pods should only have one service attached and I believe the proxy pod has two. I'm going to start looking at manual injection. Though I think automatic injection needs to be in place for notebooks as hub spawns it and it would be ideal if we don't have to mess with the spawner.

[consideRatio]

@summerswallow-whi no I haven't tried setting it up, but I mean to in a month or so.

[summerswallow-whi]

For the moment I've gone the direction of trying to encrypt through weave see issue https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/699

[aleicher]

@summerswallow @consideRatio any update on this? Is there a recommended way to get jupyterhub with the spawner up and running, when using istio? I disabled istio sidecar injection to make the jobs succeed, but hub and proxy fail to come up properly with istio sidecars. disabling the istio injection for those (sidecar.istio.io/inject: "false") allows to start them properly, but then I cannot expose jupyterhub using the istio gateway. Would love to hear your results/approach.

[consideRatio]

@aleicher I didn't focus effort into getting istio to setup mTLS between pods etc or routing traffic with istio-gateway. I have little insights to provide about this at the moment =/

I'm curious what stopped the hub / proxy pods to become ready though, if you open a thread on discourse.jupyter.org in the z2jh category where you describe what istio-usage end goal you have, and where you get stuck, i'd be happy to try help you navigate past the issues you run into

[harsimranmaan]

Open-sourced https://github.com/splunk/jupyterhub-istio-proxy to manage traffic routing via istio-gateway. More to come with examples, documentation and setup guide

[shenghu]

@aleicher have you been able to setup Istio with JupyterHub?

[shenghu]

@harsimranmaan can jupyterhub-istio-proxy be used to enforce ssl communication between JupyterHub's services?

[harsimranmaan]

@shenghu To use mTLS with JupyterHub services, you'd need to enable istio sidecar injection.I've had to patch kubespawner to get the mTLS going. Once you do that, the default configurable-http-proxy may not work for you. jupyterhub-istio-proxy can be used instead if you are using the istio service mesh in your cluster. In my setup I have the sidecar injection enabled by default. I can try to help you with your setup if you'd like and send any required patches upstream.

[shenghu]

@harsimranmaan have you run istio w/ jupyterhub in production env? Is there issue, e.g. reliability or performance? It would be appreciated that you can send me how to replace CHP w/ jupyterhub-istio-proxy and the needed patches. Thanks!

Right now I simply enable internal_ssl for jupyterhub and hub is not able to be detected as ready. Not sure if it is caused by CHP. I see these error

[W 2020-08-28 16:00:36.387 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 40964): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)
[W 2020-08-28 16:00:46.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 40978): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)
[W 2020-08-28 16:00:56.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 40992): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)
[D 2020-08-28 16:01:00.934 JupyterHub proxy:766] Proxy: Fetching GET http://10.111.56.85:8001/api/routes
[I 2020-08-28 16:01:00.938 JupyterHub proxy:320] Checking routes
[W 2020-08-28 16:01:06.380 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41008): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)
[W 2020-08-28 16:01:16.379 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41022): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)
[W 2020-08-28 16:01:26.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41036): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)
[W 2020-08-28 16:01:36.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41050): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)
[W 2020-08-28 16:01:46.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41064): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)

[shenghu]

I see this is still WIP, https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1520. But we really need enforce ssl within JupyterHub services. Could istio+jupyterhub-istio-proxy be a reasonable solution?

[harsimranmaan]

@shenghu I'll prepare the patches for upstream this week. Thanks for your patience.

[harsimranmaan]

Added some background to running JupyterHub with istio https://medium.com/@harsimran.maan/running-jupyterhub-with-istio-service-mesh-on-kubernetes-a-troubleshooting-journey-707039f36a7b

[consideRatio]

I'm closing this issue now as it does not provide a concrete action point to take. Thank you so much @harsimranmaan for sharing your work on this!

My summary of the Istio state is that z2jh/kubespawner doesn't support this yet, but @harsimranmaan has demonstrated a set of changes and tooling can make it work!

[zachwe]

An alternative which requires less configuration than @harsimranmaan's proposal is to run configurable-http-proxy with --change-origin (documented here). This works out of the box with Istio. @consideRatio would you approve a PR that enables optionally running the proxy with this setting?

[consideRatio]

@zachwe I don't fully understand the change you suggest, could you open an issue where you describe the change in a way that doesn't require me to know so much more than Istio want to add sidecards where the network traffic become routed through?

Not understanding the change better, I cannot evaluate if we can implement it without breaking changes so well, or how much knowledge is required to maintain the feature in the future etc.

[reganmcdonald]

An alternative which requires less configuration than @harsimranmaan's proposal is to run configurable-http-proxy with --change-origin (documented here). This works out of the box with Istio. @consideRatio would you approve a PR that enables optionally running the proxy with this setting?

@zachwe how did you manage to disable hub CORS?

[frobones]

Adding this single headless service to the cluster allowed me to work with istio out of the gate:

apiVersion: v1
kind: Service
metadata:
  name: single-user
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: jupyterhub
  ports:
    - port: 8888

[mallikkml]

Hi, I am trying to deploy jupyterhub on our K8s cluster in a istio enabled namespace using helm chart but seeing following failure:

helm upgrade --cleanup-on-fail --install my-jupyter jupyterhub/jupyterhub --namespace jhub --values values.yaml Release "my-jupyter" does not exist. Installing it now. Error: failed pre-install: 1 error occurred:

• timed out waiting for the condition

kubectl --namespace=jhub get pod NAME READY STATUS RESTARTS AGE hook-image-awaiter-z6v5p 1/2 NotReady 0 6m31s hook-image-puller-nwhf2 2/2 Running 0 6m32s

Deployment working without any issue if we deploy in a non-istio namespace.

@frobones, can you please provide steps using which you are able to deploy?