search

jupyterhub / zero-to-jupyterhub-k8s

Helm Chart & Documentation for deploying JupyterHub on Kubernetes
https://zero-to-jupyterhub.readthedocs.io
Other
1.56k stars 801 forks source link

Supporting launching companion Pods #830

Closed craig-willis closed 6 years ago

craig-willis commented 6 years ago

We're starting work on a new project for cybersecurity education building on the TryCybSI environment. The basic premise is that students are able to spin up a multi-pod sandbox environment to be able to walk through various security concepts. The primary environment will be Jupyter, but we need to be able to launch companion pods with each notebook environment. For example, in the ArpSpoof case we'd need the "hacker" environment (Jupyter) alongside "victim" (client, maybe Jupyter) and "server" (webserver) environments. These would need to be separate Pods with a common network/security policy, since it wouldn't be interesting to do this on locahost (i.e., a single pod). I've seen the hub.extraContainers configuration and mention of potentially allowing users to "do additional things in their namespaces (like spawning other clusters). Is this type of scenario Z2JH can support (or would be interested in supporting, if we were to work on a way)?

consideRatio commented 6 years ago

@craig-willis launching extra containers running alongside the user pods is something very realistic, but launching a full extra pod for each spawned user pod is a different matter that would be a issue for the kubespawner project. I think it is too complicated to do this, as it would force us to make very clear decisions on what to do with the pods if one fails and the other doesn't, if they should schedule on the same node or not, if one fails startup and the other doesn't, configuration for both separately etc.

But, for the extraContainers to run in the same pod as the user pod, we now have @chaoleili in #910 who is implementing the support for setting extraContainers in the config.yaml we provide helm. It would then match the support we already have (#294) for doing it for the hub pod.

craig-willis commented 6 years ago

Thanks for your response, @consideRatio. As we've explored this further, several of the scenarios could actually be handled through the user pod extraContainers, so that's a very helpful PR. Still, we do have cases where we need multiple pods and at least one where they need to schedule on the same node.

We have a system that can do some of this and were exploring whether to put more effort into it or try and add the capability to the Z2JH ecosystem. Given your reservations and that there doesn't seem to be much of a demand, we can move forward with it for now.

consideRatio commented 6 years ago

@craig-willis you are welcome!

I imagine the modern way to implement having a companion pod, would be to have some operator inspecting whats going on in a given namespace and ensuring there is always a companion pod spinning up / down based on what it finds. Not at all knowledgeable about the operator pattern though. But I figure it would make sense for it to happily side by side with a z2jh deployment without needing to live in the same repo as either kubespawner or z2jh.

I have not implemented the operator pattern yet, but by a quick google I found some resources to learn more about it, such as:

I think this would be the best path forward if you need something more than extraContainers running in the user pod.

On the other hand, I recalled and found this PR: https://github.com/jupyterhub/kubespawner/pull/115 - I don't know if I'd recommend this approach because it would be a quite unstable one where resources may be created by they are not cleaned up, and if we start adding cleanup logic for these custom companion resources along with the spawned user pods I imagine things go out of control.

On the other hand, I recalled and found this PR: https://github.com/jupyterhub/kubespawner/pull/115