search

chrisjsewell / aiida-docker-compose

Docker service infrastructure for running AiiDA
MIT License
0 stars 0 forks source link

Feedback #1

Open chrisjsewell opened 4 years ago

chrisjsewell commented 4 years ago

Hey @giovannipizzi, @sphuber just pinging you this before our meeting. If you look at the README it should be fairly self-explanatory.

Also cc'ing @ltalirz, might be interested

chrisjsewell commented 4 years ago

cc'ing @yakutovicha as well, since it may or may not be of interest for aiidalab

ltalirz commented 4 years ago

Thanks @chrisjsewell - the multi-container solution would be the obvious way to go, were it not for the fact that we currently cannot use it in the jupyterhub context. At this point in time, it means we need to duplicate our maintenance efforts, since we anyhow need to maintain a single-container version for the AiiDA lab.

Extending JupyterHUb from single container per user to docker-compose (or finding some other way of reusing this approach in the jupyterhub; perhaps with centrally managed postgresql & rabbitmq services) would be very welcome.

chrisjsewell commented 4 years ago

Extending JupyterHUb from single container per user to docker-compose (or finding some other way of reusing this approach in the jupyterhub; perhaps with centrally managed postgresql & rabbitmq services) would be very welcome.

I know there are ways to transpose docker compose systems into Kubernetes, but yeh I don't think its trivial, and I don't have really any experience (yet) with Kubernetes.

@choldgraf, as a contributor to jupyterhub and particularly zero-to-jupyterhub-k8s I wonder if you have any inside scoop on this kind of use case? (see the README on this repo for what I/we are trying to achieve)

choldgraf commented 4 years ago

In the Kubernetes world, people generally use a tool called Helm to define / orchestrate multi-container applications. They are basically just YAML files that are used by Helm to tell a Kubernetes cluster what to do.

For example, here's the BinderHub helm chart: https://github.com/jupyterhub/binderhub/tree/master/helm-chart and here is the JupyterHub helm chart: https://github.com/jupyterhub/helm-chart

I'm not sure if this is exactly what you're looking for, but Helm etc is a good tool to learn about if you're interested in kubernetes in general. Does that help?

ltalirz commented 4 years ago

@choldgraf Thanks for the note!

My question is a bit more specific to JupyterHub (be it using docker or kubernetes, we have been using both): what if a JupyterHub user needs not just one service (=a jupyter notebook server) but multiple services (e.g. a database, message broker, ...)? So far, our approach has been to bundle those in the same container but this obviously goes against the microservice concept. As far as I am aware, both the docker- and kubernetes-based versions of JupyterHub only allow to specify a singleuser image, not a singleuser docker-compose file / helm chart (in later versions you can let the user select between different images, but there is still a one-to-one correspondence between a selection and a docker image).

I see essentially three possible ways forward here

  1. continue as we do, bundling all services in one container
  2. look into adding support for something like docker-compose/helm charts/... per JupyterHub user
  3. keep the single-container approach but somehow give each user access to centrally managed DB / message broker / ... services

Is support for something like this on the roadmap or do you think this is generally out of the scope of JupyterHub?

choldgraf commented 4 years ago

Ah yeah - that's a common challenge that folks have. I suspect that your second item would be fairly complex to add within Z2JH, though I have seen several folks basically do number 3. We run JupyterHubs at Berkeley for a ton of students, and I think this is generally what we do (run centralized services as a part of the kubernetes deployment, and then students can access those services and potentially authenticate themselves w/ their user identity). I'd recommend hanging around in the JupyterHub discourse (https://discourse.jupyter.org) as well as the z2jh repository to connect with others that have experience in this space. We've also got a monthly community meeting that can be a helpful place to connect with others who are working on JupyterHub (https://jupyterhub-team-compass.readthedocs.io/en/latest/meetings.html)