search

dask / dask-gateway

A multi-tenant server for securely deploying and managing Dask clusters.
https://gateway.dask.org/
BSD 3-Clause "New" or "Revised" License
137 stars 88 forks source link

Mechanism to mount existing Kubernetes Secret for sensitive auth, e.g. `gateway.auth.kerberos.keytab`? #417

Open brews opened 3 years ago

brews commented 3 years ago

Hey y'all!

I was scoping the possibility of using dask-gateway (via Kubernetes and the daskhub chart) with Google-manged Notebooks on GCP's AI Platform (They're Jupyter notebooks). My idea was to deploy dask-gateway to a GKE cluster on the same network/VPC, with dask-gateway configured for Kerebos authentication.

I noticed that gateway.auth.kerberos.keytab takes a path to an existing keytab on the machine. I would have thought that we'd want to mount an existing Kubernetes Secret to a path with the desired permissions...? I was wondering if this is a planned feature or maybe I'm simply going about this the wrong way...?

In general, it looks like a lot of sensitive material is passed in raw through the helm configs, rather than through Kubernetes Secrets?

Thanks so much for your time and consideration! I really appreciate the work on this project.

jacobtomlinson commented 3 years ago

I don't think you want the daskhub chart for the use case you describe. Dask Hub is a meta chart which deploys Jupyter Hub and Dask gateway together. If you are using GCP AI Platform they are already providing Jupyter for you.

I'm going to move this issue over to the dask-gateway repo for further discussion about using secrets.

martindurant commented 3 years ago

I am sure you are right, and it would make sense to (optionally) embed sensitive information inside secrets rather than supplying directly. Here is the only reference I can see to something like that, I'm not sure if it's sufficient for your needs.

brews commented 3 years ago

I hadn't considered this route! Let me run with this and see what I can do.

It would be great if we could work this out. This would be helpful for submitting work from external notebooks but also orchestration jobs like Argo Workflows, etc. Thanks, Martin.

yuvipanda commented 3 years ago

/cc @consideRatio, who moved a lot of zero to jupyterhub's config from configmap to secrets.

consideRatio commented 3 years ago

About k8s ConfigMap vs k8s Secret

In the JupyterHub Helm chart we always use k8s Secrets to store sensitive information passed via Helm values (--values <my-helm-values-config.yaml>). Looking at the dask-gateway templates I note that only what is explicitly known to be secret configuration is stored in k8s Secrets rather than in the k8s ConfigMap. Something could be missed like this kind of setup.

About augmenting config from k8s Secrets

In the JupyterHub Helm chart we configure the hub pod that runs JupyterHub with various sensitive information, such as a database password. We do this via k8s Secret that the Helm chart creates. But, if you have secret information in your Helm config (passed via --values flag), then you must treat it as secret etc. It is common to want to create k8s Secret resources and then reference such existing k8s Secret resources with the sensitive parts.

In the JupyterHub Helm chart we have the Helm config called hub.existingSecret for this purpose.

To implement the equivalent of hub.existingSecret, I think it will be relevant to also coalesce details split apart into a k8s ConfigMap and a k8s Secret into one bigger k8s Secret, something that will also reduce the maintenance complexity in general I think.

Seeding secrets - a helm3 possibility

A feature in Helm 3 that can be used as a function in Helm templates is the lookup function. It allows us to lookup the value of a k8s Resource that is managed by the Helm chart already and read a value from it. This value can then be rendered by the Helm template. Like this, it becomes possible to store secret information by an initial helm install and then omit it from later helm upgrade commands for example. It also becomes possible to generate a random value and re-use it later.

Next steps?

I'm in general interested in helping out with the maintenance of this helm chart and could put in work towards the end user goal of not needing to keep passing secret information via --values when using helm upgrade.

From experience, I know that this kind of feature is quite a bit of work, so I'd like a clear signal that it would be wanted and perhaps even someone that would be willing to contribute with review efforts ahead of time so it doesn't become a large stale PR.

jacobtomlinson commented 3 years ago

I think this would be a good thing to implement and I'd be happy to take on the reviewing duties.