Open Adam-D-Lewis opened 1 month ago
viniciusdc
commented
4 weeks ago I do like this approach, and I would prefer we opted it as default behavior. We already have control over the services endpoint injection in each helm chart, as well as, the infra resources itself. So, we could have the existing deployment option default to single namespaces.
viniciusdc
commented
4 weeks ago Services accounts such as the jupyterhub one would possibly require access to the user's namespaces, so that would need to use some level of tagging system or similar if that exists.
Adam-D-Lewis
commented
4 weeks ago Copying over some internal discussion here.
Dharhas Pothina The earlier issue about doing that was the idea that some orgs wouldn't be able to do a wildcard dns entry. so it would need be optional.
Vinicius Cerutti I think the main goal is to separate the resource at the kubernetes level to separate namespaces, the endpoints entries should, in theory, be kept the same. (edited) my single concern is with deployments on existing infra, where the number of namespaces and their access might be limited. (edited)
Adam Lewis Yes, that's what I'm proposing Vinicius Cerutti. My understanding is the single namespace design originally was intended for orgs wanting to do a deployment into an existing k8s cluster. At least they could have some peace of mind that everything is going into a single namespace. However, it doesn't seem like that is very common at all. Do we know of anyone doing this currently? It seems like not the best to ignore best practices for a scenario which isn't really happening.
Adam Lewis We could continue to support single namespace deployments, but just make the default multi namespace. (edited)
Adam Lewis Most of the CRDs, cluster roles are for services like Prometheus, Argo, etc. I think making it easier for orgs doing existing k8s Nebari deployments to use their own e.g. Prometheus deployment, Argo Deployment, etc. with Nebari if they have one would be a better way. (edited)
Chuck McAndrew I am in favor, but we should expose ways to set the namespaces for each service so that if an org wants everything in one, we can do that
Adam Lewis Yeah, I think keeping namespace configurable is totally reasonable.
krassowski
commented
3 weeks ago On meeting today there was also a mention of whether this has an interaction with:
c.KubeSpawner.enable_user_namespaces https://discourse.jupyter.org/t/how-to-create-their-own-namespace-for-each-user/8039
Start moving services into their own Kubernetes Namespaces
Summary
Services integrated into Nebari often suggest putting the service in it's own namespace when used with Kubernetes (e.g. Rook Ceph, Argo Workflows). Jupyterhub also has the option to put users in their own namespace. So far we've kept everything in a single namespace. The idea being that if someone wants to deploy Nebari into an existing Kubernetes cluster they can be assured that by default everything is deployed in a single namespace.
Benefit
Putting services and potentially users in their own namespaces brings some advantages around controlling resource limits on users. It also allows us to follow the best practices recommended by the services themselves.
It can also help with some aspects of RBAC permissions. E.g. All pods in a namespace have all the the permissions granted to the default service account (assuming automountServiceToken: true). This is a smaller benefit in my mind. We currently don't use the default service account.
Design Proposal
I propose we move Argo Workflows, Keycloak, Rook-Ceph, Monitoring (Prometheus/Grafana), potentially Jupyterhub each into their own namespaces. We should ensure the namespace is configurable so users could put everything in a single namespace if they wish, but by default we separate these things out.
Alternatives or approaches considered (if any)
Best practices
Services recommend being put in their own namespace. It also gives better resource controls on various services and potentially Nebari users.
User impact
Ideally none, there may be some special instructions/manual steps required around upgrading Nebari b/c of these changes.
Unresolved questions