Discussion on limiting usage by individual users

[consideRatio]

Why open an issue here

While it may be out of scope for dask-gateway to support limiting resource usage for users in a k8s environment, I figured I'd write down an idea I got to solve an issue for dask-gateway users wanting to mitigate the risk of ending up with crazy cloud bills caused by mistakes or mischievious usage.

Old idea - ResourceQuota (+ k8s native validation admission webhook)

A dask-gateway Helm chart deployment currently can't limit the amount of cloud resources that the individual user consumes, or can it? Assuming it can't, it has been considered if k8s native ResourceQuota resources can help do this, but they can't and are only intended to collectively limit a namespace rather than separate users in a namespace.

New(?) idea - OurCustomResourceQuota (+ our developed validation admission webhook)

What if we could create a new dedicated open source project that aim to help us mitigate wild usage, while also being generally useful in k8s for other similar projects with pipelines etc created by users? The idea is to create an validating admission controller that would inspect if a pod spec should be approved to be created based on custom logic considering who wish to do so based on pod labels. I imagine we could mimic what the ResourceQuota admission controller does together with the ResourceQuota k8s resources, but let it consider labels in a suitable way and with some new custom k8s resource we define.

If we did this, we could let this resource also monitor the usage by users, which prometheus could collect, and grafana could present for administrators. As a bonus, following this, it wouldn't be too much more effort to let users of a JupyterHub or dask-gateway become more aware of their usage by providing some insights one way or another. Perhaps through a JupyterLab extension, or by injecting information to the page where before a user spawns a server.

Implementation ideas

• A Helm chart is defined to contain the k8s resources we need
• We need a pod acting as a server listening for request from the k8s-apiserver to allow/deny pod creations
• We need a ValidatingWebhookConfiguration so the k8s-apiserver knows to ask our server for pod creation approval
• We need Role/RoleBinding/ServiceAccount for our server to inspect resources to limit the creation of based on some logic
• We need some TLS stuff to speak with k8s api-server I think.
• Perhaps OPA can be of use to provide general functionality? https://www.openpolicyagent.org/docs/latest/

References

• k8s docs - ResourceQuotas
• k8s ref - ResourceQuotas
• k8s docs - Admission controllers
• k8s ref - Custom admission controllers
• k8s blog post - A guide to admission controllers

Ping @rabernat @jhamman @scottyhq @yuvipanda @choldgraf @georgianaelena

[dhirschfeld]

Could you give each user their own hierarchical namespace and just assign a ResourceQuota to that?

[consideRatio]

@dhirschfeld oh this is the first time I'm learning about that, perhaps! I need to read up a lot on it to evaluate. Thanks for the idea!

[dhirschfeld]

When I heard about hierarchical namespaces I thought this could be a perfect use-case. I haven't had time to investigate it further but I'm very interested to find out if there are any flaws in my plan!

I think it's still alpha though so it may not be a solution for right now.

[droctothorpe]

This is an awesome idea @consideRatio. At present, DG does not limit resource utilization by user though there has been some discussion about limiting max clusters per user here. A modular solution that can be leveraged by DG, JH, and other applicatons would be amazing. A helm chart could install the controller and a LabelSelectorResourceQuota CRD. CRs to map quotas to selectors could be installed on a per-application basis, e.g. no more than 5 scheduler pods per user, no more than 100 cores total per user, etc. This would be predicated on a shared, static label for all the resources associated with an application, and a label with a static key with a variable user-specific value -- that should already be configured for DG. Kubebuilder or Operator SDK could handle a lot of the boilerplate if you wanted to write it in Go. Also, regarding resource monitoring (as opposed to limiting), I highly recommend looking into Kubecost, which implements much of what the second part of your proposal describes.

[yuvipanda]

Wow, great find @dhirschfeld! I think Hierarchical Namespaces would be awesome here

[droctothorpe]

Given the LOE for this, I wonder if a more mature, thoroughly documented dynamic (one per user) namespace implementation (with or without hierarchy) leveraging vanilla ResourceQuotas might be easier and confer other benefits associated with namespace isolation. I know that it's technically possible with both JH and DG, and the R&D is actually on our road map, we are just not aware of an existing reference implementation.

[choldgraf]

I think this would be an excellent tool for a number of organizations across the ecosystem. "How can I avoid one of my users accidentally racking up a ton of $$$ in compute?" is a really common question for people deploying jupyterhub/scalable computing infrastructure for organizations or teams