Introduce Quota/Limits

[gardener-robot-ci-1]

To be spec'ed:

By @vlerenc: We want to offer trial clusters, but we need quota/limits for those.

Per IaaS account (which we model today, maybe not ideally, as a secret):

• Main Memory
• CPUs
• GPUs
• Disk Size (per type)
• Cluster LBs
• Cluster PVC Size (per type)
• Cluster Lifetime (terminate clusters after given period of time)

Alternatively, for now, we could simplify that by a VMs quota, but then we would need to restrict the machine types (since they differ significantly in their resources and prices, especially when looking at the GPU-based machine types).

Per Gardener project (which we model as namespace):

• Main Memory
• CPUs
• GPUs
• Disk Size (per type)
• Cluster LBs
• Cluster PVC Size (per type)
• Cluster Lifetime (terminate clusters after given period of time)

Note: We may have/want to add more resource types over time. Some are infrastructure specific (if not abstracted) like e.g. the disk type.

This way we could achieve the following:

• Limit the overall costs for our trial IaaS account, e.g. 400 GB Main Memory, 200 CPUs, 20 GPUs, 4 TB gp2 and 1 TB io1 Disk, 4 LB services and 200 GB gp2 and 50 GB io1 PVCs per cluster, cluster auto-termination after 28 days
• Allot each Gardener project only a tiny share, so that no single project can consume all resources, e.g. 20 GB Main Memory, 10 CPUs, 0 GPUs, 200 GB gp2 Disk, cluster auto-termination after 7 days
• But allow for exceptions like permitting one project to use GPUs, another to use more expensive io1 disks, or yet another to extend the lifetime of their clusters

Note: See also #101. Note: Most of these quotas can be enforced by an admission controller in the garden cluster, but the LBs and PVC size quotas will require similar controllers in the respective shoot clusters themselves, which we can handle with lower priority/later. Actually, it would be nice if LBs and PVC size quotas could be specified on the IaaS account level for the entire IaaS account, but that would require the shoot cluster admission controller to call back to the garden cluster, which is currently not possible (not reachable), so let's go with the per-cluster simplification. Note: We support worker count min/max, i.e. cluster auto-scaling. Check the max requirements against the quota, i.e. assume the worst case. Note: If quotas are set for the IaaS account and the Gardener project, both must comply (simplest strategy for now). Note: Make sure a cluster member can't change these quotas. Later we may allow that, e.g. in a non-trial use case a customer purchases a certain quantity of resources and allots it to its projects. Note: Secrets are currently placed into the Gardener project and must be readable, but we don't want that. They should be "somewhere else" outside the namespace, so that their contents are protected (not be directly visible in the project, but when a cluster is created, they appear in the tf-vars of the infrastructure terraform files, which is bad, but acceptable for now). That would also ease setting the overal quota (per IaaS account). One possible solution to refer them is described in #102, but it's not ellegant as it requires two different code paths in the operator and UI to handle these cases. It would be better if we find a solution where the secret feels the same, whether its shared or not.

Implementation Proposal:

• Use a similar concept like ResourceQuotas, but in our API group (see 1 and 2) to describe quotas per IaaS account and Gardener project.
• Use admission controllers for checking the quota.
• Consider having independent ResourceQuotas and admission controllers (per quota) or combine them into one in the first step, but this may make later extensions, especially when we open source, somewhat harder (still, if it speeds us up, let's go with a combined ResourceQuota and admission controller)
• The question of sharing a secret and binding quota is still somewhat unanswered. Personally, I would explore this by prototyping a few resources in the file system (mapping to namespaces):
  • Have a shared secret in the trial NS
  • Have a global quota in the trial NS
  • Assumption: No longer invite people to the trial NS, kick them out over time
  • Create a binding in a customer namespace to the shared secret
  • Create a local secret and binding in that customer namespace to the local secret
  • Motivation: The above would show local and shared secrets; the UI would always create local secret and secret binding and both the UI and the operator would only read and work with the bindings
  • Create a binding in a customer namespace to the global quota
  • Create a local quota and binding in that customer namespace to the local quota
  • Motivation: The above would show local and global quotas; the admission controllers would only read and work with the bindings

[gardener-robot-ci-1]

Comment by mliepold Tuesday Nov 07, 2017 at 10:39 GMT

---

Abstract

Currently every user or project owner has to bring its own secret(s) to create clusters. In order to increase adoption of the Gardener, we want to introduce trial clusters. But for these we need to introduce quotas to be able to limit the amount of created clusters and resources and thus the possible costs for a potential sponsor.

Requirements

The quotas/limits need to be set for (account) secrets belonging to a Gardener project. A project is represented by a K8S namespace "garden-projectName" (with a label "garden.sapcloud.io/role: project").

In order to avoid the need to create specific trial projects and to copy trial secrets, secrets need to be referable from different namespaces (i.e. projects). Also individual projects that use a trial secret should only be allowed to use a certain share of the overall quota associated with it. Thus it is needed to set quotas on project level too.

By default every project using a trial quota should have a predefined quota. But projects that have special needs can request an increase of their quota.

Also the created trial cluster should automatically be cleaned up after a defined amount of days.

Design

Metrics for quota

Metrics should be set for the most important chargeable resources of a cloud provider. Initially we specify the following metrics:

• Main Memory
• CPUs
• GPUs
• Disk Size (per type)
• Cluster LBs
• Cluster PVC Size (per type)

Defining quotas

The Gardener UI will not support the creation of quotas (in the beginning). It has to be done via the K8S CLI for already existing secrets. For this a new custom resource "Quota" of the gardener is used.

Implementation

Quota

apiVersion: garden.sapcloud.io/v1beta1
kind: Quota
metadata:
  name: trial-aws-secret-quota
  namespace: garden-trial
spec:
  scope: secret
  clusterLifetimeDays: 14
  metrics:
    cpus: "200"
    gpus: "20"
    memory: 4000Gi
    disk.basic: 8000Gi
    disk.standard: 8000Gi
    disk.premium: 2000Gi
    loadbalancers: "100"

Specification:

• scope: Defines if the quota is set for the (account) secret or for a project. Possible values: "secret" and "project"
• clusterLifetimeDays: Max. lifetime of a cluster in days.
• metrics: The individual metrics for the quota as described above.

Note: The metric for cpus uses the worst-case scenario with the maximum possible CPUs per cluster in the beginning, if auto scaling is set for clusters. This means even if the actual amount of CPUs is not reached yet, the creation of new clusters could not be possible anymore! It ensures that the total costs can never be higher than defined by the quota. But the operator of the week should monitor the actual amount of CPUs for a secret to possibly adjust the numbers.

The quota can be set for the Gardener resource Secret specified by:

• kind: Secret
• label: "garden.sapcloud.io/role: infrastructure"
• namespace: same as the ShootQuota

Controllers

Quota Controller

It is responsible for updating the status of all defined quotas. To do this, it has to monitor all existing shoot clusters which

• are using the specified secrets.
• are in the same project.

The controller also regularly checks all existing resources of kind "Shoot" for the annotation "quota.garden.sapcloud.io/creationTime". If the current date is after this date plus the expiration time of the project quota, the gardener triggers the deletion of this shoot cluster. The user will get no notification before or when the cluster is deleted!

Quota Admission Controller

It checks the current status of the quotas that are used during the creation of a shoot cluster. If I quota would be exceeded after the cluster creation, the creation will be prevented and the user gets an appropriate error message.

Binding Secrets and Quotas

For every secret in a project namespace, a PrivateSecretBinding is created. It binds the secret with a possible quota for the secret. The referenced secret and quota have to be in the same namespace. Also for the Garden projects that can use a trial secret, a CrossSecretBinding is created in the project namespace. It binds the trial secret with the secret quota and the (default) project quota. These referenced objects reside in the trial namespace.

At the secret selection during the shoot creation, the Gardener UI will use all secret bindings (PrivateSecretBinding and CrossSecretBinding) in the current project.

Open Issues

• Restrict the namespaces for trial secrets?

Extended Scope

• It should also be possible to restrict the machine types of VMs that are created for a cluster, because the costs for the different VM types can differ a lot. Setting for quota: shoots.machinetypes: List of allowed machine types for the cluster

  shoots.machinetypes: ["m4.large", "m4.xlarge"]

• Setting und modifying the (default) quota via the Gardener UI.

Out of Scope

• Quota for max. amount of VMs per user. This feature can't be implemented securely without a role concept in the gardener. But without this feature, a misuse of certain users that create (too) many clusters has to be determined by the operator of the week.

[gardener-robot-ci-1]

Comment by mliepold Friday Nov 17, 2017 at 11:11 GMT

---

Remarks/proposals during Weekly Kube:

1. @d021332: Why not just using annotations instead of an own CRD?
2. @i068969: K8S is currently discussing a more generic ResourceQuota definition. Martin will provide a link.
3. @d021770: Instead of providing a Quota CRD, make the account a CRD that contains the account and quota information

Responses (decided during Operator Sync meeting):

1. We will stick with an own CRD, because it is a more elegant way to manage quotas. Also K8S has an own resource ResourceQuota. But we will try to find a way to keep the effort for the UI low, to search and display the secrets (normal + shared).
2. Implementing a new account resource would be too much effort. Trial quotas should be available soon to increase adoption of the service.

[gardener-robot-ci-1]

Comment by mvladev Monday Nov 20, 2017 at 11:26 GMT

---

I'm still searching for the google doc for generic resources. Extended resources can provide a nice starting point on how to setup API types.

This logic should not be part of the garden-controller at all. This is a job for a External Admission WebHook or part of a custom API server. Shoot resources should not be created in the API server, if they are not accepted by the admission plugins.

[gardener-robot-ci-1]

Comment by rfranzke Monday Nov 20, 2017 at 12:00 GMT

---

Good points, external admission checks seem to be the best way of doing that.

[gardener-robot-ci-1]

Comment by vlerenc Tuesday Nov 21, 2017 at 04:42 GMT

---

Whoa, what a relief. Thank you @i068969 for the admission check proposal. That sounds much more in line with Kubernetes, hence I like it very much. Also thank you for the extended resources idea. I am not certain whether it can be leveraged, but also here my hope is, we find a well integrated mechanism for quotas. After all, that's kind of natural to expect from something like Kubernetes.

The other design aspect I am concerned with is the complexity with the shared secrets:

1. I am hopeful we can avoid increasing the complexity in the UI, which with the current design would have to make two different calls to get all the secrets
2. Shared secrets are globally shared, but it would be nicer to bind them to projects/namespaces and thereby limit their reach
3. Also, when the UI writes the secret into the cluster CRD, it needs to differentiate between a standard and a shared secret with the current design, which makes the UI more complex
4. Only cosmetic, but I don't like the namespace prefix (if it isn't used already in other Kubernetes contexts) and would prefer, only if we can't avoid (3), to use name and namespace as separate properties

[gardener-robot-ci-1]

Comment by vlerenc Thursday Nov 23, 2017 at 03:24 GMT

---

We want to offer trial clusters, but we need quota/limits for those.

Per IaaS account (which we model today, maybe not ideally, as a secret):

• Main Memory
• CPUs
• GPUs
• Disk Size (per type)
• Cluster LBs
• Cluster PVC Size (per type)
• Cluster Lifetime (terminate clusters after given period of time)

Alternatively, for now, we could simplify that by a VMs quota, but then we would need to restrict the machine types (since they differ significantly in their resources and prices, especially when looking at the GPU-based machine types).

Per Gardener project (which we model as namespace):

• Main Memory
• CPUs
• GPUs
• Disk Size (per type)
• Cluster LBs
• Cluster PVC Size (per type)
• Cluster Lifetime (terminate clusters after given period of time)

Note: We may have/want to add more resource types over time. Some are infrastructure specific (if not abstracted) like e.g. the disk type.

This way we could achieve the following:

• Limit the overall costs for our trial IaaS account, e.g. 400 GB Main Memory, 200 CPUs, 20 GPUs, 4 TB gp2 and 1 TB io1 Disk, 4 LB services and 200 GB gp2 and 50 GB io1 PVCs per cluster, cluster auto-termination after 28 days
• Allot each Gardener project only a tiny share, so that no single project can consume all resources, e.g. 20 GB Main Memory, 10 CPUs, 0 GPUs, 200 GB gp2 Disk, cluster auto-termination after 7 days
• But allow for exceptions like permitting one project to use GPUs, another to use more expensive io1 disks, or yet another to extend the lifetime of their clusters

Note: See also #101. Note: Most of these quotas can be enforced by an admission controller in the garden cluster, but the LBs and PVC size quotas will require similar controllers in the respective shoot clusters themselves, which we can handle with lower priority/later. Actually, it would be nice if LBs and PVC size quotas could be specified on the IaaS account level for the entire IaaS account, but that would require the shoot cluster admission controller to call back to the garden cluster, which is currently not possible (not reachable), so let's go with the per-cluster simplification. Note: We support worker count min/max, i.e. cluster auto-scaling. Check the max requirements against the quota, i.e. assume the worst case. Note: If quotas are set for the IaaS account and the Gardener project, both must comply (simplest strategy for now). Note: Make sure a cluster member can't change these quotas. Later we may allow that, e.g. in a non-trial use case a customer purchases a certain quantity of resources and allots it to its projects. Note: Secrets are currently placed into the Gardener project and must be readable, but we don't want that. They should be "somewhere else" outside the namespace, so that their contents are protected (not be directly visible in the project, but when a cluster is created, they appear in the tf-vars of the infrastructure terraform files, which is bad, but acceptable for now). That would also ease setting the overal quota (per IaaS account). One possible solution to refer them is described in #102, but it's not ellegant as it requires two different code paths in the operator and UI to handle these cases. It would be better if we find a solution where the secret feels the same, whether its shared or not.

Implementation Proposal:

• Use a similar concept like ResourceQuotas, but in our API group (see 1 and 2) to describe quotas per IaaS account and Gardener project.
• Use admission controllers for checking the quota.
• Consider having independent ResourceQuotas and admission controllers (per quota) or combine them into one in the first step, but this may make later extensions, especially when we open source, somewhat harder (still, if it speeds us up, let's go with a combined ResourceQuota and admission controller)
• The question of sharing a secret and binding quota is still somewhat unanswered. Personally, I would explore this by prototyping a few resources in the file system (mapping to namespaces):
  • Have a shared secret in the trial NS
  • Have a global quota in the trial NS
  • Assumption: No longer invite people to the trial NS, kick them out over time
  • Create a binding in a customer namespace to the shared secret
  • Create a local secret and binding in that customer namespace to the local secret
  • Motivation: The above would show local and shared secrets; the UI would always create local secret and secret binding and both the UI and the operator would only read and work with the bindings
  • Create a binding in a customer namespace to the global quota
  • Create a local quota and binding in that customer namespace to the local quota
  • Motivation: The above would show local and global quotas; the admission controllers would only read and work with the bindings

[gardener-robot-ci-1]

Comment by vlerenc Thursday Nov 23, 2017 at 03:29 GMT

---

Thank you @i068969 also for the external resources link. I checked it and it seems to be meant for a different use case. It allows to include more than just cpus/memory into the scheduler by advertising (it states explicitly per node) new resources that can then be requested (per pod). That's not what we are after here with quotas, I believe, but I don't know whether it can be "bend" to our needs (doesn't look like, though).

[gardener-robot-ci-1]

Comment by mliepold Friday Dec 22, 2017 at 15:12 GMT

---

Tracking the quota status

Problem: There is a problem with keeping the quota status in the status sub-resource of the quota. With the current proposal there is only one quota resource for all projects that use the default quota. Proposals:

1. Keep the quota states of all projects that use the default quota.

   apiVersion: garden.sapcloud.io/v1beta1
   kind: Quota
   metadata:
   name: trial-aws-project-quota
   namespace: garden-trial
   spec:
   scope: project
   clusterLifetimeDays: 14
   metrics:
   cpu: "20"
   memory: 400Gi
   status:
   metrics:
   - project: "garden-project1"
   cpu: "6"
   memory: 20Gi
   - project: "garden-project2"
   cpu: "15"
   memory: 230Gi

2. Only keep the status at the quota object for the secret. For the project quotas determine the status on the fly at the admission controller. Like this the expensive calculation of the quota status for the secret is done independently in the quota controller. Only the status calculation for the project quota, which is much faster, is done in the admission controller during the shoot cluster creation.

[gardener-robot-ci-1]

Comment by mliepold Friday Dec 22, 2017 at 18:14 GMT

---

For the tracking of the Shoot cluster resources specified in the quotas, a gradual implementation will be done:

1. The resources are only determined statically. This means only metrics that can be determined from the Shoot manifest can be used. Also the maximum resource consumption is assumed. E.g. when specifying auto scaling from 1 to 4 workers with a 2 CPU machine, 8 CPUs are added to the quota metric.
2. Most resources are still determined statically, but for the amount of used CPUs the NodeController is used to deliver the exact amount of current nodes and thus the used CPUs.
3. Dynamically determine all resources specified in the quota metrics. This requires the quota controller to call back to the Garden cluster, which is currently not possible.

[gardener-robot-ci-1]

Comment by vlerenc Friday Dec 22, 2017 at 20:32 GMT

---

Let's have 2 quota controllers:

• One in the garden cluster implementing (1) and then improving it with (2)
• Another one as part of the shoot cluster control plane in the seed cluster regulating the remainder:
  • project quota disk minus shoot cluster node disks is the maximum that may be used inside the shoot cluster
  • project quota LBs minus shoot cluster LBs is the maximum that may be used inside the shoot cluster That's not perfect, but good enough for our goal. We have some limits imposed, at least. Not perfect in regards to the secret quota, but with enough shoot clusters it will average out and no single cluster can get too many resources/go beyond the project quota.

[mliepold]

Event handling

Quota Events:

• Create Quota: 

  1. Verify quota specification. Also check that a correct scope is set.
  2. For quota with scope "secret": Check the existence of referred secret and duplicate definitions.
 Note: No status is set at quota creation.
• Update Quota: 

  1. Verify quota specification.
  2. Check: Only metrics and cluster lifetime are allowed to be changed. Note: No status is updated at quota update.
• Delete Quota:
  1. 
Check for CrossSecretBindings that use this quota.
     • If none exits, delete the quota resource.
     • If one or more exits, prevent the deletion.

CrossSecretBindings Events:

• Create/update CrossSecretBinding:
  1. Verify that the referenced quota(s) exits in the specified namespace.
  2. [Verify that the referenced secret exits in the specified namespace.]
  3. Verify that all namespaces are identical.
  4. Verify that at least one quota is referenced. -> Discuss!

Shoot Events:

• Create Shoot:

  1. Check if a CrossSecretBinding is used for the shoot cluster and if so, check if it has quotas referenced.

  2. If no, nothing to do. Otherwise check for the scope of the quota:
     • Scope=secret: Get the status from the quota resource. Check if the creation of the cluster would violate any metric of the quota.
     • Scope=project: Calculate the status of the quota by getting all shoot resource of the current project. Take in account the currently created and deleted ones. Use this status to check if the creation of the cluster would violate any metric of the quota.
  3. Quota violation: Prevent the creation of the shoot cluster and set an appropriate status at the shoot resource.
  4. No quota violation:
     • Quota with scope=secret: Add the resources that the new shoot cluster will use to the status of the quota.
     • Quota with scope=project: Nothing to do here?! -> Discuss (see comment Tracking the quota status)
     • Start the creation of the shoot cluster.
• Shoot Reconciliation:
  1. Check if it is using a CrossSecretBinding with quota references.
  2. If yes, check if the quota(s) have a cluster lifetime set.
  3. If yes, use the creation time of the cluster + the lifetime to determine if the cluster has to be deleted.
  4. If yes, start the deletion event for the shoot.
• Delete Shoot:

  1. Check if it is using a CrossSecretBinding with quota references.
  2. If yes, check for the scope of the quota:
     • Scope=secret: Subtract the resources that the shoot cluster used from the status of the quota.
     • Scope=project: Nothing to do here?! -> Discuss
  3. Start the deletion of the shoot cluster.
• Update Shoot:

  • Analog to "Create Shoot".
  • But for quota violation determination and quota status update: Only consider the difference of the resources that the updated shoot cluster will use!

[vlerenc]

Moving to Gardener as this is no dashboard story: https://github.com/gardener/gardener/issues/81.

@mliepold Can you please check which of your comments you would like to move to the new ticket (if you do it, it'll be under your name), e.g.:

• https://github.com/gardener/dashboard/issues/14#issuecomment-361025605
• https://github.com/gardener/dashboard/issues/14#issuecomment-361025625
• https://github.com/gardener/dashboard/issues/14#issuecomment-362249976