Support for API Priority and Fairness per tenant

[bsctl]

Describe the feature

Kubernetes introduced the API Priority and Fairness (APF) feature [beta in k8s 1.20] to control the behaviour of the Kubernetes API server in an overload situation. See here for details.

When multiple tenants are deployed, a buggy or malicious application running in a tenant can overload the APIs server or it can compete with other tenants and starve their applications.

• Each request sent to the APIs server is matched by a FlowSchema. The FlowSchema states the PriorityLevel for requests that match it, and assigns a Flow identifier to these requests. Flow identifiers are how the system determines whether requests are from the same source or not.

• The PriorityLevels may be configured to behave in several ways. Each PriorityLevel gets its own isolated concurrency pool.

• To prevent any one tenant from monopolising a PriorityLevel, they may be configured to have multiple queues. The Shuffle Sharding algorithm is used to assign each flow of requests to a subset of the queues.

• Finally, when there is capacity to service a request, a Fair Queuing algorithm is used to select the next request. Within each priority level the queues compete with even fairness.

It would be nice to implement support for APF in Capsule:

• The Capsule Controller gets its own FlowSchema with priority just one step lower than other system controllers. This assures the Capsule Operator cannot be starved by default user generated requests.

apiVersion: flowcontrol.apiserver.k8s.io/v1beta1
kind: FlowSchema
metadata:
  name: capsule-system-service-account
spec:
  distinguisherMethod:
    type: ByNamespace
  matchingPrecedence: 1000  # <-- precedence just lower than kube-system-service-accounts
  priorityLevelConfiguration:
    name: workload-high  # <-- same priority of system controllers
  rules:
  - nonResourceRules:
    - nonResourceURLs:
      - '*'
      verbs:
      - '*'
    resourceRules:
    - apiGroups:
      - '*'
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - '*'
      verbs:
      - '*'
    subjects:
    - kind: ServiceAccount
      serviceAccount:
        name: capsule # <-- Capsule ServiceAccount
        namespace: capsule-system # <-- Capsule Namespace

• Each Tenant gets its own PriorityLevel with default priority. This assures each tenant is not starved by other tenants.

apiVersion: flowcontrol.apiserver.k8s.io/v1beta1
kind: FlowSchema
metadata:
  name: oil
spec:
  distinguisherMethod:
    type: ByUser
  matchingPrecedence: 9900 
  priorityLevelConfiguration:
    name: global-default
  rules:
  - nonResourceRules:
    - nonResourceURLs:
      - '*'
      verbs:
      - '*'
    resourceRules:
    - apiGroups:
      - '*'
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - '*'
      verbs:
      - '*'
    subjects:
    - kind: Group
       group:
          name: capsule.clastix.io
    - kind: User
       user:
          name: alice

What would the new user story look like?

1. When Capsule is installed, the related FlowSchema is created.
2. When a new tenant is created/modified/deleted, the related FlowSchema is created/modified/deleted

Expected behavior

Support for API Priority and Fairness per tenant

[bsctl]

@prometherion do you have any concern about this enhancement?

[prometherion]

How to handle fairness when an Owner owns multiple Tenant resources?

We would end up with duplicates matching different values, not sure how the Kubernetes controller would deal with them.