Safeguard: Enforce min and max number of replicas

[cristiklein]

Is your feature request related to a problem? Please describe.

We have all kinds of safeguards in place, to ensure application developers don't compromise platform security and stability due to trivial mistakes. Currently, we place no restrictions regarding number of replicas. This means that users can accidentally run Deployments or StatefulSet with insufficient replication.

Describe the solution you'd like

opa:
  minMaxReplicas:
    enforcement: deny  # default dryrun

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    replicas-min: "4"
    replicas-max: "10"
  name: demo1

Should trigger an error if a Deployment or StatefulSet in the annotated Namespace has a number of replicas which does not respect the Namespace's annotation.

Describe alternatives you've considered

• Not implementing the present solution.

Additional context

• From the user's perspective, the feature should feel similar to this safeguard
• This OPA policy might be useful as a starting point.

Definition of done:

• Compliant Kubernetes supports the above configuration (please include a screenshot as evidence).
• Public docs are updated accordingly.

[viktor-f]

I think this should apply to stateful sets as well. It could be nice to give the users the option to opt-out of this policy for specific deployments/stateful sets by adding a label/annotation to the deployment/stateful set. Since I think that most users will have some pods that either cannot (apps that need more dev before it can go HA) or does not need to (non critical apps) have multiple replicas, and that should not prevent the rest of the namespace to have this useful safeguard.

[cristiklein]

Thanks @viktor-f ! I added StatefulSet as you suggested.

I'm hesitant to adding opt-out in the first iteration of this feature: It risks creating an exception culture. I can also imagine the horror scenario where people just copy-paste exception labels in Helm Charts "because it works". :smile:

I'd rather keep Namespaces as the demarcation for different enforcement of min/max number of replications.

However, let's keep your suggestion in mind, for a second iteration of this feature.

[viktor-f]

I see your point, I'm instead a bit more afraid that most users would then just go for replicas-min: "1" in most namespaces.

[cristiklein]

I see your point, I'm instead a bit more afraid that most users would then just go for replicas-min: "1" in most namespaces.

:smile:

Yeah, this feature is very much in "needs to be validated" mode. I'm imagining it being used as follows: Since labeling the Namespace is controlled by the admin, I can imagine that this is something that would be enabled after go-live testing or an incident, and never taken back. Hence, I don't think it will work as "handcuffing", more like "let's avoid that mistake even happening again".

[viktor-f]

Another solution (discussed with scrum masters) would be to allow opt-out by setting annotations to the namespace, listing which deployments and stateful sets to exclude. That would prevent the risk of just copy-pasting the opt-out on every deployment/statefulset. Though it will add some overhead in ops work if the ops team have to be the one adding these annotations (until we get hierarchical namespaces or let the users add annotations to namespaces).