This PR is a POC of adding a clusterset type that allows admins to selectively limit the Resources that a given group of clusters can consume.
Overview
Updates several items on the cluster type.
The cluster type is now namespaced.
The pods for the cluster (server/agent) are now in the same namespace as the cluster
The type contains a Limit field allowing users to specify the server/agent limits (separately) that apply to each replica. Note that this is applied separately to each replica - so with 2 agents and a limit of 1 CPU for the worker node, each agent will get a limit of 1 CPU for a cluster-wide total of 2 CPU.
The type contains a node selector field allowing users to constrain which nodes worker/servers run on.
The pods are now deployed with anti-affinity which causes agents to not be scheduled on the same host cluster node as other agents. Servers have the same anti-affinity with other servers. This does not apply to servers/agents (so a server can run on the same host node as an agent, but not on the same node as another server).
Adds a new clusterset type.
MaxLimits provides an upper limit for the limits in the cluster. Note that this works like the limits from a ResourceQuota - if no limit is specified for that resource, clusters in this set have no limit. These limits are the sum across all servers/workers.
DefaultLimits provides the defaults for new clusters which don't specify a limit (mostly a UX item so that new clusters don't need to explicitly set these values).
DefaultNodeSelector provides a starting node selector for all clusters in the set.
Adds webhook validation to enforce the clusterset values on new/updated clusters.
Updates the cluster type with field-level docs.
Adds a script to automatically generate CRDs from go types, and ports existing validation to the go-type annotations.
Using/testing
Clone the branch.
Build the binary make build.
Build/Push the image docker build -f package/Dockerfile . -t $REPO/$IMAGE:$TAG
Update values.yaml with image.repostory: $REPO/$IMAGE and tag: $TAG
Generate a key using openssl: openssl genrsa -out rootCAKey.pem 4096.
Generate a cert using openssl: openssl req -x509 -sha256 -nodes -key rootCAKey.pem -days 3650 -out rootCACert.pem --addext "subjectAltName=DNS:k3k-webhook.k3k-system.svc".
Create the namespace and upload the cert as a secret kubectl create -f ns.yaml && k create secret tls webhook-secret -n k3k-system --cert=rootCACert.pem --key=rootCAKey.pem. See below for the namespace (needs helm annotations to be imported when installing the chart).
Update charts/k3k/templates/webhooks.yaml with the value copied in the previous step (past in the caBundle field where "ReplaceMe" is for both the Valdating and Mutating Webhook).
Deploy the chart helm install k3k ./charts/k3k -n k3k-system
Right now this PR is in a POC state, and needs more testing/refinement before it could be merged. Some examples of this include:
There's an unnecessary clusterset controller
There's no clusterset webhook
The webhook needs to be manually configured with a secret
The chart doesn't tolerate an existing k3k-system namespace
Some of the fields are misnamed (cluster.Limit, cluster.Limit.WorkerLimit).
Limits and Defaults should probably be two different objects (like LimitRange and ResourceQuota) so that users can give RBAC to set defaults without giving RBAC to controller the upper limits.
Webhook needs a mutex so that it locks when calculating if a cluster exceeds the limits for a namespace.
The bootstrap secret needs an ownerref to the cluster so that it gets removed when the cluster does.
The CLI needs to be reviewed for the potential of adding new fields for various limits.
We should evaluate using --disable-agent on the server nodes by default (or even by force), which would prevent users from scheduling pods on the server node.
This PR is a POC of adding a clusterset type that allows admins to selectively limit the Resources that a given group of clusters can consume.
Overview
Limit
field allowing users to specify the server/agent limits (separately) that apply to each replica. Note that this is applied separately to each replica - so with 2 agents and a limit of 1 CPU for the worker node, each agent will get a limit of 1 CPU for a cluster-wide total of 2 CPU.MaxLimits
provides an upper limit for the limits in the cluster. Note that this works like the limits from aResourceQuota
- if no limit is specified for that resource, clusters in this set have no limit. These limits are the sum across all servers/workers.DefaultLimits
provides the defaults for new clusters which don't specify a limit (mostly a UX item so that new clusters don't need to explicitly set these values).DefaultNodeSelector
provides a starting node selector for all clusters in the set.Using/testing
make build
.docker build -f package/Dockerfile . -t $REPO/$IMAGE:$TAG
image.repostory: $REPO/$IMAGE
andtag: $TAG
openssl genrsa -out rootCAKey.pem 4096
.openssl req -x509 -sha256 -nodes -key rootCAKey.pem -days 3650 -out rootCACert.pem --addext "subjectAltName=DNS:k3k-webhook.k3k-system.svc"
.cat rootCACert.pem | base64 | tr -d '\n'
kubectl create -f ns.yaml && k create secret tls webhook-secret -n k3k-system --cert=rootCACert.pem --key=rootCAKey.pem
. See below for the namespace (needs helm annotations to be imported when installing the chart).helm install k3k ./charts/k3k -n k3k-system
Note
--disable-agent
on the server nodes by default (or even by force), which would prevent users from scheduling pods on the server node.