MShekow / azure-pipelines-k8s-agent-scaler

A Kubernetes operator for Azure DevOps Pipelines agents
Apache License 2.0
5 stars 0 forks source link

Feat: terminate idle agent pods created with a different controller-manager version #23

Open MShekow opened 8 months ago

MShekow commented 8 months ago

Whenever the cluster admin installs a newer version of our operator, this new version might fix bugs that affected how the agent pods were created. For instance, a bug could have affected the environment variables being set of the AZP agent container.

Typically, all controller-managed Pods have a short life span (they are ephemeral), but if minCount > 0 is used, they might live for a long time and be affected by a bug.

Solution approach: the controller manager attaches its own version to the Pods that it manages, via a Pod annotation, and deletes all those idle Pods that have a mismatching controller manager version. We could, for instance, define a Docker build ARG CONTROLLER_MANAGER_BUILD_ID in the Dockerfile (which is turned into an env var, so that the controller-manager can consume it) with some default value, which is overwritten by the GitHub actions workflow.