K8s slow rollout controller

[imjasonh]

GitOps is cool, but YOLOing 1000 replicas of a Deployment to a new image at once is madness.

Vanilla K8s Deployments support rolling updates, which is great, but maybe you want to roll out even slower (e.g., wait 10m between rollouts), or wait on some condition or signal before continuing rollout (metrics, smoke tests, approval).

kind: RolloutPolicy
metadata:
  name: slow-rollout-policy
spec:
  stages:
  - deploy: 10%
  - wait: 30m
  - metrics:
    prometheus.something.something/error-rate: <1%
    onFailure: Rollback
  - deploy: 50%
  - wait: 30m
  - metrics:
    prometheus.something.something/error-rate: <1%
    onFailure: Rollback
  - approval: ... # make someone click a button
  - deploy: 100%

kind: Deployment
metadata:
  name: my-deployment
  annotations:
    rollout-policy: slow-rollout-policy
spec:
  ...

• webhook to intercept Deployment PodSpec image updates (like from GitOps)
  • set an annotation for the original value (rollout/requested-image: my/new-image)
  • set the image back to the original (no immediate rollout)
  • create a RolloutRun instance to describe this instance of a rollout, and its status
• async reconciler sees the new RolloutRun
  • update some subset of Deployment's underlying Pods with the new image
  • watch for specified conditions (metrics, etc.), update RolloutRun+Deployment status to indicate why rollout is paused, emit events for rollbacks/rollforwards

This is even easier for Knative Services, since you can literally just set rollout strategy to desired %age. Maybe this should just be Knative-specific at first?