GitOps vertical pod autoscaler

[awh]

@bboreham and I were just discussing maintenance of the resource requests on the deployment manifests underpinning Weave Cloud. In the past this has been done by Bryan with pencil and paper, and more recently using information directly from the dashboard service, but ideally we would have an automated solution. Bryan pointed to the vertical pod autoscaler project, but remarked that it does not operate according to GitOps principles, so we wondered if we could make something that fits into the same model Flux uses for automatic updates: a new component that watches actual usage, and then commits resource request updates to the manifests based on configurable policy annotations.

Example policies might include:

• Updating on a regular schedule, e.g. once every N hours
• Updating N hours after a new version is deployed
• Deriving request values based on high water mark or percentile

WC UI features might include:

• Policy editor
• Reporting on resource requests vs. actual usage across all workloads
• Alerts when RR diverge wildy from actual usage

[stefanprodan]

There is a proposal to allow VPA to make pod resource updates without restarts https://docs.google.com/document/d/18K-bl1EVsmJ04xeRq9o_vfY2GDgek6B6wmLjXw-kos4/edit?ts=5b96bf40

[squaremo]

I have thought about this before in the context of horizontal pod autoscaler. I think what is needed is a slightly different (but not incompatible) mode of operation to flux.

Rather than recreating all the mechanisms of the HorizontalPodAutoscaler, or VerticalPodAutoscaler, and whatever else comes along, it might be better to let them do their thing, but make sure the cluster state can still be recovered from git by observing the given values and committing changes.

So that it doesn't flap, or otherwise constantly commit needless changes, we'd need to transform the signal into something stepwise. The idea here would be to give thresholds for commits, like "write a commit when the value has increased by 10%, or decreased by 20%".

I recommend giving this a read: https://medium.com/@mt165/latching-mutations-with-gitops-92155e84a404

[ngehani]

May want to compare it to these people - http://www.magalix.com/

[squaremo]

http://www.magalix.com/

Weave Cloud has nowhere near enough lightning coming out of it.

[jml]

Relates to #1937