Open EraYaN opened 1 year ago
Hello,
meanQuantities
is part of the --show-stats
calculations of the "global stats" for all VPAs of a namespace. The division by zero and the fact that kubectl vpa-recommendation
that returns no VPA clearly might indeed seems to be the source of the problem.
The doc states in the Limitations section:
Unlike the official VPA recommender, which is fully generic and handle any kind of "scalable" resources, the plugin recognize only some well-known controllers such as: CronJob, DaemonSet, Deployment, Job, ReplicaSet, ReplicationController, StatefulSet.
Did you get logs about the VPA target kind being unsupported ? When the plugin cannot bind a VPA with it's target, it doesn't append it to the "table". And the len of the yable is used for the states computations: https://github.com/wI2L/kubectl-vpa-recommendation/blob/8f936b6b4630eae6d913fcaaaec7696b597450d8/cli/table.go#L299
I'll fix the panic.
Regarding your use case, the plugin would need to be able to fetch any kind, tho this might cause other issues (a CRD with the /scale
subresource will probably not have the same spec for resource requests/limits, which are used for the computation of the stats):
https://github.com/wI2L/kubectl-vpa-recommendation/blob/master/vpa/target.go#L73
Do you mind sharing some information about the spec of your custom resources?
kubectl vpa-recommendation -v5
gives a ton of I0823 13:18:59.714861 13838 command.go:231] couldn't get target for vpa default/<name>
lines.
Our CRD does have a scale subresource, so the newer versions of the vpa recommender force us to update our targets from the Deployments to the CRD.
Our CRD has this in its Spec:
Resources *corev1.ResourceRequirements `json:"resources"`
So that is not the same like it is for deployments where resources are per container.
For us it would be acceptable to let us add a command line option like --kind="KindName" --resourcePath=".spec.resources". Or some way of helping it resolve to the pod selectors themselves like what the recommender does.
Maybe you can even depend on the recommender to steal this piece of code to get the selectors from the scale resource: https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/pkg/target/fetcher.go#L119-L146 then you can map to the pods and grab the current resources from that.
We define those as follows:
+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.deployment.replicas,selectorpath=.status.scale.selector
@EraYaN I'll fix the panic first, but to support custom target kinds, this will require a bit of refactor, and I won't have time to work on it immediately. If you are willing to work on it, let me know.
For us it would be acceptable to let us add a command line option like --kind="KindName" --resourcePath=".spec.resources". Or some way of helping it resolve to the pod selectors themselves like what the recommender does.
I don't think a --kind="KindName"
flag is necessary, the target reference in the VPA should have enough information to fetch the custom resource, as long as it has a /scale
subresource (same way the VPA recommender does it).
Regarding the second flag, --resourcePath=".spec.resources"
, I agree. By default the plugin should try to look into the common path .spec.template.spec
which works for well-known controllers, otherwise this should be specified manually for custom paths.
Alright I gave it a go in #4
Running: kubectl vpa-recommendation --show-stats --output wide --sort-columns mem-diff,cpu-diff
I get the following error:
It seems like meanQuantities chokes on some of our vpa's recommandations. I have dumped the vpas to yaml but there is a ton of internal info in those, so I'm not sure if I can share those.
Most of them have one container with something like this:
and a couple will have two containers.
EDIT: Alright
kubectl vpa-recommendation
doesn't actually return any VPAs. All our VPAs target a custom CRD, might that be the issue? Seems like it doesn't realize the number of matched VPAs is actually 0 because they are all targeting some unsupported kind.