kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.11k stars 3.98k forks source link

VPA assigns `minAllowed.memory` instead of actual recommendation when restarting Prometheus pod if `Limitrange` is set #7429

Open MatteoMori8 opened 1 month ago

MatteoMori8 commented 1 month ago

Which component are you using?:

What version of the component are you using?:

Component version:

What k8s version are you using (kubectl version)?:

Server Version: v1.29.8-eks

What environment is this in?:

What did you expect to happen?: I expected VPA's AdmissionController to patch my Prometheus pod with a request.memory value equal to the memory target recommendation

What happened instead?: Instead, the new pod got deployed with the following:

On the other hand, CPU got updated correctly

How to reproduce it (as minimally and precisely as possible):

VPA object
```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: prometheus-test spec: resourcePolicy: containerPolicies: - containerName: prometheus controlledResources: - cpu - memory controlledValues: RequestsAndLimits maxAllowed: memory: 30Gi minAllowed: cpu: 100m memory: 500Mi - containerName: thanos-sidecar controlledResources: - cpu - memory controlledValues: RequestsAndLimits maxAllowed: memory: 20Gi minAllowed: memory: 100Mi targetRef: apiVersion: apps/v1 kind: StatefulSet name: prometheus-test updatePolicy: updateMode: "Initial" ```
LimitRange
```yaml apiVersion: v1 kind: LimitRange metadata: name: limits spec: limits: - max: memory: 60Gi min: cpu: 1m memory: 50Mi type: Pod - default: memory: 512Mi defaultRequest: cpu: 20m memory: 256Mi min: memory: 50Mi type: Container ```

Anything else we need to know?:

We talked this out on the #sig-autoscaling Slack channel and it turns out that by deleting the LimitRange, VPA goes back on track and manages memory in the appropriate way.

Personally I do not know if this is a bug or an opportunity to enhance the docs but we thought it could be good to raise an Issue about it

adrianmoisey commented 1 month ago

/area vertical-pod-autoscaler

adrianmoisey commented 1 month ago

/assign

adrianmoisey commented 1 month ago

Does the Prometheus statefulset define any resources?

adrianmoisey commented 1 month ago

Instead, the new pod got deployed with the following:

  • limit.memory got halved
  • request.memory has been assigned the value that I specified as minAllowed.memory from the VPA object itself

Which container in the Pod got these set?