search

kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.1k stars 3.98k forks source link

CPU recommandations too low for containers #7062

Open cmotsn opened 4 months ago

cmotsn commented 4 months ago

Which component are you using?: vertical-pod-autoscaler

What version of the component are you using?: registry.k8s.io/autoscaling/vpa-recommender:

Component version: 1.1.2

What k8s version are you using (kubectl version)?:

kubectl version Output
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Comp
iler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.5", GitCommit:"05f2dca83369807327c976b24846de8687923cb6", GitTreeState:"clean", BuildDate:"2024-04-12T23:22:53Z", GoVersion:"go1.20.12", Com
piler:"gc", Platform:"linux/amd64"}

What environment is this in?: AKS

What did you expect to happen?: I expected to be able to set the recommended value as limits&request and have my pods deploy properly

What happened instead?: the recommended CPU target values are too low and the (Java Spring) containers can't even start

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

I installed the recommender using the Fairwind/VPA Helm chart and overrode the recommender image tag to use the latest version. I configured the recommender to use prometheus, with some custom args in order to avoid errors in the logs, following comments on https://github.com/kubernetes/autoscaler/issues/5031 :

 --set recommender.extraArgs.prometheus-address="http://prometheus-server.monitoring.svc.cluster.local:80" --set recommender.extraArgs.storage=prometheus --set recommender.extraArgs.metric-for-pod-labels='kube_pod_labels{job="kube-state-metrics"}[8d]' --set recommender.extraArgs.pod-namespace-label=namespace --set recommender.extraArgs.pod-name-label=pod

The Memory recommendations look more pertinent, at the very least the containers start using the given Memory target recommendations as requests and limits.

adrianmoisey commented 4 months ago

/area vertical-pod-autoscaler

voelzmo commented 4 months ago

Hey @cmotsn I'm not sure I understand your workflow properly here, as the "steps to reproduce" section is missing. What is it exactly you're doing? I understand that you are using VPA (installed with the fairwinds chart) for you workload and get some recommendations. As far as I know, Fairwinds automatically generates the VPA objects for you and runs VPA in updateMode: off, so it doesn't automatically update your resource requests, right? So how are you applying new resource requests and which values are you picking from the fairwinds UI?

Can you please provide the VPA and deployment yaml files for your workload?

Thanks!

/triage needs-information

cmotsn commented 3 months ago

Hi, Thank you for your response and taking the time to help.

I understand that you are using VPA (installed with the fairwinds chart) for you workload and get some recommendations. As far as I know, Fairwinds automatically generates the VPA objects for you and runs VPA in updateMode: off, so it doesn't automatically update your resource requests, right?

Correct. We only wanted the recommendation from VPA, that we access either by the Goldilocks/Fairwind UI or simply by consuming the metric kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_target (because Goldilocks uses this target metric for Garanteed QoS).

Here's a look to what this target metric is looking like for our Java backends: image

Basically we got the same 0.015 value for all our single-container Java backends pods

So how are you applying new resource requests and which values are you picking from the fairwinds UI?

I took the recommended values from Goldilock (which is the same as the metric value from VPA) & created a new version of my helm chart with both Memory&CPU requests&limits. But it never managed to deploy. I changed the CPU Limit to 1 and re-deployed, the deployment succeeded, which is why I think the 0.015 value is faulty.

Deployment file apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "11" creationTimestamp: "2024-06-19T17:12:23Z" generation: 11 labels: app.kubernetes.io/instance: oxx app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: oxx app.kubernetes.io/version: 1.1.0-PullRequest0037.11-6ca17a8 helm.sh/chart: oxx-1.1.0 name: oxx namespace: yyy resourceVersion: "268117519" uid: 08be245b-d1ca-48b7-a4e5-b90c942acb8b spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/instance: oxx app.kubernetes.io/name: oxx strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: container.apparmor.security.beta.kubernetes.io/oxx: runtime/default instrumentation.opentelemetry.io/inject-java: "false" creationTimestamp: null labels: app.kubernetes.io/instance: oxx app.kubernetes.io/name: oxx spec: automountServiceAccountToken: false containers: - env: # … - name: OTEL_LOGS_EXPORTER value: otlp - name: OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE value: DELTA image: oxx # Image of a JRE with our Spring Boot jar imagePullPolicy: Always livenessProbe: failureThreshold: 3 httpGet: path: /actuator/health/liveness port: http scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: oxx ports: - containerPort: 8080 name: http protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /actuator/health/readiness port: http scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: 15m memory: 814M requests: cpu: 15m memory: 814M securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsUser: 10001 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault terminationGracePeriodSeconds: 30 status: availableReplicas: 1 conditions: - lastTransitionTime: "2024-06-19T17:12:34Z" lastUpdateTime: "2024-06-19T17:12:34Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available - lastTransitionTime: "2024-06-19T17:12:23Z" lastUpdateTime: "2024-07-26T09:47:13Z" message: ReplicaSet "oxx-6f8c676868" is progressing. reason: ReplicaSetUpdated status: "True" type: Progressing observedGeneration: 11 readyReplicas: 1 replicas: 2 unavailableReplicas: 1 updatedReplicas: 1
VPA file created by Goldilocks apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: creationTimestamp: "2024-06-19T17:12:23Z" generation: 38787 labels: creator: Fairwinds source: goldilocks name: goldilocks-o namespace: yyy resourceVersion: "264572052" uid: e252f876-c3ec-4ac7-b26b-0d9d5530a621 spec: targetRef: apiVersion: apps/v1 kind: Deployment name: oxx updatePolicy: updateMode: "Off" status: conditions: - lastTransitionTime: "2024-06-19T17:12:32Z" status: "True" type: RecommendationProvided recommendation: containerRecommendations: - containerName: oxx lowerBound: cpu: 15m memory: "272040957" target: cpu: 15m memory: "272061154" uncappedTarget: cpu: 15m memory: "272061154" upperBound: cpu: 15m memory: "282159830"

The fact that the recommendations seem pertinent for memory and not for CPU is throwing me: if VPA could not access the pods data from Prometheus, it wouldn't make pertinent memory recommendations, so I'm missing something.

k8s-triage-robot commented 3 weeks ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale