VPA: admission-controller's memory limit value adjustment is much higher than the pod's memory usage

[eatwithforks]

I have vpa object's recommend mode turned to "Auto" and my pod's memory usage is ~3-4 gb but my pod's limits set by the admission-controller is 30 gb. Can anyone take a look and see if there's a bug or my configuration is incorrect.

cc @bskiba

kubectl get pod <my_pod>

    resources:
      limits:
        **cpu: 2348m
        memory: "32529702032"**
      requests:
        cpu: 587m
        memory: "4066212754"

kubectl get statefulset <foo> -o yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my_pod
  namespace: foo
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  serviceName: foo
  template:
    metadata:
    spec:
      containers:
        name: my_container
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
          requests:
            cpu: "1"
            memory: 512Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
status:
  collisionCount: 0
  currentReplicas: 1
  currentRevision: foo
  observedGeneration: 201
  readyReplicas: 1
  replicas: 1
  updateRevision: bar
  updatedReplicas: 1

VPA

recommendation:
    containerRecommendations:
    - containerName: my_container
      lowerBound:
        cpu: 475m
        memory: "3133521393"
      target:
        **cpu: 587m
        memory: "4066212754"**
      uncappedTarget:
        cpu: 587m
        memory: "4066212754"
      upperBound:
        cpu: 894m
        memory: "6107052326"

admission-controller yaml:

# From https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/deploy/admission-controller-deployment.yaml
# admission-controller-deployment.yaml
# Added project, role, team labels
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vpa-admission-controller
  namespace: kube-system
  labels:
    product: foundation
    project: vertical-pod-autoscaler
    role: admission-controller
    team: compute

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vpa-admission-controller
  namespace: kube-system
  labels:
    product: foundation
    project: vertical-pod-autoscaler
    role: admission-controller
    team: compute
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vpa-admission-controller
      project: vertical-pod-autoscaler
      role: admission-controller
  template:
    metadata:
      labels:
        app: vpa-admission-controller
        product: foundation
        project: vertical-pod-autoscaler
        role: admission-controller
        team: compute
        datadog_log_index: foundation
    spec:
      serviceAccountName: vpa-admission-controller
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534 # nobody
      containers:
      - name: admission-controller
        image: replaced_by_samson/vpa-admission-controller
        imagePullPolicy: Always
        args:
        - --register-webhook=false # we manage it manually with cert-controller
        - --client-ca-file=/etc/tls-certs/ca.crt
        - --tls-cert-file=/etc/tls-certs/tls.crt
        - --tls-private-key=/etc/tls-certs/tls.key
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: tls-certs
          mountPath: "/etc/tls-certs"
          readOnly: true
        resources:
          limits:
            cpu: 200m
            memory: 500Mi
          requests:
            cpu: 50m
            memory: 200Mi
        ports:
        - containerPort: 8000
        - name: prometheus
          containerPort: 8944
      - name: cert-controller
        image: gcr.io/docker-images-180022/apps/cert-controller-sidecar # https://github.com/zendesk/cert-controller-sidecar, it runs forever and keeps watching to see if need to cert refresh
        imagePullPolicy: Always
        samson/dockerfile: none
        args:
        - -cert-dir=/etc/tls-certs
        - -ca-name=vpa-webhook.kube-system.svc
        - -secret-name=vpa-tls-certs
        - -service-name=vpa-webhook
        - -ca-organization=fooorg
        - -namespace=kube-system
        - -dns-name=vpa-webhook.kube-system.svc
        - -webhook-name=vpa-webhook-config
        resources:
          limits:
            cpu: 200m
            memory: 500Mi
          requests:
            cpu: 25m
            memory: 128Mi
        volumeMounts:
        - name: tls-certs
          mountPath: "/etc/tls-certs"
          readOnly: true
      volumes:
      - name: tls-certs
        secret:
          secretName: vpa-tls-certs

---
apiVersion: v1
kind: Service
metadata:
  name: vpa-webhook
  namespace: kube-system
  labels:
    product: foundation
    project: vertical-pod-autoscaler
    role: admission-controller
    team: compute
spec:
  ports:
  - port: 443
    targetPort: 8000
  selector:
    project: vertical-pod-autoscaler
    role: admission-controller

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:vpa-admission-controller
  labels:
    product: foundation
    project: vertical-pod-autoscaler
    role: admission-controller
    team: compute
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - configmaps
  - nodes
  - limitranges
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "poc.autoscaling.k8s.io"
  resources:
  - verticalpodautoscalers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "autoscaling.k8s.io"
  resources:
  - verticalpodautoscalers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "coordination.k8s.io"
  resources:
  - leases
  verbs:
  - create
  - update
  - get
  - list
  - watch
- apiGroups: # for cert-controller
  - ""
  resources:
  - secrets
  verbs:
  - get
  - update
  - patch
  - list
  - watch # https://github.com/open-policy-agent/cert-controller/issues/24
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  verbs:
  - get
  - patch
  - update
  - list
  - watch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:vpa-admission-controller
  labels:
    product: foundation
    project: vertical-pod-autoscaler
    role: admission-controller
    team: compute
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:vpa-admission-controller
subjects:
- kind: ServiceAccount
  name: vpa-admission-controller
  namespace: kube-system

---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: vpa-webhook-config
  labels:
    product: foundation
    team: compute
    project: vertical-pod-autoscaler
    role: admission-controller
  annotations:
    samson/server_side_apply: "true" # do not override cabundle when deploying
webhooks:
- name: vpa.k8s.io
  objectSelector: # only looking at opted-in resources to reduce overhead and blast radius when things go wrong
    matchLabels:
      compute.zende.sk/vpa-opt-in: "true"
  failurePolicy: Ignore
  admissionReviewVersions: ["v1beta1"]
  rules:
  - apiGroups:   [""]
    apiVersions: ["v1"]
    operations:  ["CREATE"]
    resources:   ["pods"]
  - apiGroups: [ "autoscaling.k8s.io" ]
    apiVersions: [ "*" ]
    operations: [ "CREATE", "UPDATE" ]
    resources: [ "verticalpodautoscalers" ]
  clientConfig:
    service:
      namespace: kube-system
      name: vpa-webhook
  sideEffects: None
  timeoutSeconds: 15

---
apiVersion: v1
kind: Secret
metadata:
  labels:
    project: vertical-pod-autoscaler
    team: compute
    role: admission-controller
    product: foundation
  name: vpa-tls-certs
  namespace: kube-system
  annotations:
    samson/server_side_apply: 'true' # do not override data when deploying

[bskiba]

We set limit proportional to request: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#keeping-limit-proportional-to-request

Can you check your deployment settings?

[eatwithforks]

Right, I understand the theoretical 2:1 ratio for limits to requests but let's look at the behavior in actual for my pod.

My deployment has:

memory request: 512Mi
memory limit: 4 Gi

VPA request recommendations are:

target memory: 4066212754 (~= 4Gi)

My pod after it's been evicted and recreated by the admission-controller has:

memory: "32529702032"** (~=30 Gi)

That's not a 2:1 ratio of limits to requests, right?

The full manifest details are in the initial ticket description

[eatwithforks]

This datadog notebook shows the behavior before VPA's recommend mode was switched from "Off" to "Auto" You can clearly see the memory limit value rocket to 30 Gi while the actual usage and request remains constantly low.

[bskiba]

Apologies, I totally missed that the info was already there.

We are not sticking to 2:1 ratio but rather take the ratio from the deployment.

If the config is

memory request: 512Mi
memory limit: 4 Gi

That is 8:1 ratio, so if target recommendation is ~4Gi, limit will be set to ~32Gi which seems to be aligned.

[eatwithforks]

ahh i see now. that's good to know! i'll tune the deployment limit/request to be 2:1 ratio

[bskiba]

Let me know if that works and also if there's a way to improve the docs, feel free to file an issue/contribute.

[bskiba]

/close

[k8s-ci-robot]

@bskiba: Closing this issue.

In response to [this](https://github.com/kubernetes/autoscaler/issues/3930#issuecomment-794206721): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.