updateMode: Initial not applied on re-deployment

[jseletz-rave]

I'm trying to use the VPA with the Initial update mode. I'm seeing the following behavior and trying to understand if I am using this feature incorrectly, if it's not applicable to my use case, or it there is an issue:

1. Deploy a Helm chart with Pod for a Tomcat web application, no VPA.
2. Undeploy the Helm release (helm delete)
3. Deploy the Helm chart with a VPA added, configured with updateMode: Initial
4. Note timing issue - VPA resource is created, but has no recommendation. Pod is created at the same time with the resource requests configured in the Helm chart. The VPA then updates with the recommendations, but pod is not using them.
5. Manually restart the pod (delete the pod so it is automatically recreated), then the VPA recommendations are automatically applied to the pod.
6. I have tested with the VPA resource configured with a Helm pre-install hook. This does not resolve the timing issue. I did notice that when undeploying the Helm release, the VPA is not deleted (as expected), but with the pod removed, the VPA's pod selector doesn't find it anymore, and the recommendations are removed from the VPA. So next deployment does not apply them on pod creation, as the VPA doesn't get updated with recommendations until after the pod is created.
7. Deploying with updateMode: Auto shows a similar behavior - pod and vpa are created, pod has configured resource requests (not from VPA), VPA is updated with recommendations, and then a few seconds later the pod is evicted and re-deployed by the updater with the recommended resources.

Seems like a chicken-egg timing problem - even if the VPA exists before the pod is created, the VPA isn't updated with recommendations until the pod is created, and then it's too late for the initial settings to be applied. How is this expected to work?

AWS EKS 1.22

My VPA configuration:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myApp
spec:
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        controlledResources:
          - cpu
          - memory
  targetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: myApp
  updatePolicy:
    updateMode: Initial

[voelzmo]

Hey @jseletz-rave, thanks for the detailed description! What you're describing seems to be related to how VPA works: it uses PodMetrics from the metrics-server which include measurements of the Pod's current CPU and memory usage to calculate a new recommendation for requests/limits. If the Pod is not running, VPA cannot get any metrics and cannot calculate a new recommendation. As the model used for calculating a recommendation gets more accurate with more data, the Pod may even need to run for quite a while before you get "good" recommendations.

If you're looking for a way to update your Pod's resource requests on the first deploy probably the best option is to regularly write the updated recommendations back to your git repo which contains the Deployment yamls for your Pods?

[jseletz-rave]

Thanks @voelzmo for the response. I am using the VPA because we deploy the same apps to multiple namespaces and the usage/load in each namespace is different. So saving the recommendations to the yamls doesn't really help me.

I still would like to understand what is the use case/workflow for using the Initial mode and how exactly deploying a VPA with this updateMode could ever cause a pod to be updated? Based on my testing I don't see how it can ever work. Deploying the VPA before the pod doesn't work (as I have described), and deploying the VPA after the pod is too late since the pod is already running.

[jbartosik]

Something else might delete pods:

• Manual action (kubectl delete pod). You might want to do this so that pods are deleted when you know its safe to do that. For example you want to check you're not serving heavy load and you want to check recommendations before applying
• You might want recommendations to be applied only when you need to restart them anyways, for a different reason (for example update).

[kundan2707]

/kind support