Seeing `cluster_feeder.go:423] Deleting Pod {xxxx}` in logs of recommender - even if VPA updatePolicy set to `Off` and only recommender is installed

[brian-provenzano]

This may be just misunderstanding the functionality, but we have goldilocks installed in recommendation mode only (using chart with only recommender enabled) and with this configuration we are seeing log entires in the recommender logs like thiskdp-core-vpa-goldilocks-recommender-7d68656bc6-bb994 vpa I0605 17:06:48.130341 1 cluster_feeder.go:423] Deleting Pod {cema distancecalc-7b78f6c4fd-m2tmk} even though the vpa updatePolicy for this deployment is set to Off. Are these actually deleting pods?

The deployments/pods running for goldilocks (no updater only recommender):

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
kdp-core-vpa-goldilocks-controller-controller   1/1     1            1           94d
kdp-core-vpa-goldilocks-recommender             1/1     1            1           94d

The VPA for the above log:

kc get vpa goldilocks-distancecalc                                                                                                                                                         
NAME                      MODE   CPU   MEM         PROVIDED   AGE
goldilocks-distancecalc   Off    15m   272061154   True       87d
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  creationTimestamp: "2023-03-10T16:27:00Z"
  generation: 85714
  labels:
    creator: Fairwinds
    source: goldilocks
  name: goldilocks-distancecalc
  namespace: cema
  resourceVersion: "1479464947"
  uid: 33618030-c9a5-4412-96c5-eade3d873943
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: distancecalc
  updatePolicy:
    updateMode: "Off"
status:
  conditions:
  - lastTransitionTime: "2023-03-13T21:16:15Z"
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: cema-batch-distancecalc
      lowerBound:
        cpu: 15m
        memory: "109812340"
      target:
        cpu: 15m
        memory: "272061154"
      uncappedTarget:
        cpu: 15m
        memory: "272061154"
      upperBound:
        cpu: 15m
        memory: "275047622"

I can provide more information if needed but generally the question is do these log entries indicate that the recommender is deleting pods.

[brian-provenzano]

OR is this just emitting information from metrics server that are being added to the vpa model in vpa recommender regarding events on the cluster (the deleting pod event`)? in other words these are just cluster events being captured?

[brian-provenzano]

that is exactly what it is doing:

Scale down deployment:

kc scale deployment testresources --replicas=0                                                                                                                                                              
deployment.apps/testresources scaled

vpa recommender emits this in logs (based on metrics server I assume) to update the model

I0605 18:20:52.625855       1 cluster_feeder.go:423] Deleting Pod {brianprovenzano testresources-84b79c8d4f-dbrtf}
I0605 18:20:52.625880       1 cluster_feeder.go:423] Deleting Pod {brianprovenzano testresources-84b79c8d4f-l7c92}

So this is a case of stupid user :)

I'll close this and leave it here in case someone else needs this info