kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.03k stars 3.95k forks source link

VPA updater errors with messages ~"fail to get pod controller: pod=kube-scheduler-XYZ err=Unhandled targetRef v1 / Node / XYZ, last error node is not a valid owner" #7378

Open apilny-akamai opened 1 day ago

apilny-akamai commented 1 day ago

Which component are you using?: vertical-pod-autoscaler

What version of the component are you using?: 1.1.2

Component version:

What k8s version are you using (kubectl version)?: kubectl 1.25

What did you expect to happen?: VPA updater does not error with fail to get pod controller: pod=kube-scheduler-XYZ err=Unhandled targetRef v1 / Node / XYZ, last error node is not a valid owner

What happened instead?: vpa-updater log contains ` │ E1010 12:38:44.476232 1 api.go:153] fail to get pod controller: pod=kube-apiserver-x-master-1 err=Unhandled targetRef v1 / Node / x-master-1, last error node is not a valid owner │

│ E1010 12:38:44.477788 1 api.go:153] fail to get pod controller: pod=kube-controller-manager-master-1 err=Unhandled targetRef v1 / Node / x-master-1, last error node is not a valid owner │

│ E1010 12:38:44.547767 1 api.go:153] fail to get pod controller: pod=etcd-x-master-1 err=Unhandled targetRef v1 / Node / x-master-1, last error node is not a valid owner │

│ E1010 12:38:44.554646 1 api.go:153] fail to get pod controller: pod=kube-scheduler-x-master-1 err=Unhandled targetRef v1 / Node / x-master-1, last error node is not a valid owner │ `

How to reproduce it (as minimally and precisely as possible): Update VPA from 0.4 to 1.1.2 and observ the vpa-updater log.

Anything else we need to know?: I've tried to update to 1.2.1 and the error is in the log again. Did not happen with vpa 0.4. I can see this error message also in already fixed issue with panic/SIGSEGV problem but nowhere else.

Example Pod Spec

spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: kube-controller-manager
  updatePolicy:
    updateMode: Auto
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 25m
          memory: 50Mi
        maxAllowed:
          cpu: {{ index .Values.MaxAllowedVPA "controller-manager-vpa" "cpu" }}
          memory: {{ index .Values.MaxAllowedVPA "controller-manager-vpa" "memory" }}
        controlledResources: ["cpu", "memory"]
adrianmoisey commented 1 day ago

/area vertical-pod-autoscaler

adrianmoisey commented 1 day ago

Would it be possible to see the spec of the Pod that this is failing on? Which variant of Kubernetes are you running this on?

adrianmoisey commented 1 day ago

/triage needs-information

apilny-akamai commented 17 hours ago

We use standard kubeadm, K8s Rev: v1.25.16. I've updated description with an example Pod Spec.

adrianmoisey commented 14 hours ago

Hi. It seems like you added the VPA spec. I'm looking for the spec of the Pod kube-controller-manager-master-1