search

kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.05k stars 3.97k forks source link

VPA - Installer not working #1475

Closed ptaylor10 closed 5 years ago

ptaylor10 commented 5 years ago

I am trying out Vertical Pod Autoscaler and haven't had any luck with deploying it. I am using a new Cluster on Google Cloud with the following kubernetes versions:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.2-gke.18", GitCommit:"5796233393d7bc034428de15191ad3d2eaff95fb", GitTreeState:"clean", BuildDate:"2018-11-08T20:49:08Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}

However, I am seeing these as an errors and it looks like the vpa isn't working correctly as well:

customresourcedefinition.apiextensions.k8s.io "verticalpodautoscalers.autoscaling.k8s.io" created
customresourcedefinition.apiextensions.k8s.io "verticalpodautoscalercheckpoints.autoscaling.k8s.io" created
clusterrolebinding.rbac.authorization.k8s.io "system:metrics-reader" created
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-actor" created
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" created
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-updater-controllers-reader-binding" created
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-evictionter-binding" created
serviceaccount "vpa-admission-controller" created
clusterrolebinding.rbac.authorization.k8s.io "system:admission-controller" created
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:metrics-reader" is forbidden: attempt to grant extra privileges: [{[get] [metrics.k8s.io] [pods] [] []} {[list] [metrics.k8s.io] [pods] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou/rOIx33lzgER/ZrKY/aW5p9nv/qrxHG/JuYiwdrC3wtLbopLBN3Orqg7z+X+2fM6E2EvdLCBnLBTaF5nSfUPavY20lr4h9zXVMvmt38e0T1UX1zTxgQS+1/69TMDR0xYIIjV1uxVw6riiEQ++04EiiOANgc+uE9R4TGlz82nuaUNs/FVKhz3Ov7AnY6aFlBc4CJrHitMfjCx5xA06x0Rwd2DG0bTvQOCyd5Q==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-actor" is forbidden: attempt to grant extra privileges: [{[get] [] [pods] [] []} {[list] [] [pods] [] []} {[watch] [] [pods] [] []} {[get] [] [nodes] [] []} {[list] [] [nodes] [] []} {[watch] [] [nodes] [] []} {[get] [] [events] [] []} {[list] [] [events] [] []} {[watch] [] [events] [] []} {[create] [] [events] [] []} {[get] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[list] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[watch] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[patch] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[get] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[list] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[watch] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[patch] [autoscaling.k8s.io] [verticalpodautoscalers] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou/rOIx33lzgER/ZrKY/aW5p9nv/qrxHG/JuYiwdrC3wtLbopLBN3Orqg7z+X+2fM6E2EvdLCBnLBTaF5nSfUPavY20lr4h9zXVMvmt38e0T1UX1zTxgQS+1/69TMDR0xYIIjV1uxVw6riiEQ++04EiiOANgc+uE9R4TGlz82nuaUNs/FVKhz3Ov7AnY6aFlBc4CJrHitMfjCx5xA06x0Rwd2DG0bTvQOCyd5Q==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" is forbidden: attempt to grant extra privileges: [{[get] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[list] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[watch] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[create] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[patch] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[delete] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[get] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[list] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[watch] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[create] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[patch] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[delete] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[get] [] [namespaces] [] []} {[list] [] [namespaces] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou/rOIx33lzgER/ZrKY/aW5p9nv/qrxHG/JuYiwdrC3wtLbopLBN3Orqg7z+X+2fM6E2EvdLCBnLBTaF5nSfUPavY20lr4h9zXVMvmt38e0T1UX1zTxgQS+1/69TMDR0xYIIjV1uxVw6riiEQ++04EiiOANgc+uE9R4TGlz82nuaUNs/FVKhz3Ov7AnY6aFlBc4CJrHitMfjCx5xA06x0Rwd2DG0bTvQOCyd5Q==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:evictioner" is forbidden: attempt to grant extra privileges: [{[get] [extensions] [replicasets] [] []} {[create] [] [pods/eviction] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou/rOIx33lzgER/ZrKY/aW5p9nv/qrxHG/JuYiwdrC3wtLbopLBN3Orqg7z+X+2fM6E2EvdLCBnLBTaF5nSfUPavY20lr4h9zXVMvmt38e0T1UX1zTxgQS+1/69TMDR0xYIIjV1uxVw6riiEQ++04EiiOANgc+uE9R4TGlz82nuaUNs/FVKhz3Ov7AnY6aFlBc4CJrHitMfjCx5xA06x0Rwd2DG0bTvQOCyd5Q==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:controllers-reader" is forbidden: attempt to grant extra privileges: [{[get] [] [replicationcontrollers] [] []} {[list] [] [replicationcontrollers] [] []} {[watch] [] [replicationcontrollers] [] []} {[get] [apps] [statefulsets] [] []} {[list] [apps] [statefulsets] [] []} {[watch] [apps] [statefulsets] [] []} {[get] [apps] [replicasets] [] []} {[list] [apps] [replicasets] [] []} {[watch] [apps] [replicasets] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou/rOIx33lzgER/ZrKY/aW5p9nv/qrxHG/JuYiwdrC3wtLbopLBN3Orqg7z+X+2fM6E2EvdLCBnLBTaF5nSfUPavY20lr4h9zXVMvmt38e0T1UX1zTxgQS+1/69TMDR0xYIIjV1uxVw6riiEQ++04EiiOANgc+uE9R4TGlz82nuaUNs/FVKhz3Ov7AnY6aFlBc4CJrHitMfjCx5xA06x0Rwd2DG0bTvQOCyd5Q==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:admission-controller" is forbidden: attempt to grant extra privileges: [{[get] [] [pods] [] []} {[list] [] [pods] [] []} {[watch] [] [pods] [] []} {[get] [] [configmaps] [] []} {[list] [] [configmaps] [] []} {[watch] [] [configmaps] [] []} {[get] [] [nodes] [] []} {[list] [] [nodes] [] []} {[watch] [] [nodes] [] []} {[create] [admissionregistration.k8s.io] [mutatingwebhookconfigurations] [] []} {[delete] [admissionregistration.k8s.io] [mutatingwebhookconfigurations] [] []} {[get] [admissionregistration.k8s.io] [mutatingwebhookconfigurations] [] []} {[list] [admissionregistration.k8s.io] [mutatingwebhookconfigurations] [] []} {[get] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[list] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[watch] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[get] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[list] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[watch] [autoscaling.k8s.io] [verticalpodautoscalers] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou/rOIx33lzgER/ZrKY/aW5p9nv/qrxHG/JuYiwdrC3wtLbopLBN3Orqg7z+X+2fM6E2EvdLCBnLBTaF5nSfUPavY20lr4h9zXVMvmt38e0T1UX1zTxgQS+1/69TMDR0xYIIjV1uxVw6riiEQ++04EiiOANgc+uE9R4TGlz82nuaUNs/FVKhz3Ov7AnY6aFlBc4CJrHitMfjCx5xA06x0Rwd2DG0bTvQOCyd5Q==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
serviceaccount "vpa-updater" created
deployment.extensions "vpa-updater" created
serviceaccount "vpa-recommender" created
deployment.extensions "vpa-recommender" created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus

I also noticed these log errors on the vpa pod:

E1204 16:30:17.361649      10 reflector.go:134] k8s.io/autoscaler/vertical-pod-autoscaler/pkg/utils/vpa/api.go:89: Failed to list *v1beta1.VerticalPodAutoscaler: verticalpodautoscalers.autoscaling.k8s.io is forbidden: User "system:serviceaccount:kube-system:vpa-admission-controller" cannot list verticalpodautoscalers.autoscaling.k8s.io at the cluster scope: RBAC: clusterrole.rbac.authorization.k8s.io "system:admission-controller" not found
I1204 16:30:18.362369      10 reflector.go:169] Listing and watching *v1beta1.VerticalPodAutoscaler from k8s.io/autoscaler/vertical-pod-autoscaler/pkg/utils/vpa/api.go:89
bskiba commented 5 years ago

Please take a look at this excerpt from the prerequisites in readme and let me know if this fixes your problem:

If you are using a GKE Kubernetes cluster, you will need to grant your current Google identity cluster-admin role. Otherwise you won't be authorized to grant extra privileges to the VPA system components.

$ gcloud info | grep Account    # get current google identity
Account: [myname@example.org]

$ kubectl create clusterrolebinding myname-cluster-admin-binding --clusterrole=cluster-admin --user=myname@example.org
Clusterrolebinding "myname-cluster-admin-binding" created
ptaylor10 commented 5 years ago

I did run that prior to vpa-up.sh and it looks like the clusters roles were created; however, then they are denied and forbidden access. I also given the service account Kubernetes-Cluster-Admin rights as well.

Here is the full messages I get when I run vpa-up.sh

customresourcedefinition.apiextensions.k8s.io "verticalpodautoscalers.autoscaling.k8s.io" created
customresourcedefinition.apiextensions.k8s.io "verticalpodautoscalercheckpoints.autoscaling.k8s.io" created
clusterrolebinding.rbac.authorization.k8s.io "system:metrics-reader" created
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-actor" created
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" created
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-updater-controllers-reader-binding" created
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-evictionter-binding" created
serviceaccount "vpa-admission-controller" created
clusterrolebinding.rbac.authorization.k8s.io "system:admission-controller" created
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:metrics-reader" is forbidden: attempt to grant extra privileges: [{[get] [metrics.k8s.io] [pods] [] []} {[list] [metrics.k8s.io] [pods] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou+nymRus51hZrGpnKZp+wumChHD1DUAkO0V9+1+3tplSI/bcsKf9Vrmko5bjE6BdoroAgMj+UpBvPOTkN/rXsR3TW6/YDFOLTxh5tlUUjhjQKig8p1zVxfrWb8D0zGd02/vTI1b7PRSwN0LQqu3Tr2e4fuhv30Nf1SvSW0euzB+avtlzeBmzETBQYsFv58q/MNYhKQIRyuwDtMJaaxU0e3iwwB8xCeaELTWew==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-actor" is forbidden: attempt to grant extra privileges: [{[get] [] [pods] [] []} {[list] [] [pods] [] []} {[watch] [] [pods] [] []} {[get] [] [nodes] [] []} {[list] [] [nodes] [] []} {[watch] [] [nodes] [] []} {[get] [] [events] [] []} {[list] [] [events] [] []} {[watch] [] [events] [] []} {[create] [] [events] [] []} {[get] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[list] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[watch] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[patch] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[get] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[list] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[watch] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[patch] [autoscaling.k8s.io] [verticalpodautoscalers] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou+nymRus51hZrGpnKZp+wumChHD1DUAkO0V9+1+3tplSI/bcsKf9Vrmko5bjE6BdoroAgMj+UpBvPOTkN/rXsR3TW6/YDFOLTxh5tlUUjhjQKig8p1zVxfrWb8D0zGd02/vTI1b7PRSwN0LQqu3Tr2e4fuhv30Nf1SvSW0euzB+avtlzeBmzETBQYsFv58q/MNYhKQIRyuwDtMJaaxU0e3iwwB8xCeaELTWew==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" is forbidden: attempt to grant extra privileges: [{[get] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[list] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[watch] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[create] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[patch] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[delete] [poc.autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[get] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[list] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[watch] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[create] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[patch] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[delete] [autoscaling.k8s.io] [verticalpodautoscalercheckpoints] [] []} {[get] [] [namespaces] [] []} {[list] [] [namespaces] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou+nymRus51hZrGpnKZp+wumChHD1DUAkO0V9+1+3tplSI/bcsKf9Vrmko5bjE6BdoroAgMj+UpBvPOTkN/rXsR3TW6/YDFOLTxh5tlUUjhjQKig8p1zVxfrWb8D0zGd02/vTI1b7PRSwN0LQqu3Tr2e4fuhv30Nf1SvSW0euzB+avtlzeBmzETBQYsFv58q/MNYhKQIRyuwDtMJaaxU0e3iwwB8xCeaELTWew==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:evictioner" is forbidden: attempt to grant extra privileges: [{[get] [extensions] [replicasets] [] []} {[create] [] [pods/eviction] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou+nymRus51hZrGpnKZp+wumChHD1DUAkO0V9+1+3tplSI/bcsKf9Vrmko5bjE6BdoroAgMj+UpBvPOTkN/rXsR3TW6/YDFOLTxh5tlUUjhjQKig8p1zVxfrWb8D0zGd02/vTI1b7PRSwN0LQqu3Tr2e4fuhv30Nf1SvSW0euzB+avtlzeBmzETBQYsFv58q/MNYhKQIRyuwDtMJaaxU0e3iwwB8xCeaELTWew==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:controllers-reader" is forbidden: attempt to grant extra privileges: [{[get] [] [replicationcontrollers] [] []} {[list] [] [replicationcontrollers] [] []} {[watch] [] [replicationcontrollers] [] []} {[get] [apps] [statefulsets] [] []} {[list] [apps] [statefulsets] [] []} {[watch] [apps] [statefulsets] [] []} {[get] [apps] [replicasets] [] []} {[list] [apps] [replicasets] [] []} {[watch] [apps] [replicasets] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou+nymRus51hZrGpnKZp+wumChHD1DUAkO0V9+1+3tplSI/bcsKf9Vrmko5bjE6BdoroAgMj+UpBvPOTkN/rXsR3TW6/YDFOLTxh5tlUUjhjQKig8p1zVxfrWb8D0zGd02/vTI1b7PRSwN0LQqu3Tr2e4fuhv30Nf1SvSW0euzB+avtlzeBmzETBQYsFv58q/MNYhKQIRyuwDtMJaaxU0e3iwwB8xCeaELTWew==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
Error from server (Forbidden): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:admission-controller" is forbidden: attempt to grant extra privileges: [{[get] [] [pods] [] []} {[list] [] [pods] [] []} {[watch] [] [pods] [] []} {[get] [] [configmaps] [] []} {[list] [] [configmaps] [] []} {[watch] [] [configmaps] [] []} {[get] [] [nodes] [] []} {[list] [] [nodes] [] []} {[watch] [] [nodes] [] []} {[create] [admissionregistration.k8s.io] [mutatingwebhookconfigurations] [] []} {[delete] [admissionregistration.k8s.io] [mutatingwebhookconfigurations] [] []} {[get] [admissionregistration.k8s.io] [mutatingwebhookconfigurations] [] []} {[list] [admissionregistration.k8s.io] [mutatingwebhookconfigurations] [] []} {[get] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[list] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[watch] [poc.autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[get] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[list] [autoscaling.k8s.io] [verticalpodautoscalers] [] []} {[watch] [autoscaling.k8s.io] [verticalpodautoscalers] [] []}] user=&{118045764671172196470  [system:authenticated] map[user-assertion.cloud.google.com:[AK5xou+nymRus51hZrGpnKZp+wumChHD1DUAkO0V9+1+3tplSI/bcsKf9Vrmko5bjE6BdoroAgMj+UpBvPOTkN/rXsR3TW6/YDFOLTxh5tlUUjhjQKig8p1zVxfrWb8D0zGd02/vTI1b7PRSwN0LQqu3Tr2e4fuhv30Nf1SvSW0euzB+avtlzeBmzETBQYsFv58q/MNYhKQIRyuwDtMJaaxU0e3iwwB8xCeaELTWew==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]
serviceaccount "vpa-updater" created
deployment.extensions "vpa-updater" created
serviceaccount "vpa-recommender" created
deployment.extensions "vpa-recommender" created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus
....................+++++
....+++++
e is 65537 (0x010001)
Generating RSA private key, 2048 bit long modulus
...........+++++
.....................................................................................................................................................................................................+++++
e is 65537 (0x010001)
Signature ok
subject=CN = vpa-webhook.kube-system.svc
Getting CA Private Key
Uploading certs to the cluster.
secret "vpa-tls-certs" created
Deleting /tmp/vpa-certs.
deployment.extensions "vpa-admission-controller" created
service "vpa-webhook" created
ptaylor10 commented 5 years ago

I figured out the issue. It looks like it was using a different principal Email to make these changes. Instead of my service account email it was using it's uniqueID.

I had to do the following (in case someone else runs into this as well):

gcloud iam service-accounts describe [myname@example.org] | grep uniqueId