search

kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.07k stars 3.97k forks source link

GCP: InstanceTemplates are no longer `global` resources #7396

Open 4wdonny opened 3 weeks ago

4wdonny commented 3 weeks ago

Which component are you using?: Cluster Autoscaler

cluster-autoscaler What version of the component are you using?: 1.29.4

Component version: 1.29.4

What k8s version are you using (kubectl version)?:

v1.29.8-gke.1096000

kubectl version Output
$ kubectl version

What environment is this in?: Google

GCP has changed the location of instance templates...they used to be Global and now they are regional. Is this configurable in the Autoscaler?

4wdonny commented 3 weeks ago

Last night a node pool was upgraded to 1.29.8-gke.1096000 and the underlying instance template for the MIG changed from a global one to a regional one.

4wdonny commented 3 weeks ago

I've tried master, and even creating a new node spot node pool, and it looks like this is broken. The instanceTemplate regional logic is not being called correctly.

Shubham82 commented 3 weeks ago

/area provider/gce /area cluster-autoscaler

Shubham82 commented 3 weeks ago

cc @BigDarkClown @x13n

aaktaev commented 2 weeks ago

Same for k8s=1.29.9-gke.1177000 and ClusterAutoscaler appVersion=v1.29.3:

Unable to build proper template node for https://www.googleapis.com/compute/v1/projects/$PROJECT/zones/europe-west1-b/instanceGroups/gke-k8s-system-grp: googleapi: Error 404: The resource 'projects/$PROJECT/global/instanceTemplates/gke-k8s-system-7777777' was not found, notFound
x13n commented 2 weeks ago

This has been fixed in CA 1.29 (https://github.com/kubernetes/autoscaler/pull/7231), but not 1.28. I just created1.28 cherrypick here: https://github.com/kubernetes/autoscaler/pull/7409

4wdonny commented 2 weeks ago

@x13n I attempted using CA 1.29 and encountered the same issue.

x13n commented 2 weeks ago

It was backported but not released yet. The release is happening right now