search

kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.09k stars 3.98k forks source link

Unable to get cluster autoscaler to work in Azure Govcloud environment #7078

Open jmlaubach opened 3 months ago

jmlaubach commented 3 months ago

Which component are you using?:

What version of the component are you using?:

v1.27.2

What k8s version are you using (kubectl version)?:

v1.27.2

What environment is this in?:

Cluster autoscaler deployed on VMSS in Azure Govcloud

What did you expect to happen?:

When viewing the cluster autoscaler logs, I expect the cluster autoscaler to discover the VMSSs within the Azure Govcloud using the Azure Govcloud REST API endpoint at management.usgovcloudapi.net

What happened instead?:

I see in the logs the first REST API GET request it attempts is at management.azure.com. This gives a 400 auth error complaining about a cross cloud request.

How to reproduce it (as minimally and precisely as possible):

  1. Use the cluster-autoscaler-vmss.yaml in the Azure examples
  2. Along with the other ENV variables, set the ARM_ENVIRONMENT to base64 encoded "AzureUSGovernmentCloud" (have also tried us base64 encoded value of "usgovernment" which also does not work)
  3. Deploy into cluster and note the errors in the cluster autoscaler logs explained above

Anything else we need to know?:

I'm not sure why I can't get it to deploy in a cluster within Azure Govcloud. It would be great to know if I'm doing something wrong or there is an actual issue. Nothing I do seems to get the cluster autoscaler to understand it needs to hit the Govcloud REST API endpoint and not Commercial. I have confirmed the following:

  1. I have confirmed that all my ENV variables are set to the correct values in the secret (subscription ID, client id/secret, etc)
  2. Under the command section in my Deployment, I have set --cloud-provider=azure and have ensured I have the --node-group-auto-discovery set
  3. The tags set in --node-group-auto-discovery are confirmed to be set on my VMSS in Azure
Shubham82 commented 3 months ago

/area cluster-autoscaler /area provider/azure

marc-sensenich commented 1 month ago

Hey @jmlaubach I don't currently have access to an Azure US Government account to test this. But have you attempted setting the environment variable ARM_CLOUD to be equal to the value AzureUSGovernmentCloud looking at the 1.27 release for Azure config that should be the value to set to configure the Azure autoscaler provider to connect to Azure US Government.