search

kubernetes-sigs / cluster-api-provider-azure

Cluster API implementation for Microsoft Azure
https://capz.sigs.k8s.io/
Apache License 2.0
297 stars 428 forks source link

CAPZ with ASO doesn't work for AzureUSGovernment #4113

Open ionutleca opened 1 year ago

ionutleca commented 1 year ago

/kind bug

What steps did you take and what happened: The AzureManagedControlPlane supports the following parameter:

spec:
  azureEnvironment: AzureUSGovernmentCloud

The createSecretFromClusterIdentity function doesn't set azureResourceManagerEndpoint and any other cloud specific variables.

The ASO controller fails with: The subscription '***' could not be found.: PUT https://management.azure.com/subscriptions/***/resourceGroups/***

What did you expect to happen: CAPZ to also add to the *-aso-secret the values specific to what spec.azureEnvironment on the AzureManagedControlPlane resource points to.

Anything else you would like to add:

Environment:

Jont828 commented 1 year ago

/triage accepted

Jont828 commented 1 year ago

/assign @mboersma

nojnhuh commented 1 year ago

It appears ASO can only configure the ARM endpoint globally for all resources it manages. I've opened an issue here to make that capability accessible at the per-resource level to match CAPZ's capabilities from before: https://github.com/Azure/azure-service-operator/issues/3447.

If your management cluster is only managing workload clusters for one cloud, you could possibly modify the ASO deployment manually to configure it as you need as a stopgap.

ionutleca commented 1 year ago

Hi @nojnhuh, I was looking into how we could globally set the ARM endpoint url for the ASO controller but I can't find a way.

CAPZ is always setting the resource scoped secret https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/controllers/asosecret_controller.go#L235

ASO is not merging the resource and the global secret, but stops after getCredentialFromAnnotation https://github.com/Azure/azure-service-operator/blob/main/v2/internal/identity/credential_provider.go#L98-L99

Because of this, even if we do manually set the global or namespaced settings for the ASO controller, they will be ignored (unless I missed something) :(

nojnhuh commented 1 year ago

Even if changing it in aso-controller-settings doesn't work, I suppose it would be possible to edit the ASO deployment and hardcode the value there or referring to a different secret instead of deriving it from the global ASO secret.

nojnhuh commented 1 year ago

@ionutleca Did that workaround work for you? I'll reopen this to keep tracking making this more automatic.

ionutleca commented 1 year ago

Yes, I managed to make it work with the right values set in aso-controller-settings. Thank you!

CecileRobertMichon commented 11 months ago

/assign @nojnhuh /unassign @mboersma /cc @nawazkh

nojnhuh commented 11 months ago

Just opened this PR to at least allow setting these fields with environment variables when CAPZ is installed. Hopefully that's at least a step in the right direction: #4390

nojnhuh commented 9 months ago

There's still more to iron out here re: CAPZ configuring the environment per-workload cluster and ASO only configuring it globally for all resources it manages, but I don't think I'll be able to follow up with that during this milestone.

/unassign

dtzar commented 8 months ago

Blocked until https://github.com/Azure/azure-service-operator/issues/3447