Allow targeting individual resources for specific Azure clouds (AzurePublicCloud, AzureUSGovernmentCloud, etc.)

[nojnhuh]

Describe the current behavior Currently, ASO exposes a single azureResourceManagerEndpoint value in the Helm chart which applies to all resources managed by that ASO instance.

Describe the improvement ASO should expose a way to deploy individual resources to specific clouds.

Additional context Add any other context about the suggested improvement.

[nojnhuh]

A similar mechanism to per-resource clouds could also potentially apply to a particular namespace.

[theunrepentantgeek]

What's the use case where a single cluster would be overseeing/managing resources across different clouds?

To this point, my expectation has been that a cluster managing resources in one of the non-public clouds would of necessity also be running within that cloud for security reasons. At first glance, crossing the security boundary between clouds seems as though it may be an issue.

[nojnhuh]

What's the use case where a single cluster would be overseeing/managing resources across different clouds?

I admit I don't know if this is actually what we need or if the main idea is only to make it possible to use different clouds at all. This seems to be the most relevant context I can find from CAPZ making that config local to each workload cluster and I don't see anything implying this was to enable managing resources in different clouds simultaneously: https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/1244.

Being able to specify this per-resource in ASO was mostly a selfish ask because that would likely be easiest to utilize in CAPZ without introducing breaking changes.

To this point, my expectation has been that a cluster managing resources in one of the non-public clouds would of necessity also be running within that cloud for security reasons. At first glance, crossing the security boundary between clouds seems as though it may be an issue.

This makes me wonder if there is perhaps a more sensible place to expose this in CAPZ.

@CecileRobertMichon Do you know if it's a requirement from CAPZ users or even possible to deploy workload clusters to different Azure clouds from a single management cluster?

[ionutleca]

@nojnhuh, the initial issue in the CAPZ repo was there because we couldn't target Azure Gov clouds at all (without any hacks). The 1 controller:N clouds capability is not really that important to us either.

[theunrepentantgeek]

I admit I don't know if this is actually what we need or if the main idea is only to make it possible to use different clouds at all.

I think ASO already has full support for the various available Azure clouds.

As you've noted, our helm chart has the requisite variables - and a review of config.go shows that we support the following environment variables:

• AZURE_RESOURCE_MANAGER_ENDPOINT - ResourceManagerEndpoint is the Azure Resource Manager endpoint. If not specified, the default is the Public cloud resource manager endpoint.
• AZURE_RESOURCE_MANAGER_AUDIENCE - ResourceManagerAudience is the Azure Resource Manager AAD audience. If not specified, the default is the Public cloud resource manager audience https://management.core.windows.net/.
• AZURE_AUTHORITY_HOST - AzureAuthorityHost is the URL of the AAD authority. If not specified, the default is the AAD URL for the public cloud: https://login.microsoftonline.com/.

Maybe a similar approach (environment variables + public cloud defaults) would work for CAPZ?

[nojnhuh]

Maybe a similar approach (environment variables + public cloud defaults) would work for CAPZ?

That seemed to be the approach we took until a couple years ago with https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/1244. @CecileRobertMichon Is there still a requirement that Azure Cloud needs to be specified per-cluster?

[CecileRobertMichon]

@nojnhuh today it's possible (before this regression) to deploy workload clusters in different clouds from the same management cluster with CAPZ. I don't know whether there are any users who actively use this feature, but regardless removing it would be a breaking change/regression. The reason it was initially removed from environment variables in the PR you linked above seems to be to not require envsubst to install CAPZ (CAPZ can now be installed without an Azure account / credentials) and all environment substitution happens at the workload cluster manifest apply step.

Seems like the current regression is that users can't create clusters in other clouds at all which is itself more critical than the multi-cloud aspect and we should focus on getting that fixed first though.

[nojnhuh]

@theunrepentantgeek Is it possible to change those environment variables in aso-controller-settings after ASO has started up and could we expect those to take effect while using per-resource secrets for credentials? That seems like a potentially reasonable workaround for now.

[mjnovice]

@CecileRobertMichon @nojnhuh we are actually using multiple clouds from the same management plane. So, we would need this. 🙏

[matthchr]

@theunrepentantgeek Is it possible to change those environment variables in aso-controller-settings after ASO has started up and could we expect those to take effect while using per-resource secrets for credentials? That seems like a potentially reasonable workaround for now.

You can if you restart the ASO pod afterwards.

[matthchr]

We should do an ADR on this, describing how we could solve it (put cloud into credential?)

[matthchr]

We synced up with the CAPZ folks on this and determined the following:

• We need an ADR (see above).
• We need to check with CAPZ if per-namespace is good enough or if they also need per-resource. It seems likely we need per-resource, and in any case ASO currently doesn't use a different secret configuration for per-namespace or per-resource so from our side it might be harder to do only per-namespace if that's what we want.
• CAPZ's opinion was, we should do this, but it's not urgent because there is a somewhat hacky workaround.