[Bug]: Too many Federated Identity Credential requests

[KarlisAG]

Is there an existing issue for this?

• [x] I have searched the existing issues

Affected Resource(s)

resources.azure.upbound.io/v1beta1 - ResourceGroupTemplateDeployment

Resource MRs required to reproduce the bug

No response

Steps to Reproduce

1. Have ResourceGroupTemplateDeployment that creates multiple Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials (we have aprox. 300) and have it be deployed within AKS cluster
2. Have the same in multiple other AKS clusters in the same tenant, subscription and resource group

What happened?

In some cases it works as expected and all federated credentials are made. But because we have multiple clusters and many requests sometimes one of federated credential creation fails leading to the whole deployment failing which leads to the remaining federated credentials not being made and then failing our other dependencies.

Relevant Error Output Snippet

In failed cases the following error appears in Azure:
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity","message":"Too many Federated Identity Credentials are written concurrently for the managed identity '/subscriptions/xxxx/resourcegroups/xxxx/providers/microsoft.managedidentity/userassignedidentities/xxxx'. Concurrent Federated Identity Credentials writes under the same managed identity are not supported."}]}

And when in Azure looking up "Related events" for these failed ones we also noticed the following error:
"Error code: 429,
Message: Request is temporarily throttled because subscription xxxx has issued too many requests. Retry after 2 seconds."

Crossplane Version

v1.14.5

Provider Version

v0.42.0

Kubernetes Version

1.27.7

Kubernetes Distribution

AKS

Additional Info

I have a question - is there a way we can nicely avoid or spread out these requests? Current workaround, to not be blocked, was to make another such ResourceGroupTemplateDeployment with different name to force it to deploy again and that usually worked, if not, then we did couple more retries until it did. We also thought of adding timestamp to ResourceGroupTemplateDeployment, but then realized it probably wouldn't work as the Helm chart that manages it won't be remaking it by itself. Are there any suggestions on what we could do about this?

[turkenf]

Hi @KarlisAG, thank you for raising this issue. Could you please try it with the latest provider version and provide us with example MRs if the problem occurs again?

[github-actions[bot]]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions[bot]]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!