Azure / azure-service-operator

Azure Service Operator allows you to create Azure resources using kubectl
https://azure.github.io/azure-service-operator/
MIT License
738 stars 194 forks source link

Remove or relax self throttling, or make it configurable #4150

Closed matthchr closed 2 months ago

matthchr commented 3 months ago

Describe the current behavior We self-throttle to avoid ARM throttling: https://github.com/Azure/azure-service-operator/blob/807a2bb52d0cbc9f4db424fb0fc30ca2bcbacda9/v2/internal/reconcilers/generic/generic_reconciler.go#L278-L295

Describe the improvement ARM has since changed how they throttle, and we have moved to support multiple identities/multiple subscriptions, which means that this simple bucket-based throttling is no-longer sufficient.

We should either remove the self-throttling entirely, OR make it larger and possibly configurable. I tend towards removing it given that:

matthchr commented 3 months ago

We also might want to look at adding some verbose logging to make it more obvious if/when throttling is kicking in. Or possibly there already is some in controller-runtime, which if there is we should document how to get at it.

matthchr commented 3 months ago

See https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#migrating-to-regional-throttling-and-token-bucket-algorithm