search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.48k stars 4.81k forks source link

GetServiceTags throws rate limit exceptions #45317

Closed TibbsTerry closed 3 months ago

TibbsTerry commented 3 months ago

Library name and version

Azure.ResourceManager.Network Version 1.7.0

Describe the bug

Azure.ResourceManager.Network.NetworkExtensions.GetServiceTagAsync throws rate limit exceptions at way below the documented limits.

GetServiceTag /subscriptions/{subscriptionId}/providers/Microsoft.Network/locations/{location}/serviceTags throws the following exception

 Azure.RequestFailedException: A retryable error occurred. 
 Status: 429 ErrorCode: RetryableError  Content: {
   "error": {
     "code": "RetryableError",
     "message": "A retryable error occurred.",
     "details": [
       {
         "code": "RetryableErrorDueToTooManyCallsToThisOperation",
         "message": "Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 305 seconds."
       }
     ]
   }
 }

Expected behavior

Actual behavior

A call frequency of 1 per second will cause the rate limit exception to occur

Reproduction Steps

/*
Example to demonstrate rate limit issue for GetServiceTag.
Documentation shows a limit of 25 per sec should be achievable
Here a request frequency of 1 per second is used for two routes
/subscriptions/{subscriptionId}/providers/Microsoft.Network/virtualNetworks
/subscriptions/{subscriptionId}/providers/Microsoft.Network/locations/{location}/serviceTags
GetVirtualNetworks works without error
GetServiceTags shows multiple throttling limit exceptions

// https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#migrating-to-regional-throttling-and-token-bucket-algorithm
// Subscription Refill rate == 25 per sec

// https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#network-throttling
// Microsoft.Network
// read (GET) 10000 per 5 minutes == 33 calls per sec

*/

/*
<PackageReference Include="Azure.Identity" Version="1.10.4" />
<PackageReference Include="Azure.ResourceManager.Network" Version="1.7.0" />
*/

using Azure.Core;
using Azure.ResourceManager;
using Azure.ResourceManager.Network;
using Azure.ResourceManager.Resources;
using System.Diagnostics;
using System.Text.RegularExpressions;
using Azure;
using Azure.Identity;

namespace AzureGetServiceTagIssue;

internal class Program
{
    private static readonly TimeSpan Frequency = TimeSpan.FromSeconds(1);

    static async Task Main()
    {
        string tenantId = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX";
        var authorityHost = new Uri($"{AzureAuthorityHosts.AzurePublicCloud}{tenantId}/");
        string subscriptionId = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX";

        Console.WriteLine("Start");

        var subscription = await RetrieveSubscription
            (authorityHost, tenantId, subscriptionId)
            .ConfigureAwait(false);

        await TestFunction(subscription, nameof(GetVirtualNetworks), GetVirtualNetworks);
        await TestFunction(subscription, nameof(GetServiceTags), GetServiceTags);

        Console.ReadLine();
    }

    private static async Task TestFunction
        (SubscriptionResource subscription, 
        string testName,
        Func<SubscriptionResource, Task<int>> testAction)
    {
        var sw = Stopwatch.StartNew();

        var calls = 0;
        var exceptions = 0;
        for (int i = 0; i < 30; i++)
        {
            calls++;
            await Task.Delay(Frequency).ConfigureAwait(false);
            Console.WriteLine($"{sw.Elapsed}: Calling {testName}");
            try
            {
                var count = await testAction(subscription).ConfigureAwait(false);
                Console.WriteLine($"{sw.Elapsed}: {count} count");
            }
            catch (RequestFailedException e) when (e.ErrorCode == "RetryableError")
            {
                exceptions++;
                var message = e.Message;
                var match = Regex.Match(message, "\"message\": \"(Operation GetServiceTagsOperation.*\")");
                if (match.Success)
                {
                    message = match.Groups[1].Value;
                }
                Console.WriteLine($"{sw.Elapsed}: {message}");
            }
        }

        Console.WriteLine("************************");
        Console.WriteLine($"Calls:{calls} Excpetions:{exceptions} Rate:{sw.ElapsedMilliseconds / 1000 / calls}sec per call");
        Console.WriteLine();
    }

    private static async Task<SubscriptionResource> RetrieveSubscription
        (Uri authorityHost,
        string tenantId,
        string subscriptionId)
    {
        var options = new DefaultAzureCredentialOptions
        {
            AuthorityHost = authorityHost,
            TenantId = tenantId
        };
        TokenCredential token = new DefaultAzureCredential(options);
        var client = new ArmClient(token, subscriptionId);

        var subscriptionIdentifier = new ResourceIdentifier($"/subscriptions/{subscriptionId}");
        var subscriptionResponse = await client
            .GetSubscriptionResource(subscriptionIdentifier)
            .GetAsync()
            .ConfigureAwait(false);
        return subscriptionResponse.Value;
    }

    // subscriptions/{subscriptionId}/providers/Microsoft.Network/virtualNetworks
    private static async Task<int> GetVirtualNetworks
        (SubscriptionResource subscription)
    {
        var result = 0;
        var pageable = subscription.GetVirtualNetworksAsync();
        await foreach (var _ in pageable.ConfigureAwait(false))
        {
            result++;
        }
        return result;
    }

    // /subscriptions/{subscriptionId}/providers/Microsoft.Network/locations/{location}/serviceTags
    private static async Task<int> GetServiceTags
        (SubscriptionResource subscription)
    {
        AzureLocation location = AzureLocation.AustraliaEast;
        var response = await subscription
            .GetServiceTagAsync(location)
            .ConfigureAwait(false);
        return response.Value.Values.Count;
    }
}

/*
Sample output:

Start
00:00:01.0112308: Calling GetVirtualNetworks
00:00:01.2009604: 0 count
00:00:02.2040622: Calling GetVirtualNetworks
00:00:02.4171261: 0 count
00:00:03.4209379: Calling GetVirtualNetworks
00:00:03.7912528: 0 count
00:00:04.8025850: Calling GetVirtualNetworks
00:00:04.9491728: 0 count
00:00:05.9527621: Calling GetVirtualNetworks
00:00:06.2818412: 0 count
00:00:07.2892255: Calling GetVirtualNetworks
00:00:07.4500068: 0 count
00:00:08.4590672: Calling GetVirtualNetworks
00:00:08.6071784: 0 count
00:00:09.6095414: Calling GetVirtualNetworks
00:00:09.8034028: 0 count
00:00:10.8178706: Calling GetVirtualNetworks
00:00:11.0059241: 0 count
00:00:12.0208564: Calling GetVirtualNetworks
00:00:12.1576434: 0 count
00:00:13.1691694: Calling GetVirtualNetworks
00:00:13.5184397: 0 count
00:00:14.5335443: Calling GetVirtualNetworks
00:00:14.6845697: 0 count
00:00:15.6980300: Calling GetVirtualNetworks
00:00:16.0260445: 0 count
00:00:17.0334492: Calling GetVirtualNetworks
00:00:17.3973843: 0 count
00:00:18.4023447: Calling GetVirtualNetworks
00:00:18.7830186: 0 count
00:00:19.7839762: Calling GetVirtualNetworks
00:00:19.9574795: 0 count
00:00:20.9700587: Calling GetVirtualNetworks
00:00:21.1421881: 0 count
00:00:22.1508022: Calling GetVirtualNetworks
00:00:22.3378692: 0 count
00:00:23.3497904: Calling GetVirtualNetworks
00:00:23.4895643: 0 count
00:00:24.4994823: Calling GetVirtualNetworks
00:00:24.6427858: 0 count
00:00:25.6532271: Calling GetVirtualNetworks
00:00:25.8148441: 0 count
00:00:26.8211910: Calling GetVirtualNetworks
00:00:26.9771824: 0 count
00:00:27.9860753: Calling GetVirtualNetworks
00:00:28.3103158: 0 count
00:00:29.3221837: Calling GetVirtualNetworks
00:00:29.4624319: 0 count
00:00:30.4673335: Calling GetVirtualNetworks
00:00:30.7865203: 0 count
00:00:31.7897989: Calling GetVirtualNetworks
00:00:31.9287928: 0 count
00:00:32.9402895: Calling GetVirtualNetworks
00:00:33.1722715: 0 count
00:00:34.1819885: Calling GetVirtualNetworks
00:00:34.3708741: 0 count
00:00:35.3831218: Calling GetVirtualNetworks
00:00:35.7405426: 0 count
00:00:36.7473961: Calling GetVirtualNetworks
00:00:36.9170312: 0 count
************************
Calls:30 Excpetions:0 Rate:1sec per call

00:00:01.0021128: Calling GetServiceTags
00:00:01.7217958: 1062 count
00:00:02.7256515: Calling GetServiceTags
00:00:03.6174495: 1062 count
00:00:04.6202055: Calling GetServiceTags
00:00:05.2891709: 1062 count
00:00:06.3037381: Calling GetServiceTags
00:00:07.0030850: 1062 count
00:00:08.0120622: Calling GetServiceTags
00:00:08.6667014: 1062 count
00:00:09.6767260: Calling GetServiceTags
00:00:11.4114680: 1062 count
00:00:12.4232868: Calling GetServiceTags
00:00:14.2044231: 1062 count
00:00:15.2190519: Calling GetServiceTags
00:00:16.7577587: 1062 count
00:00:17.7683517: Calling GetServiceTags
00:00:19.6027322: 1062 count
00:00:20.6060023: Calling GetServiceTags
00:00:22.5117164: 1062 count
00:00:23.5229715: Calling GetServiceTags
00:00:30.6445811: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 711 seconds."
00:00:31.6518658: Calling GetServiceTags
00:00:38.3977931: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 815 seconds."
00:00:39.4090769: Calling GetServiceTags
00:00:46.1140406: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 783 seconds."
00:00:47.1276585: Calling GetServiceTags
00:00:53.9102357: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 672 seconds."
00:00:54.9154284: Calling GetServiceTags
00:01:01.4311293: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 277 seconds."
00:01:02.4340280: Calling GetServiceTags
00:01:08.2076913: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 610 seconds."
00:01:09.2094855: Calling GetServiceTags
00:01:16.0518627: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 446 seconds."
00:01:17.0540704: Calling GetServiceTags
00:01:24.0453003: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 778 seconds."
00:01:25.0603298: Calling GetServiceTags
00:01:31.7395866: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 531 seconds."
00:01:32.7480515: Calling GetServiceTags
00:01:39.8560844: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 731 seconds."
00:01:40.8676154: Calling GetServiceTags
00:01:47.3257713: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 720 seconds."
00:01:48.3307869: Calling GetServiceTags
00:01:54.9288800: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 193 seconds."
00:01:55.9434153: Calling GetServiceTags
00:02:02.9520227: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 642 seconds."
00:02:03.9664869: Calling GetServiceTags
00:02:10.7116983: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 413 seconds."
00:02:11.7218384: Calling GetServiceTags
00:02:17.5653871: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 479 seconds."
00:02:18.5718888: Calling GetServiceTags
00:02:24.6838546: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 255 seconds."
00:02:25.6969980: Calling GetServiceTags
00:02:32.4491714: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 340 seconds."
00:02:33.4502203: Calling GetServiceTags
00:02:39.9259144: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 499 seconds."
00:02:40.9407868: Calling GetServiceTags
00:02:47.0263667: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 275 seconds."
00:02:48.0411907: Calling GetServiceTags
00:02:55.0242699: Operation GetServiceTagsOperation exceeded throttling limit of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX calls within last 5 minutes. The number of calls exceeds Microsoft.Network throttling limit. The call can be retried in 262 seconds."
************************
Calls:30 Excpetions:20 Rate:5sec per call

*/

Environment

.NET SDK: Version: 8.0.302 Commit: ef14e02af8 Workload version: 8.0.300-manifests.5273bb1c MSBuild version: 17.10.4+10fbfbf2e

Runtime Environment: OS Name: Windows OS Version: 10.0.22631 OS Platform: Windows RID: win-x64 Base Path: C:\Program Files\dotnet\sdk\8.0.302\

TibbsTerry commented 3 months ago

Issue is the same for Azure.ResourceManager.Network V 1.8.0

jsquire commented 3 months ago

Hi @TibbsTerry. Thank you for reaching out and we regret that you're experiencing difficulties. The behavior that you're seeing is from the ARM service itself; the Azure SDK packages have no insight into when the service applies rate limits nor influence over them.

Unfortunately, this is not something that the maintainers of the Azure SDK packages can assist with. Because the Azure services themselves are not managed on GitHub, we're unable to transfer this on your behalf. To ensure that the right team has visibility and can help, your best path forward for would be to open an Azure support request or inquire on the Microsoft Q&A site.

I'm going to close this out; if I've misunderstood what you're describing, please let us know in a comment and we'd be happy to assist as we're able.