Get-AzSubscriptionDeploymentStack returns 429 TooManyRequests if no subscription scoped stacks exist

[maciejporebski]

Describe the bug Get-AzSubscriptionDeploymentStack command and GET /subscriptions/${SubscriptionId}/providers/Microsoft.Resources/deploymentStacks?api-version=2021-05-01-preview return 429 TooManyRequests if no subscription scoped stacks exist. This in turn results in the Get-AzSubscriptionDeploymentStack command hanging as it enters a retry loop which continuously receives 429 responses. In my test environment I was also unable to exit out of the command and had to close the shell.

To Reproduce Steps to reproduce the behavior:

1. Install Az module 6.6.0
2. Install v0.1.1 release of the Deployment Stacks module
3. Remove all existing subscription scoped deployment stacks
4. Run the Get-AzSubscriptionDeploymentStack command
5. Command runs indefinitely

Repro script used (script output below): (Instead of using the Get-AzSubscriptionDeploymentStack command in the retro script the API endpoint is called directly as -Debug does not appear to return any useful information before it hangs, example of that included below)

param(
    [Guid]$SubscriptionId
)

$emptyTemplateUri = "https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/100-blank-template/azuredeploy.json"
$token = Get-AzAccessToken
$headers = @{ Authorization = "Bearer $($token.Token)" }
$getSubStacksUri = "https://management.azure.com/subscriptions/${SubscriptionId}/providers/Microsoft.Resources/deploymentStacks?api-version=2021-05-01-preview"

function Get-SubStacks {
    Write-Host "GET ${getSubStacksUri}"
    $req = Invoke-WebRequest -Uri $getSubStacksUri -Headers $headers -SkipHttpErrorCheck

    Write-Host "STATUS: $($req.StatusCode) $($req.StatusDescription)"
    Write-Host "HEADERS:"
    foreach($headerName in $req.Headers.Keys) {
        Write-Host "$($headerName): $($req.Headers[$headerName])"
    }
    Write-Host "BODY:"
    Write-Host $req.Content
    $result = $req.Content | ConvertFrom-Json
    return $result
}

Write-Host "Creating subscription stack"
Write-Host "----------------------------------------------"
New-AzSubscriptionDeploymentStack -Name testStack -TemplateUri $emptyTemplateUri -Location northeurope

Write-Host "Getting subscription stacks"
Write-Host "----------------------------------------------"
$stacks = Get-SubStacks

Write-Host "Removing all subscription stacks"
Write-Host "----------------------------------------------"
foreach($stack in $stacks.value) {
    Write-Host "Removing $($stack.name)"
    Remove-AzSubscriptionDeploymentStack -ResourceId $stack.id -Force
}

Write-Host "Getting subscription stacks"
Write-Host "----------------------------------------------"
$stacks = Get-SubStacks

Expected behavior An empty result is returned. (Same as the behaviour of Get-AzResourceGroupDeploymentStack)

Screenshots N/A

Repro Environment Host OS: Windows 11 (22499.1010) Powershell Version:

Name                           Value
----                           -----
PSVersion                      7.2.0
PSEdition                      Core
GitCommitId                    7.2.0
OS                             Microsoft Windows 10.0.22499
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Server Debugging Information Correlation ID: 93dfa910-6a50-43a5-b8c1-0c893ba5249f Tenant ID: 5c162c0f-ddd8-49d9-9933-f99540d7ec74 Subscription ID: 9cc55246-7157-4ef7-a375-ce22fb559f66 Timestamp of issue (please include time zone): Tue, 16 Nov 2021 23:12:22 GMT Data Center (eg, West Central US, West Europe): UKSOUTH

Additional context

Output of Get-AzSubscriptionDeploymentStack -Debug

Get-AzSubscriptionDeploymentStack -Debug

DEBUG: 23:16:40 - GetAzSubscriptionDeploymentStack begin processing with ParameterSet 'ListDeploymentStacks'.
DEBUG: 23:16:40 - using account id '<upn>'...

Repro script output:

Creating subscription stack
----------------------------------------------

Id                : /subscriptions/9cc55246-7157-4ef7-a375-ce22fb559f66/providers/Microsoft.Resources/deploymentStacks/testStack
Name              : testStack
ProvisioningState : succeeded
UpdateBehavior    : detachResources
Location          : northeurope
CreationTime(UTC) : 16/11/2021 23:12:09
DeploymentId      : /subscriptions/9cc55246-7157-4ef7-a375-ce22fb559f66/providers/Microsoft.Resources/deployments/testStack-2021-11-16-23-12-09-1ab29
SnapshotId        : /subscriptions/9cc55246-7157-4ef7-a375-ce22fb559f66/providers/Microsoft.Resources/deploymentStacks/testStack/snapshots/2021-11-16-23-12-
                    09-1ab29

Getting subscription stacks
----------------------------------------------
GET https://management.azure.com/subscriptions/9cc55246-7157-4ef7-a375-ce22fb559f66/providers/Microsoft.Resources/deploymentStacks?api-version=2021-05-01-preview
STATUS: 200 OK
HEADERS:
Cache-Control: no-cache
Pragma: no-cache
x-ms-original-request-ids: 042e9d52-86ad-435b-94e3-fa77f590410c
x-ms-ratelimit-remaining-subscription-reads: 11996
x-ms-request-id: 679254f1-d6ac-4ba0-a5d8-ac1674486723
x-ms-correlation-request-id: 679254f1-d6ac-4ba0-a5d8-ac1674486723
x-ms-routing-request-id: UKSOUTH:20211116T231221Z:679254f1-d6ac-4ba0-a5d8-ac1674486723
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
Date: Tue, 16 Nov 2021 23:12:20 GMT
Content-Type: application/json; charset=utf-8
Expires: -1
Content-Length: 1029
BODY:
{"value":[{"location":"northeurope","properties":{"snapshotId":"/subscriptions/9cc55246-7157-4ef7-a375-ce22fb559f66/providers/Microsoft.Resources/deploymentStacks/testStack/snapshots/2021-11-16-23-12-09-1ab29","updateBehavior":"detachResources","deploymentId":"/subscriptions/9cc55246-7157-4ef7-a375-ce22fb559f66/providers/Microsoft.Resources/deployments/testStack-2021-11-16-23-12-09-1ab29","parameters":{},"templateLink":{"uri":"https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/100-blank-template/azuredeploy.json"},"managedResources":[],"provisioningState":"succeeded"},"systemData":{"createdBy":"maciej.porebski@cloudmdp.com","createdByType":"User","createdAt":"2021-11-16T23:12:09.4332876Z","lastModifiedBy":"maciej.porebski@cloudmdp.com","lastModifiedByType":"User","lastModifiedAt":"2021-11-16T23:12:09.4332876Z"},"id":"/subscriptions/9cc55246-7157-4ef7-a375-ce22fb559f66/providers/Microsoft.Resources/deploymentStacks/testStack","type":"Microsoft.Resources/deploymentStacks","name":"testStack"}]}
Removing all subscription stacks
----------------------------------------------
Removing testStack
True
Getting subscription stacks
----------------------------------------------
GET https://management.azure.com/subscriptions/9cc55246-7157-4ef7-a375-ce22fb559f66/providers/Microsoft.Resources/deploymentStacks?api-version=2021-05-01-preview
STATUS: 429 TooManyRequests                                                                                             
HEADERS:
Cache-Control: no-cache
Pragma: no-cache
Retry-After: 15
x-ms-failure-cause: service
x-ms-request-id: 93dfa910-6a50-43a5-b8c1-0c893ba5249f
x-ms-correlation-request-id: 93dfa910-6a50-43a5-b8c1-0c893ba5249f
x-ms-routing-request-id: UKSOUTH:20211116T231222Z:93dfa910-6a50-43a5-b8c1-0c893ba5249f
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
Date: Tue, 16 Nov 2021 23:12:22 GMT
Connection: close
Content-Type: application/json; charset=utf-8
Expires: -1
Content-Length: 256
BODY:
{"error":{"code":"ResourceCollectionRequestsThrottled","message":"Operation 'Microsoft.Resources/deploymentStacks/read' failed as server encountered too many requests. Please try after '15' seconds. Tracking Id is '93dfa910-6a50-43a5-b8c1-0c893ba5249f'."}}

[snarkywolverine]

Thanks for reporting this! We should have a fix available in the next couple weeks.

[snarkywolverine]

This should now be fixed in all regions - a GET of deploymentStacks at subscription scope should now return an empty array. Let us know if you see any further issues. Thanks!