search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.24k stars 3.84k forks source link

Restore-AzWebAppSnapshot fails when large number of resources exist (sometimes) #26268

Open danbuchal-mom opened 1 week ago

danbuchal-mom commented 1 week ago

Description

Restore-AzWebAppSnapshot can fail when a large number of resources exist in the subscription. Restore-AzWebAppSnapshot calls the "resources" API which returns paginated results.

If the Web App of the Snapshot is not returned on the first "page" of results, the restore operation will fail with an InternalServerError

Note: a "filter" is passed to this API call, however, it appears that the Resources API is first retrieving all results, then only filtering using the first page.

I suspect the above as I have tested the API call using the Azure Resource Explorer (https://resources.azure.com/subscriptions/[subscription-guid-redacted]/resources) We have 24 different apps and only have consistent issues restoring 5 of them with this call.

All 5 of the affected web apps do not appear on the first "page" of results in Azure Resource Explorer, while all working apps DO appear on the first page.

In addition, we have one web app that works some of the time. Refreshing the results in Azure Resource Explorer shows that sometimes this app appears on the first page and sometimes it does not.

Note: In the debug output below you will see the call to the Resources API https://management.azure.com/subscriptions/[redacted]/resources?$filter=resourceType eq 'Microsoft.Web/sites' and name eq '[redacted]'&api-version=2016-09-01

Observe it returns an empty array with a "nextLink"

Issue script & Debug output

$snapshot = (Get-AzWebAppSnapshot -Name $WebAppName -ResourceGroupName $ResourceGroupName)[0]

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/snapshots?api-version=2021-01-15

Headers:
x-ms-client-request-id        : 8b732e64-9590-447f-8ef0-a81b8b8f317c
Accept-Language               : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
ETag                          : "1DB0EE97209AB0B"
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-request-id               : 86bd8e45-aff8-4a31-8c31-059e0458e97e
X-AspNet-Version              : 4.0.30319
X-Powered-By                  : ASP.NET
x-ms-ratelimit-remaining-subscription-reads: 249
x-ms-ratelimit-remaining-subscription-global-reads: 3749
x-ms-correlation-request-id   : d27badb3-c7e0-474e-9c7e-44c996ba1411
x-ms-routing-request-id       : EASTUS:20241010T204514Z:d27badb3-c7e0-474e-9c7e-44c996ba1411
X-Content-Type-Options        : nosniff
X-Cache                       : CONFIG_NOCACHE
X-MSEdge-Ref                  : Ref A: E117A506F88B46559CD0664687396A12 Ref B: MNZ221060610029 Ref C: 2024-10-10T20:45:13Z
Date                          : Thu, 10 Oct 2024 20:45:14 GMT

Body:
{
  "value": [
    {
      "id": "/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/snapshots/2024-09-11T00_25_46_4226984",
      "name": "2024-09-11T00:25:46.4226984",
      "type": "Microsoft.Web/sites/snapshots",
      "properties": {
        "time": "2024-09-11T00:25:46.4226984"
      }
    },
    ...
  ]
}
Restore-AzWebAppSnapshot -Name $TargetWebAppName -ResourceGroupName $TargetResourceGroupName -InputObject $snapshot -Force -RecoverConfiguration -Debug

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resources?$filter=resourceType eq 'Microsoft.Web/sites' and name eq '[redacted]'&api-version=2016-09-01

Headers:
x-ms-client-request-id        : bb36d532-556d-4e7a-90a4-f7f55f0bafad
Accept-Language               : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-ratelimit-remaining-subscription-reads: 249
x-ms-ratelimit-remaining-subscription-global-reads: 3749
x-ms-request-id               : 52ce3c1d-ac04-4795-bd9a-84a4e949f186
x-ms-correlation-request-id   : 52ce3c1d-ac04-4795-bd9a-84a4e949f186
x-ms-routing-request-id       : EASTUS:20241010T204621Z:52ce3c1d-ac04-4795-bd9a-84a4e949f186
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
X-Cache                       : CONFIG_NOCACHE
X-MSEdge-Ref                  : Ref A: E4B43C8E16E345328AA4FB9B3127D5CB Ref B: MNZ221060618037 Ref C: 2024-10-10T20:46:21Z
Date                          : Thu, 10 Oct 2024 20:46:20 GMT

Body:
{
  "value": [],
  "nextLink": "https://management.azure.com/subscriptions/[redacted]/resources?%24filter=resourceType+eq+%27Microsoft.Web%2fsites%27+and+name+eq+%27[redacted]%27&api-version=2016-09-01&%24skiptoken=rZFNa4NAEED%2fi2cPm9VcAj20XWMNzoplzXb3JkbqR1TQFaOh%2f73bIC2BhID0OgNvHvPORp2elJ%2fXZWdszkbdtCpL0lq18bHXEyMdd738yBDj2zHG%2b9ErmpxWoSU4DHQCRXk00hEhwUrb584JsFCiEEhU0TogyRowDF6NngzT6Jr%2bMVvwENMpQcCelWRg0VeEAu6MPvscBHcUMAeBC3ZAwKYsQjM7jTvVd%2fiOsHTDUZJwLXmkaLU9Sg2FybkICyYUJTKnTOayEBYtDtUMffQHgWUGPFwBdpR2Wf24SlIO%2bg8ocDUWvxd6d5LEmWSxy65c76gK7NlAIksWiQrYSwm5Vq32lc%2b9AapIBS6sxJRYAckyIIf8wvwy%2fyoG9VtzPOiW8xDitkxbfe067ea%2fwprXWW9yl0Q156Q3gUuCmr85byKXxJwd7%2fCWhPwG"
}

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
POST

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/restoreSnapshot?api-version=2021-01-15

Headers:
x-ms-client-request-id        : bb36d532-556d-4e7a-90a4-f7f55f0bafad
Accept-Language               : en-US

Body:
{
  "properties": {
    "snapshotTime": "2024-10-10T19:49:23",
    "recoverySource": {},
    "overwrite": true,
    "recoverConfiguration": true,
    "ignoreConflictingHostNames": true,
    "useDRSecondary": false
  }
}

...

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
InternalServerError

Headers:
Cache-Control                 : no-cache
Cache-Control                 : no-cache
Pragma                        : no-cache
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-AspNet-Version              : 4.0.30319
X-Powered-By                  : ASP.NET
x-ms-failure-cause            : service
x-ms-ratelimit-remaining-subscription-writes: 199
x-ms-ratelimit-remaining-subscription-global-writes: 2999
x-ms-request-id               : e65257ab-3bba-4119-8811-d379bf759616
x-ms-correlation-request-id   : e65257ab-3bba-4119-8811-d379bf759616
x-ms-routing-request-id       : EASTUS:20241010T204644Z:e65257ab-3bba-4119-8811-d379bf759616
X-Content-Type-Options        : nosniff
X-Cache                       : CONFIG_NOCACHE
X-MSEdge-Ref                  : Ref A: D5FB6060669A470A849BE4CBC9E966BF Ref B: MNZ221060610047 Ref C: 2024-10-10T20:46:44Z
Date                          : Thu, 10 Oct 2024 20:46:43 GMT

Body:
{
  "Message": "An error has occurred."
}

Environment data

Name                           Value
----                           -----
PSVersion                      7.2.23
PSEdition                      Core
GitCommitId                    7.2.23
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     3.0.2                 Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     5.1.0                 Az.KeyVault                         {Add-AzKeyVaultCertificate, Add-AzKeyVaultCertificateContact, Add-AzKeyVaultKey, Add-AzKe… 
Script     6.13.0                Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}   
Script     1.2.2                 Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, A… 
Script     3.1.2                 Az.Websites                         {Add-AzWebAppAccessRestrictionRule, Add-AzWebAppTrafficRouting, Edit-AzWebAppBackupConfig…

Error output

Message        : Operation returned an invalid status code 'InternalServerError'
StackTrace     :    at Microsoft.Azure.Management.WebSites.WebAppsOperations.BeginRestoreSnapshotWithHttpMessagesAsync(String resourceGroupName, String name,   
                 SnapshotRestoreRequest restoreRequest, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.WebSites.WebAppsOperations.RestoreSnapshotWithHttpMessagesAsync(String resourceGroupName, String name,        
                 SnapshotRestoreRequest restoreRequest, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.RestoreSnapshotAsync(IWebAppsOperations operations, String
                 resourceGroupName, String name, SnapshotRestoreRequest restoreRequest, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.RestoreSnapshot(IWebAppsOperations operations, String
                 resourceGroupName, String name, SnapshotRestoreRequest restoreRequest)
                    at Microsoft.Azure.Commands.WebApps.Utilities.WebsitesClient.RestoreSnapshot(String resourceGroupName, String webSiteName, String
                 slotName, SnapshotRestoreRequest restoreReq)
                    at Microsoft.Azure.Commands.WebApps.Cmdlets.BackupRestore.RestoreAzureWebAppSnapshot.<>c__DisplayClass17_0.<ExecuteCmdlet>b__0()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(Boolean force, String continueMessage, String
                 processMessage, String target, Action action, Func`1 useShouldContinue)
                    at Microsoft.Azure.Commands.ResourceManager.Common.AzureRMCmdlet.ConfirmAction(Boolean force, String continueMessage, String
                 processMessage, String target, Action action)
                    at Microsoft.Azure.Commands.WebApps.Cmdlets.BackupRestore.RestoreAzureWebAppSnapshot.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException
InvocationInfo : {Restore-AzWebAppSnapshot}
Line           : Restore-AzWebAppSnapshot -Name [redacted] -ResourceGroupName [redacted] -InputObject $snapshot -Force
                 -RecoverConfiguration -Debug
microsoft-github-policy-service[bot] commented 17 hours ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @antcp, @AzureAppServiceCLI.

microsoft-github-policy-service[bot] commented 17 hours ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @antcp, @AzureAppServiceCLI.