Backup-AzApiManagement: BadRequest

pim-simons commented 8 months ago

Description

We have been using Backup-AzApiManagement for quite a long time to backup our API Management instance, since this week this script has started failing with this error:

Operation returned an invalid status code 'BadRequest'

When I add debug logging I find the following information:

{
  "error": {
    "code": "InvalidParameters",
    "message": "Invalid parameter: Requested value 'GET,HEAD,POST,PUT,DELETE,OPTIONS,PATCH' was not found.\r\nParameter name: backupStorageAccount",
    "details": null,
    "innerError": null
  }
}

Nothing on our side has changed as far as I know, what could be causing this issue?

Issue script & Debug output

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
POST

Absolute Uri:
https://management.azure.com/subscriptions/{subscription}/resourceGroups/{resourcegroup}/providers/Microsoft.ApiManagement/service/{apimname}/backup?api-version=2021-08-01

Headers:
Accept-Language               : en-US
x-ms-client-request-id        : a050d5b5-6b96-4478-a023-8cb52be674be

Body:
{
  "storageAccount": "{storageaccount}",
  "containerName": "{containername}",
  "backupName": "{backupname}",
  "accessType": "AccessKey",
  "accessKey": "{token}"
}

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
BadRequest

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
Strict-Transport-Security     : max-age=31536000; includeSubDomains
Server                        : Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1199
x-ms-request-id               : 9cde0310-e539-455a-b5cd-b8e7bcf85a91
x-ms-correlation-request-id   : 9cde0310-e539-455a-b5cd-b8e7bcf85a91
x-ms-routing-request-id       : WESTEUROPE:20231114T125043Z:9cde0310-e539-455a-b5cd-b8e7bcf85a91
X-Content-Type-Options        : nosniff
Date                          : Tue, 14 Nov 2023 12:50:42 GMT

Body:
{
  "error": {
    "code": "InvalidParameters",
    "message": "Invalid parameter: Requested value 'GET,HEAD,POST,PUT,DELETE,OPTIONS,PATCH' was not found.\r\nParameter name: backupStorageAccount",
    "details": null,
    "innerError": null
  }
}

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.9
PSEdition                      Core
GitCommitId                    7.3.9
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     2.13.2                Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzConte…
Script     4.0.2                 Az.ApiManagement                    {Add-AzApiManagementApiToGateway, Add-AzApiManage…
Script     6.0.0                 Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorage…

Error output

HistoryId: 2

Message        : Operation returned an invalid status code 'BadRequest'
StackTrace     :    at Microsoft.Azure.Management.ApiManagement.ApiManagementServiceOperations.BeginBackupWithHttpMessa
                 gesAsync(String resourceGroupName, String serviceName, ApiManagementServiceBackupRestoreParameters
                 parameters, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ApiManagement.ApiManagementServiceOperations.BackupWithHttpMessagesAs
                 ync(String resourceGroupName, String serviceName, ApiManagementServiceBackupRestoreParameters
                 parameters, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ApiManagement.ApiManagementServiceOperationsExtensions.BackupAsync(IA
                 piManagementServiceOperations operations, String resourceGroupName, String serviceName,
                 ApiManagementServiceBackupRestoreParameters parameters, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.ApiManagement.ApiManagementServiceOperationsExtensions.Backup(IApiMan
                 agementServiceOperations operations, String resourceGroupName, String serviceName,
                 ApiManagementServiceBackupRestoreParameters parameters)
                    at Microsoft.Azure.Commands.ApiManagement.ApiManagementClient.BackupApiManagement(String
                 resourceGroupName, String serviceName, String storageAccountName, String storageAccountKey, String
                 backupContainer, String backupBlob, String accessType, String identityClientId)
                    at Microsoft.Azure.Commands.ApiManagement.Commands.BackupAzureApiManagement.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronously
                 OrAsJob>b__3_0(T c)
                    at
                 Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T
                 cmdlet, Action`1 executor)
                    at
                 Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T
                 cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : Microsoft.Azure.Management.ApiManagement.Models.ErrorResponseException
InvocationInfo : {Backup-AzApiManagement}
Line           : Backup-AzApiManagement -ResourceGroupName "{resourcegroup}" -Name "{apimname}" -StorageContext
                 $StorageContext -TargetContainerName "{containername}" -TargetBlobName "{backupname}" -verbose -debug
Position       : At C:\Users\Pim Simons\Desktop\test.ps1:4 char:1
                 + Backup-AzApiManagement -ResourceGroupName "vg-acc-esb-infra" -Name "v …
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 2

isra-fel commented 8 months ago

The API management PowerShell module was released in January. This is an issue on the service side. Let me loop in their team.

microsoft-github-policy-service[bot] commented 8 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @solankisamir, @mikebudzynski, @KedarJoshi, @yingru97.

pim-simons commented 6 months ago

Any update on this? We are still experiencing this issue and haven't been able to create a backup of our API Mangement for a while now...

o-l-a-v commented 4 months ago

I've gotten this cmdlet to work by using managed identity on the ~Cosmos DB~ APIM, instead of storage account SAS key.

You make sure the ~Cosmos DB~ APIM has system assigned managed identity enabled, then give it (for instance) "Storage Blob Data Contributor" on the Storage Account container you want the backup to be written to. Then do something like this to trigger the backup job.

# Assets
$ApimResourceId           = [string] ''
$StorageAccountResourceId = [string] ''
$ContainerName            = [string] 'test-apim-backup'
$BackupName               = [string] 'test-apim-backup-{0}.apimbackup' -f [datetime]::UtcNow.ToString('yyyyMMdd-HHmmss')

# Get storage account context
$StorageAccount = Get-AzResource -ResourceId $StorageAccountResourceId
$StorageContext = New-AzStorageContext -StorageAccountName $StorageAccount.'Name' -Protocol 'Https' -UseConnectedAccount

# Get Id of managed identity
$Apim = Get-AzResource -ResourceId $ApimResourceId
$ApimManagedIdentity = $Apim.'Identity'.Where{$_.'Type' -eq 'SystemAssigned'}
if ([string]::IsNullOrEmpty($ApimManagedIdentity.'PrincipalId')) {
    Write-Error -ErrorRecord 'Stop' -Message ('Failed to find system assigned managed identity of "{0}".' -f $Apim.'Name')
}

# Do backup
Backup-AzApiManagement -ResourceGroupName $Apim.'ResourceGroup' -Name $Apim.'Name' -AccessType 'SystemAssignedManagedIdentity' `
    -StorageContext $StorageContext -TargetContainerName $ContainerName -TargetBlobName $BackupName `
    -IdentityClientId $ApimManagedIdentity.'PrincipalId'

The principal/user running the script must obviously be authenticated, and have permissions to trigger an APIM backup.

pim-simons commented 4 months ago

@o-l-a-v thank you for your response! However I don't quite understand since we do not use Cosmos DB, why is this necessary when we want to backup the APIM instance to a storage account?

o-l-a-v commented 4 months ago

@o-l-a-v thank you for your response! However I don't quite understand since we do not use Cosmos DB, why is this necessary when we want to backup the APIM instance to a storage account?

Oops, sorry, I ment APIM. This was about APIM, not Cosmos DB.

pim-simons commented 4 months ago

@o-l-a-v thank you for your response! However I don't quite understand since we do not use Cosmos DB, why is this necessary when we want to backup the APIM instance to a storage account?

Oops, sorry, I ment APIM. This was about APIM, not Cosmos DB.

Ah that makes a bit more sense 👍🏻 Will give this a try, thanks!!

pim-simons commented 3 months ago

Got it working with the use of managed identity, however using the storage account keys still seems broken.

For now I have a way to fix this, but it still is strange that the storage account keys version is not working anymore. @solankisamir, @mikebudzynski, @KedarJoshi, @yingru97 any feedback on that?

Nealsaha1007 commented 2 months ago

I've gotten this cmdlet to work by using managed identity on the ~Cosmos DB~ APIM, instead of storage account SAS key.

You make sure the ~Cosmos DB~ APIM has system assigned managed identity enabled, then give it (for instance) "Storage Blob Data Contributor" on the Storage Account container you want the backup to be written to. Then do something like this to trigger the backup job.
# Assets
$ApimResourceId           = [string] ''
$StorageAccountResourceId = [string] ''
$ContainerName            = [string] 'test-apim-backup'
$BackupName               = [string] 'test-apim-backup-{0}.apimbackup' -f [datetime]::UtcNow.ToString('yyyyMMdd-HHmmss')

# Get storage account context
$StorageAccount = Get-AzResource -ResourceId $StorageAccountResourceId
$StorageContext = New-AzStorageContext -StorageAccountName $StorageAccount.'Name' -Protocol 'Https' -UseConnectedAccount

# Get Id of managed identity
$Apim = Get-AzResource -ResourceId $ApimResourceId
$ApimManagedIdentity = $Apim.'Identity'.Where{$_.'Type' -eq 'SystemAssigned'}
if ([string]::IsNullOrEmpty($ApimManagedIdentity.'PrincipalId')) {
    Write-Error -ErrorRecord 'Stop' -Message ('Failed to find system assigned managed identity of "{0}".' -f $Apim.'Name')
}

# Do backup
Backup-AzApiManagement -ResourceGroupName $Apim.'ResourceGroup' -Name $Apim.'Name' -AccessType 'SystemAssignedManagedIdentity' `
    -StorageContext $StorageContext -TargetContainerName $ContainerName -TargetBlobName $BackupName `
    -IdentityClientId $ApimManagedIdentity.'PrincipalId'
The principal/user running the script must obviously be authenticated, and have permissions to trigger an APIM backup.

I am still getting the below error using managed identities, i followed the above mentioned steps.

@pim-simons @o-l-a-v

o-l-a-v commented 2 months ago

@Nealsaha1007

You have a different error.

Did you create a container in your storage account that matches the name set in the PowerShell you ran?
What info do you get if running the command with -Debug?

Edit: Updated the sample PowerShell with more failproofing.

Click to expand

```powershell <# Prerequirements: * APIM exists with managed identity enabled. * Storage Account and container exists. * APIM managed identity has suitable RBAC data-plane role on target Storage Account container. * For instance: "Storage Blob Data Contributor". #> # PowerShell preferences $ErrorActionPreference = 'Stop' # Assets $ApimResourceId = [string] '/subscriptions//resourceGroups//providers/Microsoft.ApiManagement/service/' $StorageAccountResourceId = [string] '/subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts/' $StorageAccountContainerName = [string] 'test-apim-backup' $BackupName = [string] 'test-apim-backup-{0}.apimbackup' -f [datetime]::UtcNow.ToString('yyyyMMdd-HHmmss') # Get ## APIM and ID of its' managed identity $Apim = Get-AzResource -ResourceId $ApimResourceId $ApimManagedIdentity = $Apim.'Identity'.Where{$_.'Type' -eq 'SystemAssigned'} ## Storage account and its' context $StorageAccount = Get-AzResource -ResourceId $StorageAccountResourceId $StorageContext = New-AzStorageContext -StorageAccountName $StorageAccount.'Name' -Protocol 'Https' -UseConnectedAccount # Failproofing ## APIM managed identity exists if ([string]::IsNullOrEmpty($ApimManagedIdentity.'PrincipalId')) { Write-Error -ErrorAction 'Stop' -Message ('Failed to find system assigned managed identity of "{0}".' -f $Apim.'Name') } ## Storage Account container exists if (-not $(Try{$null = Get-AzStorageContainer -Context $StorageContext -Name $StorageAccountContainerName -ErrorAction 'Stop'; $?}Catch{$false})) { Write-Error -ErrorAction 'Stop' -Message ('Failed to find Storage Account "{0}" container "{1}".' -f $StorageAccount.'Name', $StorageAccountContainerName) } # Do backup Backup-AzApiManagement -ResourceGroupName $Apim.'ResourceGroup' -Name $Apim.'Name' -AccessType 'SystemAssignedManagedIdentity' ` -StorageContext $StorageContext -TargetContainerName $StorageAccountContainerName -TargetBlobName $BackupName ` -IdentityClientId $ApimManagedIdentity.'PrincipalId' ```

Nealsaha1007 commented 2 months ago

It worked now thanks @o-l-a-v

Azure / azure-powershell