Assigning VM to recovery fault in different rg fails

guidovanbrakel commented 2 weeks ago

Bicep version run bicep --version via the Bicep CLI, az bicep version via the AZ CLI or via VS code by navigating to the extensions tab and searching for Bicep

Describe the bug We have created a module to add an vm to a recovery services vault and use that in template but then it fails with message that only a recovery services vault can be assigned from the current resource group, while in the portal it is possible to assign a vault in a different resource group

{"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/subid/providers/Microsoft.Resources/deployments/ApplicationDeployment","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see
https://aka.ms/arm-deployment-operations
for usage details.","details":[{"code":"ResourceDeploymentFailure","target":"/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Resources/deployments/deploy-fileserver","message":"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.","details":[{"code":"DeploymentFailed","target":"/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Resources/deployments/deploy-fileserver","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see
https://aka.ms/arm-deployment-operations
for usage details.","details":[{"code":"ResourceNotFound","message":"The Resource 'Microsoft.RecoveryServices/vaults/rg' under resource group 'rg' was not found. For more details please go to
https://aka.ms/ARMResourceNotFoundFix"}]}]}]}}

To Reproduce We use the following template:

param location string
param subnetName string
param virtualNetworkId string

// param virtualMachineName string

param code string // parameter
param vmenviroment string // parameter
param vmnumber string // parameter
param vmfunction string // parameter
param vmsku string // default 2022-datacenter-azure-edition
// param zones string

var VMComputerName = 'variable' 
var VMNICName = 'variable'

param vmAcceleratedNetworking bool
param virtualMachineSize string
param adminUsername string
@secure()
param adminPassword string

var vnetId = virtualNetworkId

var subnetRef = '${vnetId}/subnets/${subnetName}'

resource networkInterface 'Microsoft.Network/networkInterfaces@2021-08-01' = {
  name: VMNICName 
  location: location
  properties: {
    enableAcceleratedNetworking: vmAcceleratedNetworking
    ipConfigurations: [
      {
        name: 'ipconfig1'
        properties: {
          subnet: {
            id: subnetRef
          }
          privateIPAllocationMethod: 'Dynamic'
        }
      }
    ]
  }
  dependsOn: []
}

resource virtualMachine 'Microsoft.Compute/virtualMachines@2022-03-01' = {
  name: VMComputerName
  // zones: [
  //   zones
  // ]
  location: location
  properties: {
    hardwareProfile: {
      vmSize: virtualMachineSize
    }
    storageProfile: {
      osDisk: {
        name: '${VMComputerName}_osdisk_1'
        createOption: 'fromImage'
        managedDisk: {
          storageAccountType: 'Premium_LRS'
        }
        deleteOption: 'Detach'
      }
      imageReference: {
        publisher: 'MicrosoftWindowsServer'
        offer: 'WindowsServer'
        sku: vmsku
        version: 'latest'
      }
    }
    networkProfile: {
      networkInterfaces: [
        {
          id: networkInterface.id
          properties: {
            deleteOption: 'Detach'
          }
        }
      ]
    }
    securityProfile: {
      encryptionAtHost: true
      securityType: 'TrustedLaunch'
      uefiSettings: {
        secureBootEnabled: true
        vTpmEnabled: true
      }
    }
    osProfile: {
      computerName: VMComputerName
      adminUsername: adminUsername
      adminPassword: adminPassword
      windowsConfiguration: {
        enableAutomaticUpdates: false
        provisionVMAgent: true
        patchSettings: {
          enableHotpatching: false
          assessmentMode: 'ImageDefault'
          patchMode: 'Manual'
        }
      }
    }
  }
}

resource virtualMachineName_GuestAttestation 'Microsoft.Compute/virtualMachines/extensions@2018-10-01' = {
  parent: virtualMachine
  name: 'GuestAttestation'
  location: location
  properties: {
    publisher: 'Microsoft.Azure.Security.WindowsAttestation'
    type: 'GuestAttestation'
    typeHandlerVersion: '1.0'
    autoUpgradeMinorVersion: true
    settings: {
      AttestationConfig: {
        MaaSettings: {
          maaEndpoint: ''
          maaTenantName: 'GuestAttestation'
        }
        AscSettings: {
          ascReportingEndpoint: ''
          ascReportingFrequency: ''
        }
        useCustomToken: 'false'
        disableAlerts: 'false'
      }
    }
  }
}

output adminUsername string = adminUsername

alex-frankel commented 1 week ago

@guidovanbrakel - I might be missing something, but where in this file is the code that adds the VM to the recovery services vault. I can't find any reference to that in what you shared.

guidovanbrakel commented 1 week ago

Hello @alex-frankel

This should be the correct code, we created a module called virtualmachine for it

@description('Deployment Location')
param pLocation string = resourceGroup().location

@description('Username for the Virtual Machine.')
param adminUsername string

@description('Password for the Virtual Machine.')
@secure()
param adminPassword string

@minLength(5)
@maxLength(5)
@description('Customer/Shared code')
param pCustcode string // c0001

@minLength(2)
@maxLength(3)
@description('Instance number')
param pInstanceNumber string // 001

// param AddressPrefix string
param privateIPAddress string
param subnetId string
param vmNumber int
param vmType string
param vmSize string

param createDataDisk bool
param dataDiskSize int
param dataDiskSku string

var vmName = vmType == 'img' ? 'vm-${vmType}-${pCustcode}-${padLeft(string(vmNumber + 1), 2, '0')}' : 'vm-${vmType}-${pCustcode}-${padLeft(string(vmNumber + 1), 3, '0')}'
var vmNicName = '${vmName}-nic-${pInstanceNumber}'
var asgName = 'asg-${vmType}-${pCustcode}-${pInstanceNumber}'
var networkAcceleratedVmSizes = ['Standard_D2ds_v5', 'Standard_DS12_v2','Standard_E4bds_v5','Standard_B4as_v2']
var imageReference = {
  dc: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-core-g2'
    version: 'latest'
  }
  cf: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-g2'
    version: 'latest'
  }
  cc: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-g2'
    version: 'latest'
  }
  cs: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-core-g2'
    version: 'latest'
  }
  fl: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-core-g2'
    version: 'latest'
  }
  img: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-g2'
    version: 'latest'
  }
  li: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-g2'
    version: 'latest'
  }
  db: {
    publisher: 'MicrosoftSQLServer'
    offer: 'SQL2022-WS2022'
    sku: vmNumber == 0 ? 'standard-gen2' : 'sqldev-gen2'
    version: 'latest'
  }
  ut: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-g2'
    version: 'latest'
  }
  xa: {
    publisher: 'MicrosoftWindowsServer'
    offer: 'WindowsServer'
    sku: '2022-datacenter-g2'
    version: 'latest'
  }
}
var dataDisks = createDataDisk == true && vmType == 'db' ? [
  {
    name: '${vmName}-dbdisk${pInstanceNumber}'
    lun: 1
    diskSizeGB: 256
    caching: 'ReadOnly'
    createOption: 'Empty'
  }
  {
    name: '${vmName}-logdisk${pInstanceNumber}'
    lun: 2
    diskSizeGB: 128
    caching: 'None'
    createOption: 'Empty'
  }
  {
    name: '${vmName}-tempdisk${pInstanceNumber}'
    lun: 3
    diskSizeGB: 128
    caching: 'ReadWrite'
    createOption: 'Empty'
  }
] : [
  {
    name: '${vmName}-datadisk${pInstanceNumber}'
    lun: 0
    diskSizeGB: dataDiskSize
    caching: 'None'
    createOption: 'Empty'
  }
]
var vmDataDisks = [for i in range(0, length(dataDisks)): {
  lun: dataDisks[i].lun
  name: dataDisks[i].name
  managedDisk: {
    storageAccountType: dataDiskSku
    id: dataDisk[i].id
  }
  caching: dataDisks[i].caching
  createOption: 'Attach'
}]

resource asg 'Microsoft.Network/applicationSecurityGroups@2023-11-01' existing = {
  name: asgName
}

resource networkInterface 'Microsoft.Network/networkInterfaces@2023-11-01' = {
  name: vmNicName
  location: pLocation
  properties: {
    ipConfigurations: [
      {
        name: 'ipconfig1'
        properties: {
          privateIPAddress: privateIPAddress
          privateIPAllocationMethod: 'Static'
          subnet: {
            id: subnetId
          }
          applicationSecurityGroups: [
            {
              id: asg.id
              location: pLocation
              properties: {}
            }
          ]
        }
      }
    ]
    dnsSettings: {
      dnsServers: []
    }
    enableAcceleratedNetworking: contains(networkAcceleratedVmSizes, vmSize) ? true : false
    enableIPForwarding: false
  }
}

resource dataDisk 'Microsoft.Compute/disks@2024-03-02' = [for i in dataDisks: if (createDataDisk) {
  name: i.name
  zones: vmType == 'dc' || vmType == 'cf' || vmType == 'cc' || vmType == 'cs' ? [string(vmNumber + 1)] : []
  location: pLocation
  sku: {
    name: dataDiskSku
  }
  properties: {
    creationData: {
      createOption: i.createOption
    }
    diskSizeGB: i.diskSizeGB
  }
}]

resource virtualMachine 'Microsoft.Compute/virtualMachines@2024-07-01' = {
  name: vmName
  zones: vmType == 'dc' || vmType == 'cf' || vmType == 'cc' || vmType == 'cs' ? [string(vmNumber + 1)] : []
  location: pLocation
  properties: {
    hardwareProfile: {
      vmSize: vmSize
    }
    licenseType: 'Windows_Server'
    osProfile: {
      computerName: vmName
      adminUsername: adminUsername
      adminPassword: adminPassword
      windowsConfiguration: {
        provisionVMAgent: true
        patchSettings: {
          patchMode: 'AutomaticByPlatform'
          automaticByPlatformSettings: {
            bypassPlatformSafetyChecksOnUserSchedule: true
          }
          enableHotpatching: false
          assessmentMode: 'AutomaticByPlatform'
        }
      }
      secrets: []
    }
    storageProfile: {
      imageReference: imageReference[vmType]
      osDisk: {
        osType: 'Windows'
        name: '${vmName}-osdisk${pInstanceNumber}' //vmName
        createOption: 'FromImage'
        caching: 'ReadWrite'
        diskSizeGB: vmType == 'img' ? 512 : 127
      }
      dataDisks: createDataDisk ? vmDataDisks : []
    }
    securityProfile: { 
      encryptionAtHost: true
      securityType: 'TrustedLaunch'
      uefiSettings: {
        secureBootEnabled: true
        vTpmEnabled: true
      }
    }
    networkProfile: {
      networkInterfaces: [
        {
          id: networkInterface.id
        }
      ]
    }
  }
}

resource sqlVirtualMachine 'Microsoft.SqlVirtualMachine/sqlVirtualMachines@2023-10-01' = if (vmType == 'db') {
  name: vmName
  location: pLocation
  properties: {
    sqlManagement: 'Full'
    sqlServerLicenseType: 'PAYG'
    virtualMachineResourceId: virtualMachine.id
  }
}

param vaultName string

var backupFabric = 'Azure'
var backupPolicy = imageReference == 'fl' ? 'fileServerBackupPolicy' : 'vmServerBackupPolicy'
var protectionContainer = 'iaasvmcontainer;iaasvmcontainerv2;${resourceGroup().name};${vmName}'
var protectedItem = 'vm;iaasvmcontainerv2;${resourceGroup().name};${vmName}'

resource backupVault 'Microsoft.RecoveryServices/vaults@2024-04-30-preview' existing = {
  name: vaultName
}

resource protectedItems 'Microsoft.RecoveryServices/vaults/backupFabrics/protectionContainers/protectedItems@2024-04-30-preview' = {
  name: '${vaultName}/${backupFabric}/${protectionContainer}/${protectedItem}'
  location: pLocation
  properties: {
    protectedItemType: 'Microsoft.Compute/virtualMachines'
    policyId: '${backupVault.id}/backupPolicies/${backupPolicy}'
    sourceResourceId: resourceId(subscription().subscriptionId, resourceGroup().name, 'Microsoft.Compute/virtualMachines', vmName)
  }
  dependsOn: [
    backupVault
  ]
}

resource runCommand 'Microsoft.Compute/virtualMachines/runCommands@2024-07-01' = if (vmType == 'dc' || vmType == 'fl' || vmType == 'img' || vmType == 'db' || vmType == 'ut' || vmType == 'li') {
  name: 'RunPowerShellScript'
  location: pLocation
  parent: virtualMachine
  properties: {
    source: {
      script: vmType == 'dc' ? '''
        #Add-WindowsFeature AD-Domain-Services;
        Install-WindowsFeature -Name 'AD-Domain-Services' -IncludeManagementTools;
        Install-WindowsFeature 'FS-DFS-Namespace', 'RSAT-DFS-Mgmt-Con';
        reg add HKLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters /v MaxConcurrentApi /t REG_DWORD /d 00000050 /f;
        reg add HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters /v AlwaysExpectDomainController /t REG_DWORD /d 00000001 /f;
        # Set the DVD/CD drive to Q:
        $drv = Get-WmiObject win32_volume -filter 'DriveType = 5'
        if ($drv) {
            $drv.DriveLetter = 'Q:'
            $drv.Put() | Out-Null
        }
        # Check if the drive letter E: already exists
        $existingVolume = Get-Volume -DriveLetter E
        # If the volume does not exist, proceed with initialization and formatting
        if (-not $existingVolume) {
            $disk = Get-Disk | Where-Object { $_.PartitionStyle -eq 'Raw' }
            if ($disk) {
                $disk | Initialize-Disk -PartitionStyle GPT
                New-Partition -DiskNumber $disk.Number -UseMaximumSize -DriveLetter E
                Format-Volume -DriveLetter E -FileSystem NTFS -NewFileSystemLabel DATA
            }
        }
      ''' : vmType == 'fl' ? '''
        Install-WindowsFeature 'RSAT-DFS-Mgmt-Con', 'FS-FileServer'
        # Set the DVD/CD drive to Q:
        $drv = Get-WmiObject win32_volume -filter 'DriveType = 5'
        if ($drv) {
            $drv.DriveLetter = 'Q:'
            $drv.Put() | Out-Null
        }
        # Check if the drive letter E: already exists
        $existingVolume = Get-Volume -DriveLetter E

        # If the volume does not exist, proceed with initialization and formatting
        if (-not $existingVolume) {
            $disk = Get-Disk | Where-Object { $_.PartitionStyle -eq 'Raw' }
            if ($disk) {
                $disk | Initialize-Disk -PartitionStyle GPT
                New-Partition -DiskNumber $disk.Number -UseMaximumSize -DriveLetter E
                Format-Volume -DriveLetter E -FileSystem NTFS -NewFileSystemLabel DATA
            }
        }
      ''' : vmType == 'img' ? '''
        $drv = Get-WmiObject win32_volume -filter 'DriveType = 5'
            if ($drv) {
            $drv.DriveLetter = 'Q:'
            $drv.Put() | Out-Null
        }
        # Check if the drive letters already exist
        $existingVolumes = Get-Volume | Where-Object { $_.DriveLetter -in 'E' }

        # If the volumes do not exist, proceed with initialization and formatting
        if ($existingVolumes.Count -eq 0) {
            $disk = Get-Disk | Where-Object { $_.PartitionStyle -eq 'Raw' }
            if ($disk) {
                $disk | Initialize-Disk -PartitionStyle GPT
                New-Partition -DiskNumber 2 -UseMaximumSize -DriveLetter E
                Format-Volume -DriveLetter E -FileSystem NTFS -AllocationUnitSize 65536 -NewFileSystemLabel DATA
            }
        }
        # Check if the feature is already installed
        $feature = Get-WindowsFeature -Name 'RDS-RD-Server'
        if ($feature.Installed -eq $false) {
            # Install the feature if it's not installed
            #Install-WindowsFeature 'Remote-Desktop-Services'
            Install-WindowsFeature RDS-RD-Server -IncludeManagementTools
            # Restart the computer
            Restart-Computer -Force
        }
      ''' : vmType == 'db' ? '''
        # Set the DVD/CD drive to Q:
        $drv = Get-WmiObject win32_volume -filter 'DriveType = 5'
        if ($drv) {
            $drv.DriveLetter = 'Q:'
            $drv.Put() | Out-Null
        }
        # Check if the drive letters already exist
        $existingVolumes = Get-Volume | Where-Object { $_.DriveLetter -in 'E', 'F', 'G' }

        # If the volumes do not exist, proceed with initialization and formatting
        if ($existingVolumes.Count -eq 0) {
            $disk = Get-Disk | Where-Object { $_.PartitionStyle -eq 'Raw' }
            if ($disk) {
                $disk | Initialize-Disk -PartitionStyle GPT
                New-Partition -DiskNumber 2 -UseMaximumSize -DriveLetter E
                New-Partition -DiskNumber 3 -UseMaximumSize -DriveLetter F
                New-Partition -DiskNumber 4 -UseMaximumSize -DriveLetter G
                Format-Volume -DriveLetter E -FileSystem NTFS -AllocationUnitSize 65536 -NewFileSystemLabel DATA
                Format-Volume -DriveLetter F -FileSystem NTFS -AllocationUnitSize 65536 -NewFileSystemLabel LOG
                Format-Volume -DriveLetter G -FileSystem NTFS -AllocationUnitSize 65536 -NewFileSystemLabel TMP
            }
        }
        # Create directories on different drives
        $drives = @("E:", "F:", "G:")
        $subPaths = @("MSSQL16.MSSQLSERVER\MSSQL\DATA", "MSSQL16.MSSQLSERVER\MSSQL\LOGS", "MSSQL16.MSSQLSERVER\MSSQL\TMP")

        for ($i = 0; $i -lt $drives.Length; $i++) {
            $fullPath = "$($drives[$i])\$($subPaths[$i])"
            if (-Not (Test-Path -Path $fullPath)) {
                New-Item -Path $fullPath -ItemType Directory -Force
            }
        }
        # Create Azure Workload Backup directories
        $azureBackupPath = "C:\Program Files\Azure Workload Backup\bin"
        if (-Not (Test-Path -Path $azureBackupPath)) {
            New-Item -Path $azureBackupPath -ItemType Directory -Force
        }
        # Create JSON file
        $jsonContent = '{"DefaultBackupTasksThreshold": 1}'
        $jsonFilePath = "$azureBackupPath\ExtensionSettingsOverrides.json"
        Set-Content -Path $jsonFilePath -Value $jsonContent
        # Set registry property
        New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQLServer\Parameters\' -Name 'SQLArg3' -Value '-t2546' -PropertyType 'String' -Force
        # Set SQL Server Agent service to start automatically
        Set-Service -Name "SQLSERVERAGENT" -StartupType Automatic
      ''' : vmType == 'li' ? '''
        # Set the DVD/CD drive to Q:
        $drv = Get-WmiObject win32_volume -filter 'DriveType = 5'
        if ($drv) {
            $drv.DriveLetter = 'Q:'
            $drv.Put() | Out-Null
        }
        Install-WindowsFeature -Name 'RDS-Licensing' -IncludeAllSubFeature -IncludeManagementTools
      ''' : '''
        # Set the DVD/CD drive to Q:
        $drv = Get-WmiObject win32_volume -filter 'DriveType = 5'
        if ($drv) {
            $drv.DriveLetter = 'Q:'
            $drv.Put() | Out-Null
        }
        # Check if the drive letter E: already exists
        $existingVolume = Get-Volume -DriveLetter E

        # If the volume does not exist, proceed with initialization and formatting
        if (-not $existingVolume) {
            $disk = Get-Disk | Where-Object { $_.PartitionStyle -eq 'Raw' }
            if ($disk) {
                $disk | Initialize-Disk -PartitionStyle GPT
                New-Partition -DiskNumber $disk.Number -UseMaximumSize -DriveLetter E
                Format-Volume -DriveLetter E -FileSystem NTFS -NewFileSystemLabel DATA
            }
        }
      '''
    }
  }
}

alex-frankel commented 1 week ago

If this is the error message: The Resource 'Microsoft.RecoveryServices/vaults/rg' under resource group 'rg' was not found

Can you confirm that a Recover Service Vault with the name rg exists? That seems like a strange name for this resource type, so I'm wondering if something is wrong with the parameters you provided.

but then it fails with message that only a recovery services vault can be assigned from the current resource group

Is this based on the above error message about the vault not being found? If so, then I don't think the error message is saying this is not possible.

guidovanbrakel commented 1 week ago

No we would like to reference to a vault in a diffetent resource group but then it gives this error message

Sent from Outlook for iOShttps://aka.ms/o0ukef

From: Alex Frankel @.> Sent: Thursday, October 24, 2024 9:27:51 PM To: Azure/bicep @.> Cc: Guido van Brakel @.>; Mention @.> Subject: Re: [Azure/bicep] Assigning VM to recovery fault in different rg fails (Issue #15331)

LET OP: Deze e-mail is afkomstig van buiten de organisatie. Klik niet op links en open geen bijlagen als u de afzender niet herkent.

If this is the error message: The Resource 'Microsoft.RecoveryServices/vaults/rg' under resource group 'rg' was not found

Can you confirm that a Recover Service Vault with the name rg exists? That seems like a strange name for this resource type, so I'm wondering if something is wrong with the parameters you provided.

but then it fails with message that only a recovery services vault can be assigned from the current resource group

Is this based on the above error message about the vault not being found? If so, then I don't think the error message is saying this is not possible.

— Reply to this email directly, view it on GitHubhttps://github.com/Azure/bicep/issues/15331#issuecomment-2436181355, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BLFPV7YWSGVK7VZY2DVKOMDZ5FC3PAVCNFSM6AAAAABQBUXYCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWGE4DCMZVGU. You are receiving this because you were mentioned.Message ID: @.***>

alex-frankel commented 1 week ago

This code is referencing an already deployed vault resource:

resource backupVault 'Microsoft.RecoveryServices/vaults@2024-04-30-preview' existing = {
  name: vaultName
}

But the error message is saying that the vault resource with the name "rg" does not exist. That's why we first need to confirm if that vault does or does not exist. Can you confirm that first?

If this resource is in a different resource group than the target of the deployment, then you need to use the scope property like so:

resource backupVault 'Microsoft.RecoveryServices/vaults@2024-04-30-preview' existing = {
  scope: resourceGroup('otherRG')
  name: vaultName
}

guidovanbrakel commented 1 week ago

To be fully complete. the vau;t is dynamically built and looked up this way: name: '${vaultName}/${backupFabric}/${protectionContainer}/${protectedItem}'

And can't find in the parameters the vault, I will double check with a coworker tomorrow during GMT+1 and come back to you asap @alex-frankel

Azure / bicep

Assigning VM to recovery fault in different rg fails #15331