[AVM Module Issue]: avm/ptn/lz/sub-vending: Re-running tries to delete a subnet (and possibly more)

[cspring86]

Check for previous/existing GitHub issues

• [X] I have checked for previous/existing GitHub issues

Issue Type?

I'm not sure

Module Name

avm/ptn/lz/sub-vending

(Optional) Module Version

0.1.2

Description

I've got an Azure subscription vending repository using Bicep and Azure DevOps pipelines to manage the creation, update and cancelling of Azure subscriptions.

I'm using the avm/ptn/lz/sub-vending module to handle the subscription creation/update part of the process.

I've created a subscription using the module, which went fine.

Another team has then proceeded to populate the VNet with subnets and other resources, as expected. One of these resources is an AKS cluster.

I've since re-run the pipeline to update the same subscription by enabling more resource providers in the avm/ptn/lz/sub-vending module configuration and it fails trying to delete one of the subnets.

The bottom-most error message in a nested stack of messages is this:

{
    "code": "InUseSubnetCannotBeDeleted",
    "message": "Subnet alz-spoke-t2awudh7kzl5o-eastus2-subnet-AKS is in use by /subscriptions/xxx/resourceGroups/MC_rg-alz-landingzones-online_xxx-akscluster_eastus2/providers/Microsoft.Network/networkInterfaces/|providers|Microsoft.Compute|virtualMachineScaleSets|aks-agentpool-xxx-vmss|virtualMachines|0|networkInterfaces|aks-agentpool-xxx-vmss/ipConfigurations/ipconfig1 and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.",
    "details": []
}

Is this expected behaviour?

I thought Bicep ran in incremental mode by default, not complete mode, so it shouldn't be attempting to delete anything.

I need the avm/ptn/lz/sub-vending module to be able to create and update subscriptions when needed, such that it only updates the configuration/resources that it owns. I don't want it to interfere with resources created outside of it.

(Optional) Correlation Id

No response

[avm-team-linter[bot]]

@cspring86, thanks for submitting this issue for the avm/ptn/lz/sub-vending module!

[!IMPORTANT] A member of the @Azure/avm-ptn-lz-subvending-module-owners-bicep or @Azure/avm-ptn-lz-subvending-module-contributors-bicep team will review it soon!

[sebassem]

Please switch to the latest version where we have updated the network resource API which overcomes the problem of rebuilding the subnets.

[cspring86]

Please switch to the latest version where we have updated the network resource API which overcomes the problem of rebuilding the subnets.

@sebassem I've upgraded to 0.1.4 and I'm still getting the error.

It's in the "lz-vend-vnet-create-[random]" deployment.

{"code":"DeploymentFailed","target":"/subscriptions/xxx/resourceGroups/rg-alz-landingzones-online/providers/Microsoft.Resources/deployments/lz-vend-vnet-create-c7vzwtpwxvg7e","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"InUseSubnetCannotBeDeleted","message":"Subnet alz-spoke-t2awudh7kzl5o-eastus2-subnet-AKS is in use by /subscriptions/xxx/resourceGroups/MC_rg-alz-landingzones-online_xxx-akscluster_eastus2/providers/Microsoft.Network/networkInterfaces/|providers|Microsoft.Compute|virtualMachineScaleSets|aks-agentpool-xxx-vmss|virtualMachines|0|networkInterfaces|aks-agentpool-xxx-vmss/ipConfigurations/ipconfig1 and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.","details":[]}]}

[jtracey93]

Hey @cspring86,

Could you share an example module declaration and the values you are setting for this deployment?

Please redact any sensitive information 👍

Myself & @sebassem can then look further into this

RR

[cspring86]

@jtracey93 sure, here you go.

module subVending 'br/public:avm/ptn/lz/sub-vending:0.1.4' = {
  name: 'subVending_${deploymentNameRandom}'
  params: {
    existingSubscriptionId: existingSubscriptionId
    subscriptionAliasEnabled: subscriptionAliasEnabled
    subscriptionAliasName: subscriptionName
    subscriptionDisplayName: subscriptionName
    subscriptionBillingScope: subscriptionBillingScope
    subscriptionWorkload: subscriptionWorkload
    subscriptionManagementGroupAssociationEnabled: true
    subscriptionManagementGroupId: 'alz-landingzones-online'
    subscriptionTags: {}
    resourceProviders: {
      'Microsoft.Advisor': []
      'Microsoft.AlertsManagement': []
      'Microsoft.Automation': []
      'Microsoft.Blueprint': []
      'Microsoft.Cache': []
      'Microsoft.Capacity': []
      'Microsoft.Compute': []
      'Microsoft.EventGrid': []
      'Microsoft.GuestConfiguration': []
      'Microsoft.Insights': []
      'Microsoft.KeyVault': []
      'Microsoft.ManagedIdentity': []
      'Microsoft.ManagedServices': []
      'Microsoft.Management': []
      'Microsoft.Network': []
      'Microsoft.OperationalInsights': []
      'Microsoft.OperationsManagement': []
      'Microsoft.PolicyInsights': []
      'Microsoft.RecoveryServices': []
      'Microsoft.ResourceHealth': []
      'Microsoft.Security': []
      'Microsoft.SecurityInsights': []
      'Microsoft.Sql': []
      'Microsoft.Storage': []
    }
    deploymentScriptLocation: location
    deploymentScriptManagedIdentityName: 'id-sv-ds-${subscriptionNameUnique}-${location}'
    deploymentScriptName: 'ds-sv-ds-${subscriptionNameUnique}'
    deploymentScriptNetworkSecurityGroupName: 'nsg-sv-ds-${subscriptionNameUnique}-${location}'
    deploymentScriptResourceGroupName: 'rg-sv-ds-${subscriptionNameUnique}-${location}'
    deploymentScriptStorageAccountName: 'stsvds${subscriptionNameUnique}${locationShortNameMap[location]}'
    deploymentScriptVirtualNetworkName: 'vnet-sv-ds-${subscriptionNameUnique}-${location}'
    roleAssignmentEnabled: true
    roleAssignments: [
      {
        definition: '/providers/Microsoft.Authorization/roleDefinitions/xxx'  // Reader BuiltInRole
        principalId: roleAssignmentPrincipalId  // Should be a group
        relativeScope: '/'
      }
    ]
    virtualNetworkAddressSpace: [
      virtualNetworkAddressSpace
    ]
    virtualNetworkEnabled: true
    virtualNetworkResourceGroupName: 'rg-alz-landingzones-online'
    virtualNetworkLocation: location
    virtualNetworkName: 'alz-spoke-${subscriptionNameUnique}-${location}'
    virtualNetworkPeeringEnabled: true
    hubNetworkResourceId: '/subscriptions/xxx/resourceGroups/rg-alz-connectivity/providers/Microsoft.Network/virtualNetworks/alz-hub-${location}'
    virtualNetworkResourceGroupLockEnabled: false
    virtualNetworkUseRemoteGateways: false
  }
}

Let me know if you need anything else.

[sebassem]

@cspring86 Thanks Chris, we have identified the problem and a PR is on the way

[cspring86]

@cspring86 Thanks Chris, we have identified the problem and a PR is on the way

Amazing, thanks @sebassem !

[cspring86]

@sebassem Unrelated to this issue, but I've just tried to create a new subscription using 0.1.4 version (the upgrade you requested as part of this issue) of the sub-vending module and I'm getting an InternalServerError.

If I downgrade to 0.1.2 (my previous version), it works.

The error also happens in 0.1.3.

Is this a known issue or shall I raise another issue here?

[sebassem]

@sebassem Unrelated to this issue, but I've just tried to create a new subscription using 0.1.4 version (the upgrade you requested as part of this issue) of the sub-vending module and I'm getting an InternalServerError.

If I downgrade to 0.1.2 (my previous version), it works.

The error also happens in 0.1.3.

Is this a known issue or shall I raise another issue here?

Hi @cspring86 , this is a current issue in Azure and on the way to be resolved. Once it's resolved I will submit the PR for this GitHub issue