New-AzureRmResourceGroupDeployment: loop dependency

[akuryan]

Is there a way to add a dependendency in a loop? I am trying to add several allow all rules to NSG via loop and it fails. I am using such a template:

 {
            "copy": { 
                    "name": "allowCopy", 
                    "count": "[length(parameters('allowedCIDRs'))]" 
                    },  
            "apiVersion": "[variables('apiVersionString')]",
            "type": "Microsoft.Network/networkSecurityGroups/securityRules",
            "name": "[concat(parameters('networkSecurityGroupName'), '/', parameters('allowedCIDRsNames')[copyIndex()])]",
            "location": "[resourceGroup().location]",
              "dependsOn": [
                "[concat('Microsoft.Network/networkSecurityGroups/', parameters('networkSecurityGroupName'))]"
              ],
                "properties": {                     
                    "description": "[concat('Allow everything from ', parameters('allowedCIDRsNames')[copyIndex()])]",
                    "protocol": "*",
                    "sourcePortRange": "*",
                    "destinationPortRange": "*",
                    "sourceAddressPrefix": "[parameters('allowedCIDRs')[copyIndex()]]",
                    "destinationAddressPrefix": "*",
                    "access": "Allow",
                    "priority": "[copyIndex(100)]",
                    "direction": "Inbound"
                }

    }

where allowedCIDRs and allowedCIDRsNames are arrays of 9 elements each.

And it fails with following error: New-AzureResourceGroupDeployment : 13:55:12 PM - Resource Microsoft.Network/networkSecurityGroups/securityRules 'NSGName/RuleName' failed with message 'Another operation on this or dependent resource is in progress. Toretrieve status of the operation use uri: '

Each time on different rules

[markcowl]

@vivsriaus Any ideas?

[yangl900]

it appears the security rules does not allow concurrent update (copies will update the rule in concurrent by default). We will follow up with owner of this API.

An workaround is to set a dependency on the previous rule, so it does not run in parallel. { // provision rule[0] }, { "copy": {} "name": "[copyIndex(1)]" //offset 1 so start with second in the array "dependsOn": ".../rules/copyIndex()" //set dependency on previous rule }

[akuryan]

This solution works, but requires a lot of time to provision though :( My Powershell session ends, and Get-AzureRmResourceGroupDeployment is confirming that deployment took around 20 minutes. Here is samples I used (just in case they will be useful for someone): Template itself - https://github.com/akuryan/ConfigurationHelpers/blob/master/Azure/ArmTemplates/nsg-provisioning.json Some explanation - https://github.com/akuryan/ConfigurationHelpers/blob/master/Azure/ArmTemplates/nsg-provisioning.md

[ravbhatnagar]

@markjbrown - Any ideas on this issue?

[akuryan]

Also, I shall note, that provisioning ends with error in powershell console, but in portal it demonstrates that deployment was ok

[ravbhatnagar]

This should be fixed now. please let us know if you still run into this issue.

[nik-shornikov]

i don't believe this issue is about a specific resource type and i don't believe it's fixed. my experience has been a max of four concurrent subnet writes to a vnet, and anything beyond that, the error above. the hack to force serial does lead to very long deployments, and is a hack to begin with. if arm templates have a value proposition of being declarative, then working out degrees of parallelism and/or observing "retryable" is something this api should learn to do.

Multiple instances of Microsoft.Network/virtualNetworks/subnets/write reliably produce:

{
  "error": {
    "code": "AnotherOperationInProgress",
    "message": "Another operation on this or dependent resource is in progress. To retrieve status of the operation use uri: https://management.azure.com/subscriptions/*/providers/Microsoft.Network/locations/centralus/operations/*?api-version=2016-06-01.",
    "details": []
  }
}

[yangl900]

@nik-shornikov The issue is mostly seen in network resources, it is specific to the resource provider. @ravbhatnagar could you loop the network folks see if there is plan to fix this?

[ravbhatnagar]

@franksiegemund - Can you please take a look?

[laughtonsm]

I too am still seeing this issue when trying to perform a similar operation. I'm creating several instances of type Microsoft.Network/networkSecurityGroups/securityRules, which are dependent on a single instance of type Microsoft.Network/networkSecurityGroups. The returned error is as follows:

Template deployment returned the following errors: Resource Microsoft.Network/networkSecurityGroups/securityRules 'nsgazuDEVsvly01/80.80.80.80' failed with message '{ "error": { "code": "AnotherOperationInProgress", "message": "Another operation on this or dependent resource is in progress. To retrieve status of the operation use uri: https://management.azure.com/subscriptions/d76012fa-f8c1-480e-94bf-ffb9ce0e0321/providers/Microsoft.Network/locations/northeurope/operations/5979d8a6-2bdc-40ec-820f-fead1153592d?api-version=2015-06-15.", "details": [] } }' Template output evaluation skipped: at least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.

[jensmh]

I'm having the same issue trying to create multiple subnets in an ARM template copy loop- randomly fails on one or more subnets with the "AnotherOpertionInProgress" error message. Looks like there is a maximum concurency of around 4 writes, as mentioned by @nik-shornikov above.

[ravbhatnagar]

@jensmh @laughtonsm @nik-shornikov - While we try to get the right fix for this issue, may be you can try the serial copies support in ARM as documented here - https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-multiple#serial-copy

Sample - https://github.com/rjmax/Build2017/blob/master/Act1.TemplateEnhancements/Chapter03.SerialCopies.json

@markjbrown to comment on the original issue around parallelism.

[AlexKeySmith]

Unfortunately "serial" works at first for a small amount, but if you have a large batch to add it fails.

[AlexKeySmith]

Scrap my last comment, I think it was due to a deployment that had failed to stop and was still running stagnant.

However "serial" is incredibly slow when adding a large batch of rules (although I do see it's a new feature which is cool).

[shadewlkr]

Has this item been resolved? I'm getting the same errors when I deploy subnets that have a dependency on a network security group with a large number of rules. Sometimes the deployment completes successfully, other times it fails with the "Another Operation in Progress" error. All of my subnets have dependencies, but they seem to be ignored or there are too many parallel operations going on at once. Thanks!

[cormacpayne]

@ravbhatnagar ping

[HQJaTu]

Has this item been resolved? I'm getting the same errors when I deploy subnets that have a dependency on a network security group with a large number of rules. Sometimes the deployment completes successfully, other times it fails with the "Another Operation in Progress" error. All of my subnets have dependencies, but they seem to be ignored or there are too many parallel operations going on at once. Thanks!

I'm experiencing exactly the same.

The worst part is, that given timings, this can fail or succeed. I'm trying to add enough dependencies to disable any parallelism, but the security rules are truly the headache here.

[mahmoodr786]

A workaround I did is made one subnet depend on the one before it. This is slower however it does not fail. "dependsOn": [ "[resourceId('Microsoft.Network/virtualNetworks/subnets', 'VNET','theSubnetBeforeThisOne')]" ]

[HQJaTu]

A workaround I did is made one subnet depend on the one before it. This is slower however it does not fail. "dependsOn": [ "[resourceId('Microsoft.Network/virtualNetworks/subnets', 'VNET','theSubnetBeforeThisOne')]" ]

Yes, I realized the same. Given my setup, which consists of 3 subnets, I did exactly what you describe there. In my template two of the subnets depend on previous subnet to force them to be created in a chain.

[CharmanderJieniJieni]

I am hitting same issue, triggering nested ARM deployment through CLI with multiple subnets randomly received error: "AnotherOperationInProgress". Do we have an estimate on fix of this one?

[HQJaTu]

If I'd have to guess, the limitation of max. 4 operations at a time originates from some sort of network appliance where the settings are set into. Failing to account for this in ARM-template manager is weird.

Maybe Microsoft should test their systems before going public?