Changing VirtualNetwork causes Azure to silently re-create child resources and fail

Problem

This is an unfortunate situation where a change to a azure.network.VirtualNetwork resource causes Azure (not Pulumi) to attempt to re-create the network and child resources out of band of Pulumi causing a scary and confusing error message from Azure. Even worse; if the dependent subnet in this case is not in use, Azure will silently re-create the subnet and Pulumi's state will be inaccurate.

In particular changing addressSpaces on azure.network.VirtualNetwork causes this situation.

Suggestions for a fix

Mark addressSpaces as "force new" so that Pulumi does the re-creation and appropriately takes care of dependencies.

Steps to reproduce

git clone https://gist.github.com/3cb019fdeda291dd3f618cc432e536f7.git && cd 3cb019fdeda291dd3f618cc432e536f7
pulumi stack init test1
pulumi up -y
pulumi config set --path 'addressSpaces[0]' "10.0.4.0/22"
pulumi up -y

Error Output

% pulumi up -y --skip-preview                               
Updating (test)

View Live: https://app.pulumi.com/clstokes/demo-azure-ts-webserver/test/updates/5

     Type                             Name                          Status                  Info
     pulumi:pulumi:Stack              demo-azure-ts-webserver-test  **failed**              1 error
 ~   └─ azure:network:VirtualNetwork  main                          **updating failed**     [diff: ~addressSpaces]; 1 error

Diagnostics:
  pulumi:pulumi:Stack (demo-azure-ts-webserver-test):
    error: update failed

  azure:network:VirtualNetwork (main):
    error: 1 error occurred:
        * updating urn:pulumi:test::demo-azure-ts-webserver::azure:network/virtualNetwork:VirtualNetwork::main: Error Creating/Updating Virtual Network "main1c78d7d7" (Resource Group "main8a8521d5"): network.VirtualNetworksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InUseSubnetCannotBeDeleted" Message="Subnet mainb6c0c94c is in use by /subscriptions/32b9cb2e-69be-4040-80a6-02cd6b2cc5ec/resourceGroups/main8a8521d5/providers/Microsoft.Network/networkInterfaces/maina24484ee/ipConfigurations/main and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet." Details=[]

Resources:
    2 unchanged

Duration: 7s

Workaround

Not a workaround, but a way to guard against this would be to use protect: true on dependent resources. Docs on that are here

We have experienced exactly the same issue and here are some of my findings.

Consider the following 2 versions of Stack implementation:

Version 1

public MyStack()
    {
        var resourceGroup = new ResourceGroup("exampleResourceGroup", new ResourceGroupArgs
        {
            Name = "iac-pulumi-poc2-rg"
        });
        var exampleNetworkSecurityGroup = new NetworkSecurityGroup("nsg", new NetworkSecurityGroupArgs
        {
            Location = resourceGroup.Location,
            ResourceGroupName = resourceGroup.Name,
        });
        var virtualNetwork = new VirtualNetwork("vnet", new VirtualNetworkArgs
        {
            Name = "iac-pulumi-poc2-vnet",
            Location = resourceGroup.Location,
            ResourceGroupName = resourceGroup.Name,
            AddressSpaces = 
            {
                "10.0.0.0/16",
            },
            Tags = 
            {
                { "environment", "dev" },
            },
        });

        var aksSubnet = new Subnet("aks", new SubnetArgs
        {
            Name = "aks",
            VirtualNetworkName = virtualNetwork.Name,
            ResourceGroupName = resourceGroup.Name,
            AddressPrefixes = "10.0.0.0/20"
        });

        var nic = new NetworkInterface("nic", new NetworkInterfaceArgs
        {
            Location = resourceGroup.Location,
            ResourceGroupName = resourceGroup.Name,
            IpConfigurations = 
            {
                new NetworkInterfaceIpConfigurationArgs
                {
                    Name = "internal",
                    SubnetId = aksSubnet.Id,
                    PrivateIpAddressAllocation = "Dynamic",
                }
            }
        });        
    }

First time pulumi up will provision vnet, subnet and allocate private IP for nic. Changing vnet tags and run pulumi up will give the following error message

Error Creating/Updating Virtual Network "iac-pulumi-poc2-vnet" (Resource Group "iac-pulumi-poc2-rg"): network.VirtualNetworksClient#CreateOrUpda
te: Failure sending request: StatusCode=400 -- Original Error: Code="InUseSubnetCannotBeDeleted" Message="Subnet aks is in use by /subscriptions/.../resourceGroups/iac-pulumi-poc2-rg/providers/Micr
osoft.Network/networkInterfaces/nic0238c9b2/ipConfigurations/internal and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet." Details=[]

Version 2

using Pulumi;
using Pulumi.Azure.ContainerService;
using Pulumi.Azure.ContainerService.Inputs;
using Pulumi.Azure.Core;
using Pulumi.Azure.Network;
using Pulumi.Azure.Network.Inputs;
using Pulumi.Azure.Storage;

class MyStack : Stack
{
    public MyStack()
    {
        var resourceGroup = new ResourceGroup("exampleResourceGroup", new ResourceGroupArgs
        {
            Name = "vipps-pulumi-poc2-rg"
        });
        var exampleNetworkSecurityGroup = new NetworkSecurityGroup("nsg", new NetworkSecurityGroupArgs
        {
            Location = resourceGroup.Location,
            ResourceGroupName = resourceGroup.Name,
        });
        var virtualNetwork = new VirtualNetwork("vnet", new VirtualNetworkArgs
        {
            Name = "vipps-pulumi-poc2-vnet",
            Location = resourceGroup.Location,
            ResourceGroupName = resourceGroup.Name,
            AddressSpaces = 
            {
                "10.0.0.0/16",
            },
            Subnets = 
            {
                new VirtualNetworkSubnetArgs
                {
                    Name = "aks",
                    AddressPrefix = "10.0.0.0/20",
                }
            }, 
            Tags = 
            {
                { "environment", "dev" },
            },
        });

        var aksSubnet = virtualNetwork.Subnets.First();

        var nic = new NetworkInterface("nic", new NetworkInterfaceArgs
        {
            Location = resourceGroup.Location,
            ResourceGroupName = resourceGroup.Name,
            IpConfigurations = 
            {
                new NetworkInterfaceIpConfigurationArgs
                {
                    Name = "internal",
                    SubnetId = aksSubnet.Apply(x => x.Id),
                    PrivateIpAddressAllocation = "Dynamic",
                }
            }
        });        
    }
}

Provision it with pulumi up, update vnet tags and update it again with pulumi up will work without any problem.

Some thoughts

The difference between these 2 versions is that in the second version, subnet configuration is part of the VirtualNetwork resource.

Actually, if we extract ARM template for the VNet from our Stack, the aks subnet resource exists both as part of Microsoft.Network/virtualNetworks resource and as a separate Microsoft.Network/virtualNetworks/subnets resource.

{
    "type": "Microsoft.Network/virtualNetworks",
    "apiVersion": "2020-05-01",
    "name": "[parameters('virtualNetworks_iac_pulumi_poc2_vnet_name')]",
    "location": "westeurope",
    "tags": {
        "environment": "dev"
    },
    "properties": {
        "addressSpace": {
            "addressPrefixes": [
                "10.0.0.0/16"
            ]
        },
        "dhcpOptions": {
            "dnsServers": []
        },
        "subnets": [
            {
                "name": "aks",
                "properties": {
                    "addressPrefix": "10.0.0.0/20",
                    "serviceEndpoints": [],
                    "delegations": [],
                    "privateEndpointNetworkPolicies": "Enabled",
                    "privateLinkServiceNetworkPolicies": "Enabled"
                }
            }
        ],
        "virtualNetworkPeerings": [],
        "enableDdosProtection": false,
        "enableVmProtection": false
    }
},
{
    "type": "Microsoft.Network/virtualNetworks/subnets",
    "apiVersion": "2020-05-01",
    "name": "[concat(parameters('virtualNetworks_iac_pulumi_poc2_vnet_name'), '/aks')]",
    "dependsOn": [
        "[resourceId('Microsoft.Network/virtualNetworks', parameters('virtualNetworks_iac_pulumi_poc2_vnet_name'))]"
    ],
    "properties": {
        "addressPrefix": "10.0.0.0/20",
        "serviceEndpoints": [],
        "delegations": [],
        "privateEndpointNetworkPolicies": "Enabled",
        "privateLinkServiceNetworkPolicies": "Enabled"
    }
}

Maybe pulumi should also treat subnets both as part of VirtualNetwork resource and as a separate resource. That is - if I create subnet and "attach" it to the VNet by using VirtualNetworkName (this is very useful and needed use-case), this subnet should be added as an item to the subnets array of VirtualNetwork resource.

pulumi / pulumi-azure