Changing a vnet & subnet's CIDR fails

roy-work commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.14.2
+ provider registry.terraform.io/hashicorp/azuread v1.4.0
+ provider registry.terraform.io/hashicorp/azurerm v2.52.0

Your version of Terraform is out of date! The latest version
is 0.14.9. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

azurerm_virtual_network
azurerm_subnet

Terraform Configuration Files

Note the comments below; the previous values are shown in the comments, and are part of the bug here.

resource "azurerm_virtual_network" "example_vnet" {
  name = "example"
  location = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  address_space = ["10.200.0.0/16"]  # changed from 10.0.0.0/16
}

resource "azurerm_subnet" "example_main_subnet" {
  name = "main"
  resource_group_name = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes = ["10.200.0.0/24"]  # changed from 10.0.0.0/24
}

Example Output

Error: creating/updating Virtual Network: (Name "example" / Resource Group "example"): network.VirtualNetworksClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="NetcfgSubnetRangeOutsideVnet" Message="Subnet 'main' is not valid because its IP address range is outside the IP address range of virtual network 'example'." Details=[]

Expected Behaviour

Ideally, the subnet & vnet should have been updated to the new values.

Actual Behaviour

This is a complex change, and Terraform / the azurerm provider is executing the wrong sequence of API calls. Terraform is first trying to update the vnet to the new range that's been provided in the configuration. But we can't do that: there's still a subnet in the old range (which what Azure is trying to say in the error).

The change is possible, but it needs to happen over multiple API calls, involving multiple resources:

The new range has to be first added to the vnet
The subnet has to then be updated to use the new range
The old range has to be removed from the vnet

And that's in the simple case of not having anything using the subnet. I'm not sure if it is possible in the case that there are actually any resources in the subnet. (But that wasn't part of what I tried to do today, thankfully.)

Steps to Reproduce

Change a vnet (and associated resources) from one CIDR to another.
terraform apply

amasover commented 2 years ago

I agree the provider should handle the "simple" case where there's no resources actually in any of the vnet's subnets. Maybe we could get the ip allocations / ip configurations of the subnets from the API to determine whether they're empty?

I'm not sure if it is possible in the case that there are actually any resources in the subnet.

If there are resources in the subnet it seems like things get very complicated very quickly. I tried doing this via terraform today with some resources I had deployed, and I had to manually taint the following resources:

All NICs in the subnets
All VMs attached to the NICs
All other resources in the subnets (I also had an azurerm_virtual_network_gateway)

(After tainting those resources, I still had to follow @roy-work's steps 1-3 because we don't yet handle the "simple" case.)

It would be nice if Terraform could help with some of the heavy lifting here with calculating which resources need to be force-replaced during a CIDR change (for example: currently tainting a subnet will trigger terraform to update-in-place the subnet_id of the ip_configuration of the NIC, rather than force-replacing the NIC).

Perhaps we should split this into two separate issues. One for the "simple" case, and one for the more complicated one (if it's even possible to tackle the complicated one).

rcskosir commented 11 months ago

Thanks for opening this issue. This was a problem in the 2.x version of the provider which is no longer actively maintained. If this is still an issue with the 3.x version of the provider please do let us know by opening a new issue, thanks!