Connection Monitor fails to create/update when attached VM is recreated

[stusklinar]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.6.2

AzureRM Provider Version

3.77.0

Affected Resource(s)/Data Source(s)

azurerm_network_connection_monitor

Terraform Configuration Files

provider "azurerm" {
  features {
  }
}

resource "azurerm_resource_group" "rg" {
  name     = "tf_bug"
  location = "uksouth"
}

resource "azurerm_virtual_network" "network" {
  name                = "example-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "subnet" {
  name                 = "internal"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.network.name
  address_prefixes     = ["10.0.2.0/24"]
}

resource "azurerm_network_interface" "nic" {
  name                = "nic"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.subnet.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_network_security_group" "nsg" {
  name                = "tfbugnsg"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_network_interface_security_group_association" "nic_nsg" {
  network_interface_id      = azurerm_network_interface.nic.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

resource "azurerm_linux_virtual_machine" "vm" {
  name                = "tfbuglocalvm"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  network_interface_ids = [azurerm_network_interface.nic.id]
  size                  = "Standard_B2s"

  disable_password_authentication = false

  os_disk {
    name                 = "tfbuglocalvmdisk"
    storage_account_type = "Standard_LRS"
    caching              = "ReadWrite"
    disk_size_gb         = 30
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "latest"
  }

  admin_username = "myadmin"
  admin_password = "mySupersecurepassword3"

  computer_name = "tfbuglocalvm"
}

resource "azurerm_virtual_machine_extension" "networkwatcher" {

  name                       = "${azurerm_linux_virtual_machine.vm.name}-netwatcherextension"
  virtual_machine_id         = azurerm_linux_virtual_machine.vm.id
  publisher                  = "Microsoft.Azure.NetworkWatcher"
  type                       = "NetworkWatcherAgentLinux"
  type_handler_version       = "1.4"
  auto_upgrade_minor_version = true

}

resource "azurerm_storage_account" "nsg_logs" {
  name                     = "tfbugnsglogs"
  location                 = azurerm_resource_group.rg.location
  resource_group_name      = azurerm_resource_group.rg.name
  account_tier             = "Standard"
  account_replication_type = "GRS"
}

resource "azurerm_log_analytics_workspace" "law" {
  name                = "tfbug"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  sku                 = "PerGB2018"
}

resource "azurerm_network_watcher_flow_log" "this" {

  network_watcher_name = "NetworkWatcher_uksouth"
  resource_group_name  = azurerm_resource_group.rg.name
  location             = "uksouth"
  name                 = "tfbug_logs"

  network_security_group_id = azurerm_network_security_group.nsg.id
  storage_account_id        = azurerm_storage_account.nsg_logs.id
  enabled                   = true
  version                   = 2

  retention_policy {
    enabled = true
    days    = 365
  }

  traffic_analytics {
    enabled               = true
    workspace_id          = azurerm_log_analytics_workspace.law.workspace_id
    workspace_region      = azurerm_log_analytics_workspace.law.location
    workspace_resource_id = azurerm_log_analytics_workspace.law.id
    interval_in_minutes   = 10
  }

}

resource "azurerm_network_watcher" "nw" {
  name                = "acctestnw"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_network_connection_monitor" "this" {
  name = "tf-bug-conn-mon"

  network_watcher_id = azurerm_network_watcher.nw.id
  location           = azurerm_network_watcher.nw.location

 endpoint {
    name                 = "www.google.com"
    address              = "www.google.com"
    target_resource_type = "ExternalAddress"
  }

  endpoint {
    name               = azurerm_linux_virtual_machine.vm.name
    target_resource_id = azurerm_linux_virtual_machine.vm.id

    filter {
      item {
        address = azurerm_linux_virtual_machine.vm.id
        type    = "AgentAddress"
      }

      type = "Include"
    }
  }

  test_configuration {
    name                      = "Https"
    protocol                  = "Http"
    test_frequency_in_seconds = 60

    http_configuration {
      method       = "Get"
      port         = 443
      prefer_https = true
    }
  }

  test_group {
    name                     = "tg-outbound-internet"
    destination_endpoints    = ["www.google.com"]
    source_endpoints         = [azurerm_linux_virtual_machine.vm.name]
    test_configuration_names = ["Https"]
  }

}

Debug Output/Panic Output

azurerm_virtual_machine_extension.networkwatcher: Creating...
azurerm_virtual_machine_extension.networkwatcher: Still creating... [10s elapsed]
azurerm_virtual_machine_extension.networkwatcher: Still creating... [20s elapsed]
azurerm_virtual_machine_extension.networkwatcher: Still creating... [30s elapsed]
azurerm_virtual_machine_extension.networkwatcher: Creation complete after 32s [id=/subscriptions/-----------/resourceGroups/tf_bug/providers/Microsoft.Compute/virtualMachines/tfbuglocalvm/extensions/tfbuglocalvm-netwatcherextension]
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for azurerm_network_connection_monitor.this to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/azurerm" produced an invalid new value for .endpoint: planned set element
│ cty.ObjectVal(map[string]cty.Value{"address":cty.StringVal("www.google.com"), "coverage_level":cty.NullVal(cty.String), "excluded_ip_addresses":cty.SetValEmpty(cty.String),
│ "filter":cty.ListValEmpty(cty.Object(map[string]cty.Type{"item":cty.Set(cty.Object(map[string]cty.Type{"address":cty.String, "type":cty.String})), "type":cty.String})), "included_ip_addresses":cty.SetValEmpty(cty.String),
│ "name":cty.StringVal("www.google.com"), "target_resource_id":cty.StringVal(""), "target_resource_type":cty.StringVal("ExternalAddress")}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behaviour

Connection monitor and all resources dropped

Actual Behaviour

Looks like some residual resources may be left over on the VM destroy

Steps to Reproduce

Set the VM SKU/Offer to be:

offer     = "0001-com-ubuntu-server-focal"
sku       = "20_04-lts"

And run a terraform apply

Then once it's completed. change the VM SKU/Offer to another working SKU/Offer (example below) that will force a recreation of the VMs and connection monitors:

offer     = "0001-com-ubuntu-server-jammy"
sku       = "22_04-lts"

Rerun terraform apply, which should force a destroy and present the issye

Important Factoids

No response

References

https://github.com/hashicorp/terraform-provider-azurerm/issues/22659 - possibly same issue, this just has a more details and steps to re-create.

[teowa]

Hi @stusklinar , thanks for taking the time to submit this! I am still debuging on the source code, could you try use terraform apply -replace azurerm_network_connection_monitor after VM recreation, which might work around this for now.

[stusklinar]

@teowa - Thanks, will give it a shot, but that won't work for our automated pipelines - may help resolve the immediate though!