"OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" when adding a secondary IP to a NIC in the backend pool of an Azure Load Balancer

BrandonStiff commented 4 years ago

Summary

I can't add a secondary IP configuration on a NIC if it's part of an Azure Load Balancer back end with an outbound rule assigned using Terraform.

It's important to note that you can manually go into the Azure Portal and add a secondary IP configuration exactly as configured in Terraform without issue.

Terraform (and AzureRM Provider) Version

Terraform v0.13.0

registry.terraform.io/hashicorp/azurerm v2.33.0

Affected Resource(s)

azurerm_network_interface.primary

Terraform Configuration Files

variable "tenant_id" {
  type    = string
}

variable "subscription_id" {
  type    = string
}

variable "location" {
  type    = string
  default = "eastus"
}

variable "resource_group_name" {
  description = "Specifies the RG to create and put all the resources in."
  type    = string
  default = "test-rg"
}

variable "virtual_network_name" {
  description = "Specifies the vNet the subnet the VM network interface will be attached to.  This must already exist."
  type    = string
}

variable "subnet_name" {
  description = "Specifies the subnet the VM network interface will be attached to.  This must already exist."
  type    = string
}

variable "subnet_resource_group_name" {
  description = "Specifies the RG the subnet is inside.  This must already exist."
  type    = string
}

provider "azurerm" {
  subscription_id = var.subscription_id
  tenant_id       = var.tenant_id
  features {}

  skip_provider_registration = true
}

data "azurerm_subnet" "subnet" {
  name                 = var.subnet_name
  resource_group_name  = var.subnet_resource_group_name
  virtual_network_name = var.virtual_network_name
}

resource "azurerm_resource_group" "rg" {
  name     = var.resource_group_name
  location = var.location
}

resource "azurerm_network_interface" "primary" {
  name                = "primary-nic"
  location            = var.location
  resource_group_name = azurerm_resource_group.rg.name

  ip_configuration {
    primary                       = true
    name                          = "primary"
    subnet_id                     = data.azurerm_subnet.subnet.id
    private_ip_address_allocation = "Dynamic"
  }

  #  Run this Terraform once, then comment this in and run again to reproduce the issue.
  ip_configuration {
    primary                       = false
    name                          = "secondary-01"
    subnet_id                     = data.azurerm_subnet.subnet.id
    private_ip_address_allocation = "Dynamic"
  }

  enable_accelerated_networking = true
}

resource "azurerm_linux_virtual_machine" "vm" {
  name                = "test-vm"
  resource_group_name = azurerm_resource_group.rg.name
  location            = var.location
  size                = "Standard_D3_v2"

  admin_username                  = "adminroot"
  admin_password                  = "Welcome01"
  disable_password_authentication = false

  network_interface_ids = [
    azurerm_network_interface.primary.id
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  plan {
    name      = "byol"
    product   = "vmseries-flex"
    publisher = "paloaltonetworks"
  }

  source_image_reference {
    publisher = "paloaltonetworks"
    offer     = "vmseries-flex"
    sku       = "byol"
    version   = "9.1.3"
  }
}

resource "azurerm_public_ip" "public_lb_ip" {
  name                = "test-pip"
  sku                 = "Standard"
  location            = var.location
  resource_group_name = azurerm_resource_group.rg.name
  allocation_method   = "Static"
}

resource "azurerm_lb" "public_lb" {
  name                = "test-public-lb"
  location            = var.location
  resource_group_name = azurerm_resource_group.rg.name
  sku                 = "Standard"

  frontend_ip_configuration {
    name                 = "test-fw-public-lb"
    public_ip_address_id = azurerm_public_ip.public_lb_ip.id
  }
}

resource "azurerm_lb_backend_address_pool" "public_lb_backend" {
  name                = "test-backend"
  resource_group_name = azurerm_resource_group.rg.name
  loadbalancer_id     = azurerm_lb.public_lb.id
}

resource "azurerm_network_interface_backend_address_pool_association" "backend" {
  network_interface_id    = azurerm_network_interface.primary.id
  ip_configuration_name   = "primary"
  backend_address_pool_id = azurerm_lb_backend_address_pool.public_lb_backend.id
}

resource "azurerm_lb_outbound_rule" "public_lb_outbound_rule" {
  name                    = "test-outbound"
  resource_group_name     = azurerm_resource_group.rg.name
  loadbalancer_id         = azurerm_lb.public_lb.id
  protocol                = "All"
  backend_address_pool_id = azurerm_lb_backend_address_pool.public_lb_backend.id

  frontend_ip_configuration {
    name = "test-fw-public-lb"
  }
}

Debug Output

https://gist.github.com/BrandonStiff/4bdc1fcc1e5b7bc237e4754617bba8fb

Expected Behavior

Both Terraform apply commands should succeed and add a second IP configuration to the network interface.

Actual Behavior

The first Terraform apply succeeds, but after commenting in the secondary IP configuration and running again, the second apply fails with the following error:

Error: Error updating Network Interface "primary-nic" (Resource Group "test-rg"): network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="OutboundRuleCannotBeUsedWithBackendAddressPoolThatIsReferencedBySecondaryIpConfigs" Message="OutboundRule /subscriptions/xyz-123/resourceGroups/test-rg/providers/Microsoft.Network/loadBalancers/test-public-lb/outboundRules/test-outbound cannot be used with Backend Address Pool /subscriptions/xyz-123/resourceGroups/test-rg/providers/Microsoft.Network/loadBalancers/test-public-lb/backendAddressPools/test-backend that contains Secondary IPConfig /subscriptions/xyz-123/resourceGroups/test-rg/providers/Microsoft.Network/networkInterfaces/primary-nic/ipConfigurations/secondary-01" Details=[]

on main.tf line 55, in resource "azurerm_network_interface" "primary": 55: resource "azurerm_network_interface" "primary" {

Steps to Reproduce

Run the terraform as shown above: terraform apply
Verify it runs successfully
Comment in the commented out code on lines 68-73 to add a secondary IP configuration
Run the terraform again: terraform apply

Manual Workaround

You can do this manually via the portal without issue:

Navigate to the network interface in the Azure Portal
Click IP configurations in the lefthand menu.
Click the Add button.
Fill in the name of the IP configuration
Specify Dynamic
Specify Disassociate
Click OK

No errors occur.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

ricardoedo commented 4 years ago

Same issue here, but I already have the secondary IP and the backend pool defined, but I get the error adding the outbound rule.

jdelforno commented 2 years ago

https://docs.microsoft.com/en-us/azure/load-balancer/outbound-rules#limitations

Outbound rules can only be applied to primary IP configuration of a NIC. You can't create an outbound rule for the secondary IP of a VM or NVA. Multiple NICs are supported

This is a limitation in Azure, that's been known about for years unfortunately.

rcskosir commented 7 months ago

Thank you for taking the time to raise this! I am going to close this with @jdelforno‘s response as an answer. If you have future questions, I suggest using the Community Resources, such as the Azure Provider forum.

github-actions[bot] commented 6 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm