Terraform wipes out AGIC's backends on each deploy

[justinmchase]

Describe the bug When I create a App Gateway with terraform I have to include a default backend pool, listener, settings, rule, etc.

Later when I apply application gateway to AKS it takes control of the app gateway and replaces all of these. The next time I deploy terraform detects its settings are missing and recreates them all. When this happens it leaves the app gateway in a broken state until I manually recreate an Ingress object.

To Reproduce

1. Create an app gateway with terraform
2. Create aks and attach the gateway to aks
3. Deploy a 2nd time with terraform

Ingress Controller details The default that comes with aks right now

• Output of kubectl logs <ingress controller>. NA`
• Any Azure support tickets associated with this issue. NA

[audunsolemdal]

Had this issues a couple of years ago, simply add ignore_changes to all the resources you mentioned and it works OK. Don't think there is a better way to go about it.

[justinmchase]

I tried to set the ignore_changes also but now its erorring on the unexpected backend pools rather than just deleting them. Do you happen to have an example of the terraform for this?

bp-service-api-8080-8080-api was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.

This resource is generated by the ingress controller and is naturally not in my terraform template.

module.shared.module.resource_group.azurerm_resource_group.this: Refreshing state... [id=/subscriptions/0816a7b7-daf6-4f6a-8d35-0297a9da1f73/resourceGroups/testservice-stpr99]
╷
│ Error: updating Application Gateway: (Name "api-appgateway" / Resource Group "testservice-stpr99-common"): network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidResourceReference" Message="Resource /subscriptions/0816a7b7-daf6-4f6a-8d35-0297a9da1f73/resourceGroups/testservice-stpr99-common/providers/Microsoft.Network/applicationGateways/api-appgateway/probes/pb-service-api-8080-api referenced by resource /subscriptions/0816a7b7-daf6-4f6a-8d35-0297a9da1f73/resourceGroups/testservice-stpr99-common/providers/Microsoft.Network/applicationGateways/api-appgateway/backendHttpSettingsCollection/bp-service-api-8080-8080-api was not found. Please make sure that the referenced resource exists, and that both resources are in the same region." Details=[]
│ 
│   with module.common.module.compute.module.appgateway.azurerm_application_gateway.api,
│   on ../common/compute/appgateway/api.tf line 1, in resource "azurerm_application_gateway" "api":
│    1: resource "azurerm_application_gateway" "api" {

[audunsolemdal]

resource "azurerm_application_gateway" "agic" {
  name                = var.agic_name
  resource_group_name = azurerm_resource_group.waf_test_agic_rg.name
  location            = azurerm_resource_group.waf_test_agic_rg.location
  firewall_policy_id  = azurerm_web_application_firewall_policy.waf_policy.id

  sku {
    name     = var.application_gateway_sku.name
    tier     = var.application_gateway_sku.tier
    capacity = 1
  }

  ssl_certificate {
    name                = "mycert"
    key_vault_secret_id = "myid"
  }
  zones = var.application_gateway_zones

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.managed_agic_id.id]
  }

  gateway_ip_configuration {
    name      = "gateway-ip-configuration-test"
    subnet_id = "/mysubnetid"
  }

  frontend_port {
    name = local.frontend_port_name
    port = 80
  }

  frontend_port {
    name = local.frontend_https_port_name
    port = 443
  }

  frontend_ip_configuration {
    name                 = local.frontend_ip_configuration_name
    public_ip_address_id = azurerm_public_ip.pip_agic.id
  }

  backend_address_pool {
    name         = "aks-internal-lb"
    ip_addresses = ["123.123.123.123"]
  }

  backend_http_settings {
    name                  = "dummy_required_setting"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 60
  }

  http_listener {
    name                           = "dummy_required_listener"
    frontend_ip_configuration_name = local.frontend_ip_configuration_name
    frontend_port_name             = local.frontend_port_name
    protocol                       = "Http"
  }

  request_routing_rule {
    name                       = "dummy_required_rule"
    rule_type                  = "Basic"
    http_listener_name         = "dummy_required_listener"
    backend_address_pool_name  = "aks-internal-lb"
    backend_http_settings_name = "dummy_required_setting"
    priority                   = 6000
  }
  lifecycle {
    ignore_changes = [
      http_listener,
      probe,
      tags,
      request_routing_rule,
      backend_address_pool,
      backend_http_settings,
      url_path_map,
      frontend_port,
      redirect_configuration,
    ]
  }
}

If the issue bp-service-api-8080-8080-api was not found I would attempt trying to delete it via the azure portal and see if that solves things

[justinmchase]

Yup I needed to add more things to the ignore and that did the trick.

I expected this to cause everything to be updated still even if a non-ignored field is updated but it doesn't this works as needed, thanks!