search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.55k stars 4.62k forks source link

azurerm_application_gateway creation fails without descriptive error when access to Key Vault fails. #12910

Closed mitchellkingsley closed 1 year ago

mitchellkingsley commented 3 years ago

Terraform (and AzureRM Provider) Version

Terraform v1.0.1 on linux_amd64

Affected Resource(s)

Expected Behaviour

When creating an application gateway with SSL termination, if there are insufficient access permissions to the key vault containing a target certificate then the application gateway creation fails promptly with a descriptive error.

Actual Behaviour

When creating an application gateway with SSL termination, when there are insufficient access permissions to a key vault that contains a target certificate creation fails with the following error after 25-30 minutes:

╷
│ Error: waiting for create/update of Application Gateway: (Name "example_gateway" / Resource Group "example"): Code="InternalServerError" Message="An error occurred." Details=[]
│ 
│   with azurerm_application_gateway.unit,
│   on application_gateway.tf line XX, in resource "azurerm_application_gateway" "example":
│   XX: resource "azurerm_application_gateway" "example" {
│

I assume that the "insufficient access permissions" in my case is that my host IP is being blocked by the Key Vault even though it is specified as an exception in the network_acls block. See the Workaround section below for more detail.

The config for the example Key Vault and Access Policy:

resource "azurerm_key_vault" "example" {
  name                       = "example"
  resource_group_name        = azurerm_resource_group.example.name
  location                   = azurerm_resource_group.example.location
  tenant_id                  = data.azurerm_client_config.example.tenant_id
  soft_delete_retention_days = 90
  purge_protection_enabled   = false
  sku_name                   = "standard"

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
    # This would be the public IP of the executing machine.
    ip_rules       = ["xxx.xxx.xxx.xxx/32"]
  }
}

resource "azurerm_key_vault_access_policy" "example" {
  key_vault_id = azurerm_key_vault.example.id
  tenant_id    = data.azurerm_client_config.example.tenant_id
  object_id    = data.azurerm_client_config.example.object_id

  certificate_permissions = [
    "Backup",
    "Create",
    "Delete",
    "DeleteIssuers",
    "Get",
    "GetIssuers",
    "Import",
    "List",
    "ListIssuers",
    "ManageContacts",
    "ManageIssuers",
    "Purge",
    "Recover",
    "Restore",
    "SetIssuers",
    "Update"
  ]
}

The config for the example Application Gateway:

resource "azurerm_application_gateway" "example" {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location

  sku {
    name     = "Standard_v2"
    tier     = "Standard_v2"
    capacity = 2
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.example.id]
  }

  gateway_ip_configuration {
    name      = "gipconfig"
    subnet_id = azurerm_subnet.example.id
  }

  frontend_port {
    name = local.frontend_port_name_80
    port = 80
  }

  frontend_port {
    name = local.frontend_port_name_443
    port = 443
  }

  ssl_certificate {
    name                = azurerm_key_vault_certificate.example.name
    key_vault_secret_id = azurerm_key_vault_certificate.example.secret_id
  }

  frontend_ip_configuration {
    name                 = local.frontend_ip_configuration_name
    public_ip_address_id = azurerm_public_ip.example.id
  }

  backend_address_pool {
    name = local.backend_address_pool_name_admin
  }

  backend_address_pool {
    name = local.backend_address_pool_name_report
  }

  # Backend for report access.
  backend_http_settings {
    name                  = local.http_setting_name_report
    cookie_based_affinity = "Disabled"
    port                  = 3000
    protocol              = "Http"
    request_timeout       = 60
  }

  # Backend for admin access.
  backend_http_settings {
    name                  = local.http_setting_name_admin
    cookie_based_affinity = "Disabled"
    port                  = 8086
    protocol              = "Http"
    request_timeout       = 60
  }

  http_listener {
    name                           = local.listener_name_http
    frontend_ip_configuration_name = local.frontend_ip_configuration_name
    frontend_port_name             = local.frontend_port_name_80
    protocol                       = "Http"
  }

  http_listener {
    name                           = local.listener_name_https
    frontend_ip_configuration_name = local.frontend_ip_configuration_name
    frontend_port_name             = local.frontend_port_name_443
    protocol                       = "Https"
    ssl_certificate_name           = "example"
  }

  request_routing_rule {
    name               = local.request_routing_rule_name
    rule_type          = "PathBasedRouting"
    http_listener_name = local.listener_name_https
    backend_address_pool_name  = local.backend_address_pool_name_report
    backend_http_settings_name = local.http_setting_name_report
    url_path_map_name = "url_path_map"
  }

  # Route requests based off the URL path provided.
  url_path_map {
    name                               = "url_path_map"
    default_backend_address_pool_name  = local.backend_address_pool_name_report
    default_backend_http_settings_name = local.http_setting_name_report

    path_rule {
      name                       = "path2"
      paths                      = ["/path2"]
      backend_address_pool_name  = local.backend_address_pool_name_report
      backend_http_settings_name = local.http_setting_name_report
    }

    path_rule {
      name                       = "path1"
      paths                      = ["/path1"]
      backend_address_pool_name  = local.backend_address_pool_name_admin
      backend_http_settings_name = local.http_setting_name_admin
    }
  }

  # Redirect all HTTP traffic to HTTPS.
  redirect_configuration {
    name                 = "httpToHttps"
    redirect_type        = "Permanent"
    include_path         = true
    include_query_string = true
    target_listener_name = local.listener_name_https
  }

  request_routing_rule {
    name                        = "httpToHttpsRule"
    rule_type                   = "Basic"
    http_listener_name          = local.listener_name_http
    redirect_configuration_name = "httpToHttps"
  }

}

Steps to Reproduce and Workaround

In my specific case, with the above configurations, the noted error occurred after a terraform apply.

My workaround was to temporarily set the network_acls block in the azurerm_key_vault to a defaut_action of Allow during creation:

  network_acls {
    default_action = "Alllow"
    bypass         = "AzureServices"
    # This would be the public IP of the executing machine.
    ip_rules       = ["xxx.xxx.xxx.xxx/32"]
  }

Making this change allowed the terraform apply to complete successfully, removing the error quoted above. This indicates that the source of the failure arises from the allowed ip_rules I've specified. While the failure itself isn't the bug that I am looking to report (the lack of an appropriate error message is), any comments that shed light on why this failure occurs would be greatly appreciated.

Community Note

naimadswdn commented 3 years ago

I was facing the same issue. In my case, it was related to not whitelisting application gateway subnet on allowed KeyVault virtual network subnets. There was no descriptive error and it took me like 2 days to figure out what is going on.

lasoster commented 3 years ago

I was facing the same issue. In my case, it was related to not whitelisting application gateway subnet on allowed KeyVault virtual network subnets. There was no descriptive error and it took me like 2 days to figure out what is going on.

Thanks!!!!!!

spr0ut commented 2 years ago

Similar issue to @naimadswdn, but although the app gateway subnet was allowed in keyvault, the logs showed requests coming from the public ip address of the gateway which also needed to be whitelisted(?!). Recommend anyone deploying app gateway and keyvault and experiencing issues uses the logging described here https://docs.microsoft.com/en-us/azure/key-vault/general/logging?tabs=Vault.

rcskosir commented 1 year ago

Thanks for opening this issue. This was a problem in the 2.x version of the provider which is no longer actively maintained. If this is still an issue with the 3.x version of the provider please do let us know by opening a new issue, thanks!

github-actions[bot] commented 4 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.