azurerm_app_service_certificate fails to replace with "The service does not have access to ... Key Vault." error

graememeyer commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.3.2

AzureRM Provider Version

3.30.0

Affected Resource(s)/Data Source(s)

azurerm_app_service_certificate

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.30"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name = "myresourcegroup"
  location = "UK South"
}

data "azurerm_client_config" "current" {}

resource "random_id" "kv" {
  byte_length = 8
}

resource "azurerm_key_vault" "key_vault" {
  name                       = "keyvault${random_id.kv.hex}"
  location                   = azurerm_resource_group.rg.location
  resource_group_name        = azurerm_resource_group.rg.name
  tenant_id                  = data.azurerm_client_config.current.tenant_id
  sku_name                   = "standard"
  soft_delete_retention_days = 7

  access_policy {
    tenant_id = data.azurerm_client_config.current.tenant_id
    object_id = data.azurerm_client_config.current.object_id

    certificate_permissions = [
        "Backup",
        "Create",
        "Delete",
        "DeleteIssuers",
        "Get",
        "GetIssuers",
        "Import",
        "List",
        "ListIssuers",
        "ManageContacts",
        "ManageIssuers",
        "Purge",
        "Recover",
        "Restore",
        "SetIssuers",
        "Update"
    ]
  }
}

resource "azurerm_key_vault_certificate" "certificate" {
  name         = "mycert"
  key_vault_id = azurerm_key_vault.key_vault.id

  certificate_policy {
    issuer_parameters {
      name = "Self"
    }

    key_properties {
      exportable = true
      key_size   = 2048
      key_type   = "RSA"
      reuse_key  = true
    }

    lifetime_action {
      action {
        action_type = "AutoRenew"
      }

      trigger {
        days_before_expiry = 30
      }
    }

    secret_properties {
      content_type = "application/x-pkcs12"
    }

    x509_certificate_properties {
      # Server Authentication = 1.3.6.1.5.5.7.3.1
      # Client Authentication = 1.3.6.1.5.5.7.3.2
      extended_key_usage = ["1.3.6.1.5.5.7.3.1"]

      key_usage = [
        "cRLSign",
        "dataEncipherment",
        "digitalSignature",
        "keyAgreement",
        "keyCertSign",
        "keyEncipherment",
      ]

      subject_alternative_names {
        dns_names = ["internal.contoso.com", "domain.hello.world"]
      }

      subject            = "CN=hello-world"
      validity_in_months = 12
    }
  }
}

# Grant the azure application service access to our keyvault to read the certificate data
# https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service_certificate#:~:text=If%20using%20key_vault_secret_id,every%20AAD%20Tenant%3A
# https://azure.github.io/AppService/2016/05/24/Deploying-Azure-Web-App-Certificate-through-Key-Vault.html
data "azuread_service_principal" "MicrosoftWebApp" {
  application_id = "abfa0a7c-a6b6-4736-8310-5855508787cd"
}

resource "azurerm_key_vault_access_policy" "keyvault_aad_access" {
  key_vault_id   = azurerm_key_vault.key_vault.id
  tenant_id      = data.azurerm_client_config.current.tenant_id
  object_id      = data.azuread_service_principal.MicrosoftWebApp.id

  key_permissions = [
  ]

  secret_permissions = [
    "Get"
  ]

  certificate_permissions = [
    "Backup",
    "Create",
    "Delete",
    "DeleteIssuers",
    "Get",
    "GetIssuers",
    "Import",
    "List",
    "ListIssuers",
    "ManageContacts",
    "ManageIssuers",
    "Purge",
    "Recover",
    "Restore",
    "SetIssuers",
    "Update"
  ]
}

resource "azurerm_app_service_certificate" "app_service_certificate" {
  name                = "exampleappservicecert"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location

  key_vault_secret_id = azurerm_key_vault_certificate.certificate.id
}

Debug Output/Panic Output

Creating...
╷
│ Error: creating/updating Certificate: (Name "exampleappservicecert" / Resource Group "mysupertestrg"): web.CertificatesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="The service does not have access to '/subscriptions/103907c2-mysub-fed0a9332b09/resourcegroups/mysupertestrg/providers/microsoft.keyvault/vaults/keyvaultb3510cb2e0a426be' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation." Details=[{"Message":"The service does not have access to '/subscriptions/103907c2-mysub-fed0a9332b09/resourcegroups/mysupertestrg/providers/microsoft.keyvault/vaults/keyvaultb3510cb2e0a426be' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","ExtendedCode":"59716","Message":"The service does not have access to '/subscriptions/103907c2-mysub-fed0a9332b09/resourcegroups/mysupertestrg/providers/microsoft.keyvault/vaults/keyvaultb3510cb2e0a426be' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.","MessageTemplate":"The service does not have access to '{0}' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.","Parameters":["/subscriptions/103907c2-mysub-fed0a9332b09/resourcegroups/mysupertestrg/providers/microsoft.keyvault/vaults/keyvaultb3510cb2e0a426be"]}}]
│
│   with azurerm_app_service_certificate.app_service_certificate,
│   on main.tf line 152, in resource "azurerm_app_service_certificate" "app_service_certificate":
│  152: resource "azurerm_app_service_certificate" "app_service_certificate" {
│
╵

Expected Behaviour

This Terraform configuration should create an Azure App Service Certificate, using a certificate created and stored by and Azure Key Vault.

Actual Behaviour

Terraform successfully creates the certificate the first time, but when asked to replace the certificate, it fails with an access error. If asked a third time, Terraform then successfully creates the certificate. It alternates between success and failure thereafter.

I use terraform apply -auto-approve to create the infrastructure originally

Then use terraform apply -auto-approve -replace="azurerm_app_service_certificate.app_service_certificate" to test iterations of create/destroy.

Steps to Reproduce

terraform init
terraform apply -auto-approve

This should succeed.

terraform apply -auto-approve -replace="azurerm_app_service_certificate.app_service_certificate"

This should fail with the error I provided.

terraform apply -auto-approve -replace="azurerm_app_service_certificate.app_service_certificate"

This should then succeed.

Important Factoids

No response

References

Note the requirement to grant the Azure App Service access to your Key Vault so that it can read the Certificate data.

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service_certificate#:~:text=If%20using%20key_vault_secret_id,every%20AAD%20Tenant%3A

https://azure.github.io/AppService/2016/05/24/Deploying-Azure-Web-App-Certificate-through-Key-Vault.html

graememeyer commented 1 year ago

Note: I used granted every certificate permission available here both to myself and the Azure App Service, just to rule out permissions issues (since that's what it's intermittently complaining about).

graememeyer commented 1 year ago

I've just realised, TF may be failing to actually create the App Service Certificate in the first place - it's not showing up in the Azure Portal, so unless it's being created with permissions that mean I can't see it (I'm a global admin) then I think it's actually the create operation that must be failing

Edit: disregard this comment. The azurerm_app_service_certificate does show up in the Azure Portal, it's just a hidden type called microsoft.web/certificates. I was misunderstanding the resource mapping of the API to the portal. For anyone else in future that gets confused:

Azure Portal (GUI) Name	Azure API Resource	Terraform Resource
"Azure App Service Certificate"	Microsoft.CertificateRegistration/certificateOrders	azurerm_app_service_certificate_order
Hidden Type: Microsoft.Web/certificates	Microsoft.Web/certificates	azurerm_app_service_certificate

My bug report about the alternating success/failure to create azurerm_app_service_certificate_order still stands.

xiaxyi commented 1 year ago

Thanks @graememeyer for raising this issue.

I noticed that you are using the access_policy block in azurerm_key_vault and using azurerm_key_vault_access_policy at the same time. There will be conflicts using these two resources at the same time as mentioned in our official guidance here.

Can you try using one of them and see issue can be resolved? It works fine by using only one of these two resources from my side.

Thanks, feel free to let me know if there is anything needed.

xiaxyi commented 1 year ago

@catriona-m This is not a bug for both app_service and key_vault. :)

graememeyer commented 1 year ago

Good catch @xiaxyi, I've reconfigured it with only azurerm_key_vault_access_policy resource blocks and that does indeed seem to have fixed it:

Terraform Configuration

``` Terraform terraform { required_providers { azurerm = { source = "hashicorp/azurerm" version = ">= 3.30" } } } provider "azurerm" { features {} } resource "azurerm_resource_group" "rg" { name = "myresourcegroup" location = "UK South" } data "azurerm_client_config" "current" {} resource "random_id" "kv" { byte_length = 8 } resource "azurerm_key_vault" "key_vault" { name = "keyvault${random_id.kv.hex}" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name tenant_id = data.azurerm_client_config.current.tenant_id sku_name = "standard" soft_delete_retention_days = 7 } resource "azurerm_key_vault_certificate" "certificate" { name = "mycert" key_vault_id = azurerm_key_vault.key_vault.id certificate_policy { issuer_parameters { name = "Self" } key_properties { exportable = true key_size = 2048 key_type = "RSA" reuse_key = true } lifetime_action { action { action_type = "AutoRenew" } trigger { days_before_expiry = 30 } } secret_properties { content_type = "application/x-pkcs12" } x509_certificate_properties { # Server Authentication = 1.3.6.1.5.5.7.3.1 # Client Authentication = 1.3.6.1.5.5.7.3.2 extended_key_usage = ["1.3.6.1.5.5.7.3.1"] key_usage = [ "cRLSign", "dataEncipherment", "digitalSignature", "keyAgreement", "keyCertSign", "keyEncipherment", ] subject_alternative_names { dns_names = ["internal.contoso.com", "domain.hello.world"] } subject = "CN=hello-world" validity_in_months = 12 } } } resource "azurerm_key_vault_access_policy" "terraform_user_access" { key_vault_id = azurerm_key_vault.key_vault.id tenant_id = data.azurerm_client_config.current.tenant_id object_id = data.azurerm_client_config.current.object_id certificate_permissions = [ "Backup", "Create", "Delete", "DeleteIssuers", "Get", "GetIssuers", "Import", "List", "ListIssuers", "ManageContacts", "ManageIssuers", "Purge", "Recover", "Restore", "SetIssuers", "Update" ] } # Grant the azure application service access to our keyvault to read the certificate data # https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service_certificate#:~:text=If%20using%20key_vault_secret_id,every%20AAD%20Tenant%3A # https://azure.github.io/AppService/2016/05/24/Deploying-Azure-Web-App-Certificate-through-Key-Vault.html data "azuread_service_principal" "MicrosoftWebApp" { application_id = "abfa0a7c-a6b6-4736-8310-5855508787cd" } resource "azurerm_key_vault_access_policy" "keyvault_aad_access" { key_vault_id = azurerm_key_vault.key_vault.id tenant_id = data.azurerm_client_config.current.tenant_id object_id = data.azuread_service_principal.MicrosoftWebApp.id key_permissions = [ ] secret_permissions = [ "Get" ] certificate_permissions = [ "Backup", "Create", "Delete", "DeleteIssuers", "Get", "GetIssuers", "Import", "List", "ListIssuers", "ManageContacts", "ManageIssuers", "Purge", "Recover", "Restore", "SetIssuers", "Update" ] depends_on = [ azurerm_key_vault_access_policy.terraform_user_access ] } resource "azurerm_app_service_certificate" "app_service_certificate" { name = "exampleappservicecert" resource_group_name = azurerm_resource_group.rg.name location = azurerm_resource_group.rg.location key_vault_secret_id = azurerm_key_vault_certificate.certificate.id } ```

I ran the replace operation 5 times with no errors.

I did however have to add depends_on resource dependency on the terraform_user_access block - would you say that is to be expected? I'm not sure if Terraform should be expected to be able to determine the access permission dependency of its own context or not, perhaps that's unreasonable. Without the depends_on block, I was intermittently getting errors like:

Error print-out

``` Terraform Error: creating/updating Certificate: (Name "exampleappservicecert" / Resource Group "myresourcegroup"): web.CertificatesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="The service does not have access to '/subscriptions/103907c2-mysubscriptionid-fed0a9332b09/resourcegroups/myresourcegroup/providers/microsoft.keyvault/vaults/keyvault38af82a76225da27' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation." Details=[{"Message":"The service does not have access to '/subscriptions/103907c2-mysubscriptionid-fed0a9332b09/resourcegroups/myresourcegroup/providers/microsoft.keyvault/vaults/keyvault38af82a76225da27' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","ExtendedCode":"59716","Message":"The service does not have access to '/subscriptions/103907c2-mysubscriptionid-fed0a9332b09/resourcegroups/myresourcegroup/providers/microsoft.keyvault/vaults/keyvault38af82a76225da27' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.","MessageTemplate":"The service does not have access to '{0}' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.","Parameters":["/subscriptions/103907c2-mysubscriptionid-fed0a9332b09/resourcegroups/myresourcegroup/providers/microsoft.keyvault/vaults/keyvault38af82a76225da27"]}}] ```

xiaxyi commented 1 year ago

Thanks @graememeyer for the update, the depends_on is used when there is no explicit dependency. If you need the dependency relation between two access policies, you'll need to use the depends_on as there is no explicit dependency. May I know why you are configuring dependency between two policies?

rcskosir commented 11 months ago

Thanks for taking the time to submit this issue. It looks like this has been resolved with the insight from @xiaxyi. As such, I am going to mark this issue as closed.

github-actions[bot] commented 1 month ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm