azurerm_windows_function_app does not resolve storage_key_vault_secret_id:

LennDG commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.4

AzureRM Provider Version

3.21.1

Affected Resource(s)/Data Source(s)

azurerm_windows_function_app

Terraform Configuration Files

provider "azurerm" {
  features {
    key_vault {
      purge_soft_delete_on_destroy = true
    }
  }
}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

data "azurerm_client_config" "current" {}

resource "azurerm_storage_account" "example" {
  name                     = "windowsfunctionappsa"
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_service_plan" "example" {
  name                = "example-app-service-plan"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  os_type             = "Windows"
  sku_name            = "Y1"
}

resource "azurerm_key_vault" "example" {

  name                = "examplekeyvault"
  tenant_id           = data.azurerm_client_config.current.tenant_id
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku_name            = "standard"
  soft_delete_retention_days  = 7
  purge_protection_enabled    = false
}

resource "azurerm_user_assigned_identity" "example" {

  name                = "exampleid"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location

}

resource "azurerm_key_vault_access_policy" "owner" {

  key_vault_id = azurerm_key_vault.example.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = data.azurerm_client_config.current.object_id

  secret_permissions = [
    "Delete",
    "Get",
    "Set",
    "List",
    "Purge",
    "Recover",
    "Restore"
  ]
}

resource "azurerm_key_vault_access_policy" "funcapp" {

  key_vault_id = azurerm_key_vault.example.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = azurerm_user_assigned_identity.example.principal_id

  secret_permissions = [
    "Delete",
    "Get",
    "Set",
    "List"
  ]
}

resource "azurerm_key_vault_secret" "azure_files_connection_string" {
  key_vault_id = azurerm_key_vault.example.id
  name         = "azure-files-connection-string"
  value        = azurerm_storage_account.example.primary_connection_string

  content_type    = "text/plain"
  expiration_date = local.expiration_date

  # We must delay the secret creation until the policy of the vault owner has
  # been created
  depends_on = [azurerm_key_vault_access_policy.owner]
}
data "azurerm_key_vault_secret" "azure_files_connection_string" {

  name         = azurerm_key_vault_secret.azure_files_connection_string.name
  key_vault_id = azurerm_key_vault_secret.azure_files_connection_string.key_vault_id
}

resource "azurerm_windows_function_app" "funcapp" {
  name                = "example-functionapp"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location

  # Service plan is shared between all function apps:
  service_plan_id = azurerm_service_plan.example.id

  storage_key_vault_secret_id = data.azurerm_key_vault_secret.azure_files_connection_string.id

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.example.id]
  }

  key_vault_reference_identity_id = azurerm_user_assigned_identity.example.id

  app_settings = {
    AzureWebJobsSecretStorageType             = "keyvault"
    AzureWebJobsSecretStorageKeyVaultUri      = azurerm_key_vault.example.vault_uri
    AzureWebJobsSecretStorageKeyVaultClientId = azurerm_user_assigned_identity.example.client_id

    # Disable default landing page:
    AzureWebJobsDisableHomepage = true

    # Disable editing in Azure portal:
    FUNCTION_APP_EDIT_MODE = "readonly"

  }

  site_config {
    application_stack {
      node_version = "~14"
    }
    # Disable FTP(s) deployments as per CKV_AZURE_78:
    ftps_state = "Disabled"
  }
}

Debug Output/Panic Output

╷
│ Error: creating Windows Function App: (Site Name "func-p-10007318-cdbfx-D0yy" / Resource Group "rg-p-app-10007318"): web.AppsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="BadRequest" Message="Unable to resolve Azure Files Settings from Key Vault. Details: Unable to resolve setting: WEBSITE_CONTENTAZUREFILECONNECTIONSTRING with error: OtherReasons." Details=[{"Message":"Unable to resolve Azure Files Settings from Key Vault. Details: Unable to resolve setting: WEBSITE_CONTENTAZUREFILECONNECTIONSTRING with error: OtherReasons."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","ExtendedCode":"04603","Message":"Unable to resolve Azure Files Settings from Key Vault. Details: Unable to resolve setting: WEBSITE_CONTENTAZUREFILECONNECTIONSTRING with error: OtherReasons.","MessageTemplate":"Unable to resolve Azure Files Settings from Key Vault. Details: {0}","Parameters":["Unable to resolve setting: WEBSITE_CONTENTAZUREFILECONNECTIONSTRING with error: OtherReasons."]}}]
│ 
│   with azurerm_windows_function_app.funcapp["AC-IOT-P-INFR-NanoController"],
│   on funcapps.tf line 217, in resource "azurerm_windows_function_app" "funcapp":
│  217: resource "azurerm_windows_function_app" "funcapp" {
│ 
│ creating Windows Function App: (Site Name "func-p-10007318-cdbfx-D0yy" /
│ Resource Group "rg-p-app-10007318"): web.AppsClient#CreateOrUpdate: Failure
│ sending request: StatusCode=400 -- Original Error: Code="BadRequest"
│ Message="Unable to resolve Azure Files Settings from Key Vault. Details:
│ Unable to resolve setting: WEBSITE_CONTENTAZUREFILECONNECTIONSTRING with
│ error: OtherReasons." Details=[{"Message":"Unable to resolve Azure Files
│ Settings from Key Vault. Details: Unable to resolve setting:
│ WEBSITE_CONTENTAZUREFILECONNECTIONSTRING with error:
│ OtherReasons."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","ExtendedCode":"04603","Message":"Unable
│ to resolve Azure Files Settings from Key Vault. Details: Unable to resolve
│ setting: WEBSITE_CONTENTAZUREFILECONNECTIONSTRING with error:
│ OtherReasons.","MessageTemplate":"Unable to resolve Azure Files Settings
│ from Key Vault. Details: {0}","Parameters":["Unable to resolve setting:
│ WEBSITE_CONTENTAZUREFILECONNECTIONSTRING with error: OtherReasons."]}}]

Expected Behaviour

The function app should be deployed with the environment variable WEBSITE_CONTENTAZUREFILECONNECTIONSTRING set to the keyvault identifier: @Microsoft.KeyVault(SecretUri=${azurerm_key_vault_secret.azure_files_connection_string})

Actual Behaviour

The function app does not get created because the connection string does not get resolved. An error is thrown with a poor error message ("OtherReasons") that according to this Medium post might have something to do with incorrect versions. Using outputs I was able to confirm that the correct version of the secret is being used (it also being the only one).

Steps to Reproduce

terraform apply

Important Factoids

No response

References

No response

xiaxyi commented 2 years ago

Thanks @LennDG for raising this issue, I'll work on the fix to properly generate the WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and the content share

xiaxyi commented 2 years ago

@LennDG , I checked the code and it turns out a restrictions from the api side. _"When using Key Vault references for this setting, this validation check will fail by default, as the secret itself cannot be resolved while processing the incoming request. To avoid this issue, you can skip the validation by setting WEBSITE_SKIP_CONTENTSHAREVALIDATION to "1". This will bypass all checks, and the content share will not be created for you. You should ensure it is created in advance."

Also please notice that you need to ensure the file share gets created in advance, otherwise, the app cannot be started and you'll get the 500 error

Can you add the KVP in your app_setting and try again? WEBSITE_SKIP_CONTENTSHARE_VALIDATION = 1

my config works fine after passing the file check:


  app_settings = {
    WEBSITE_CONTENTAZUREFILECONNECTIONSTRING = "@Microsoft.KeyVault(SecretUri=https://xxx.vault.azure.net/secrets/azure-files-connection-string/xxx)"
    AzureWebJobsStorage = "@Microsoft.KeyVault(SecretUri=https://xxx.vault.azure.net/secrets/azure-files-connection-string/xxx)"
    WEBSITE_CONTENTSHARE = "consumption-functionapp-7fea"
    AzureWebJobsDisableHomepage = true
    WEBSITE_SKIP_CONTENTSHARE_VALIDATION  = 1
  }

More details are mentioned in this docs

mcollier commented 2 years ago

Is there a way to make this work when using a SystemAssigned identity for the Function app? When trying to use SystemAssigned, I seem to be caught in a where the Function app can't resolve the storage key vault reference because the Key Vault access policy isn't set until after the function is created (needs the identity details).

LennDG commented 2 years ago

@xiaxyi Hi, thanks for the information. As the docs specify, I should ensure the content share is created in advance. What exactly does that mean? I seem to have relied on the automatic creation of it so far, and now that it is not created automatically I cannot deploy source code to the funcapp.

Also, when I add the AzureWebJobsStorage to app settings, the create fails with error: Error: creating Windows Function App: (Site Name "func-d-10007318-cdbfx-T17L" / Resource Group "rg-d-app-10007318--regionpoc"): web.AppsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: Code="Conflict" Message="Parameter with name AzureWebJobsStorage already exists." Details=[{"Message":"Parameter with name AzureWebJobsStorage already exists."},{"Code":"Conflict"},{"ErrorEntity":{"Code":"Conflict","ExtendedCode":"01013","Message":"Parameter with name AzureWebJobsStorage already exists.","MessageTemplate":"Parameter with name {0} already exists.","Parameters":["AzureWebJobsStorage"]}}]

Seems like this app setting is created by the storage_account_name parameter? Not sure how it does not error for you with those settings.

xiaxyi commented 1 year ago

@mcollier Thanks for the comment, can you confirm if the issue you are seeing matches with this issue - https://github.com/hashicorp/terraform-provider-azurerm/issues/13490?

xiaxyi commented 1 year ago

@LennDG The predefined file share would be creating the file share in storage account in advance because app service won't be able to create one for you. You may not try adding the AzureWebJobsStorage because it's auto-generated when you specifying the storage account related settings via:https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/windows_function_app#storage_account_name

Have you try settingWEBSITE_SKIP_CONTENTSHARE_VALIDATION to "1" in app_setting block?

xiaxyi commented 1 year ago

Hey @LennDG , may I know if you have tried the solution mentioned in my last comment?

LennDG commented 1 year ago

I did but was not able to really get it working. Did not try very hard however, since we moved these functions to containers in an AKS cluster.

hashicorp / terraform-provider-azurerm