Unable to access KV from ASP.NET Core application

[Kralizek]

This issue is a: (mark with an x)

• [x] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Issue description

I have a ASP.NET Core application running on Azure Container Instance and uses KV to fetch some secrets by using the configuration provider. The container instance is configured to use a user-assigned managed identity.

Running the same container in ACA with the same managed identity fails to start because the application is unable to access KV.

Steps to reproduce

Here are some Terraform constructs used to define the different parts. Since I'm copying from different areas, names might not match.

resource "azurerm_user_assigned_identity" "this" {
  resource_group_name = var.resource_group.name
  location = var.resource_group.location
  name = "usi-${module.namer.base_name}-001"
}

resource "azurerm_key_vault" "secrets" {
  // ####

  access_policy {
    tenant_id = azurerm_user_assigned_identity.this.tenant_id
    object_id = azurerm_user_assigned_identity.this.principal_id

    key_permissions = ["Get", "WrapKey", "UnwrapKey"]

    secret_permissions = ["Get", "List"]
  }

  // ###
}

resource "azurerm_log_analytics_workspace" "backend" {
  name                = "law-${module.namer.base_name}-001"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  sku                 = "PerGB2018"
  retention_in_days   = 30
}

resource "azurerm_container_app_environment" "backend" {
  name                = "env-${module.namer.base_name}-001"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location

  log_analytics_workspace_id = azurerm_log_analytics_workspace.backend.id
}

locals {
  container_environment_variables = {
    "ConnectionStrings__Database"                = "something"
    # "AZURE_CLIENT_ID"                            = azurerm_user_assigned_identity.this.principal_id
    "ConnectionStrings__AzureKeyVault"           = azurerm_key_vault.secrets.vault_uri
    "ASPNETCORE_ENVIRONMENT"                     = module.dotnet_environment.name
    "ASPNETCORE_LOGGING__CONSOLE__DISABLECOLORS" = "true"
  }
}

resource "azurerm_container_app" "backend" {
  name                         = "capp-${module.namer.base_name}-001"
  container_app_environment_id = azurerm_container_app_environment.backend.id
  resource_group_name          = azurerm_resource_group.this.name
  revision_mode                = "Single"

  template {
    min_replicas = 1
    container {
      name   = "service"
      image  = "${data.terraform_remote_state.cloud_ops.outputs.container_registry_login_server}/${local.application}/api:latest"
      cpu    = 1.0
      memory = "2Gi"

      liveness_probe {
        path             = "/_health"
        port             = 8080
        transport        = "HTTP"
        initial_delay    = 10
        interval_seconds = 30
      }

      dynamic "env" {
        for_each = local.container_environment_variables
        content {
          name  = env.key
          value = env.value
        }
      }
    }
  }

  registry {
    server   = data.terraform_remote_state.cloud_ops.outputs.container_registry_login_server
    identity = azurerm_user_assigned_identity.this.id
  }

  ingress {
    target_port      = 8080
    external_enabled = true
    traffic_weight {
      percentage      = 100
      latest_revision = true
    }
  }

  identity {
    identity_ids = [azurerm_user_assigned_identity.this.id]
    type         = "UserAssigned"
  }
}

resource "azurerm_role_assignment" "containerapp" {
  scope                = data.terraform_remote_state.cloud_ops.outputs.container_registry_id
  role_definition_name = "acrpull"
  principal_id         = module.api.service_identity.principal_id

  depends_on = [module.api]
}

And this is the snippet used in my ASP.NET Core 8 RC2 app to access the KV.

if (builder.Configuration.GetConnectionString("AzureKeyVault") is { } vaultUrl)
{
    builder.Configuration.AddAzureKeyVault
    (
        new Uri(vaultUrl),
        new DefaultAzureCredential()
    );
}

This works correctly in ACI.

Expected behavior [What you expected to happen.]

Be able to access a KV vault from my ASP.NET Core application using the configuration provider.

Actual behavior [What actually happened.]

The revision is not provisioned and removed. These are some entries in the console log of the revision.

Unhandled exception. Azure.Identity.AuthenticationFailedException: ManagedIdentityCredential authentication failed: Service request failed.

---> Azure.RequestFailedException: Service request failed.
Status: 400 (Bad Request)

[Kralizek]

Update:

Adding an environment variable called AZURE_CLIENT_ID set to the value azurerm_user_assigned_identity.this.client_id solved the issue. I can't find the page now, but there was a page suggesting to use the principal id value for the environment variable. That didn't work.

[vturecek]

@Kralizek you have to specify the client ID when using a user-assigned managed identity. new DefaultAzureCredential() without any parameters will try to use a system-assigned identity and will fail if it's not there. It will also use the AZURE_CLIENT_ID environment variable if it's set.

See here for more info: https://learn.microsoft.com/en-us/dotnet/api/overview/azure/identity-readme?view=azure-dotnet#specify-a-user-assigned-managed-identity-with-defaultazurecredential

[Kralizek]

Adding the AZURE_CLIENT_ID environment variable fixed the issue. Since the container app has an identity attached, I was expecting the system to take care of providing what's needed for the application to use the identity to make calls to Azure services.

I recently started working with Azure after years at AWS and I had the conceptual map managed identities are the equivalent of the IAM execution role assigned to an instance or a container. Under that assumption, as mentioned, I was expected that either the environment was providing the credentials needed or the SDK could fetch them short-lived ones.