Key Vault does not work with container registries with passwordSecretRef

pengxo commented 1 year ago

This issue is a: (mark with an x)

[x] bug report -> please search issues before submitting
[ ] documentation issue or request
[ ] regression (a behavior that used to work and stopped in a new release)

Issue description

The key vault integration for aca app works well as stated in https://github.com/microsoft/azure-container-apps/issues/608. But there is an exception. The key vault could not be used for private container registries with passwordSecretRef.

Steps to reproduce

A key vault with RBAC based policy is created with secrets in it;
Define a managed identity and proper role for accessing key vault;

Define a bicep template for aca app using the managed identity and having the key vault integration(deployed with Azure CLI):

resource myContainerApp 'Microsoft.App/containerApps@2022-11-01-preview' = {
location: 'northeurope'
name: 'my-container-app-name'
identity: 'my identity'
properties: {
managedEnvironmentId: 'managed environment id'
configuration: {
  ...
  secrets: [
    {
      name: 'password1'
      keyVaultUrl: 'key vault url to password1'
      identity: 'id of managed identity'
    }
    {
      name: 'password2'
      keyVaultUrl: 'key vault url to password2'
      identity: 'id of managed identity'
    }
    ...
  ]
  registries: [
    {
      server: 'fqdn of the private container registry'
      username: 'username'
      passwordSecretRef: 'password1'
    }
  ]
}
template: {
  containers: [
    {
      ...
    }
  ]

}
}
}

Expected behavior [What you expected to happen.] All the secrets from key vault could be used and referenced in the container app. Actual behavior [What actually happened.] All the secrets except password1 work well. The secret of password1 could not be found when used for registries with passwordSecretRef. The error message is: password password1 for my registry not found

Additional context

I double checked the key vault url to the secret above and there should be no problem with it. The private container registry is a custom registry instead of Azure Container Registry. Version of Azure CLI: 2.45.0

msevastian commented 1 year ago

Can confirm that issue still exist. Also can be reproduced from the Portal with the same error. From terraform AzAPI provider I get next error

│ RESPONSE 400: 400 Bad Request
│ ERROR CODE: WebhookInvalidParameterValue
│ --------------------------------------------------------------------------------
│ {
│   "error": {
│     "code": "WebhookInvalidParameterValue",
│     "message": "The following field(s) are either invalid or missing. Invalid value: \"xxxxxx.xxxxxx.com/xxxxxx/xxxxxx/xxxxxx/xxxxxx-app\": password registry-pass for xxxxxx.xxxxxx.com registry not found: template.containers.app2.image."
│   }
│ }

chriswue commented 1 year ago

I can also confirm that this fails for me with WebhookInvalidParameterValue. Supplying the secret value directly instead of via KeyVault reference works.

I understand that this feature is still in preview but it's quite annoying that this is so broken (it's been broken in a different way just a few weeks ago)

torosent commented 1 year ago

Hi, AKV support with container registry secret is still being worked on. ETA: End of June.

afdezl commented 1 year ago

Hi @torosent, it looks like this issue is still present on the latest preview api, still on track for end of June release?

torosent commented 1 year ago

Yes. Rollout to all regions will be completed in 24 hours. I tested it and it works

param appName string = 'someapp'
param subscriptionId string = ''
param resourceGroupName string = 'somerg'

resource containerApp 'Microsoft.App/containerApps@2022-11-01-preview' = {
  name: '${appName}'
  location: 'North Central US'
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '/subscriptions/${subscriptionId}/resourcegroups/kube-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/testmi': {}
    }
  }
  properties: {
    environmentId: '/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.App/managedEnvironments/someenv'
    configuration: {
      secrets: [
        {
          name: 'testsecret'
          keyVaultUrl: 'https://sanmehtakv.vault.azure.net/secrets/testsecret1'
          identity: '/subscriptions/${subscriptionId}/resourcegroups/kube-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/testmi'
        }
      ]
      activeRevisionsMode: 'Single'
      ingress: {
        external: true
        targetPort: 80
        transport: 'Auto'
        traffic: [
          {
            weight: 100
            latestRevision: true
          }
        ]
      }
      registries: [
        {
          server: 'test.azurecr.io'
          username: 'cappsinttestregistryprivate'
          passwordSecretRef: 'testsecret'
        }
      ]
    }
    template: {
      containers: [
        {
          image: 'test.azurecr.io/nginx:latest'
          name: 'simple-hello-world-container'
          resources: {
            cpu: '0.25'
            memory: '0.5Gi'
            ephemeralStorage: '1Gi'
          }
        }
      ]
      scale: {
        minReplicas: 0
        maxReplicas: 10
      }
    }
  }
}

pengxo commented 1 year ago

I could confirm that it works now. Thanks @torosent

afdezl commented 1 year ago

@torosent Many thanks, works fine.

chriswue commented 1 year ago

@torosent Your example shows a user managed identity and I'm having trouble getting it to work with the system managed identity. Is this supposed to work yet or is MSI support for referencing KV secret still in the works? In #733 you mentioned validation was disabled but that doesn't seem to be the case since my deployments fail with "unable to fetch secret using Managed identity 'system'"

vegardgs-ksat commented 1 year ago

When testing this feature out, I was also met with the unable to fetch secret using Managed identity '...'. I verified that:

user managed identity was created and had get secrets privileges.
the reference to the keyvault was correct

In the end, the issue was that my keyvault had the allow access from = Allow public access from specific virtual networks and IP addresses, without having added the VNet where my container app was running to the allow-list in the keyvault networking tab. Adding this allowed the bicep to validate.

KelvisGama commented 5 months ago

It is still happening. It only works when the KV secret name does not have dashes.

compulim commented 2 months ago

I am able to get this working with user-assigned managed identity and it can be deployed from scratch.

Deployed from scratch, maintains idempotency, single/complete Bicep
Secret saved to Key Vault
Secret retrieved in Container Apps using "secret refs" and exposed as an environment variable
No string concatenation to build IDs or URLs

Excerpts from my Bicep:

// Creates/updates a User-assigned Managed Identity to be used by the Container Apps.
resource containerAppIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-07-31-preview' = {
  // ... 
}

// Creates/updates a Key Vault and allow the managed identity to get secrets.
resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = {
  // ...
  properties: {
    accessPolicies: [
      {
        objectId: containerAppIdentity.properties.principalId
        permissions: {
          secrets: ['get']
        }
        // ...
      }
    ]
    // ...
  }
}

// Creates/updates a Key Vault secret, will be retrieved by Container Apps.
resource myPasswordKeyVaultSecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = {
  // ...
  parent: keyVault
  properties: {
    value: 'DUMMY'
  }
}

// Creates/updates the Container Apps and retrieve secrets from Key Vault as environment variable.
resource containerApp 'Microsoft.App/containerApps@2024-03-01' = {
  // ...
  identity: {
    // Uses the User-assigned Managed Identity.
    userAssignedIdentities: {
      '${containerAppIdentity.id}': {}
    }
    type: 'UserAssigned'
  }
  secrets: [
    // Gets the Key Vault secrets.
    {
      name: 'my-password'
      identity: containerAppIdentity.id
      keyVaultUrl: myPasswordKeyVaultSecret.properties.secretUri
    }
  ]
  template: {
    containers: [
      {
        // ...
        env: [
          // Sets environment variables "MY_PASSWORD" with the secret from Key Vault.
          {
            name: 'MY_PASSWORD'
            secretRef: 'my-password'
          }
        ]
      }
    ]
  ]
}

I didn't try System-assigned Managed Identity. Feels like it could be a chicken-and-egg problem.

microsoft / azure-container-apps