Deployment using secret and `keyVaultUrl` reference fails "ERROR: (ContainerAppSecretKeyVaultUrlInvalid)"

[andmos]

This issue is a: (mark with an x)

• [X] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [X] regression (a behavior that used to work and stopped in a new release)

Issue description

Our Container App setup contains a simple application using a secret referencing a key vault:

        "configuration": {
            "secrets": [
                {
                    "name": "ai-connection-string",
                    "keyVaultUrl": "https://XXXXXX.vault.azure.net/secrets/ai-connection-string",
                    "identity": "system"
                },

Which was created after the container app itself, as noted in this issue: https://github.com/microsoft/azure-container-apps/issues/608

To update and deploy the app, we use the following az containerapp command:

          az containerapp update \
          --resource-group ${{ env.AZURE_CONTAINER_APP_RESOURCE_GROUP }} \
          --name ${{ env.AZURE_CONTAINER_APP_NAME }} \
          --image "${{ env.CONTAINER_IMAGE_APP }}:${{ steps.version.outputs.version }}" \
          --set-env-vars ${{ env.ASPNETCORE_APP_ENVIRONMENT }} ${{ steps.deployment-secretrefs.outputs.variables }}

Where ${{ steps.deployment-secretrefs.outputs.variables }} extracts to APPLICATIONINSIGHTS_CONNECTION_STRING=secretref:ai-connection-string, hence a reference to the above mentioned secret referencing the key vault.

This has worked perfectly up until yesterday. Then we started to receive the following error:

ERROR: (ContainerAppSecretKeyVaultUrlInvalid) Invalid Request: Container app secret(s) with name(s) 'ai-connection-string' are with invalid keyVaultUrl values, the value should be at format 'https://mykeyvault.vault.azure.net/secrets/mysecret', or 'https://mykeyvault.vault.azure.net/secrets/mysecret/994e967bb9b5468b92a4737aeee97276', and Container App could not reference keyvault secret from different cloud.

The last working timestamp is Fri, 21 Apr 2023 12:11:09 GMT, while first broken timestamp is Mon, 24 Apr 2023 05:20:49 GMT.

Steps to reproduce

1. Create an container app environment
2. Create an container app in this environment
3. Create a key vault with a test secret in the same resource group as container app
4. Create a secret with Key Vault Reference pointing to the secret in the key vault on the correct format, like https://my-vault.vault.azure.net/secrets/ai-connection-string
5. Add a reference to the secret when adding environmental variables to the app with --set-env-vars APPLICATIONINSIGHTS_CONNECTION_STRING=secretref:ai-connection-string or select it from the dropdown in the portal
6. Try to deploy

Expected behavior [What you expected to happen.] Expected the az containerapp update command or revision update in the portal to execute without error.

Actual behavior [What actually happened.] Resulting in the errors above:

ERROR: (ContainerAppSecretKeyVaultUrlInvalid) Invalid Request: Container app secret(s) with name(s) 'ai-connection-string' are with invalid keyVaultUrl values, the value should be at format 'https://mykeyvault.vault.azure.net/secrets/mysecret', or 'https://mykeyvault.vault.azure.net/secrets/mysecret/994e967bb9b5468b92a4737aeee97276', and Container App could not reference keyvault secret from different cloud.

Additional context

Occur both with CLI and API via Terraform.

Container App Environment and Container App was created before the key vault and secret referencing it.

Also see our previous error: https://github.com/microsoft/azure-container-apps/issues/722

[floriankoch]

I hit the same issue, this was working fine last week

 secrets: [
      {
        name: 'cosmosdb-connection-string'
        keyVaultUrl: '${keyVault.properties.vaultUri}secrets/${cosmosDBAccount.name}-connectionstring-primary'
        identity: appIdentity.outputs.id
      }
]

re with invalid keyVaultUrl values, the value should be at format 'https://mykeyvault.vault.azure.net/secrets/mysecret', or 'https://mykeyvault.vault.azure.net/secrets/mysecret/994e967bb9b5468b92a4737aeee97276', and Container App could not reference keyvault secret from different cloud.

[xwang971]

Hi sorry for the inconvenience. The issue was that secret name with dashes would be ruled out during validation which is not correct. We have fixed it and the fix would be deployed to everywhere in one or two weeks. To mitigate the issue for now, we can use the secret name without dashes. Again sorry for the inconvenience and we would update here once fix is deployed.

[floriankoch]

@xwang971 thx for the Info

[andmos]

Can confirm that changing the key vault secret name to a value without dashes works.

[dwlabcube]

@xwang971 Such a change should be rolled back immediately i think. everyone using container apps and having this issue can not deploy any new software. in my eyes 1 or two weeks is just way too much.

[torosent]

@dwlabcube, We are working to release a fix and unblock affected customers.

[dwlabcube]

@torosent deploying secrets without dash is still working. but changing all my secrets and depending deployments to non-dashed secrets is not an option. imagine you don't have write access to the key-vault. or you have hundreds of microservices which all use one or two of these secrets.

since it is only a validation-issue of how the secrets can be named and only in deployment of the container-app there should be somehow an option to bypass this validation. the validation has been added after everything was running smoothly for one or two weeks. and then suddenly BAMM...

and there is no easy way to rename existing variables in the keyvault even if one had write permission. but maybe there i am just not finding the feature.

i don't want to be unpolite, but being not able to deploy latest releases for one or two weeks is a problem. and the workaround with renaming all secrets and then reconfigure all the apps is just a lot of work. i wouldn't say anything if things were all for free. but we are paying for this stuff ;-)

[JGrzybowski]

I second to that! Expecting us to rename the keys to be able to update already deployed applications is not a solution. It's a workaround for the issue, if you create new environments and can start from scratch. But not for existing, previously working environments. Renaming every single key vault secret and changing all the urls just to be able to work for the next two weeks and the potentially reverting it back is out of option.

[torosent]

Hi @dwlabcube, @JGrzybowski . I understand the issue and I'm sorry for the inconvenience.

1. The fix rollout will be completed to all regions by 4/27.
2. We added more tests regarding AKV secrets to reduce the chances of regressions.
3. I added "renaming existing secrets/variables" to to planning backlog.

[dwlabcube]

Thanks a lot. do you mean 4/27 or really MAY/27 ?

[torosent]

oh sorry. 4/27

[torosent]

Hi @dwlabcube , Please confirm that this issue is solved for you now.

[dwlabcube]

yes it is solved! thanks a lot. even i did not open the issue i think you can close it.

[chriswue]

@torosent This feature has been broken again for a month now: https://github.com/microsoft/azure-container-apps/issues/739 could someone please investigate?

[torosent]

@chriswue, can you send your subscription id and app name to acasupport at microsoft dot com? I'm not able to reproduce it.

[chriswue]

@torosent I've created a self-contained reproducer:

First create the basic infra (log analytics, ACA env, key vault, ACR with an imported hello world image + scope map + token):

az group create --location australiaeast -g aca-git-733-test
lawid=$(az monitor log-analytics workspace create -n acagit733-law --location australiaeast -g aca-git-733-test --query customerId -o tsv)
lawkey=$(az monitor log-analytics workspace get-shared-keys -n acagit733-law -g aca-git-733-test --query primarySharedKey -o tsv)
az containerapp env create -n acagit733-env --location australiaeast -g aca-git-733-test  --logs-workspace-id "$lawid"  --logs-workspace-key "$lawkey"
az keyvault create --location australiaeast -g aca-git-733-test -n acagit733-kv
az acr create --location australiaeast -g aca-git-733-test -n acagit733acr --sku standard
az acr import -n acagit733acr --source mcr.microsoft.com/mcr/hello-world:latest --image hello-world:latest
az acr scope-map create -r acagit733acr -n helloworldscope --repository hello-world 'content/read'
acrpwd=$(az acr token create -r acagit733acr -n helloworldtoken --scope-map helloworldscope --query 'credentials.passwords[0].value' -o tsv)
az keyvault secret set --vault-name acagit733-kv -n tokenpass --value "$acrpwd"

Ok, now the following bicep to deploy the ACA:

param location string = resourceGroup().location

resource kv 'Microsoft.KeyVault/vaults@2023-02-01' existing = {
  name: 'acagit733-kv'
}

resource kvSecret 'Microsoft.KeyVault/vaults/secrets@2023-02-01' existing = {
  name: 'tokenpass'
  parent: kv
}

resource cr 'Microsoft.ContainerRegistry/registries@2022-12-01' existing = {
  name: 'acagit733acr'
}

resource acaEnv 'Microsoft.App/managedEnvironments@2022-10-01' existing = {
  name: 'acagit733-env'
}

resource aca 'Microsoft.App/containerApps@2022-11-01-preview' = {
  name: 'acagit733'
  location: location

  identity: {
    type: 'SystemAssigned'
  }

  properties: {
    configuration: {
      ingress: {
        external: true
        targetPort: 80
        allowInsecure: false
        traffic: [
          {
            latestRevision: true
            weight: 100
          }
        ]
      }
      secrets: [
        {
          name: 'secretpass'
          keyVaultUrl: kvSecret.properties.secretUri
          identity: 'system'
        }
      ]
      registries: [
        {
          server: cr.properties.loginServer
          username: 'helloworldtoken'
          passwordSecretRef: 'secretpass'
        }
      ]
    }
    template: {
      containers: [
        {
          name: 'hello-world'
          image: '${cr.properties.loginServer}/hello-world:latest'
          resources: {
            cpu: json('0.5')
            memory: '1.0Gi'
          }
        }
      ]
    }
    managedEnvironmentId: acaEnv.id
  }
}

var kvSecretUserRole = '4633458b-17de-408a-b874-0445c86b69e6'
resource userKvAccessRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(resourceGroup().id, aca.id, kvSecretUserRole)
  scope: resourceGroup()
  properties: {
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', kvSecretUserRole)
    principalId: aca.identity.principalId
    principalType: 'ServicePrincipal'
  }
}

deploy with

az deployment group create -g aca-git-733-test --template-file acagit733.bicep 

Fails with

{"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/REDACTED/resourceGroups/aca-git-733-test/providers/Microsoft.Resources/deployments/acagit733","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"InvalidParameterValueInContainerTemplate","message":"The following field(s) are either invalid or missing. Field 'template.containers.hello-world.image' is invalid with details: 'Invalid value: \"acagit733acr.azurecr.io/hello-world:latest\": password secretpass for acagit733acr.azurecr.io registry not found';."}]}}

[chriswue]

No it dawned on me that possibly there could be a chicken-and-egg problem here: The ACA identity will not exist before the ACA resource is deployed so it may not in a state where it can use it's identity to access the KV since the role assignment will only happen after the ACA is deployed.

So I switched to a user assigned identity:

param location string = resourceGroup().location

resource kv 'Microsoft.KeyVault/vaults@2023-02-01' existing = {
  name: 'acagit733-kv'
}

resource kvSecret 'Microsoft.KeyVault/vaults/secrets@2023-02-01' existing = {
  name: 'tokenpass'
  parent: kv
}

resource cr 'Microsoft.ContainerRegistry/registries@2022-12-01' existing = {
  name: 'acagit733acr'
}

resource acaEnv 'Microsoft.App/managedEnvironments@2022-10-01' existing = {
  name: 'acagit733-env'
}

resource userId 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'acagit733-identity'
  location: location
}

var kvSecretUserRole = '4633458b-17de-408a-b874-0445c86b69e6'
resource userKvAccessRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(resourceGroup().id, userId.id, kvSecretUserRole)
  scope: resourceGroup()
  properties: {
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', kvSecretUserRole)
    principalId: userId.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

resource aca 'Microsoft.App/containerApps@2022-11-01-preview' = {
  name: 'acagit733'
  location: location

  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${userId.id}': {}
    }
  }

  properties: {
    configuration: {
      ingress: {
        external: true
        targetPort: 80
        allowInsecure: false
        traffic: [
          {
            latestRevision: true
            weight: 100
          }
        ]
      }
      secrets: [
        {
          name: 'secretpass'
          keyVaultUrl: kvSecret.properties.secretUri
          identity: userId.id
        }
      ]
      registries: [
        {
          server: cr.properties.loginServer
          username: 'helloworldtoken'
          passwordSecretRef: 'secretpass'
        }
      ]
    }
    template: {
      containers: [
        {
          name: 'hello-world'
          image: '${cr.properties.loginServer}/hello-world:latest'
          resources: {
            cpu: json('0.5')
            memory: '1.0Gi'
          }
        }
      ]
    }
    managedEnvironmentId: acaEnv.id
  }
  dependsOn: [
    userKvAccessRole
  ]
}

However that fails with:

{"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/REDACTED/resourceGroups/aca-git-733-test/providers/Microsoft.Resources/deployments/acagit733","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"InvalidParameterValueInContainerTemplate","message":"The following field(s) are either invalid or missing. Field 'configuration.secrets' is invalid with details: 'Invalid value: \"secretpass\": Unable to get value using Managed identity /subscriptions/REDACTED/resourceGroups/aca-git-733-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acagit733-identity for secret secretpass. Error: unable to fetch secret 'secretpass' using Managed identity '/subscriptions/REDACTED/resourceGroups/aca-git-733-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acagit733-identity'';."}]}}

[chriswue]

Even if the chicken-and-egg problem is what is causing the reported issue - it means the ability configure the ACA's system managed identity to access the key vault is useless because it cannot possibly work.

[torosent]

@chriswue , I mentioned it in another issue, the akv support with registry secret is not working yet. We are actively working on it. Regarding the chicken and egg problem with system identity, you are right but we removed the validation so the secret pull will fail at first but succeed on retry.

[chriswue]

@torosent Ok thanks for the update. Slightly unrelated: I'm almost sure that one used to be able to deploy an ACA instance without configuring an image but this is no longer possible? Is my memory or did this change?

[torosent]

You cannot create a container app without an image. It was never an option.

[mmancu]

Hi, I am having the same issue. I am getting an error like that code":"ContainerAppSecretInvalid","message":"Invalid Request:value or keyVaultUrl and identity should be provided.

{ "name": "logiscore-sb-connection-string", "keyVaultUrl": "[concat('https://', variables('key_vault_name'),'.vault.azure.net','/secrets','/mysecret--ConnectionString')]", "identity": "System"

        }

[Promise-Igbo]

Hi i am having issues creating an aca This is the command i ran

az containerapp create --name $API_NAME --resource-group $RESOURCE_GROUP --environment $ENVIRONMENT --image $ACR_NAME.azurecr.io/$API_NAME --target-port 3500 --ingress 'external' --registry-server $ACR_NAME.azurecr.io --query properties.configuration.ingress.fqdn

This is the error i got

(InvalidParameterValueInContainerTemplate) The following field(s) are either invalid or missing. Field 'template.containers.album-api.image' is invalid with details: 'Invalid value: "/album-api": GET https:: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:album-api Type:repository]]';.

[Promise-Igbo]

I have authenticated to the registry but it still returns that error

[Promise-Igbo]

I have authenticated to the registry but it still returns that error

Hello! were you able to create the app? I am facing the same issue.

I was able to rectify the issue, the command did not pick the ACR name did i saved as an env variable, i had to specify the exact acr names for each flags that require the ACR name

[Promise-Igbo]

'Invalid value: "/album-api"

You can see it didn't pick the ACR name here

[GregDSmith]

@torosent deploying secrets without dash is still working. but changing all my secrets and depending deployments to non-dashed secrets is not an option. imagine you don't have write access to the key-vault. or you have hundreds of microservices which all use one or two of these secrets.

since it is only a validation-issue of how the secrets can be named and only in deployment of the container-app there should be somehow an option to bypass this validation. the validation has been added after everything was running smoothly for one or two weeks. and then suddenly BAMM...

and there is no easy way to rename existing variables in the keyvault even if one had write permission. but maybe there i am just not finding the feature.

i don't want to be unpolite, but being not able to deploy latest releases for one or two weeks is a problem. and the workaround with renaming all secrets and then reconfigure all the apps is just a lot of work. i wouldn't say anything if things were all for free. but we are paying for this stuff ;-)

As of today 11/30/2023 I am still unable to add secrets to a container app that has dashes in the name? It does not matter if I use the system identity or a user assigned identity. Based on all the comments above and the fact that this bug is closed I thought that the issue has been fixed? I can't find any other bug logged, what am I missing?

Update also the name of the secret has to be all lower case in-order for it to be used.

[MohammedFadin]

Same issue here as well.

[Vairamlakshumanasamy]

I am also facing the same issue

Failed to update secrets: The following field(s) are either invalid or missing. Field 'configuration.secrets' is invalid with details: 'Invalid value: "dataset": Unable to get value using Managed identity

[riteshbxr]

Same here... is there some way to escape dashes?

[mpigram1]

I'm encountering this issue when trying to create a secret using the below:

  secret {
    name                = "servicebusconnectionstring"
    identity            = azurerm_user_assigned_identity.aca-brevo-interface.id
    key_vault_secret_id = azurerm_key_vault_secret.Shared--ConnectionStrings--ServiceBus.id
  }

It errors with a similar error to @chriswue :

Error: unable to fetch secret 'secretpass' using Managed identity '/subscriptions/REDACTED/resourceGroups/aca-git-733-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acagit733-identity'';."}]}}

If I comment out this secret, create the ACA and then uncomment, it will modify the ACA and create the secret and the env var.

At this point that is not ideal and surely there has to be a fix for this soon as its been almost a year since this has been reported,

[catinodeh]

If anybody gets here looking for answers, I was able to make it work after I changed the (very long) identityref string to simply "system". Like this:

az containerapp secret set --name "NAME" --resource-group "WRE.DEV" --secrets "queueconnectionstring=keyvaultref:https://my-keyvault-url,identityref:system"