Provisioning state failed from private ACR with User managed identity

[nextdarius]

This issue is a: (mark with an x)

• [x] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Issue description

I have a private ACR without admin access enabled from which I pull images for my azure container app. I've created a user managed identity for which I granted AcrPull and is assigned to my Azure Container App. I try to update the revision of my container app using AZ CLI (OIDC Login) but I simply receive "provisioningState": "failed" without any additional information. I tried to check in both ContainerAppSystemLogs_CL and ContainerAppConsoleLogs_CL but could not find anything.

As soon as I enable admin access on ACR, then everything works normal and I can see logs (creating new revision, deprovisioning of old one etc.)

Doing this from Portal with the same user managed identity is OK as well.

Steps to reproduce

1. Use Az CLI with OIDC authentication
2. Prepare an ACR without admin access
3. Prepare a User Identity with AcrPull for previous ACR created
4. Assign the user identity to the container app
5. Perform an az containerapp update with an image from the ACR
6. Receive provisioning state failed

Expected behavior A new revision to be created

Actual behavior Provisioning state failed without any information or logs in any tables.

[redging-very-well]

I'm facing the exact same issue.

Incidentally, I've also tried setting the registry on the container app to my managed identity, but this also fails:

az containerapp registry set -n example -g $RG --server $ACR.azurecr.io --identity $ID_NAME
User identity /subscriptions/<subid>/resourcegroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mi-acr-puller is already assigned to containerapp
- Running ..Failed to provision revision for container app 'example'. Error details: The following field(s) are either invalid or missing. Field 'configuration.Registries<acr>.azurecr.io.Identity' is invalid with details: 'Invalid value: "mic-acr-puller": Managed Identity does not exist';..

(I've blocked out the sub and rg intentionally - those are correctly populated with the expected sub and rg in the console output.

[simonjj]

Thank you for raising this. @nextdarius and @redging-very-well. We've labeled this as Backlog. If this is of high priority please go ahead an raise a support ticket and feel free to mention this issue in your ticket.

[simongottschlag]

I was trying this out in my lab and noticed the same issue. In my case, I have an Azure Firewall that blocks everything to the internet. I saw that it was blocking traffic to two different FQDNs for login (the normal one and a region specific).

After opening that traffic it started working for me.

These two was needed for me:

login.microsoftonline.com
swedencentral.login.microsoft.com

[redging-very-well]

I've figured out that you can set the registry if you specify the --identity parameter as a fully qualified ID.

e.g.

FQID=$(az identity show -n ${identityName} -g ${RG} --query id --output tsv)
az containerapp registry set -n example -g $RG --server $ACR.azurecr.io --identity $FQID

[nextdarius]

Thanks @redging-very-well, I confirm as well that this does the trick!

We're using terraform to create the resources and discovered in the meantime that adding the registry block solves it as well.

However, it's still very hard to tackle a deployment failure in such case, as from what I experienced, there's no information at all. Also the fact that az cli does not throw an error in case of a failure is not ideal for CI/CD.

[redging-very-well]

@nextdarius glad that helped!

I totally agree - the container app deployment experience isn't great. It would be good if there was a way to wait for a deployment to succeed, as is possible with tools like helm.

[Greedygre]

Hi @nextdarius What is the error you got for Perform an az containerapp update with an image from the ACR? Base on the step process, you didn't execute command to assign user identity to the registry as following: az containerapp registry set -n example -g $RG --server $ACR.azurecr.io --identity $Identity-resource-id

[Greedygre]

Hi @redging-very-well Need to input resource id for a user-defined identity, did you occur this error when input a user-defined identity's name? Thanks.

az containerapp registry set -h

Command
    az containerapp registry set : Add or update a container registry's details.

Arguments
    --identity          : The managed identity with which to authenticate to the Azure Container
                          Registry (instead of username/password). Use 'system' for a system-defined
                          identity or a resource id for a user-defined identity. The managed
                          identity should have been assigned acrpull permissions on the ACR before
                          deployment (use 'az role assignment create --role acrpull ...').

[Greedygre]

I will give a more friendly error message for command: az containerapp registry set when --identity input not system and not a resource id for a user-defined identity.