Bicep Deployment Error - `WebhookInvalidParameterValue` (Private Azure Container Registry)

mumby0168 commented 2 years ago

[x] bug report -> please search issues before submitting
[ ] documentation issue or request
[ ] regression (a behavior that used to work and stopped in a new release)

Issue description

Failure to pull container image from private azure container registry when deploying azure container app via bicep.

"WebhookInvalidParameterValue\\\",\\r\\n  \\\"message\\\": \\\"The following field(s) are either invalid or missing. 
Invalid value: \\\\\\\"cosmossdkidentitydemoacr.azurecr.io/apps/books-api:18a49dc9e045714edaafad60a3f2f9ae67b542e3\\\\\\\": 
GET https:?scope=repository%3Aapps%2Fbooks-api%3Apull&service=cosmossdkidentitydemoacr.azurecr.io: 
UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.: template.containers.books-api.image.\\\"\\r\\n***\"\r\n          ***\r\n        ]\r\n      ***\r\n    ]\r\n  ***\r\n***"***]***

Steps to reproduct

Provision azure container apps environment.
Provision azure container registry (store admin password in KV).
Build & Push docker image to ACR tagged with GitHub sha.
Deploy azure container apps passing in the image name with tag as the image to deploy (get admin pass from KV) via bicep.

Actual behavior

An error was thrown as shown above, the example error can also be seen in this build here: https://github.com/mumby0168/cosmos-repository-sdk-identity/runs/6628955590?check_suite_focus=true#step:5:20

Screenshots
n/a

Additional context

There is a full working example of the issue available on this GitHub repository:

https://github.com/mumby0168/cosmos-repository-sdk-identity

The GitHub Actions file can be found here: https://github.com/mumby0168/cosmos-repository-sdk-identity/blob/main/.github/workflows/deploy.yml

mumby0168 commented 2 years ago

I have recently updated this to use managed identities to pull the image and I get the exact same error see below

{
    "authorization": {
        "action": "Microsoft.App/containerApps/write",
        "scope": "/subscriptions/7c6cf4f2-b5e5-433c-8e68-4ba91fb2d6c9/resourcegroups/cosmos-repository-sdk-identity/providers/Microsoft.App/containerApps/books-api"
    },
    "caller": "367f3975-e2a4-400d-b572-496a69970814",
    "channels": "Operation",
    "claims": {
        "aud": "https://management.core.windows.net/",
        "iss": "https://sts.windows.net/16e04e4f-42c3-445b-9884-605e3bacbeee/",
        "iat": "1654546222",
        "nbf": "1654546222",
        "exp": "1654550122",
        "aio": "E2ZgYPitsDVgaoGngle9gWeg8R9TAA==",
        "appid": "5ebdc9f5-8218-4276-b948-4338e7178d6c",
        "appidacr": "1",
        "http://schemas.microsoft.com/identity/claims/identityprovider": "https://sts.windows.net/16e04e4f-42c3-445b-9884-605e3bacbeee/",
        "idtyp": "app",
        "http://schemas.microsoft.com/identity/claims/objectidentifier": "367f3975-e2a4-400d-b572-496a69970814",
        "rh": "0.ATAAT07gFsNCW0SYhGBeO6y-7kZIf3kAutdPukPawfj2MBMwAAA.",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "367f3975-e2a4-400d-b572-496a69970814",
        "http://schemas.microsoft.com/identity/claims/tenantid": "16e04e4f-42c3-445b-9884-605e3bacbeee",
        "uti": "yASbGzgETU6YnySCJ4IgAA",
        "ver": "1.0",
        "xms_tcdt": "1517084385"
    },
    "correlationId": "1f97834d-d8d0-41ed-b9ec-471f9a2776ea",
    "description": "",
    "eventDataId": "3ded3823-7b24-4e2a-bb57-4beb2aea155b",
    "eventName": {
        "value": "EndRequest",
        "localizedValue": "End request"
    },
    "category": {
        "value": "Administrative",
        "localizedValue": "Administrative"
    },
    "eventTimestamp": "2022-06-06T20:16:12.6187064Z",
    "id": "/subscriptions/7c6cf4f2-b5e5-433c-8e68-4ba91fb2d6c9/resourcegroups/cosmos-repository-sdk-identity/providers/Microsoft.App/containerApps/books-api/events/3ded3823-7b24-4e2a-bb57-4beb2aea155b/ticks/637901433726187064",
    "level": "Error",
    "operationId": "357aeba9-e356-43bd-87bf-bb591cc44cdf",
    "operationName": {
        "value": "Microsoft.App/containerApps/write",
        "localizedValue": "Create or Update Container App"
    },
    "resourceGroupName": "cosmos-repository-sdk-identity",
    "resourceProviderName": {
        "value": "Microsoft.App",
        "localizedValue": "Microsoft.App"
    },
    "resourceType": {
        "value": "Microsoft.App/containerApps",
        "localizedValue": "Microsoft.App/containerApps"
    },
    "resourceId": "/subscriptions/7c6cf4f2-b5e5-433c-8e68-4ba91fb2d6c9/resourcegroups/cosmos-repository-sdk-identity/providers/Microsoft.App/containerApps/books-api",
    "status": {
        "value": "Failed",
        "localizedValue": "Failed"
    },
    "subStatus": {
        "value": "BadRequest",
        "localizedValue": "Bad Request (HTTP Status Code: 400)"
    },
    "submissionTimestamp": "2022-06-06T20:17:50.139927Z",
    "subscriptionId": "7c6cf4f2-b5e5-433c-8e68-4ba91fb2d6c9",
    "tenantId": "16e04e4f-42c3-445b-9884-605e3bacbeee",
    "properties": {
        "statusCode": "BadRequest",
        "serviceRequestId": null,
        "statusMessage": "{\"code\":\"WebhookInvalidParameterValue\",\"message\":\"The following field(s) are either invalid or missing. Invalid value: \\\"cosmossdkidentitydemoacr.azurecr.io/apps/books-api:49feb3713bccacaf8832268bb7691e71dac77290\\\": GET https:?scope=repository%3Aapps%2Fbooks-api%3Apull&service=cosmossdkidentitydemoacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.: template.containers.books-api.image.\"}",
        "eventCategory": "Administrative",
        "entity": "/subscriptions/7c6cf4f2-b5e5-433c-8e68-4ba91fb2d6c9/resourcegroups/cosmos-repository-sdk-identity/providers/Microsoft.App/containerApps/books-api",
        "message": "Microsoft.App/containerApps/write",
        "hierarchy": "7c6cf4f2-b5e5-433c-8e68-4ba91fb2d6c9"
    },
    "relatedEvents": []
}

vturecek commented 2 years ago

@mumby0168 I think the problem here is that the provided registry server does not match the registry server in the image name. This is the transpiled ARM request from the bipec deployment:

  "registries": [
  {
    "server": "cosmossdkidentitydemoacr",
    ...
  }
  ...
  "template": {
      "containers": [
        {
          "image": "cosmossdkidentitydemoacr.azurecr.io/apps/books-api:49feb3713bccacaf8832268bb7691e71dac77290",
          ...
        }
     ]
  }

The registry server needs to include the full domain: cosmossdkidentitydemoacr.azurecr.io. Right now what's happening is that we look for a registry named cosmossdkidentitydemoacr.azurecr.io based on the image, which is not found in the provided registry list, so we try to pull the image using anonymous pull (which is a valid configuration, as long as the registry has anonymous pull enabled. In your case it doesn't, so you get an unauthorized response).

mumby0168 commented 2 years ago

@mumby0168 I think the problem here is that the provided registry server does not match the registry server in the image name. This is the transpiled ARM request from the bipec deployment:
  "registries": [

  {

    "server": "cosmossdkidentitydemoacr",

    ...

  }

  ...

  "template": {

      "containers": [

        {

          "image": "cosmossdkidentitydemoacr.azurecr.io/apps/books-api:49feb3713bccacaf8832268bb7691e71dac77290",

          ...

        }

     ]

  }
The registry server needs to include the full domain: cosmossdkidentitydemoacr.azurecr.io. Right now what's happening is that we look for a registry named cosmossdkidentitydemoacr.azurecr.io based on the image, which is not found in the provided registry list, so we try to pull the image using anonymous pull (which is a valid configuration, as long as the registry has anonymous pull enabled. In your case it doesn't, so you get an unauthorized response).

Brilliant, thanks for that! I will retry this tomorrow. Then I'll update here and close the issue.

Many thanks

mumby0168 commented 2 years ago

Hi @vturecek I tried this just now and I set the ACR name to have the azure.io suffix, you can see this in the below inputs to the ace module:

The error is the same:

{
    "status": "Failed",
    "error": {
        "code": "WebhookInvalidParameterValue",
        "message": "The following field(s) are either invalid or missing. Invalid value: \"cosmossdkidentitydemoacr.azurecr.io/apps/books-api:228380a04acb0fc091ec117c18cda53d7f3cda5b\": GET https:?scope=repository%3Aapps%2Fbooks-api%3Apull&service=cosmossdkidentitydemoacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.: template.containers.books-api.image."
    }
}

Azure correlation ID: a0e48484-d61e-4b00-80fe-5c6fce72c823 and operation ID 0BC59E506D6F9B51 Any ideas?

Thanks, Billy

mumby0168 commented 2 years ago

@vturecek any updates on this ? :)

AndiHahn commented 2 years ago

I have the same issue. Could still not find any solution.

mumby0168 commented 2 years ago

I have the same issue. Could still not find any solution.

Hi I am still stuck on this also.

iamdanthedev commented 2 years ago

@mumby0168 it probably should be azurecr.io, not azure.io

However I am stuck with the same error

AndiHahn commented 2 years ago

I have found a way to access azure container registry with username and password:

Bicep module:

param location string
param containerAppsEnvironmentId string
param imageVersion string
param containerRegistryUsername string
@secure()
param containerRegistryPassword string

var environmentConfig = [
  {
    name: 'ASPNETCORE_ENVIRONMENT'
    value: 'Development'
  }
  {
    name: 'ASPNETCORE_URLS'
    value: 'http://0.0.0.0:80'
  }
]

resource containerApp 'Microsoft.App/containerApps@2022-03-01' = {
  name: 'shopping-api'
  location: location
  properties: {
    managedEnvironmentId: containerAppsEnvironmentId
    template: {
      containers: [
        {
          name: 'shopping-api'
          image: 'andaha.azurecr.io/andaha/services/shopping:${imageVersion}'
          env: environmentConfig
          probes: [
            {
              httpGet: {
                port: 80
                path: '/hc'
              }
              type: 'Readiness'
            }
            {
              httpGet: {
                port: 80
                path: '/liveness'
              }
              type: 'Liveness'
            }
          ]
        }
      ]
      scale: {
        minReplicas: 0
        maxReplicas: 2
      }
    }
    configuration: {
      activeRevisionsMode: 'single'
      dapr: {
        enabled: true
        appId: 'shopping-api'
        appPort: 80
      }
      ingress: {
        external: true
        targetPort: 80
        allowInsecure: true
      }
      registries: [
        {
          server: 'andaha.azurecr.io'
          username: containerRegistryUsername
          passwordSecretRef: 'container-registry-password'
        }
      ]
      secrets: [
        {
          name: 'container-registry-password'
          value: containerRegistryPassword
        }
      ]
    }
  }
}

Bicep main module:

param location string = resourceGroup().location
param version string
param containerRegistryUsername string
param containerRegistryPassword string

module coreInfrastructure 'core-infrastructure.bicep' = {
  name: 'andaha-core-infrastructure'
  params: {
    location: location
  }
}

module shoppingService 'service.bicep' = {
  name: 'andaha-shopping-service'
  params: {
    location: location
    containerAppsEnvironmentId: coreInfrastructure.outputs.containerAppEnvironmentId
    imageVersion: version
    containerRegistryUsername: containerRegistryUsername
    containerRegistryPassword: containerRegistryPassword
  }
}

From the pipeline i am reading the username and password from azure container registry and pass it to the main bicep file:

- task: AzureCLI@2
      displayName: 'Deploy to dev'
      inputs:
        azureSubscription: $(azureServiceConnection)
        scriptType: bash
        scriptLocation: inlineScript
        inlineScript: |
          acrUser=$(az acr credential show --name andaha --resource-group andaha --query username -o tsv)
          acrPassword=$(az acr credential show --name andaha --resource-group andaha --query "passwords[0].value" -o tsv)
          az deployment group create --resource-group andaha-dev --template-file $(bicepTemplateFile) --parameters containerRegistryUsername="$acrUser" containerRegistryPassword="$acrPassword" version="$(Build.SourceBranchName)"

ThorstenHans commented 2 years ago

Also blocked because of this.

This leads to using ACR premium with dedicated scope maps and tokens in the mean time 😞

CezaryKlus commented 2 years ago

I am also experiencing this error. I have registry configuration set properly:

            "secrets": [
                {
                    "name": "container-registry-password"
                }
            ],
            "registries": [
                {
                    "server": "docker.io",
                    "username": "...",
                    "passwordSecretRef": "container-registry-password",
                    "identity": ""
                }
            ]

when trying to apply the image from the private Docker registry

{
    "properties": {
        "template": {
            "containers": [
                {
                    "image": "docker.io/....:v1",
                    "name": "api",
                    "resources": {
                        "cpu": 0.5,
                        "memory": "1Gi",
                        "ephemeralStorage": ""
                    }
                }
            ]
        }
    }
}

it ends up with an error:

{
    "code": "WebhookInvalidParameterValue",
    "message": "The following field(s) are either invalid or missing. Invalid value: \"docker.io/...:v1\": GET https:: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:... Type:repository]]: template.containers.api.image."
}

AndiHahn commented 2 years ago

@CezaryKlus I have tested my solution only for azure container registry. Not sure if this is working on docker registry aswell. Maybe you can try another registry url, some examples are here: https://stackoverflow.com/questions/34198392/docker-official-registry-docker-hub-url

Have you forgot to set the secret value for 'container-registry-password'?

kendallroden commented 1 year ago

Hi all- please let me know if you are still having issues with this

zbuchheit commented 1 year ago

This has become an issue for me.

vturecek commented 1 year ago

@zbuchheit if you have a correlation ID from the failed attempt to create/update a Container App, I can take a look for you.

andrewfabrizi commented 1 year ago

@vturecek, here is a correlation ID that you can look at: a319c1ec-fcee-467f-aaa0-16dbb9f3645d

andrewfabrizi commented 1 year ago

We are no longer experiencing this issue. Thanks for the help @vturecek.

For anyone interested, we found that azureADAuthenticationAsArmPolicy was 'disabled'. This was causing the error: "WebhookInvalidParameterValue". Setting azureADAuthenticationAsArmPolicy to 'enabled', resolved this issue.

vturecek commented 1 year ago

In case anyone else runs into this, you can use the following command to check if ARM tokens are allowed to access your ACR:

az acr config authentication-as-arm show -r <registry>

If ARM tokens are disallowed, you can allow them with the following command:

az acr config authentication-as-arm update -r <registry> --status [enabled/disabled]

microsoft / azure-container-apps