search

microsoft / azure-container-apps

Roadmap and issues for Azure Container Apps
MIT License
362 stars 29 forks source link

Container app deployment with environment variables formed by combining concat/union and reference function fail validation #391

Open maskati opened 2 years ago

maskati commented 2 years ago

This issue is a:

Issue description

Deploying a Container App with an environment variable value calculated during deployment using the reference function and composed using the concat or union array functions fails validation with the following error: ContainerAppInvalidSchema: Invalid request body for container app. Does not conform to Container App schema

Steps to reproduce

Below is an example Bicep template showing variations of environment variables, as well as the corresponding ARM template for the env property in comments:

This example shows referencing the Container App environment static IP property, but the same issue results when referencing any other resource property using the reference function. The problem does not manifest if only referencing the resource id of a resource, since this maps to the resourceId (instead of reference) function. This seems to be an issue with the way the Container Apps resource provider handles ARM template functions.

param location string = resourceGroup().location
resource managedEnvironment 'Microsoft.App/managedEnvironments@2022-03-01' existing = {
  name: 'containerappenvironment'
}
// Inline array with reference() function works
resource inlineArrayWithReferenceFunctionWorks 'Microsoft.App/containerApps@2022-03-01' = {
  name: 'inlineArrayWithReferenceFunctionWorks'
  location: location
  properties: {
    template: {
      containers: [
        {
          name: 'containerapps-helloworld'
          image: 'mcr.microsoft.com/azuredocs/containerapps-helloworld:latest'
          /* ARM template env block
          "env": [
            {
              "name": "staticIp",
              "value": "[reference(resourceId('Microsoft.App/managedEnvironments', 'containerappenvironment'), '2022-03-01').staticIp]"
            }
          ]
          */
          env: [
            {
              name: 'staticIp'
              value: managedEnvironment.properties.staticIp
            }
          ]
        }
      ]
    }
    managedEnvironmentId: managedEnvironment.id
  }
}
// Concat array with no reference() function works
resource concatWithNoReferenceFunctionWorks 'Microsoft.App/containerApps@2022-03-01' = {
  name: 'concatWithNoReferenceFunctionWorks'
  location: location
  properties: {
    template: {
      containers: [
        {
          name: 'containerapps-helloworld'
          image: 'mcr.microsoft.com/azuredocs/containerapps-helloworld:latest'
          /* ARM template env block
          "env": "[concat(createArray(createObject('name', 'staticIp', 'value', 'managedEnvironment.properties.staticIp')))]"
          */
          env: concat(
            [
              {
                name: 'staticIp'
                value: 'managedEnvironment.properties.staticIp' // dummy string instead of reference function
              }
            ]
          )
        }
      ]
    }
    managedEnvironmentId: managedEnvironment.id
  }
}
// Concat array with reference() function FAILS
// Code: ContainerAppInvalidSchema
// Message: Invalid request body for container app. Does not conform to Container App schema, please visit for more information https://docs.microsoft.com/azure/container-apps/azure-resource-manager-api-spec?tabs=arm-template#container-app
resource concatWithReferenceFunctionFails 'Microsoft.App/containerApps@2022-03-01' = {
  name: 'concatWithReferenceFunctionFails'
  location: location
  properties: {
    template: {
      containers: [
        {
          name: 'containerapps-helloworld'
          image: 'mcr.microsoft.com/azuredocs/containerapps-helloworld:latest'
          /* ARM template env block
          "env": "[concat(createArray(createObject('name', 'staticIp', 'value', reference(resourceId('Microsoft.App/managedEnvironments', 'containerappenvironment'), '2022-03-01').staticIp)))]"
          */
          env: concat(
            [
              {
                name: 'staticIp'
                value: managedEnvironment.properties.staticIp
              }
            ]
          )
        }
      ]
    }
    managedEnvironmentId: managedEnvironment.id
  }
}

Expected behavior [What you expected to happen.] Reference function derived environment variable values should be composable using the concat or union array functions.

Actual behavior [What actually happened.] Reference function usage combined with concat or union array functions cause template validation to fail.

Additional context

This issue occurs during template validation of both Bicep and ARM templates, both through the Portal and Azure CLI.

cwe1ss commented 2 years ago

I tried to set environment variables dynamically by using a configuration set-variable but I received the same error.

Extract from my bicep file:

// My configuration set
var allSettings = {
  http: {
    env: [
      {
        name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
        value: appInsights.properties.ConnectionString
      }
    ]
  }
  // grpc: { ... }
}

// Settings are loaded based on another variable
var settings = allSettings[serviceDefaults.appType]

// Usage in container app resource
resource containerApp 'Microsoft.App/containerApps@2022-03-01' = {
  name: appName
  properties: {
    template: {
      containers: [
        {
          name: 'app'
          env: settings.env
        }
      ]
    }
  }
}

Compiled ARM JSON:

"env": "[createObject('http', createObject('env', createArray(createObject('name', 'APPLICATIONINSIGHTS_CONNECTION_STRING', 'value', reference(extensionResourceId(format('/subscriptions/{0}/resourceGroups/{1}', subscription().subscriptionId, parameters('envGroupName')), 'Microsoft.Insights/components', parameters('appInsightsName')), '2020-02-02').ConnectionString))))[variables('serviceDefaults').appType].env]"

Deployment error message:

"statusMessage": "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'svc-20220905201959' is not valid according to the validation procedure. The tracking id is '35cd00b2-20d3-4672-be7b-b2160c98d95a'. See inner errors for details.\",\"details\":[{\"code\":\"ValidationForResourceFailed\",\"message\":\"Validation failed for a resource. Check 'Error.Details[0]' for more information.\",\"details\":[{\"code\":\"ContainerAppInvalidSchema\",\"message\":\"Invalid request body for container app. Does not conform to Container App schema, please visit for more information https://docs.microsoft.com/azure/container-apps/azure-resource-manager-api-spec?tabs=arm-template#container-app\"}]}]}}",
kendallroden commented 2 years ago

Thanks for reporting

corwestermaniddink commented 1 year ago

@kendallroden Update on this?

colbylwilliams commented 1 year ago

@ruslany the same error happens when you use concat with secrets property

maskati commented 1 year ago

A workaround is to deploy using a Bicep module / ARM nested template, passing in the precalculated array as a parameter. In this way the Container App resource provider does not "see" the functions as they are handled by ARM.

I am surprised that resource providers get unexpanded functions as I always assumed template expansion at the individual template/resource level was also handled by ARM.

ruslany commented 1 year ago

@maskati - you are correct. The container app RP receives an unexpanded function when validating the deployment and that breaks the JSON deserialization which results in the ContainerAppInvalidSchema error. It looks like an inconsistent behavior in the ARM, considering it expands the function to an array when no resource references are used in any of array elements. We are following up with ARM team to investigate further.

wterpstra commented 1 year ago

In a response on https://github.com/Azure/bicep/issues/8405 it said this would most likely be resolved within a month from November 9th, is there an updated timeline on this?

wterpstra commented 1 year ago

Conditional expressions also fail when using reference functions:


resource managedEnvironment 'Microsoft.App/managedEnvironments@2022-03-01' existing = {
  name: managedEnvironmentName
  scope: resourceGroup(resourceGroupname) 
}

var minReplicas = managedEnvironment.properties.zoneRedundant ? 3 : 2

resource containerApp 'Microsoft.App/containerApps@2022-03-01' = {
  name: appName
  properties: {
    template: {
      containers: [
        {
          name: 'app'
          image: image
        }
      ]
     scale: {
       minReplicas: minReplicas
     }
  }
}
matsisak commented 1 year ago

A workaround is to deploy using a Bicep module / ARM nested template, passing in the precalculated array as a parameter. In this way the Container App resource provider does not "see" the functions as they are handled by ARM.

I am surprised that resource providers get unexpanded functions as I always assumed template expansion at the individual template/resource level was also handled by ARM.

@maskati Can you elaborate on the workaround using a Bicep module? I tried passing the env array with dynamic values to a module responsible for the containerApps creationg, but received the same error as mention above, i.e. Invalid template....

maskati commented 1 year ago

It's been a while so don't recall the details. Might also have something to do with ARM what-if backing out in certain situations (see https://github.com/Azure/arm-template-whatif/issues/157) resulting in the ACA RP not validating.

matsisak commented 1 year ago

It's been a while so don't recall the details. Might also have something to do with ARM what-if backing out in certain situations (see Azure/arm-template-whatif#157) resulting in the ACA RP not validating.

@maskati Thanks for getting back, I managed to workaround it with an "app specific" module, passing all the "dynamic" env variables as params to the module. Looking forward to a fix for this issue :-)