Feature Request: Separating infrastructure and application deployment

[maskati]

A Container App is defined almost entirety by the Microsoft.App/containerApps resource and its properties, including the associated environment, identity, secrets, container and scaling configuration. Exceptions are authConfigs and sourcecontrols sub-resources as well as revisions history.

Templated deployment through ARM only support creating containerApps by declaring the full resource specification. This is equivalent to the Create or Update REST API operation.

A consequence of this is that infrastructure deployments cannot be easily separated from application deployments in terms of responsibility (separate infrastructure and application teams and/or repos) and security (infrastructure pipelines with wider permissions and application pipelines with minimum permissions to the specific application resource). For example a Container App application repo with a deployment pipeline doing Update of template.container.image or performing az containerapp update --image. Such a setup is complex when combined with updates of desired state infrastructure through ARM, which would typically be out of sync with the application definition.

Azure VMs, App Service, Azure Functions, Static Web Apps, and Azure Kubernetes Service support separating deployment and configuration of associated Azure resource infrastructure from the applications hosted on the infrastructure. Azure Container Instances is like Azure Container Apps in this regard.

Conceptually the problem with likening Microsoft.App/containerApps resources to Kubernetes deployments is that one is an Azure resource and the other isn't. And Azure application hosting resources have a range of other dimensions that are typically in the area of "infrastructure", such as managed identities and associated role assignments.

What is the best practice to separating infrastructure from app deployment when using Container Apps?

Ideally it would be nice to be able to deploy the Microsoft.App/containerApps as well as application-scope changes (properties.configuration) through desired state ARM deployments without impacting revision-scope properties (properties.template) of the apps. One way that this could be done is separating the revision scope template specification into a sub-resource that could be deployed separately. This would allow separating "infrastructure deployments" to target the Microsoft.App/containerApps resource and "application deployments" would instead target the Microsoft.App/containerApps/template resource. Since Container Apps cannot exist as "stub" resources without an template specification, the Container Apps platform could require that the template sub-resource be initially defined in conjunction with the parent containerApps.

// application scoped specification (Microsoft.App/containerApps except properties.template)
// deploying this will not change child Microsoft.App/containerApps/template specification
resource containerApp 'Microsoft.App/containerApps@2023-01-01-preview' = {
  name: 'containerapp'
  properties: {
    configuration: {
      activeRevisionsMode: 'Single'
    }
  }
}

// revision scoped specification
// this can be deployed separately from the application scope
// updates to this would be equivalent to PATCH of containerApps.properties.template
resource template 'Microsoft.App/containerApps/template@2023-01-01-preview' = {
  name: 'template'
  parent: containerApp
  properties: {
    containers: [
      {
        name: 'helloworld'
        image: 'mcr.microsoft.com/azuredocs/containerapps-helloworld:latest'
      }
    ]
    scale: {
      minReplicas: 0
      maxReplicas: 1
    }
  }
}

[mustafazeya]

Is this part of roadmap yet?

[duglin]

@maskati can you take a step back and elaborate on why this is an issue? For example, today I believe you can modify activeRevisionsMode w/o impacting the currently running revisions - so I think that's the type of separation you're interested in - even though they're all part of the same Bicep/ARM file.

A consequence of this is that infrastructure deployments cannot be easily separated from application deployments in terms of responsibility

I've heard this from time to time, and in some type of infrastructure it makes sense... e.g. VM separate from app running on the VM - because each has their own lifecycle. However, for ACA I'd like to better understand why you think this is needed. Why would someone want to setup the skeleton of an app w/o also giving it things like the image name? What would be the purpose of the "skeleton"? What's the benefit when the important parts (that define an app) are all things that the devs will be able to touch ... like image name. In other words, saying "only certain people can create an app, but other people (devs) can set the image" feels like a false security blanket when the 2nd set of people can change just about every aspect of the app. Would love to learn more about the needs here...

[maskati]

I agree that often this is not required. It has mainly to do with scenarios, where there is a separation of responsibility regarding resources external to the app itself.

One example is authorization of the application identity. Such authorizations cannot be made to the Container App system assigned managed identity until the app itself is created and running (somewhat related). User assigned identities are of course an alternative, but they have a slightly different (worse) security profile as well as requiring associated implementation in the codebase (codebases which assume system assigned identity may need to be refactored for user assigned support).

Another example is controlling which applications can be joined to a specific Container App environment. Currently deploying a Microsoft.App/containerApps and specifying environmentId requires the microsoft.app/managedenvironments/join/action permission (somewhat related) which you might not want to delegate in larger environments.

[duglin]

The identity stuff should be handled by ACA, not the user - there are ways to tweak ACA so that it can do that. Just not implemented yet - AFAIK.

As for separation of authority, sure in general I understand it's a nice idea, but the higher authority should control all risky actions not just one. It might feel good to only let certain people associate an app with an ACA env, but if the less trusted devs can change the image of an authorized app to something dangerous then what have people really prevented? Closing one hole but leaving many others open is a false security blanket.

Imagine someone saying their system is more secure because they only allow certain people to install executables into the VM, but they let a lot of people change the bytes of those exes - e.g. cp /usr/sbin/sshd $HOME/trusted-exe. In fact, I think this would be much harder to detect than a new exe (app) appearing - and the option I would choose if I were up to no good :-)

When this level of concern is present, the entire ACA env should be locked down and only a few people (or bots) are given access, then the gatekeeper controls are handled in the CI/CD review processes, long before it gets to ACA

[maskati]

there are ways to tweak ACA so that it can do that. Just not implemented yet - AFAIK.

Looking forward to whatever this is :)

I believe in a layered approach to security, and that zero-trust does not do away with the benefits of other security measures. Introducing a new malicious workload and attacking an existing workload are two distinct though similar attack vectors. The separation of concerns is also not necessarily to prevent malicious intent but also to prevent inadvertent mistakes.

Overall, I agree that security is probably not the driving factor for this request, and the bigger challenge is to do with the ACA system assigned identity where "identity comes into existence" and "the app is already running" occur at the same moment. Like I mentioned in my previous comment, there are workarounds, but these tend to dictate specific deployment strategies or identity management practices. However, since there are workarounds I am also happy to close this feature request and wait for whatever related features are coming up in the ACA pipeline.

[duglin]

Definitely don't close it because of me ;-) I just wanted to better understand the ask.