Add support for sidecar containers

[wiktork]

Deploy an application with a primary container as well as secondary sidecar containers. Something like:

var builder = DistributedApplication.CreateBuilder(args);
builder.AddProject<Projects.WebApplication3>("webapplication")
  .WithContainer("sidecar")

This should translate to

containers: [
  { 
    image: 'webpplication'
  },
  {
    image: 'sidecar'
  }
]

[davidfowl]

@karolz-ms This might require a new resource type in dcp

[karolz-ms]

@wiktork what would be the differences between a "sidecar" container "attached" to some other application project (this is my interpretation of your code snippet) vs. just another Container instance that is part of the workload, but not attached to any specific project?

[wiktork]

@karolz-ms When I deploy my aspire app I want to have 1 Azure Container App with 2 Containers and 1 Volume Mount shared between the two. Currently if I do both AddProject and AddContainer, and then azd init my app from code, the Container specification overrides the Project.

[karolz-ms]

@wiktork thanks. Can you tell a bit more what these two containers do and why there is a shared volume mount between them? And in general, what would be your preference for running your app locally vs, say, in an Azure Container App environment?

E.g. when running locally, would you want to run the "WebApplication3" project as a regular OS process, or as a container? Does the "sidecar" need to be a container, or can it be an OS process too? If the sidecar must be a container, we could presumably still run the WebApplication3 as a standalone process using bind mounts for shared filesystem between the two.

@mitchdenny FYI

[mitchdenny]

I think we should break this down into two problems. Sidecar containers and volume mounts. For sidecar containers, I think that this is something that is fairly common amongst orchestrators with some question marks over container lifetime (e.g. init containers etc).

Volume mounts is more problematic. Once you get to the cloud hosting providers there isn't really a viable way to use a host path as a volume mount, so you need to consider what volume mounting solutions the storage system for your particular compute platform supports. For this reason, it might make sense to model out the concept of a volume as a first-class resource type - in ACA it could be provisioned as an Azure Storage Files endpoint which is then mounted into the ACA deployed container (this would just be down to the way that AZD interprets a volume resource).

Another angle on this problem is what is the initial content of the volume mount. A lot of people will use volume mounts to inject configuration which cannot be done via simple environment variables. Prometheus is a pretty good example of this, as is Grafana. For these scenarios it might be better for folks to inject these configs via building a customized container.

How we handle staging loose files from the build directory into a volume mount is another challenge. Different IaC solutions support this to varying degrees. Using Bicep for example it is possible to load a text file into memory and then set that to the content of a blob by using a deployment script resource ... but it is wildly inefficient and there are size constraints.

[mitchdenny]

... it might be that we have to treat priming a volume mount the same way that we treat initialzing a database in that we have a contianer which runs which populates the content. Where it gets the files from is another question.

[wiktork]

For our scenario, we are trying to do something similar to https://github.com/Azure-Samples/dotNET-FrontEnd-to-BackEnd-with-dotnet-monitor-on-Azure-Container-Apps/blob/main/Azure/container_app.bicep. We use the volume mount primarily as a means of sharing information between two containers. (In this particular case, it's dotnet diagnostic information from the process that is only accessible across unix domain sockets / named pipes on windows). As such, the volume mounting has no initial content, follows the lifetime of the replica or pod, and uses the emptyDir type. It does not require a specific storage provider such as Azure Files to function.

I do see complications for local vs. cloud in this case, since we communicate differently with a Windows app vs. a container.

[karolz-ms]

Makes sense, thanks to both of you. I agree that both "sidecar container" and "volume mount" concepts make sense, but I am wondering if it makes sense to raise the abstraction level a notch in the CAM.

For example, we could have "multi-project service" object type, and its instances would be deployed as multi-container apps to ACA and pods to Kubernetes. In the local dev environment we might not need to do anything different for multi-project services because all projects share the same OS environment.

For volume mounts my inclination is to treat "configuration via files" vs "scratchpad filesystem" as different use cases/different model entities. Wiktor's case is the scratchpad filesystem one. The CAM could have a type for it, and in the ACA environment it would result in a volume, plus volume mount for each container that consumes it. In the local dev environment a scratchpad filesystem instance would be rendered as a temporary folder (allocated by DCP). The mount path might be a bit tricky to keep uniform between the ACA/K8s case and the local dev case because it is not always easy (and potentially conflicting) to create a link at the root of the filesystem as in Wiktor's example. We could mitigate this by using extra level of indirection via env. vars (i.e. the scratchpad filesystem path would not be well-known but instead provided via an environment variable). One wishes cgroups/namespaces was an universal OS concept ☺️

[TeddyAlbina]

On AWS you can mount S3 as a volume in your container not sure if we can do that on Azure, so volume mount is a challenge by itself.

[davidfowl]

We should figure out how to support this scenario in dev and deployment in the manifest. Requires more brainstorming.

[samsp-msft]

Do we have the notion of launching projects as docker (or other container orchestrator) containers rather than running locally? Concepts like sidecars and volume mounts (tempfs, secrets, shared mounts) etc make much more sense once running as a container, so maybe only offer them in that environment. It would then be an option in the appmodel as to how the project is run?

[DamianEdwards]

Do we have the notion of launching projects as docker (or other container orchestrator) containers rather than running locally?

Not currently, no. That's part of the issue here.