Publish complex compose projects

[mike-sul]

Description

The linked feature here is a valuable addition, allowing users to distribute a Compose application using container image infrastructure (registries/hubs). However, it appears incomplete. The current alpha implementation only supports publishing a single Compose project file, whereas a complete project may consist of:

1. Multiple compose files (utilizing the extends attribute).

2. Complementary files containing necessary data/configurations for one or more Compose app services, like a bind mount of a config file or secrets.

The feature request is to enhance the current publishing functionality to support complex Compose projects. This extension should enable the publication of multi-file Compose projects along with associated complementary files.

[ndeloof]

For such scenario, using a git remote is way more convenient and flexible AFAICT

[doanac]

A git server is great for developers. However, one of the most powerful features of Docker is container image distribution. Treating a compose project as its own entity and teaching compose how to consume this bundle/app/entity could be a really powerful means of application distribution. Not totally unlike the old work done with Docker Apps.

Another benefit - authentication/authorization for private registries. You distribute one set of creds to your fleet (for an OCI registry) rather than creds for both docker and git.

[ndeloof]

for sure having OCI support for complex project would be nice, but this require Compose to discover all the involved files and this is far from being trivial. That's obviously something we consider for improvement of the publish command and, as you noted, this would come with benefits. My point is, short terms, use of git makes this way simpler and should be considered as a workaround.

Not totally unlike the old work done with Docker Apps

I used to work on docker/app some years ago :) While I hated the CNAB approach, the main concept is something I'd like to resurrect with publish, but this we are still in a very early stage.

[doanac]

While I hated the CNAB approach,

No comment :) but y'all had the right big vision which was why we got interested.

After Docker Apps went away, my team wound up needing something. We built a pretty low-tech tool/concept called "compose apps". Basically we tar up a directory like:

./<proj>/docker-compose.yml
./<pro>/nginx.conf # or whatever

We also pin each container image reference in the compose.yml to its sha256 reference giving us an "immutable app". We push that tarball to the registry along with a "compose-app" manifest that we had to hack up before all the good OCI standards had emerged.

We've had some challenges/shortfalls such multiple compose files and pinning the container images with the compose-ref libraries.

Anyway - if y'all every want to talk about this, Mike or I would be happy to discuss.

[ndeloof]

I created https://github.com/docker/compose/pull/11136 as an attempt to offer a feature closer to docker/app

[oliora]

One way to simply the publishing would be by publishing of the “final” compose.yaml - i.e. the file after merge, extend and include of all the compose project yamls is done. This single file plus configs and secrets it references should be pretty much enough to run the published app

[ndeloof]

One way to simply the publishing would be by publishing of the “final” compose.yaml

in most cases variable in compose file must be adapted to the user's target environment, interpolating before application is published as OCI artifact would make those hard-coded, including application name. Ability to publish a parametrizable compose file is a MUST HAVE feature. Also, if you only want to publish resulting compose model, you don't need a new command. Just run docker compose config and run any registry client to publish as an OCI artifact.

This single file plus configs and secrets it references should be pretty much enough to run the published app This assumes both secrets and configs are local, which is not always the case (many users have those file living on remote docker host. Also, publishing secrets would be a terrible anti-pattern. I don't want I have to manage a CVE with this feature