Future of podman-compose

[jmauro]

Hello,

Since the latest Podman release (v3), the docker-compose cli is natively supported (Podman Release Note). So I was wondering what is the roadmap regarding this project? Will it be slowly put into maintenance mode and deprecated in favor of docker-compose?

Kind regards, J.M.

[athrunsun]

Reference: Using Podman and Docker Compose

For podman to work with docker-compose, we must start podman in root mode right? But that defeat the purpose of being rootless!

podman-compose is able to work in rootless mode, which I think is better than docker-compose. So I wish we could carry on with this project.

[rhatdan]

BTW I am fine with carrying on this effort, especially if podman-compose supports more features of podman then docker-compose.
Docker-compose support in Podman right now requires rootfull containers. We plan on adding support for rootless in the future, but it is not there yet. Also running docker-compose requires users to enable to podman.socket systemd unit file. In rootless mode, the same requirement will be there, once it is supported.

The main issue with rootless docker-compose is around handling of networks, since rootless networking is very different then rootfull networking.

[KangoV]

We are about to look into moving to Podman and this would be part of it. Do I take it that development of this has not stalled?

[markstos]

If it's not stalled, it's not exactly active either. I submitted a simple PR 9 months ago that has yet to even receive a comment from a developer:

https://github.com/containers/podman-compose/pull/200

If you only use "rootful" containers, I would use the docker-compose support linked above.

[markstos]

From the lack of merges in the last 6 months but the active interest expressed through Issues and Pull Requests, it appears time for a "call for maintainers" to encourage someone to step up to maintain this. Otherwise, it seems ripe to be forked and maintained elsewhere if interested parties are not able to get commit access here.

[rhatdan]

I would prefer that this stay under the containers umbrella, and if the originators of this project have disappeared, we could give it over to new maintainers.

[NicolasCARPi]

I agree with @markstos. This repository does not appears to be maintained. To be honest, I'm suprised that Red Hat, who is pushing for podman use, is not developing this tool themselves. From the git logs it looks like this project was initially created by @muayyad-alsadi, who seems to have lost interest in this project (which is fine, that's just the life of an open source project for which you are not paid, thank you for the work already done!), and @debrutal is Author but has not commited in 2021 anywhere.

Consequently, it is indeed the time for someone else to step in as maintainer/developer, or else podman will probably never be able to compete with docker: hear me out. Are users expected to always use a bunch of run commands instead of having portable yml files describing everything nicely? Similarly, are users expected to have a PhD in containerization to figure out around iptables/dns/port mapping/volumes/network/random issues/bugs that arise all the time if they chose to use docker-compose or podman-compose in the state they are in?

IMHO, a "compose" tool goes hand in hand with the container tool. Note that docker-compose is created/maintained by the docker team, so why isn't podman-compose given love by the podman creators?

It looks like users are using docker-compose with podman containers, but you have to forget about the two majors features of podman: being rootless and socketless (see message above by @rhatdan).

As @KangoV, I was considering trying out my "docker+docker-compose" setup on centos 8 with podman, and see if it works, but after looking at the state of this project, I decided it's probably not worth it to invest time in it. And looking at the issues related with docker-compose + podman also stopped me in my tracks (not that anyone cares though :p).

That being said, I understand fully the scope and difficulty of such a project and I am aware that my slight rant doesn't really bring anything to the table. Unfortunately, I am not familiar enough with the code base, nor a user of podman-compose to propose myself as a maintainer, but I hope someone else can step up, because docker is great and all, but suffers from flaws that hopefully can be alleviated by podman. But as I said, if no "compose" tool is available, few will switch (I might be wrong, but it is my opinion). And if Docker is not a first class citizen in the red hat world, people will move to distributions that don't try to push you into podman by force.

Or maybe the podman project is still too young and lacking maturity? After all, it is not a trivial project!

@rhatdan Maybe you can discuss with other members of the containers org and see what to do with this project? Or maybe ask the redhat execs to actually pay someone to work on this project? Would love to get your insight about that subject, too! It looks like there is a strong direction[1] towards K8s, so maybe the "intermediate" usage isn't a priority for the podman project? Or maybe it's silly to try and have a podman-compose, and we should rethink how to deploy several containers in a complex environment?

Sorry for the wall of text, but I had to express myself after stupendously realizing that the de facto tool for doing composition with podman is sorta kinda abandoned!

Cheers, ~Nico

[rhatdan]

Nico, The Red Hat Podman team has been focused on two ways of doing local orchestration of containers.

1. kubernetes - yaml support We believe that Podman is a great gateway tool to move users into the kubernetes world. Basically built and test pods/containers locally and then move them kubernetes. Allowing them to get familiar with the kubernetes yaml content. Also allow them to use the same files whether in kubernetes as well as on a local system, where they just want the containers to run, without the overhead of kubernetes.
2. docker-compose This is more about just the installed base and institutional knowledge of compose. The goal here is to allow users who have built large compose apps to work with Podman as a server. This also aligns with us attempting to support the huge base of CI/CD systems that use the docker.sock via docker-python. This is one of our number one requirements to help Red Hat customers migrate from RHEL7->RHEL8 where Red Hat does not support Docker engine.

We have supported the Podman-compose effort and would love to continue to support it, since it has some nice features and we would love to see it continue to evolve. If maintainers stup up to support it, we will them get permissions, I don't see Red Hat directly funding this, but any Red Hat engineer is welcome to step up and work on any Open Source project, and this is heavily encouraged, if the engineer has time.

[NicolasCARPi]

Thank you @rhatdan for your insights about podman's goals! :)

[khimaros]

For podman to work with docker-compose, we must start podman in root mode right?

i have successfully used docker-compose with the podman service running under an unprivileged account. this was on Debian Bullseye. instructions at https://khimaros.com/articles/docker-compose-podman/

[rhatdan]

Sure it works, except for cases where network configuration is happening, I believe.

[nickavem]

+1 on keeping this developed.

docker-compose on podman can still be fiddly, and isn't nearly as well "integrated" with podman as this project. If RH is not providing direct funding, I'm sure many of us would be willing to pitch in with an open collective, LibrePay etc.

[WhyNotHugo]

What's the benefits of improving podman-compose rather than just make podman work with docker-compose? E.g.: In what way do the goals differ?

[muayyad-alsadi]

you can use podman with docker-compose like this

https://fedoramagazine.org/use-docker-compose-with-podman-to-orchestrate-containers-on-fedora/

but podman-compose have different goals that focus on podman benefits over docker that it can work without daemon and in use user containers

[WhyNotHugo]

docker-compose works with podman without any daemon (e.g.: you can use podman instead of docker).

However, this requires using podman-as-root, and won't work with rootless podman.

Personally, I'd love to run my docker-compose.yaml files with rootless podman. There's two approaches to this: fixing whatever it is that fails in rootless-podman right now, or focus on work on podman-compose.

So my question of how goals differ is because there's two approaches, and work can go it two directions, but I'd like to understand if it's not just duplicate work. If podman (rootless) get the necessary fixes to work with docker-compose, wouldn't that negate the need for podman-compose?

[muayyad-alsadi]

let me give you an example to see how podman-compose and docker-compose approach the problem differently

let's say you run the podman socket rootless

export DOCKER_HOST=///run/user/$UID/podman/podman.sock

and you enter a directory and typed docker-compose build since docker-compose is designed to talk to remote docker daemon, it will create build context tarball and send it to the daemon via the socket

on the other hand podman-compose does not create tarballs nor send them to anybody, it will just run buildah on the directory it's standing in.

[WhyNotHugo]

Ah, thanks for the explanation. That seems like it would be a great improvement, especially building large repositories / contexts.

[Himura2la]

Let me start the new "thread" in discussion about the future of podman-compose. As we can see in docker/compose@master, the docker-compose is being rewritten in Golang, so we can't even use their python parser of docker-compose.yml. This project is dying without full support of docker-compose.yml schema, and I'm sure that maintainers should abandon the existing Python implementation ASAP, and switch to forking the docker/compose@v2.

[muayyad-alsadi]

so we can't even use their python parser

currently I'm not using their python parser, so this change does not impact us in any way

the docker-compose is being rewritten in Golang

this change has no impact on us, because this project is not a fork of their project. and it does not even use the same idea of talking to a daemon via remote API. it just executes podman (process model, not remote daemon model)

[Himura2la]

currently I'm not using their python parser

I see this as a major problem. Without forking the original project, podman-compose will always lack of support for some new or rarely-used features. For instance, even now I can't use object as a port mapping, which is very useful because I always forget which port is inner and which is outer. And I don't know how many docker-compose.yml features are not supported by podman-compose... This porject should inherit AT LEAST the upstream tests, and ideally, the upstream parser...

[WhyNotHugo]

I see this as a major problem. Without forking the original project, podman-compose will always lack of support for some new or rarely-used features. For instance, even now I can't use object as a port mapping, which is very useful because I always forget which port is inner and which is outer. And I don't know how many docker-compose.yml features are not supported by podman-compose... This porject should inherit AT LEAST the upstream tests, and ideally, the upstream parser...

You're assuming that upstream docker-compose won't diverge in conflicting ways, which, IMHO, is rather unlikely. Even the recent docker-compose v2 no longer works with podman, and I can imagine that docker will want to continue diverging their own way.

However, I do see some value in starting off with a fork of upstream's go implementation, to have a solid starting point. Given the history of conflicts with distributing Python to various distributions and platforms, it might be a sensible step, but it ultimately requires someone to grab the docker-compose go code, and port it to work in golang. That's a lot of work, but I'm sure someone would step up for the right price.

[muayyad-alsadi]

I see this as a major problem.

it's a standard YAML, we are not reinventing the wheel here.

This porject should inherit AT LEAST the upstream tests,

I have different goals, our goals diverge so our implementation and priorities.

my priority is user-space containers and process model (ex. podman-compose build does not send gigs of content tarball to the daemon, it just execute the build)

if you want to use podman man just like docker, you can open a podman socket and use docker-compose as in this article

https://fedoramagazine.org/use-docker-compose-with-podman-to-orchestrate-containers-on-fedora/

[Himura2la]

it's a standard YAML, we are not reinventing the wheel here.

I mean the schema. Key names and types allowed in them.

my priority is user-space containers and process model

And that's the most awesome thing about this project. But the overall task can be roughly decomposed in two parts:

1. Deserialize YAML and represent the result in some unified inner form (like OOP objects).
2. Convert a unified inner representation of docker-compose.yml to a series of podman CLI commands.

Two steps are required, because docker-compose.yml allows several forms of the same statements, like ports and volumes, and we should merge them anyway. As I already said, the absence of support of ports long syntax makes me sad.

Part one should be inherited from the mainstream, IMHO, because it saves tons of work for full ongoing Compose file format support.

[rhatdan]

@muayyad-alsadi Great point on the podman build side... It is too bad this was not optimized for the local case. If compose could send over the location of the context directory rather then copying it locally.

[muayyad-alsadi]

For instance, even now I can't use object as a port mapping,

I've just solved this, you can now use object port mapping.

rather then copying it locally.

would you please elaborate on this, as far as I know, podman-compose do not copy context, it just point to the location of the context (around 1332).

    build_args.append(ctx)
    compose.podman.run([], "build", build_args, sleep=0)

[Himura2la]

Looks like there is a separate project aimed for standardizing the compose-file spec: https://github.com/compose-spec/compose-spec/blob/master/schema/compose-spec.json They provide the JSON spec, so looks like we can avoid rewriting the existing python code, and simply use the JSON schema for parsing.

At least, it's a good idea to run their conformance-tests.

You're assuming that upstream docker-compose won't diverge in conflicting ways

@WhyNotHugo, yes

[muayyad-alsadi]

If compose could send over the location of the context directory rather then copying it locally.

as far as I know and as far as podman-compose is concern, compose does not copy, there is no copying in my code would you please give me more details about your case

They provide the JSON spec, so looks like we can avoid rewriting the existing python code, and simple use the JSON scheme for parsing.

this json schema might help validating the yaml before running it

https://github.com/compose-spec/compose-spec/blob/master/schema/compose-spec.json

but I see no value byond that.

On Wed, Nov 17, 2021 at 11:36 AM Himura Kazuto @.***> wrote:

Looks like there is a separate project aimed for standardizing the compose-file spec: https://github.com/compose-spec/compose-spec. They provide the JSON spec, so looks like we can avoid rewriting the existing python code, and simple use the JSON scheme for parsing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/containers/podman-compose/issues/276#issuecomment-971401428, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKAPK7ESOQL4QDWRFBC7NDUMNZTRANCNFSM4XU5LJIA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Himura2la]

this json schema might help validating the yaml before running it but I see no value byond that.

If podman-compose uses its own compose-file parser, then it might fail to recognize some valid compose-file statements. My greatest concern is that podman-compose might fail on a valid docker-compose.yml. You should validate your parser against this schema, not user YAMLs...

[markstos]

Validating against the schema seems like a great idea. That would allow podman-compose to provide diagnostics like "That feature is supported in spec, but podman-compose doesn't implement it". That would be better than having some option be ignored and wondering why it didn't appear to have an effect.

On Wed, Nov 17, 2021 at 7:43 AM Himura Kazuto @.***> wrote:

this json schema might help validating the yaml before running it but I see no value byond that.

If podman-compose uses its own compose-file parser, then it might fail to recognize some valid compose-file statements. My great concern is that podman-compose might fail on a valid docker-compose.yml. You should validate your parser against this schema, not user YAMLs...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/containers/podman-compose/issues/276#issuecomment-971544085, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAGJZLG4MBMEBR76RPPSETUMOPPVANCNFSM4XU5LJIA .

--

Mark Stosberg (he/him)

Director of Systems & Security

@.*** | 765.277.1916

https://www.rideamigos.com https://rideamigos.com/

Changing the way the world commutes.

https://www.linkedin.com/company/rideamigos https://www.twitter.com/rideamigos https://www.facebook.com/rideamigos https://www.instagram.com/rideamigos https://rideamigos.com/newsletter-sign-up/