search

containers / podman-compose

a script to run docker-compose.yml using podman
GNU General Public License v2.0
5.08k stars 485 forks source link

Rewrite to go #445

Closed matejsp closed 2 years ago

matejsp commented 2 years ago

I found out that there is another project that is written in Go and with also better (closer to podman) license. https://github.com/aramean/go-podman-compose

Aposhian commented 2 years ago

I am interested in helping rewrite podman-compose in Go as CLI plugin for Podman (similar to docker compose v2), so that it can be more pluggable with other Podman library functionality, such as systemd generation, etc.

See: https://github.com/containers/podman/discussions/13425

muayyad-alsadi commented 2 years ago

the way "pluggable" works in docker can be done in any language because it's just an executable placed in /usr/libexec/docker/cli-plugins/ or ~/.docker/cli-plugins named docker-<COMMAND>. Go is not a dynamic language (runtime loadable plugins is not a common thing in go).

Aposhian commented 2 years ago

I used "pluggable" a little loosely (incorrectly). What I meant is allowing more flexibility for re-use between Podman libraries and this project. Here are the primary benefits of rewriting podman-compose in Go:

Aposhian commented 2 years ago

For my selfish reason, I will end up duplicating some of the logic of podman-compose in Go anyway for my project systemd-compose, so I would at least like that effort to not go to waste. If it is good enough, then maybe it can be fleshed out into a feature-complete Go rewrite of `podman-compose. So maybe I can report back once I have some code to show.

muayyad-alsadi commented 2 years ago

podman-compose could use the compose-go library to parse the compose spec YAML files

docker-compose was implemented in python, yet I did not found any value for using their parser (I'm using plain PyYaml).

For example, I would like to develop a variant of podman-compose that converts compose files directly into systemd unit files via podman generate systemd.

You can do this today. A pod or a container created by podman-compose can be passed to podman generate systemd

I'm working in better integration between podman-compose and systemd but I want a way that cover all cases. For example, if we passed --infra=true with --share= then podman pod ps would show degraded, if we remove --share= then two containers can't have different hostname (which compose allow). On the other hand if we pass --infra=false then systemd won't work.

matejsp commented 2 years ago

I prefer minimal setup on host machine running podman. I am using alpinelinux that does not even have python installed. Not being dependent on python is a bonus. That's why I started following go-podman-compose as alternative solution. But feel free to ignore this issue :) Just wanted to get conversation started if it even makes sense since podman is in Go anyway with zero dependencies. There could even be some code share/collaboration with docker-compose since it is in Go too.

Aposhian commented 2 years ago

You can do this today. A pod or a container created by podman-compose can be passed to podman generate systemd

Not exactly. Today, you have to do it in two steps: podman-compose up and then podman generate systemd. That is, you have to run the containers once before you can generate systemd unit files off of them. I would like to have a systemd generator that converts compose YAML files directly to systemd unit files on boot. I plan to use github.com/containers/podman/v2/cmd/podman/generate to convert libpod.Container struct instances directly to unit files. With a Go implementation, I can convert compose YAML files directly to libpod.Container structs, and then convert those to unit files. This makes it so unit files can be generated without having to start the containers first. The benefit to this approach is that a compose file can be the source of truth for running containers on a system with systemd: when the compose file is updated, the service definitions will be updated on next boot.

docker-compose was implemented in python, yet I did not found any value for using their parser (I'm using plain PyYaml).

I can understand this for Python, but since I would like to rewrite the compose -> podman translation in Go, it makes sense to use the compose-go library, which not only provides parsing and strong typing, but also provides validation of compose files. That is not the reason why I want to rewrite in Go though. The main reason is reusability and interoperability with the libpod and Podman Go libraries, as mentioned above.

Aposhian commented 2 years ago

I understand that my motivation for this big change comes from something that may not be of interest to many users. But before I go and re-implement in Go, I wanted to see if this project would have interest in sharing that effort. But this can be revisited once I actually have some code to show (maybe forking off go-podman-compose).

muayyad-alsadi commented 2 years ago

That is, you have to run the containers once before you can generate systemd unit files off of them. I would like to have a systemd generator tha

Make it

podman-compose up --no-start

Maybe this week end I'll automate that using a systemd command inside podman-compose

Back in the good old unix days composing multiple cli commands is not considered something bad.

muayyad-alsadi commented 2 years ago

I prefer minimal setup on host machine running podman. I am using alpinelinux that does not even have python installed.

@matejsp I understand the value of this use case, I myself use busybox image for most of the tests. I'm not against go, I might consider it in the future of this project.

Having said that, I would like to point something regarding alpine, libmusl ..etc. please take those as my personal preference (not as facts) I'm trying not to start a disto war here, but to state my personal reasoning (as an expert in the field). feel free to disagree with me and you would still have full support.

I know that alpine, musl (as opposed to gnu libc) and statically linked small binaries is marketed as faster, minimal and secure because the claimed smaller attack surface.

This is no true at all, I personally don't trust it for production environment (feel free to disagree with me, this is just my personal preference).

Docker have shipped alpine with empty root password multiple times and they keep opening this loophole even after it's reported to them (for example in 2015, it's reported in October, fixed in November, then broken again in December) They don't know how to lock/protect root passwords (This part is history not my personal opinion) Alpine and Docker are affiliated because Natanael Copa (founder of alpine) works for docker inc.

One reason that GNU libc is too big, is that it's full of extra sanity checks and tons of binary backward comparability.

I personally would rather run a fedora-minimal (35MB) and get full GNU/Linux system that using a 6MB alpine linux who can't sort their root password even after they are taught how do that feel free to disagree with me (it's just my distro of choice, my favorite color, my favorite football team)

unless you plane to run it on the edge or a CASIO calulator, ~29MB is not a big deal.

$ podman pull registry.fedoraproject.org/fedora-minimal:latest

now let's get into statically linked binaries. Long, long, ago, we used to treat statically linked binaries as security threat, why? Because it won't get patched (security patches) when you update your system. if there was a buffer overflow vulnerability in some YAML parsing library, it will got system-wide patching if dynamically linked. if it's statically linked then good luck.

Nowadays there are tooling that reduce this risk by scanning (ex. quay.io) ..etc. but the point is still valid.

Last but not least, calling CLI tools (as opposed to a library) is a good old UNIX philosophy not a bad thing (in its own), docker-compose itself is a cli binnary not a library. in docker you place binaries with exec flag in /usr/libexec/docker/cli-plugins/ or ~/.docker/cli-plugins named docker-<COMMAND>

All of the above is my personal reasoning for my own choices feel free to disagree with the above, after all I do understand that my personal favorite color or football team is not a universal fact.

if this project would have interest in sharing that effort.

@Aposhian when I started this project docker-compose was in python, and rootless podman did not have cross-containers networking was impossible (I did many workarounds to make it work, like sharing network namespace and faking host using --add-host to 127.0.0.1, check branch 0.1)

my focus now is to be feature complete. later I might consider go. my priority is aligned with podman, that is root-less and daemon-less (process model rather than daemon model) that is I don't want to talk to a remote podman daemon, I want to execute podman. for example podman-compose build wont' transfer a huge tarball to the remote daemon.

matejsp commented 2 years ago

@muayyad-alsadi I am well aware of the points that you made.

Well Alpine we use is because of Colima and Lima for Mac. It is mostly for development. Don't really have a preference. Now Fedora minimal does not contain python. Not to mention some distros still having Python 2 as default (like AmazonLinux2). And even my Mac came with Python 2 only. And the attack surface is incomparably increased by installing whole Python in addition to statically linked podman written in Go that you have installed anyway.

Are you also aware that python had 6 CVE in 2021? One in 2022. It adds extra 100 MB size. And it is pain to maintain its dependencies. Especially for non python users. pip3 then sometimes pip, virtualenv, pyenv or perhaps venv, workon ... Horrible if you don't want to clash deps with another python project.

And updating a pip installed project is harder than just doing brew update.

Yes I program in python daily and would still prefer such tool in Go. Sry :D But yeah let's agree to disagree.

Aposhian commented 2 years ago

Make it podman-compose up --no-start

Thanks for that suggestion. I'll play around with that and see how well it fits my use-case.

muayyad-alsadi commented 2 years ago

Are you also aware that python had 6 CVE in 2021?

I can query them and patch globally them using a single command dnf/apt/...etc (you patch the dynamic library not each individual binary that binds to it) but with statically linked a flaw in yaml parser embedded inside a binary can't be identified or patched globally.

clash deps with another python project.

this project has only two python deps (yaml and dotenv)

and would still prefer such tool in Go. Sry :D But yeah let's agree to disagree.

don't get me wrong, I'm not against go, I might move to it in the future. I just liked to state my preference and reasoning behind them for other people who keep asking me same questions.

muayyad-alsadi commented 2 years ago

@Aposhian check my comment here

https://github.com/containers/podman-compose/issues/307#issuecomment-1062234830