search

portainer / portainer

Making Docker and Kubernetes management easy.
https://www.portainer.io
zlib License
30.52k stars 2.44k forks source link

DISCUSSION: Compose > v2 support #3750

Closed ghost closed 3 years ago

ghost commented 4 years ago

Is your feature request related to a problem? Please describe. Libcompose, the library we currently use for deploying stacks on docker standalone is deprecated and is causing a TONNE of issues. Something needs to be done, as a broken implementation of stack deploy on standalone is not user-friendly nor is it a feasible for support reasons.

Issues that will be solved by this outlined below:

Describe the solution you'd like I think there are two obvious paths we can take, but there could also be alternate solutions to this (feel free to comment with suggestions);

  1. Replace Libcompose with the docker-compose binary and wait for the docker-compose go library to be released tracked here. The limitation of this approach is the docker-compose binary is only compiled for Linux and Windows.
  2. Hide the stack deploy UI for standalone endpoints while we wait for the docker-compose go library to be released tracked here
Codelica commented 4 years ago

Option 2 "Hide the stack deploy UI for standalone endpoints" would definitely require us to freeze at a previous release until things are sorted out. :) At this point Portainer's standalone compose implementation is "usable" but with a variety of quirks and limitations (as mentioned), some of which can be worked around. So personally I like option 1, assuming building things out against binary for now isn't too involving. If that's more work than it's worth, I'd say leave things as they are until the Go lib is available. Just my .02

deviantony commented 4 years ago

@Codelica agree. Actually our idea was to hide the stack deploy UI only for specific plaforms.

The docker-compose binary is only built for Linux, Mac and Windows (see https://github.com/docker/compose/releases). This means that if we want to leverage it and embed it in Portainer, we'd only support stack deployment on these platforms which would mainly impact arm, arm64 and our other platform users. This is why we wanted to hide the UI in that specific case.

Another alternative would be to fallback to libcompose if the platform is not supported by docker-compose.

mateuszdrab commented 4 years ago

@deviantony Considering the other issue we talked about recently when it comes to the stack feature pulling from Git hosted on VSTS - it seems like it might be just better to go with executables and disable certain features on certain platforms. No point making some x64 users sacrifice useful features like this so that someone can run Portainer on the Raspberry Pi - I'd say a Raspberry Pi user is less likely to need stack deployment anyway.

Codelica commented 4 years ago

Not that our use is necessarily typical, but we do deploy under linux across x86_64, arm32v7, and arm64v8 with custom app templates for each arch that include v2 compose recipes (working around limitations we've hit). So personally I'd love to see this approach for now:

"Another alternative would be to fallback to libcompose if the platform is not supported by docker-compose."

IMO the current compose lib isn't useless for ARM -- just a little "special" :) So if it's possible to keep around for ARM (with disclaimer?) until the Docker Go lib is available, that would be great for us.

BTW, the recent issue triage is noticeable. Thanks!

mateuszdrab commented 4 years ago

@Codelica I agree, automatic fallback and a little red disclaimer at the stack deployment page about limited functionality due to library issue sounds like a good solution.

xZero707 commented 4 years ago

I would disagree that stack deployment is completely broken as is. It certainly has own quirks, but can serve really well if used properly.

  1. Hiding stack functionality would prevent some users from updating Portainer. Usage of stacks on a single host is beneficial, simply because it makes easy to manage microservices within the stack and also makes things organized which can turn really messy when you have hundreds of containers running. If users were forced to withhold the update of Portainer, it could also pose a security risk for them due to potential vulnerabilities. The alternative is to hide, but allow activation by setting Portainer environment variable or by supplying argument to the command.

  2. How would be the environment be different for docker-compose versus libcompose? Isn't libcompose executed within container anyway and uses docker API? Why would be docker-compose different? It can run inside the container. Pardon me if I'm lacking deeper knowledge about docker-compose that prevents me from seeing issue in your way.

deviantony commented 4 years ago

@xZero707 libcompose is a library approach to provide the same functionality a docker compose but it is outdated and not maintainted, as such it only supports (and partially) docker compose format version 2.0.

Using the latest version of the docker-compose binary would provide the latest docker-compose features and fixes as well as support for the latest docker compose format version.

xZero707 commented 4 years ago

@deviantony That's pretty clear for me, but what isn't is how docker-compose is different if used inside the container? It supports different CPU architectures (like ARM). Here I'm calling for alleged problems on different platforms.

deviantony commented 4 years ago

@xZero707 See https://github.com/docker/compose/releases

Binaries are available for Linux and Windows (x86_64) and not for ARM (which is an architecture that is supported by Portainer). Of course you can use it to execute docker-compose to remote environments but we also need to keep in mind the use case of Portainer being deployed on arm.

guillaumelamirand commented 4 years ago

@xZero707 See https://github.com/docker/compose/releases

Binaries are available for Linux and Windows (x86_64) and not for ARM (which is an architecture that is supported by Portainer). Of course you can use it to execute docker-compose to remote environments but we also need to keep in mind the use case of Portainer being deployed on arm.

Hi, I am realy interesting by the support for compose >2. I have been able to install docker-compose on armbian, using alternative procedure describes here : https://docs.docker.com/compose/install/#install-using-pip. I am using this procedure for basic debian too.

pedrolamas commented 4 years ago

Another one using docker-compose in armbian with no issues at all (I too sed pip to install it)!

Mind you, you can actually run docker-compose from a container as indicated here!

sudo curl -L --fail https://github.com/docker/compose/releases/download/1.25.5/run.sh -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
deviantony commented 4 years ago

This would be another alternative although we'd now need to maintain another Linux image instead of having a generic Linux image (we would need to have pip available in the image).

crazy-max commented 4 years ago

@deviantony I think https://github.com/compose-spec/compose-go could be a good candidate in this case.

Awatatah commented 4 years ago

Almost a month since last comment. Any progress on this? #3382 in this list is completely blocking us. Is there a work around?

ghost commented 4 years ago

So, we now have a development timeline out for v2.* versions of Portainer, you can read about it here. Support for compose version3 is slated for Q3/Q4 2020

ghost commented 4 years ago

Is there a work around

Not that I am currently aware of

xZero707 commented 4 years ago

@Awatatah There is no workaround unless moving to (single node?) docker swarm is an option. In that case, Portainer uses docker stack which works well with compose version 3.

lemmy04 commented 4 years ago

I'm very much in favor of just calling docker-compose.

Also, hiding the "stack deploy" ui on standalone setups would make people start looking for other tools to use instead of portainer - a LOT of docker images on docker hub provide their "how to run this" in form of a compose file, usually in version 3.

Hecatron commented 4 years ago

Just to mention I'm currently using docker / portainer / docker-compose on a Rpi4 / ARM64 (gentoo) with hard disks and 8Gb Ram. docker-compose appears to use python3

It would be nice if there was a way for this to be rolled into portainer at some point, or just call the docker-compose on the actual host to save referencing a linux image containing python3

I had a quick look if extensions / plugins were possible but it looks like that's not an option ether. So I'll probably look at using Yacht for deploying docker-compose templates. https://github.com/SelfhostedPro/Yacht

mygrexit commented 3 years ago

Are you planning to release the business version before adding docker compose support?

Inglebard commented 3 years ago

Hi, @Kourgiantakis , there is a roadmap here : https://www.portainer.io/2020/06/portainer-2-0-and-the-development-balancing-act/

ncresswell commented 3 years ago

Yes we are.

Business edition is released soon, the next CE release will include compose v3 support.

Rgds,

Neil Cresswell

lemmy04 commented 3 years ago

Yes we are. Business edition is released soon, the next CE release will include compose v3 support. Rgds, Neil Cresswell

...where do I send the beer?

siddjellali commented 3 years ago

Hi guys,

If I am not mistaken, support of v3 compose is not part of the new release CE, right ? Maybe it's only for Business edition ?

lemmy04 commented 3 years ago

Hi guys,

If I am not mistaken, support of v3 compose is not part of the new release CE, right ? Maybe it's only for Business edition ?

new release? did I miss something?

ncresswell commented 3 years ago

Its earmarked for CE 2.1

From: Sid-Ahmed notifications@github.com Sent: Sunday, 8 November 2020 10:07 PM To: portainer/portainer portainer@noreply.github.com Cc: Neil Cresswell neil@cresswell.net.nz; Comment comment@noreply.github.com Subject: Re: [portainer/portainer] DISCUSSION: Compose > v2 support (#3750)

Hi guys,

If I am not mistaken, support of v3 compose is not part of the new release CE, right ? Maybe it's only for Business edition ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_issues_3750-23issuecomment-2D723549245&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=ALLVYs0_h7o_jxQ9Uo9-h-621-Je2vN1yF8YcR1cucU&s=YWNczAA1RkTXQbdSmj1oPZj_TKotcFzlcMyw1Rg7kLg&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AFQ2XFIGFKKVBKOTA6AOMH3SOZNSNANCNFSM4MP2SO4Q&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=ALLVYs0_h7o_jxQ9Uo9-h-621-Je2vN1yF8YcR1cucU&s=wlw51BREY8-XHyTg-EAaMm0vQRiwCVFTdDETWu3vJuw&e=.

BeatLink commented 3 years ago

eagerly awaiting its release

dwrightpulse commented 3 years ago

eagerly awaiting its release

Yup, me to...!

siddjellali commented 3 years ago

@ncresswell May i ask for a release date ? :) Is it something you can share in public ?

ksurl commented 3 years ago

the roadmap had it at end of oct/early nov. no date. hopefully before end of the year

siddjellali commented 3 years ago

Appreciate thanks @ksurl

mygrexit commented 3 years ago

Any news? Is a release of 2.1 before the new year realistic?

ncresswell commented 3 years ago

We actually had a meeting on this yesterday…

There are a number of complications regarding adding the compose binary, so this is likely not going to make it this year… it will however be January.

We are still likely to embed a copy of the compose binary in Portainer container and the agent containers.. we are just trying to find a reliable producer of the binary as we need it come from a reliable/trusted source that is regularly maintained. Docker don’t provide it except for x86.

xZero707 commented 3 years ago

Appreciate the effort. Have nice holidays and take it easy. :rocket: :tada:

deviantony commented 3 years ago

Update on this topic:

As this one was of our main requirements to continue using the scratch image, we're now thinking about using the following solution:

We'd be happy to accept some help as well regarding the static compilation of the docker-compose binary on arm64 as we've not been able to crack that one.

Codelica commented 3 years ago

Just a couple random questions / comments...

How would the .tar.gz packaged releases (like portainer-2.x.x-linux-amd64.tar.gz) be handled? Asking for a friend. :)

I'm hoping they would just included linked binaries? Linux server does supply Linux and Alpine releases also, separate from their Docker images. (FWIW the ARM64 Linux release from them seems to work fine here under Ubuntu 20 on a Raspberry Pi 4)

If docker-compose isn't bundled in the .tar.gz releases, then just to be clear, this...

Always fallback to libcompose if the docker-compose binary is not available in the image (default scratch image for the linux/arm, linux/arm64, linux/s390x and linux/ppcl64e platforms).

...means that Portainer would still honor using docker-compose if it found it was available, or in some specific location, etc? Basically we could make it available ourselves.

Thanks!... and Happy Holidays..

lemmy04 commented 3 years ago

...couldn't you just base the docker images on a linux flavor that provides the right docke-compose version, and install the package?

deviantony commented 3 years ago

@Codelica the binary archive releases for 2.0.0 were a mistake on our side as we do not support them anymore starting from version 2.0.0 but we forgot to remove them from our CI pipeline.

We're currently thinking about removing them for the existing 2.0.0 release but if you think we should keep them, I'm keen to discuss it. You just have to be aware that this deployment is not tested by us anymore.

Regarding the Linux server releases, these are dynamically linked binaries, which means they require some libraries to be available in your runtime environment (and that is not the case for the scratch image which is why one of our option is to use a alpine/ubuntu based image).

@lemmy04 what do you mean? Also the scratch base image is important to us as it allows Portainer to be very portable and to run in any kind of environment without exposing too much attack surface.

Codelica commented 3 years ago

@Codelica the binary archive releases for 2.0.0 were a mistake on our side as we do not support them anymore starting from version 2.0.0 but we forgot to remove them from our CI pipeline.

We're currently thinking about removing them for the existing 2.0.0 release but if you think we should keep them, I'm keen to discuss it. You just have to be aware that this deployment is not tested by us anymore.

:( I'd certainly vote for keeping them around un-tested/un-supported if possible. I realize 99+% of people probably use your images directly, but we bundle the binary into a custom (Ubuntu based) image along with a variety of other tooling for support. (Yes, not very Docker-like, but super helpful ;) ). I guess I could migrate to try and pull things out of your image(s), just depending. It's just quite clean currently, and we test anyway. I won't hijack the thread here though.

Regarding the Linux server releases, these are dynamically linked binaries, which means they require some libraries to be available in your runtime environment (and that is not the case for the scratch image which is why one of our option is to use a alpine/ubuntu based image).

Yep, I understand. I was speaking more about inclusion of compose in your binary releases which may be going away. :/ But for ARM64, if static isn't an option for compose, I'd think Alpine would be fine for Portainer itself. It's not scratch, but at under 3MB it's not toooo far off. :)

mygrexit commented 3 years ago

@ncresswell, are you guys still on track for a January release? I don't want to build up any pressure with this, but am simply following the topic with interest.

ncresswell commented 3 years ago

We have a dev meeting today actually, so i will report back in 12 hours.

Rgds,

Neil Cresswell

ncresswell commented 3 years ago

Hi everyone.... we need your help... is there anyone out there that can create a "statically compiled binary" of compose that runs in a scratch container on ARM/ARM64/AMD64?

We are having an increasingly difficult time building an image that works in scratch, and unless we can do this, we will have no choice but to change our base from scratch to alpine (which we really dont want to do).

Is there anyone out there that can help us with this?

tcurdt commented 3 years ago

@ncresswell can you elaborate on the problems a bit?

xZero707 commented 3 years ago

I will try and experiment a bit and let you know if I find a way to do it. Let's be optimistic; If several people try to do this, at least one may succeed.

tcurdt commented 3 years ago

I guess the question is whether the binary really is statically linked (hence me asking for more details) or whether there is some form of dependency to libc. In Rust you usually need to link against lib musl instead of libc to get a proper static binary that works. Just my 2 cents.

lemmy04 commented 3 years ago

We are having an increasingly difficult time building an image that works in scratch, and unless we can do this, we will have no choice but to change our base from scratch to alpine (which we really dont want to do).

Why alpine specifically, and what is the problem with it? There are other linux base images for ARM...

xZero707 commented 3 years ago

Alpine is extremely lightweight. When you run a microservice infrastructure with a lot of different images, you really start to appreciate every megabyte less as images can quickly add up and consume hundreds of gigabytes.

Edo78 commented 3 years ago

Alpine is extremely lightweight. When you run a microservice infrastructure with a lot of different images, you really start to appreciate every megabyte less as images can quickly add up and consume hundreds of gigabytes.

This is why they want to start from scratch without even the alpine layer (IMO)

deviantony commented 3 years ago

Multiple concerns here regarding the use of the alpine image:

Regarding the details about the statically compiled binary, it would be required to have the binary without any form of dependency I believe as there is no lib dependency available in the scratch image.

To give more details on the technical side: To run in a scratch image docker-compose binary needs to be statically compiled and not depend on any system library. docker-compose source is in python and then need to be first compiled and then statically linked to build a standalone binary. We were not able to achieve the last step for arm64 and getting segment fault error.

If you need more technical details we can also describe what we have achieved.

tcurdt commented 3 years ago

To run in a scratch image docker-compose binary needs to be statically compiled and not depend on any system library. docker-compose source is in python and then need to be first compiled and then statically linked to build a standalone binary. We were not able to achieve the last step for arm64 and getting segment fault error.

I'd recommend to try the exact same steps on a musl based system (e.g. on Alpine). If that doesn't help I'd suggest you post the exact steps you have taken.