Releases of Atomic Docker do not keep pace with Docker CE

[joekohut]

Docker stable release is around 17.09. I still see Atomic at 1.12/1.13. Just curious, what is the driver/constraints here?
Atomic is a great project, but the lag in Docker version is a bit unexpected and unsettling from an end-user perspective.

[rhatdan]

Our version of Docker we have kept locked to the version required by Kubernetes. We back port any serios fixes to it. In the past updating versions of Docker has broken Kubernetes. Are there new features that you are looking for?

[joekohut]

I think Atomic Host is the best platform on which run containers. I also think Atomic Host is the best platform on which to run Docker. We use Compose and want to work up to multi-stage builds. We use Swarm for now and anticipate the announced Docker/Kubernetes melding.

I no longer see Atomic Host as the best platform on which to run Docker if Atomic is only destined to support K8s and OC.

I respectfully ask for your feedback on these questions:

1) What is the best RedHat platform for hosting the latest stable version of Docker? 2) Is Atomic still a viable hosting platform for Docker stable releases? 3) Is it possible/recommended to use tree pkg_add {docker} to install a newer Docker? 4) "docker-latest" in RedHat's AH is supposed to be the latest Docker and not interfere with K8s. How/when/why did this change?

[rhatdan]

We are moving to running container runtimes inside of containers rather then running them native on the host. This would allow atomic host the flexibility of running different versions of docker or other container runtimes like CRI-O.

We have not updated docker-latest since the Moby/Docker split.

Confusion over what Docker wants/does not want distributions to ship.

I think packaging up latest stable docker into a System Container would be the best way to go for atomic host, but adding it later docker using rpm-ostree install to atomic host.

Running these on Standard RHEL is also viable.

We don't support Swarm.

Atomic is not destined to only support k8s and OC, we want Atomic to be able to support all container runtimes, but not be tied to one at a particular version.

[joekohut]

I think the container-agnostic view is a good direction. I think running Docker in a System Container makes sense, as Docker-in-Docker is already a pattern. However, I have not seen a recipe for that implementation to bake it myself(?) and this does not seem to be something RedHat is going to offer.

As far as running the latest stable Docker, it seems like RHEL, not Atomic, is the only method at-hand today.

"We have not updated docker-latest since the Moby/Docker split." - why would RH not keep docker-latest at the latest stable release? That would allow a more current Docker offering until such a time when Docker would be available in a System Container.

Lastly, is there any plan to raise docker-current to 1.13.1 in the next RH AH release?

[cgwalters]

In the latest rpm-ostree release v2017.11, the override command is no longer experimental, which means you can also install a different docker via RPM, while still gaining the benefits of a fully transactional hybrid image/package system.

[joekohut]

Thanks, cgwalters. When would this capability make it into RedHat Atomic Host?

And, Dan, if you would please comment on these previous questions:

1) "We have not updated docker-latest since the Moby/Docker split." - why would RH not keep docker-latest at the latest stable release? That would allow a more current Docker offering until such a time when Docker would be available in a System Container.

2) Lastly, is there any plan to raise docker-current to 1.13.1 in the next RH AH release?

[hedayat]

Just wanted to note that Docker 17.03.x is also officially supported by Kubernetes 1.9 & 1.10. (And I've tried running a sample kuberentes cluster using docker 17.12 successfully).

Docker in Fedora is not just for Atomic host, so one expects Fedora to provide the latest Docker-ce too. If there is no point in continuing the docker-latest branch, I'd suggest packaging latest docker-ce as docker-latest or maybe as docker-ce.

The current status of docker vs docker-latest in Fedora is .... fun! docker is actually newer than docker-latest in F28. It is better removed completely if is going to remain like that. :P

Well, actually, I'd suggest to provide latest docker by default (through docker or docker-ce package), and provide a kubernetes specific package, e.g. docker-kube. It'd be also much more clear why Fedora has 2 versions of docker in its repos.

Thanks anyway

[rlogiacco]

Sorry lads, all this doesn't sound clear to me... I have something (Jenkins plugin) requiring docker API 1.31, but the currently installed API version is 1.26: is there a way for me to satisfy such dependency?

[hedayat]

Use upstream docker repository and install the latest one. Fedora, unlike most other packages, provides a very old docker version for no good reason!

[rlogiacco]

Apologies for my noobness, but as there is no yum available, how should I pull and install it?

[hedayat]

Just use dnf instead of yum in the same way.

[FranklinYu]

Docker is still at 1.13 in F29. Both docker and docker-latest. In comparison, Docker on Ubuntu has version 17.12.

I don't use Atomic, so why should it affect ordinary Fedora user?

[bka2004]

There's this problem with bad performance when mounting a big named volume on container start, reported here: https://forums.docker.com/t/large-named-volume-slows-down-container-start/44689 and here: https://forums.docker.com/t/docker-run-weirdly-slow-with-volumes-containing-many-files/13456

This problem exists with version 1.13.1 and is hitting me atm. The first report says it's fixed with version 17.12.

Hence I would really love a new version. Unfortunately I found no patch or commit for that, so I don't know if you could backport that...

[rhatdan]

@bka2004 Any chance you could try this with podman to see if it has issues?

[bka2004]

I gave it a try today (didn't even have to change anything besides the named volume, just used "podman" instead of "docker") and it seemed to work fine. Container was up in an instant and all files have been accessible. So now I have to think about moving all my images...

[rhatdan]

Awesome.

[eseiker]

now docker 18.06 has been officially validated with kubernetes since 1.12, still no plan to update projectatomic docker? are there any other reason sticking with docker 1.13.1?

[rhatdan]

We are moving to CRI-O for mainline support under OpenShift. Fedora no longer can Ship newer versions of Docker, since Docker project changed their licensing. Fedora now ships Moby-engine for the upstream project.

We are also shipping Podman as a Docker CLI Replacement.

[FranklinYu]

@rhatdan What’s wrong with the license? Isn’t Docker CLI released in Apache License?

[rhatdan]

We are no longer allowed to ship "Docker" since that is owned by the company. Moby is the upstream project. Docker Inc made it clear they did not want any distro's shipping tools called Docker without a coorporate agreement with them. So Fedora is not going to ship it.

[FranklinYu]

We are no longer allowed to ship "Docker" since that is owned by the company.

If I understand correctly, Docker CLI is released in Apache License. See https://github.com/docker/docker-ce/blob/master/components/cli/LICENSE

[rhatdan]

The letters Docker versus the code is my understanding. IAMNAL but my understanding is legal felt it was better to not ship anything named Docker.

[FranklinYu]

Then how did other distributions make it? Ubuntu, Arch Linux, openSUSE?

[rhatdan]

Don't know and I am not sure they all are, or the corporate entity entered an agreement with Docker. Or they are willing to take a chance on not being sued.

[hedayat]

We are no longer allowed to ship "Docker" since that is owned by the company. Moby is the upstream project. Docker Inc made it clear they did not want any distro's shipping tools called Docker without a coorporate agreement with them. So Fedora is not going to ship it.

That's great. I didn't know that we have moby-engine in Fedora. And I don't care what it is called.

[eseiker]

Trying moby-engine and seems working good :) I didn't know that fedora is shipping upstream moby-engine package instead of docker. thanks

[FranklinYu]

Sorry for being a jerk. I didn’t know moby-engine also carries the docker binary (I thought it’s only the daemon). That’s more than enough.

[rhatdan]

@eseiker @FranklinYu We would love to have you play with podman and tell us what you think it is missing as compared to Docker cli.

[rlogiacco]

I'm sorry, but what of the above mentioned tools (moby-engine, podman) applies to Project Atomic and how can I upgrade my system to give those a test before taking more drastic actions?

[FranklinYu]

@rhatdan Is there shell completion (like Bash or Zsh) for Podman? The one for Moby engine works very well.

[rhatdan]

Yes podman has Bash completion and there is a PR for Zsh completion that just opened.

We would love to have you try it out and tell us how it works. https://github.com/containers/libpod/pull/2637

[rhatdan]

@rlogiacco Atomic Host is going away and being replaced with Fedora CoreOS, which has both podman and Moby-engine on it.

[FranklinYu]

Atomic Host is going away and being replaced with Fedora CoreOS

Is that the same thing as the CoreOS I know?

[dustymabe]

Atomic Host is going away and being replaced with Fedora CoreOS

Is that the same thing as the CoreOS I know?

Not exactly. CoreOS Inc. was acquired by Red Hat last year. CoreOS Container Linux and Red Hat's Project Atomic Atomic Host are getting combined into a new offering known as Fedora CoreOS (upstream) and Red Hat Enterprise Linux CoreOS (downstream). We're in heavy development mode right now but you can find some details at https://coreos.fedoraproject.org/ and follow along with our development at https://github.com/coreos/fedora-coreos-tracker

[keithy]

So how do you upgrade to moby-engine and still have cockpit working?

[FranklinYu]

@dustymabe How long will Atomic be supported? According to the FAQ page:

The current plan is for Fedora Atomic Host to have at least a 29 version and 6 months of lifecycle.

I assume it means “6 month after release of Fedora 29” which is the end of this April? Will Fedora CoreOS be ready before that?

[dustymabe]

@FranklinYu, we currently plan to make the first release of Fedora CoreOS a preview release and declare it stable sometime after that. We will continue to test/release F29 Atomic Host until Fedora 29 EOL (i.e. shortly after Fedora 31 is released)

More details in https://github.com/coreos/fedora-coreos-tracker/issues/145

[keithy]

Could someone please explain how to get a more recent version of docker into fedora 29. Just add moby-engine doesn't work. The old docker and cockpit get in the way. Some/many people are new to the atomic way, documentation leaves a lot to be desired. If I had known I can't build in docker compose I would have used another os but now I'm committed. I'm astounded that it's not clear that the os I specifically choose to use docker for, is not only unsupported, undocumented, no one replies in irc, and has a core deliverable so out of date.

Word of wisdom to your managers, if it isn't documented well it may as well not exist.

[dustymabe]

Could someone please explain how to get a more recent version of docker into fedora 29.

I assume you mean Fedora 29 Atomic Host?

Just add moby-engine doesn't work. The old docker and cockpit get in the way.

[root@vanilla-f29-atomic ~]# rpm-ostree override remove docker cockpit-docker
...
[root@vanilla-f29-atomic ~]# rpm-ostree install moby-engine
...
[root@vanilla-f29-atomic ~]# reboot

Should work, but i'm getting an error when doing that (reported it here).

Some/many people are new to the atomic way, documentation leaves a lot to be desired. If I had known I can't build in docker compose I would have used another os but now I'm committed. I'm astounded that it's not clear that the os I specifically choose to use docker for, is not only unsupported, undocumented, no one replies in irc, and has a core deliverable so out of date.

Yes. The Atomic Host project was maturing and closing gaps on documentation at the time that we pivoted to start working on Fedora CoreOS (there was an acquisition of CoreOS Inc. early last year). It is lacking some documentation, I agree, but it's also a community project (i.e. no one is paying for Fedora Atomic Host) so there is no guarantee of support. If you have paid anyone for Fedora Atomic Host (which they probably shouldn't be doing) then you should contact them to open feature requests and get support.

I'm in the IRC channel but I admit that there is a lot going on these days so I don't read the scrollback. Feel free to ping my name in there or to ask questions in #fedora-coreos where a lot of people are hanging out now.

As for docker compose, podman has podman pod functionality but it's definitely not a 1:1 with docker compose. See https://discussion.fedoraproject.org/t/podman-and-docker-compose/766/4 for some discussion. Hopefully we can get https://github.com/projectatomic/rpm-ostree/issues/1722 fixed soon, in the mean time you could just override remove docker and then download and extract the moby-engine package components to /usr/local/bin or something to try to get it moving along.

[keithy]

Thanks for your prompt reply. Can you reload cockpit, I showed it to the new boss, would rather not loose it just yet.? Does it reload afterwards? I'm using podman for image builds, hope it works.

[dustymabe]

Thanks for your prompt reply. Can you reload cockpit, I showed it to the new boss, would rather not loose it just yet.? Does it reload afterwards? I'm using podman for image builds, hope it works.

yeah we should be able to just not remove cockpit at all using a command like rpm-ostree override remove docker --install moby-engine, but that currently doesn't work because of https://github.com/projectatomic/rpm-ostree/issues/1722 - hopefully we'll get that fixed soon.

[dustymabe]

ok turns out we weren't grabbing everything. Here is the command you can use to get moby-engine installed!

[root@vanilla-f29-atomic ~]# rpm-ostree override remove docker docker-common --install moby-engine 
Checking out tree 42e8e52... done
Enabled rpm-md repositories: updates fedora
rpm-md repo 'updates' (cached); generated: 2019-03-19T04:43:55Z
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
Importing rpm-md... done
Resolving dependencies... done
Applying 2 overrides and 1 overlay
Processing packages... done
Running pre scripts... done
Running post scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Removed:
  docker-2:1.13.1-65.git1185cfd.fc29.x86_64
  docker-common-2:1.13.1-65.git1185cfd.fc29.x86_64
Added:
  moby-engine-18.06.0-2.ce.git0ffa825.fc29.x86_64
Run "systemctl reboot" to start a reboot
[root@vanilla-f29-atomic ~]# 
[root@vanilla-f29-atomic ~]# reboot

[FranklinYu]

@rhatdan When I run podman run my-image is it running the image from Docker Hub? IIUC the image format is different (“Docker Image Formats” v.s. “OCI Image Formats”).

[rhatdan]

Podman supports both formats.

[FranklinYu]

@rhatdan Does Podman plan to support Docker Swarm?

[rhatdan]

Nope. We are firmly in the Kubernetes camp.

[keithy]

btw @dustymabe wanted to thank you for your invaluable help, much needed.

[jcberthon]

Hi @rhatdan

I am a fairly advanced and experienced Docker user and I am quite interested in using Podman.

Apart from a few glitches (like on Ubuntu, the "official" podman PPA provides a slirp4netns package slightly too old as far as I understood and "root-less" port mapping seems broken), I have already the following feedback:

The good

1. Very easy to switch to Podman from Docker
2. User namespaces out of the box 👍
3. No daemon running
4. Can be run by a user without privileges

The to-be-improved

1. Documentation is very minimal, apart from the manual page of each command there isn't much.
2. Corollary of the previous point is that information on doing networking with Podman is very scarse. There seems to be as much functionalities (except overlay network) + slirp4netns, but no documentation I could find on how to configure it, create it, manage it, investigate, etc.
3. I use seldom the docker cli, only for testing or a quick "run". I'm a big fan of compose and I keep my running configuration in Git repositories, and that includes building and running. I haven't find any ways of doing this with podman. Searching for podman and compose was not conclusive either. Is there something equivalent?
4. Focusing only on K8s is a bit sad. I mean for a hobbyist who would like to put 3 Raspberry Pis in a small cluster, Fedora is a perfect fit with Docker Swarm. For the enterprise world, on many projects we need to create small clusters (3-6 machines) for which K8s is way over-complicated (same apply for OpenShift). Swarm is extremely cool for such projects. And I have big hopes for K3s as well. We do not all need to build data centers 😉. Maybe OpenShift is really for data centers. But RHEL, or even Fedora, could be considered for lightweight clusters. I understand that Swarm might not get Podman supports. But what about K3s (maybe it works, but again some documentation would be very welcome)?
5. Is there a way to only allow certain users to create containers? Perhaps that's a Kernel question and not a Podman question. And it is how to disallow the access to user namespaces to some users.
6. As I understand the root-less implementation, containers run by user1 are not modifiable by user2. But how can we share containers within an organisation? Obviously I could create a machine userZ, and everyone would need to do sudo -u userZ podman .... Is there another more "podman" way to do that?

[rhatdan]

@jcberthon Thanks for the feedback.

The to-be-improved

Documentation is very minimal, apart from the manual page of each command there isn't much. We are working to fix this and appreciate any help. What would you like to see in additonal documentation other then the Gihub pages, Man Pages, and Blogs at podman.io?

Corollary of the previous point is that information on doing networking with Podman is very scarse. There seems to be as much functionalities (except overlay network) + slirp4netns, but no documentation I could find on how to configure it, create it, manage it, investigate, etc.

Ok, were did you look for this? @giuseppe Should be able to help document it. I would point out that most of the work in Podman was originally in root podman, but our users tend to want to play with rootless, so we are working on fixing as many issues as possible. Hopefully the podman 1.2 release which should be out shortly will help here as well.

I use seldom the docker cli, only for testing or a quick "run". I'm a big fan of compose and I keep my running configuration in Git repositories, and that includes building and running. I haven't find any ways of doing this with podman. Searching for podman and compose was not conclusive either. Is there something equivalent? Well docker-compose is a separate project from Docker. There is a project podman-compose. https://github.com/muayyad-alsadi/podman-compose But no one on my team is currently working on this. Focusing only on K8s is a bit sad. I mean for a hobbyist who would like to put 3 Raspberry Pis in a small cluster, Fedora is a perfect fit with Docker Swarm. For the enterprise world, on many projects we need to create small clusters (3-6 machines) for which K8s is way over-complicated (same apply for OpenShift). Swarm is extremely cool for such projects. And I have big hopes for K3s as well. We do not all need to build data centers wink. Maybe OpenShift is really for data centers. But RHEL, or even Fedora, could be considered for lightweight clusters. I understand that Swarm might not get Podman supports. But what about K3s (maybe it works, but again some documentation would be very welcome)? Podman/libpod are fully open source projects. If people want to add support for podman play swarm, or podman generate swarm or podman play compose or podman generate compose, we would look into adding it as long as people would want to support it.

We added the Kube Support to make it easier for someone to move from podman to OpenShift.

Is there a way to only allow certain users to create containers? Perhaps that's a Kernel question and not a Podman question. And it is how to disallow the access to user namespaces to some users.

Just don't put their names in /etc/subuid and /etc/subgid. This will prevent them from building containers with multiple UIDs. There is no way in the kernel right now to prevent a user from using USER Namespace that I know of.

As I understand the root-less implementation, containers run by user1 are not modifiable by user2. But how can we share containers within an organisation? Obviously I could create a machine userZ, and everyone would need to do sudo -u userZ podman .... Is there another more "podman" way to do that?

The traditional way would be for the user1 to commit his container to an image and then allow him to push to a registry and then user2 could pull.

There is currently no way to share containers/storage between the two users, unless they are running in the same UID.

If you wanted to get clever you could cause their /etc/subuild and /etc/subgid files to have overlapping UID Ranges. then map the containers/storage different then the default podman does. If you look at the toolbox project they map the first UID in the range of UIDs as root and then just map the UID->UID

SO I would have a range of UIDS like

0:100000:MYUID MYUID:MYUID:1 MYUID+1:MYUID+1:65536-MYUID If You did a similar mapping for the second user, then they could share their storage except for files owned by their REALUIDs.

[jcberthon]

Wow, thank you for the very quick reply, and sorry to have disturb your Sunday I did not expect to get an answer today.

Documentation is very minimal, apart from the manual page of each command there isn't much.

We are working to fix this and appreciate any help. What would you like to see in additonal documentation other then the Gihub pages, Man Pages, and Blogs at podman.io?

Some ideas: some more blogs/tutorials would be great with other things than just the basics. So for example, publishing port on a bridge network, or sharing a common private network between 2 containers, using pod with 2-3 containers, etc. There is an example with httpd, it could be expanded. I recall when I started using Docker some 2-3y ago, there was a great tutorial from Jérôme Petazzoni, it was made in many steps from building a simple image, to deploying an example service with front- and backend, something like that for Podman could help new comers to the technology. Examples/tutorials on using podman with existing self-hosted CI would be great too. There is one about Cirrus CI, but before that blog post I did not even know that CI 😉

Corollary of the previous point is that information on doing networking with Podman is very scarse. There seems to be as much functionalities (except overlay network) + slirp4netns, but no documentation I could find on how to configure it, create it, manage it, investigate, etc.

Ok, were did you look for this? @giuseppe Should be able to help document it. I would point out that most of the work in Podman was originally in root podman, but our users tend to want to play with rootless, so we are working on fixing as many issues as possible. Hopefully the podman 1.2 release which should be out shortly will help here as well.

I've looked mainly on podman.io which has always redirected me to the GitHub documentation directory of the project. I limited my search to that project. I also searched online a bit but only found a few blogs with really easy steps, nothing a bit more advanced or complex.

I could also try podman as root, but there I'm also missing the steps for using bridge or other network mode. I know how to create a bridge manually, but is there some recommended settings for podman, etc. this is a bit missing. I mean it is possible than on Fedora when installing it the experience is a better simpler, but on Ubuntu after installation there are no special networking made.

For instance, I like the convenience of the docker network create ... or the default bridge with its embedded DNS resolver. I might be a bit lazy here, but to quickly start using the technology this is great to have.

I use seldom the docker cli, only for testing or a quick "run". I'm a big fan of compose and I keep my running configuration in Git repositories, and that includes building and running. I haven't find any ways of doing this with podman. Searching for podman and compose was not conclusive either. Is there something equivalent?

Well docker-compose is a separate project from Docker. There is a project podman-compose.
https://github.com/muayyad-alsadi/podman-compose
But no one on my team is currently working on this.

OK, thanks for the feedback. It would not require to be compose, but I find it cleaner to keep a description of a service like compose can provide rather than a shell script which execute the low level command. But that a personal opinion of course.

Focusing only on K8s is a bit sad. I mean for a hobbyist who would like to put 3 Raspberry Pis in a small cluster, Fedora is a perfect fit with Docker Swarm. For the enterprise world, on many projects we need to create small clusters (3-6 machines) for which K8s is way over-complicated (same apply for OpenShift). Swarm is extremely cool for such projects. And I have big hopes for K3s as well. We do not all need to build data centers wink. Maybe OpenShift is really for data centers. But RHEL, or even Fedora, could be considered for lightweight clusters. I understand that Swarm might not get Podman supports. But what about K3s (maybe it works, but again some documentation would be very welcome)?

Podman/libpod are fully open source projects. If people want to add support for podman play swarm, or podman generate swarm or podman play compose or podman generate compose, we would look into adding it as long as people would want to support it.

We added the Kube Support to make it easier for someone to move from podman to OpenShift.

Understood.

Is there a way to only allow certain users to create containers? Perhaps that's a Kernel question and not a Podman question. And it is how to disallow the access to user namespaces to some users.

Just don't put their names in /etc/subuid and /etc/subgid. This will prevent them from building containers with multiple UIDs. There is no way in the kernel right now to prevent a user from using USER Namespace that I know of.

That's what I thought, thanks for confirming it.

As I understand the root-less implementation, containers run by user1 are not modifiable by user2. But how can we share containers within an organisation? Obviously I could create a machine userZ, and everyone would need to do sudo -u userZ podman .... Is there another more "podman" way to do that?

The traditional way would be for the user1 to commit his container to an image and then allow him to push to a registry and then user2 could pull.

There is currently no way to share containers/storage between the two users, unless they are running in the same UID.

If you wanted to get clever you could cause their /etc/subuid and /etc/subgid files to have overlapping UID Ranges. then map the containers/storage different then the default podman does. If you look at the toolbox project they map the first UID in the range of UIDs as root and then just map the UID->UID

SO I would have a range of UIDS like

0:100000:MYUID MYUID:MYUID:1 MYUID+1:MYUID+1:65536-MYUID If You did a similar mapping for the second user, then they could share their storage except for files owned by their REALUIDs.

Then perhaps using a "common" user (my userZ in the above example) is perhaps the easier. 😄 But thank you for this clever hack!