illogical docker installation method

When testing 2022-04-04-raspios-bullseye-arm64-lite.img (using qemu):

install.sh (and scripts/install_docker.sh) installs docker and docker-compose with:

function install_docker() {
  if command_exists docker; then
    echo "Docker already installed" >&2
  else
    echo "Install Docker" >&2
    curl -fsSL https://get.docker.com | sh
    sudo usermod -aG docker $USER
  fi

  if command_exists docker-compose; then
    echo "docker-compose already installed" >&2
  else
    echo "Install docker-compose" >&2
    sudo apt install -y docker-compose
  fi

Script from get.docker.com adds a repository and installs docker-ce:

iotstack@raspberrypi:~/IOTstack $ curl -fsSL https://get.docker.com | sh
# Executing docker install script, commit: b2e29ef7a9a89840d2333637f7d1900a83e7153f
+ sudo -E sh -c apt-get update -qq >/dev/null
+ sudo -E sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sudo -E sh -c mkdir -p /etc/apt/keyrings && chmod -R 0755 /etc/apt/keyrings
+ sudo -E sh -c curl -fsSL "https://download.docker.com/linux/debian/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
+ sudo -E sh -c chmod a+r /etc/apt/keyrings/docker.gpg
+ sudo -E sh -c echo "deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian bullseye stable" > /etc/apt/sources.list.d/docker.list
+ sudo -E sh -c apt-get update -qq >/dev/null
+ sudo -E sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends docker-ce docker-ce-cli containerd.io docker-compose-plugin >/dev/null
+ version_gte 20.10
+ [ -z  ]
+ return 0
+ sudo -E sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce-rootless-extras >/dev/null
...SNIP....

This installs:

iotstack@raspberrypi:~/IOTstack $ docker -v
Docker version 20.10.17, build 100c701

But then when installing docker-compose the previously installed docker-ce is removed and Debian's version replaces it:

iotstack@raspberrypi:~/IOTstack $ sudo apt install -y docker-compose
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libslirp0 slirp4netns
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  apparmor cgroupfs-mount containerd docker.io git git-man liberror-perl libintl-perl
  libintl-xs-perl libmodule-find-perl libmodule-scandeps-perl libproc-processtable-perl
  libsort-naturally-perl libterm-readkey-perl needrestart python3-attr
  python3-cached-property python3-distutils python3-docker python3-dockerpty
  python3-docopt python3-importlib-metadata python3-jsonschema python3-lib2to3
  python3-more-itertools python3-pyrsistent python3-setuptools python3-texttable
  python3-websocket python3-yaml python3-zipp runc tini
Suggested packages:
  apparmor-profiles-extra apparmor-utils containernetworking-plugins docker-doc
  aufs-tools btrfs-progs debootstrap rinse rootlesskit xfsprogs zfs-fuse | zfsutils-linux
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb
  git-cvs git-mediawiki git-svn needrestart-session | libnotify-bin iucode-tool
  python-attr-doc python-jsonschema-doc python-setuptools-doc
Recommended packages:
  criu
The following packages will be REMOVED:
  containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras
The following NEW packages will be installed:
  apparmor cgroupfs-mount containerd docker-compose docker.io git git-man liberror-perl
  libintl-perl libintl-xs-perl libmodule-find-perl libmodule-scandeps-perl
  libproc-processtable-perl libsort-naturally-perl libterm-readkey-perl needrestart
  python3-attr python3-cached-property python3-distutils python3-docker python3-dockerpty
  python3-docopt python3-importlib-metadata python3-jsonschema python3-lib2to3
  python3-more-itertools python3-pyrsistent python3-setuptools python3-texttable
  python3-websocket python3-yaml python3-zipp runc tini
0 upgraded, 34 newly installed, 4 to remove and 36 not upgraded.
Need to get 53.6 MB/55.9 MB of archives.
After this operation, 85.2 MB disk space will be freed.

This results in:

iotstack@raspberrypi:~/IOTstack $ docker -v
Docker version 20.10.5+dfsg1, build 55c4c88

It's useless to install the updated docker-ce just to remove it immediately!?

Oh, I so agree!!!

All the IOTstack install methods have always been a bit of a mess. I gave up trying to persuade an approach of a single script that was not either embedded in the menu or ever called by it, and which simply did the job of installing (not the double-duty of updating).

And all that daft version checking - gives me the heebies.

This is, pretty much, why I went my own way with PiBuilder, particularly the 04_setup.sh script.

Recently, the goal-posts moved again. Now, the docker convenience script also installs docker-compose-plugin which is "modern" docker-compose. A command like docker compose up -d will work right out of the box. Getting the traditional hyphenated form to work just needs a symlink to the plugin binary.

The good thing, now, is that routine apt update/upgrade will upgrade both docker and compose so you no longer need anything more than routine system maintenance.

As it should always have been!

Just to call it out - the last element on this line of your output:

+ sudo -E sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends docker-ce docker-ce-cli containerd.io docker-compose-plugin >/dev/null

the last element on this line of your output:

Yup, this brings Compose v2, while the docker-compose package is v1.

I'm thinking, for IOTstack, the 'correct' (=simplest that just works) thing to do, is to just use Debian's bundled docker and docker-compose packages, and not mess around with get.docker.com. At least if the 20.10.5+dfsg1 version won't cause any known bugs?

Both packages do have support for the no-hypen docker compose style. I guess it might be time to start migrating to this style?

I think I'd disagree that "simplest" equals installing docker and docker-compose-plugin from the convenience script, followed by installing docker-compose from apt which promptly uninstalls docker and installs a different version. I'd see "simpler" as just running the convenience script - full stop.

Aside from futzing with docker, the problem with the apt version of docker-compose is that it does actually have a number of known issues. They're mostly arcane issues, to be sure, and to the best of my knowledge only affect corner cases such as the device cgroup rules I mention from time to time.

For a good 18 months the interim solution (whenever one of those bugs/limitations showed up) was to explain how to remove the obsolete version of docker-compose in favour of the python version (1.29.2). To begin with that (a) didn't futz with docker and (b) resolved the corner cases - at least, all the ones I knew about. More recently, zapping the apt-installed docker-compose left you with a broken docker so the only solution was to nuke everything, run the convenience script to get docker back, then run pip to get docker-compose.

The modern version (written in go) started as a port of 1.29.2. It's evolving quite rapidly solving plenty of bugs and issues as it goes along (and, no doubt, introducing new ones - that being the nature of programming).

For a time, PiBuilder always pulled down the latest and greatest docker-compose from the releases page. While that never caused any problems for me, and while nobody ever opened an issue on the PiBuilder repo complaining that a "bleeding edge" version had broken something, I was conscious that it was somewhat heroic to assume the latest and greatest would always be 100% reliable. It also depended on me updating the scripts to hard-code the version number because there was (and is) no concept like "latest". I really didn't want to be in that kind of loop because me bumping a version number kinda implied I'd done some basic testing. I did but I it would hardly qualify as "comprehensive".

Right now, the "release" version is 2.6.1 but what you get from installing from the convenience script as docker-compose-plugin is still 2.6.0. My understanding is that new releases only get into the apt repositories when they have gone through some additional testing. I see that as all to the common good, which is why I've removed the "upgrade docker-compose" script from PiBuilder, in favour of relying on the due diligence of whoever decides a version is appropriate for apt release. Takes me out of the loop.

There are quite a few advantages to the modern version. It uses BuildKit and is a heckofalot faster if you're developing your own containers. It also does away with the doubling-up you get with base images plus local images if Dockerfiles are involved, and that reduces confusion among people who see "unused" in Portainer and worry about it. Pruning seems to be more efficient too, particularly if you're iterating doing test builds of containers.

The main reason I'd like to see IOTstack just use the convenience script - full stop - is because of the mess you get if you do anything else. It's people who have a perfectly functioning system, hit a limitation caused by using an obsolete version, get told how to fix that (nuke and reinstall), get the heebies and then come back and ask the perfectly valid question about why the heck IOTstack didn't do it "right" in the first place? Other than shrugging my shoulders, I have no answer to that.

I'm not here to persuade you (or anyone) of anything. Whatever floats your boat. If you think it's best to continue with the current scheme, go for it. My view is that it creates a platform that is frozen in time. Whether that's good or bad depends on each person's needs and viewpoint. PiBuilder is always there for anyone who wants to opt-in to regular updates from the folks at Docker.

I'd see "simpler" as just running the convenience script - full stop.

Needs the script plus symlink to docker-compose.

I was suggesting option 3, but now that you mentioned pip, did some some testing and I'll go with option 2. When I have the installation scripts cleaned up, let's see about upgrading to option 1 with compose v2. One step at a time.

Options being:

curl -fsSL https://get.docker.com | sudo sh and manually adding the docker-compose symlink (as PiBuilder does)

$ docker -v
Docker version 20.10.17, build 100c701
$ /usr/libexec/docker/cli-plugins/docker-compose -v
Docker Compose version v2.6.0

curl -fsSL https://get.docker.com | sudo sh and sudo pip install docker-compose.

$ docker -v
Docker version 20.10.17, build 100c701
$ docker-compose -v
docker-compose version 1.29.2, build unknown

sudo apt-get install docker-compose i.e. the bundled docker and docker-compose.

$ docker -v
Docker version 20.10.5+dfsg1, build 55c4c88
$ docker-compose -v
docker-compose version 1.25.0, build unknown

They're mostly arcane issues, to be sure, and to the best of my knowledge only affect corner cases such as the device cgroup rules I mention from time to time.

Can't find an issue describing this. Wasn't this a pretty complex issue where a) cgroup rules require a certain version header in the compose file and b) the required compose file version requires a certain docker-engine and docker-compose versions? This is just a faint memory and might be completely off-the-mark.

From https://docs.docker.com/compose/compose-file/compose-file-v3/

The latest Compose file format is defined by the Compose Specification and is implemented by Docker Compose 1.27.0+.

Hence only options 1 and 2 support the latest compose file version.

The downside of option 2 being both compose versions (v1 and v2) are installed:

$ docker compose version
Docker Compose version v2.6.0
$ docker-compose -v
docker-compose version 1.29.2, build unknown

Hopefully this won't be confusing, as only the latter form is used in the docs.

I don't remember all the details either. It was just that each time "x doesn't seem to work - it's in the compose spec - what gives?" would pop up in an issue or on Discord, the solution would be 1.29.2.

I think the version header is also deprecated in the spec but, at least for a time (possibly even now) some features seemed to need "a version greater than yeah" to activate.

Incidentally, I'm not sure how far back you can go before x- prefixing doesn't work. I use that myself, mostly when I'm fiddling about with host mode. Rather than comment-out the ports lines, I'll put x- on the ports or the network mode to take them in/out. Sometimes I think of suggesting that as a quasi standard for IOTstack. If every host mode container also had its ports described in an x-ports clause, it would (a) help the user to know which ports a container exposes, and (b) be easier to write tools to list ports in use, check for conflicts etc.

We are destined to disagree on many things if you equate creating a symlink with what happens with an apt-installed compose and the problems it stores up for the future as being on the same level of complexity. Just sayin' ...

The main problem I foresee with assuming/accepting 1.29.2 plus 2.x on the same system on the basis that all our doco uses the hyphenated form is that I suspect the plug-in form is going to become more common elsewhere on the Internet. People following recipes are then going to get different behaviours and, eventually, will run version on each and ask WTF? Accepting what the convenience script does as "truth" and gaining backwards compatibility with a symlink seems to me to be the simplest and most compatible way forward.

I'm not really sure what's bugging you about 2.x. I've been using it on all my systems (live and test) since it became a thing. There was one brief regression when the much-mentioned device cgroup rules stopped working (I actually use that for OctoPrint so I smacked into it when it happened) plus it exposed a subtle issue with volumes mappings at one point. Both of those were fixed "toot sweet" (as Terry Pratchett would say) plus I put in a PR on IOTstack to regularise all our volumes mappings as a fail-safe (just removing any trailing slashes). But, aside from those two, 2.x has Just Worked. I have zero doubts about recommending it as the tool to use. Zero.

Okay, I'm convinced. Any change to docker-compose is anyway a source for possible bugs and it's safest to use what is known and working in PiBuilder.

Another annoyance/problem is supporting two installation methods. Lots of semi-duplicated code and double the bug-surface and testing needs. I think choosing just one and making it as smooth and reliable is the way to go.

Options in my mind:

install.sh
manual git clone and the various menu.sh checks and installations
same as option 2, but separate prompts and installations to a new scripts/setup.sh file.

Here options 1 and 2 are the current existing install methods. Option 3 is what I think would be good for a bit of clarity.

There is a direct Docker install script: https://github.com/docker/docker-install

I'll go back a few steps to set the scene.

The link you have provided is something I wasn't aware of so thanks for that.

What we've been using to install docker is the so-called "convenience script" which is documented here, and which boils down to:

$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh

You might note that that has a sudo in it while the example at your link doesn't, but that doesn't really matter because the script figures out for itself whether it needs sudo or not. In short: same thing.

Historically, the convenience script only installed docker and related components like docker-ce and containerd.io. It didn't install docker-compose and neither did it do anything else needed to make docker fully usable on the Raspberry Pi, such as add the current user to the docker group. In the past, you had to do that yourself.

More recently, the convenience script has started including docker-compose-plugin which tracks github.com/docker/compose (albeit with a slight lag) but that has created a minor problem of its own in that, if your Pi is "green fields" then:

$ docker compose «command»

will work while the historical form:

$ docker-compose «command»

will fail. If your Pi isn't green fields and you have an older version of docker-compose then you'll have two versions installed. So you really need to guard against that possibility and create a symlink so both command styles work and point to the same thing.

If you open an issue against docker-compose you'll be asked to include output from both docker-compose version and docker compose version - they're acutely aware of this problem too.

Sorting out docker-compose versions isn't straightforward because, historically, there have been binaries (pre 1.29), then a Python version (1.29), then binaries again; and the pre-1.29 binaries can get so tangled up with docker that your only solution is to nuke both docker and docker-compose and start over.

And that still doesn't get you all the scaffolding you really need for usability.

Part of IOTstack's "problem" is the installation, version-checking and dependency handling has grown like topsy. It's scattered all over the place, is inconsistent, and isn't really fit for purpose.

Just one example is in install.sh:

If docker isn't installed, it calls the convenience script (good). Right now, that installs docker and docker-compose-plugin but nothing creates a symlink for the docker-compose command.
Then it checks for docker-compose. The absence of a symlink means this fails so the script invokes apt to install docker-compose. This gets pre-1.29 docker-compose and that actually breaks docker in some weird way which means it returns garbage version numbers and that, in turn, stores up trouble for when the menu runs and you get stuck in a loop.
Down at line 140, it's still talking about docker-compose rather than docker-compose-plugin and it's also talking about docker on its lonesome rather than thinking about what might happen if docker gets out of sync with docker-ce, containerd.io and anything else it depends on.

My own view of all this is that IOTstack should state its dependencies but should not install, update or check anything. The menu should just do its job of creating docker-compose.yml files - period.

docker, docker-compose-plugin and their immediate dependencies will be fine if either just left alone, or updated together via a general apt update/apt upgrade. The timing of that is best left to each user rather than forced on menu runs.

My own approach to solving this problem is PiBuilder. Given media (SD/SSD) with a fresh Raspberry Pi OS image, running the PiBuilder scripts in order gets you a rock solid platform, with docker, docker compose plugin, and IOTstack installed, all dependencies satisfied, all scaffolding in place, all ready for the menu to run. My original goal was to do everything to stop the menu from trying to do anything else to the Pi, and let it concentrate on its job of building compose files. That goal took a bit of a smack around the earhole when the menu was changed to do all the Python work in a venv but I've adapted around that.

At least, for Buster and Bullseye - I've had reports that the Python stuff goes berserk on Bookend, and the workaround is to revert to old-menu (git switch old-menu).

Getting agreement on the way forward is the real problem. Despite the age of this issue (June 2022), discussions are ongoing:

PiBuilder also has a bunch of problem-solving scripts like:

Maintaining docker + docker-compose; and
Menu maintenance for when the Python venv goes berserk.

I always say "your Pi, your rules" so it's always up to you how you install anything.

Anyway, it isn't just the convenience script. That's merely the lid on a massive can of worms. 😎

Slyke (this site's maintainer) has been pushing me to look at Ansible - essentially to replicate what PiBuilder does in Ansible playbooks. I'm looking into that now.

As an aside, Andreas recommending cloning PiBuilder into $HOME certainly works but I prefer the approach of copying (rather than cloning) onto the boot volume. That way, your boot volume always contains a record of how your Pi was built (auditable). Also, because of the way PiBuilder supports customisation, it's probably better to clone PiBuilder to your support host (PC/Mac/whatever) and do all the customisation there - with appropriate commits to your local branch - before copying to the boot volume. That way you get your own version history and, if you have multiple Pis, you can have custom configurations for each all maintained within git.

That way you get your own version history and, if you have multiple Pis, you can have custom configurations for each all maintained within git.

Too muck work for a lazy people. I think we can write github actions workflow and run these tests on virtual machines. But from my point of view I see great need for CLI for system testing. I think I saw a couple of PRs on this topic here.

SensorsIot / IOTstack

illogical docker installation method #585