Open paddy-hack opened 6 years ago
--- wrong thread
@agilob looks like you're running on arm
, which is a different issue; see https://github.com/moby/moby/issues/38175
Ah thanks, Google pointed me here and timing and sympotms were similar.
I found the reason for my problem on CentOS,
I was setting the host option in daemon.json
"hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2375"],
In docker 18.06 the systemd service docker started the daemon via the line
ExecStart=/usr/bin/dockerd
With 18.09 this has been changed into
ExecStart=/usr/bin/dockerd -H unix://
Now the host is defined in the command line and in the configuration file, which is not allowed and make the daemon stop.
In my case removing the -H option from the command line made the configuration work again.
"tcp://0.0.0.0:2375"
@grafra make sure that your machine is not publicly accessible; exposing the Docker API unprotected gives root
access to the machine where the daemon runs for anyone that can access the API
@thaJeztah : Thanks for your hint. This configuration is from an internal development environment. There i think it is OK to allow the developers unprotected access.
I have checked the new version's postinst and prerm scripts and noticed that the sysvinit support has been removed 😒 Putting back the support that was there until 18.06.1~ce~3-0~debian should fix the problem as far as I understand. While I understand the package targets Debian, please be advised that there are derivatives/forks that do not use systemd, Devuan being one of them. It would be nice if your Debian packages could be used as is on (at least some of) these as well. I do not understand why the sysvinit support had to be removed.
On a related note, the containerd.io package does not support sysvinit based systems either. Perhaps it should.
@paddy-hack I think the containerd.io
package may actually be the reason here (but also checking internally with the packaging team).
Starting with Docker 18.09, containerd
is now shipping as a separate package, and will be running as a separate service (previously, dockerd
was running as a service, and ran containerd
as child process).
containerd
does not currently have a sysvinit script (upstream only has a containerd.service
unit file). Given that sysvinit is still an option on Debian, it would be good if containerd also has support for sysvinit; contributions for that would likely be accepted upstream by the containerd project (if anyone is willing to help with this, that would be appreciated!), which would be a first step.
Following that, packaging changes would have to be made to the docker-ce
/ docker-ee
packages, to bring back a sysvinit script.
@thaJeztah i dug a little more and found that debian stretch does support sysvinit with the sysvinit-core package but it is not full support and it is not required:
Debian packages are not required to provide sysvinit start scripts.
(@thaJeztah asked me to take a look at this, so here's the results of my own research/experience :innocent:)
Looking at systemctl cat containerd
on one of my updated systems, we could probably utilize https://manpages.debian.org/init-d-script and get away with something as simple as this:
#!/lib/init/init-d-script
### BEGIN INIT INFO
# Provides: containerd
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: containerd container runtime
# Description: Debian init script to start containerd.
### END INIT INFO
DAEMON=/usr/bin/containerd
do_start_prepare() {
/sbin/modprobe overlay
}
Shipping this by default is a bit iffy though, especially if it won't get heavily tested (I certainly haven't tested this -- this is just from reading the docs and trying to convert the containerd.service
definition over as best as I can while still being overly minimal).
Thanks @tianon! Yes, we should definitely be careful with including in the packages, as including it may set the wrong expectation.
Perhaps an (example) init script in the upstream contrib directory (https://github.com/containerd/containerd/tree/master/contrib) that users can use. There may be options in the systemd containerd.service
unit-file that may not translate (easily) to a sysvinit script. Missing those options may result in problems that are not immediately apparent.
Thanks for taking a look!
FYI, for me, running sudo invoke-rc.d docker restart
started both dockerd
and whatever containerd
processes it thought necessary. I guess I should mention that my servers were running containers during the upgrade.
@andrewhsu Quoting from the wiki you linked to
In
jessie
andstretch
, the default init system issystemd
, but switching tosysvinit
is supported.Since
jessie
, onlysystemd
is fully supported;sysvinit
is mostly supported, but Debian packages are not required to providesysvinit
start scripts.runit
is also packaged, but has not received the same level of testing and support as the others, and is not currently supported as PID 1.
So, yes, technically you are right, but why actively remove working support for sysvinit
from docker-ce
? It still ships with /etc/init.d/docker
and I very much prefer to keep it that way.
@thaJeztah There has been some recent activity between Debian and Devuan developers as well as others that are interested in Debian init diversity. One of the more immediate concerns is keeping sysvinit
"alive-and-kicking" in Debian. A somewhat longer term effort is maybe making the init
system an install-time option. If that materializes, your docker-ce
and containerd.io
packages would need some work. Either supporting extra init
systems or, and I most sincerely hope not!, a dependency on the supported init
systems.
@tianon If containerd.io
needs to get its daemon (re)started at installation/upgrade, I'll be happy to test a sysvinit
script, but don't really have time until the end of this month :cry:
:thumbsup:
The docker-ce
package needs to get a copy of the sysvinit
script that shipped with 18.06.1~ce~3-0~debian
. It also needs to resurrect the {pre,post}{inst,rm}
scripts snippets that invoke-rc.d
said script.
So far, I have not found any evidence that containerd.io
needs a sysvinit
script.
Using a Devuan ascii
machine that had its docker-ce
upgraded regularly since 17.10 or thereabouts until 18.09.
$ dpkg -S init.d/docker
docker-ce: /etc/init.d/docker
$ dpkg-query -W docker-ce
docker-ce 5:18.09.0~3-0~debian-stretch
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ ps -HAf | grep container
paddy-hack 5733 3321 0 19:46 pts/0 00:00:00 grep --color=auto container
root 5523 5499 0 19:43 ? 00:00:00 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
$ sudo invoke-rc.d docker stop
[ ok ] Stopping Docker: docker.
$ ps -HAf | grep container
paddy-hack 5775 3321 0 19:46 pts/0 00:00:00 grep --color=auto container
$ sudo invoke-rc.d docker start
[ ok ] Starting Docker: docker.
$ ps -HAf | grep container
paddy-hack 5999 3321 0 19:46 pts/0 00:00:00 grep --color=auto container
root 5853 5829 2 19:46 ? 00:00:00 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
Next, I purged the docker-ce
package.
$ sudo apt purge -y docker-ce
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
containerd.io* docker-ce* docker-ce-cli* libseccomp2* pigz*
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 243 MB disk space will be freed.
(Reading database ... 127207 files and directories currently installed.)
Removing docker-ce (5:18.09.0~3-0~debian-stretch) ...
Removing containerd.io (1.2.0-1) ...
Removing docker-ce-cli (5:18.09.0~3-0~debian-stretch) ...
Removing libseccomp2:amd64 (2.3.1-2.1+deb9u1) ...
Removing pigz (2.3.4-1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Processing triggers for man-db (2.7.6.1-2) ...
(Reading database ... 126988 files and directories currently installed.)
Purging configuration files for docker-ce (5:18.09.0~3-0~debian-stretch) ...
Purging configuration files for containerd.io (1.2.0-1) ...
[master be556a2] Committing changes in /etc after APT run
Author: paddy-hack <paddy-hack@...>
10 files changed, 5354 deletions(-)
delete mode 120000 alternatives/distribution_based_engine.json
delete mode 120000 alternatives/dockerd
delete mode 100644 bash_completion.d/docker
delete mode 100644 containerd/config.toml
delete mode 100644 default/docker
delete mode 100755 init.d/docker
delete mode 100644 init/docker.conf
delete mode 120000 systemd/system/multi-user.target.wants/containerd.service
delete mode 120000 systemd/system/multi-user.target.wants/docker.service
Note, I have set up apt
with a fairly aggressive dependent package removal policy (so libseccopm2
and pigz
are purged as well) and use etckeeper
to track changes below /etc
(which produces the part starting with [master
and continuing until the end).
This did not stop the containerd
process. This is not really a surprise because the prerm
script does not run invoke-rc.d docker stop
anymore starting as of 18.09.0~3-0~debian-stretch
:unamused: .
$ invoke-rc.d docker status
bash: invoke-rc.d: command not found
$ sudo invoke-rc.d docker status
invoke-rc.d: unknown initscript, /etc/init.d/docker not found.
$ ps -HAf | grep container
paddy-hack 7160 3321 0 20:01 pts/0 00:00:00 grep --color=auto container
root 5853 5829 0 19:46 ? 00:00:02 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
$ sudo kill -HUP 5853
$ ps -HAf | grep container
paddy-hack 7180 3321 0 20:05 pts/0 00:00:00 grep --color=auto container
Hmm, some time later containerd
"reappeared". Turned out dockerd
needs a kick in the head as well ...
$ ps -HAf | grep -E '(docker|container)d'
root 5829 1 1 19:46 ? 00:00:42 /usr/bin/dockerd -p /var/run/docker.pid
root 7274 5829 0 20:07 ? 00:00:02 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
$ sudo kill -TERM 5829
$ ps -HAf | grep -E '(docker|container)d'
Next, I reinstalled docker-ce
(taking care not to pull in aufs-tools
for which I have no use).
$ sudo apt install -y docker-ce aufs-tools-
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'aufs-tools' is not installed, so not removed
The following additional packages will be installed:
containerd.io docker-ce-cli libseccomp2 pigz
Recommended packages:
aufs-tools
The following NEW packages will be installed:
containerd.io docker-ce docker-ce-cli libseccomp2 pigz
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/50.5 MB of archives.
After this operation, 243 MB of additional disk space will be used.
Selecting previously unselected package pigz.
(Reading database ... 126982 files and directories currently installed.)
Preparing to unpack .../pigz_2.3.4-1_amd64.deb ...
Unpacking pigz (2.3.4-1) ...
Selecting previously unselected package libseccomp2:amd64.
Preparing to unpack .../libseccomp2_2.3.1-2.1+deb9u1_amd64.deb ...
Unpacking libseccomp2:amd64 (2.3.1-2.1+deb9u1) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../containerd.io_1.2.0-1_amd64.deb ...
Unpacking containerd.io (1.2.0-1) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../docker-ce-cli_5%3a18.09.0~3-0~debian-stretch_amd64.deb ...
Unpacking docker-ce-cli (5:18.09.0~3-0~debian-stretch) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_5%3a18.09.0~3-0~debian-stretch_amd64.deb ...
Unpacking docker-ce (5:18.09.0~3-0~debian-stretch) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up libseccomp2:amd64 (2.3.1-2.1+deb9u1) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up docker-ce-cli (5:18.09.0~3-0~debian-stretch) ...
Setting up pigz (2.3.4-1) ...
Setting up containerd.io (1.2.0-1) ...
Setting up docker-ce (5:18.09.0~3-0~debian-stretch) ...
update-alternatives: using /usr/bin/dockerd-ce to provide /usr/bin/dockerd (dockerd) in auto mode
Processing triggers for libc-bin (2.24-11+deb9u3) ...
[master a2f9d2a] Committing changes in /etc after APT run
Author: olaf <olaf@quark>
6 files changed, 23 insertions(+)
create mode 120000 alternatives/distribution_based_engine.json
create mode 120000 alternatives/dockerd
create mode 100644 containerd/config.toml
create mode 120000 systemd/system/multi-user.target.wants/containerd.service
create mode 120000 systemd/system/multi-user.target.wants/docker.service
First thing to note is that the sysvinit
script, /etc/init.d/docker
, is gone :rage:! Post facto, I assume that that script was marked as a conffile
so it survived upgrades but got zapped by the purge. For proper sysvinit
support you will need to add that script back to the docker-ce
package.
I resurrected it from the git
repo that etckeeper
faithfully kept for me. Another approach is to temporarily downgrade to 18.06.1~ce~3-0~debian
and upgrade back again to 5:18.09.0~3-0~debian-stretch
.
Next thing to note is that, unsurprisingly, dockerd
is not running. Fortunately, with the resurrected sysvinit
script that is easily rectified.
$ sudo invoke-rc.d docker status
[FAIL] Docker is not running ... failed!
invoke-rc.d: initscript docker, action "status" failed.
$ docker info
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
$ sudo invoke-rc.d docker start
[ ok ] Starting Docker: docker.
$ sudo invoke-rc.d docker status
[ ok ] Docker is running.
$ ps -HAf | grep -E '(docker|container)d'
root 14463 1 1 20:38 ? 00:00:00 /usr/bin/dockerd -p /var/run/docker.pid
root 14487 14463 0 20:38 ? 00:00:00 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
As a bonus, said script somehow triggers the start of containerd
as well. Looks like containerd
may not need a sysvinit
script ... unless I missed something. If so, I'd appreciate a cluebat as to what else needs checking.
So far, I have not found any evidence that containerd.io needs a sysvinit script.
Docker 18.09 an up run containerd as a separate service. If you run ps auxf
on Docker 18.09, you'll see that containerd
is no longer a child-process of dockerd
;
root 24838 0.3 2.4 483200 24708 ? Ssl Nov09 167:26 /usr/bin/containerd
root 3329 0.0 0.1 9336 1828 ? Sl Nov15 1:04 \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/06ff646ec5de90a52734e752851b8b432d4c694646c7190c039df1a02d0bc
root 3383 0.0 0.2 13740 2960 ? Ss Nov15 0:00 \_ nginx: master process nginx -g daemon off;
systemd+ 3478 0.0 0.1 14188 1512 ? S Nov15 0:07 \_ nginx: worker process
root 24883 0.2 8.1 557156 83256 ? Ssl Nov09 138:18 /usr/bin/dockerd -H fd://
Whereas on Docker 18.06, it is;
root 3302 5.8 2.3 1040672 95204 ? Ssl 12:04 0:01 /usr/bin/dockerd -H fd://
root 3325 0.5 0.8 800776 34896 ? Ssl 12:04 0:00 \_ docker-containerd --config /var/run/docker/containerd/containerd.toml
root 3614 0.0 0.0 8780 3608 ? Sl 12:04 0:00 \_ docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/4f729f09ad4a36f2fd652dadb351398f8fcb
root 3635 0.8 0.0 13832 3492 ? Ss 12:04 0:00 \_ nginx: master process nginx -g daemon off;
systemd+ 3685 0.0 0.0 14288 1820 ? S 12:04 0:00 \_ nginx: worker process
Doing as you suggested on my Devuan ascii setup running 18.09 with the sysvinit
script from 18.06
$ ps auxf | grep -E '(docker|container)d'
root 14463 0.2 0.8 457960 68460 ? Sl 20:38 0:07 /usr/bin/dockerd -p /var/run/docker.pid
root 14487 0.2 0.5 343324 40808 ? Ssl 20:38 0:08 \_ containerd --config /var/run/docker/containerd/containerd.toml --log-level info
$ ps -HAf | grep -E '(docker|container)d'
root 14463 1 0 20:38 ? 00:00:07 /usr/bin/dockerd -p /var/run/docker.pid
root 14487 14463 0 20:38 ? 00:00:08 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
Clearly, the containerd
process is a child process of dockerd
on my setup. Whether that is "courtesy" of the 18.06 sysvinit
or not I don't know. However, using that 18.06 sysvinit
seems to obviate the need for a separate containerd
sysvinit
script.
It's a fallback mechanism if the containerd socket is not available
I see. That explains my observations. It also implies that the containerd
socket needs to be available before dockerd
is started. Any init.d
script would have to take that into account.
Rethinking this fallback mechanism, if /etc/init.d/docker
is added back together with the *.postinst
that ran invoke-rc.d restart
, then my use case will be fixed again. As to my use case, I use the docker
command-line and rely on docker-compose
for development purposes as well as deploy purposes on a small bunch of on-premises Debian and Devuan hosts .
The /etc/init.d/docker
that I'm using traces all the way back to docker-engine_17.04.0~ce-0~debian-jessie
. My etckeeper
logs indicate no changes since that version until I purged docker-ce_5:18.09.0~3-0~debian-stretch
on my Devuan machine for testing purposes. Since upgrading to 17.04.0~ce-0~debian-jessie
, that Devuan machine has gone through the following versions without a hitch:
docker-engine 17.05.0~ce-0~debian-jessie amd64
docker-ce 17.06.0~ce-0~debian amd64
docker-ce 17.06.1~ce-0~debian amd64
docker-ce 17.07.0~ce-0~debian amd64
docker-ce 17.09.0~ce-0~debian amd64
docker-ce 17.10.0~ce-0~debian amd64
docker-ce 17.12.0~ce-0~debian amd64
docker-ce 18.01.0~ce-0~debian amd64
docker-ce 18.02.0~ce-0~debian amd64
docker-ce 18.03.0~ce-0~debian amd64
docker-ce 18.04.0~ce~3-0~debian amd64
docker-ce 18.05.0~ce~3-0~debian amd64
docker-ce 18.06.0~ce~3-0~debian amd64
docker-ce 18.06.1~ce~3-0~debian amd64
That's just to give you an idea of how "disruptive" dropping sysvinit
support was for me :wink:
My initial report mentioned 17.10 but that was for "small bunch of on-premises Debian and Devuan hosts" at the office. The above is for my laptop at home.
So while resurrecting /etc/init.d/docker
and the *.postinst
(and *.prerm
?) would rescue my use case, what other use cases are there to consider? The one where only containerd.io
gets installed or is upgraded? I'm not clear on how that would be useful and would appreciate a cluebat.
I have been playing with @tianon's suggestion but it seems that at a minimum that script needs to make sure that containerd
is put into the background. Looking at building my own debs now, was a bit ruffled at there not being a deb-src
I could use but found docker-ce-packaging
and will take things from there.
Urk! Don't tell me I need systemd
just to build a package :disappointed: :rage:
$ git clone git@github.com:docker/docker-ce-packaging.git
[...]
$ cd docker-ce-packaging/deb
$ git clone git@github.com:docker/docker-ce.git
[...]
$ make ENGINE_DIR=docker-ce/components/engine CLI_DIR=docker-ce/components/cli debian-stretch
make: systemctl: Command not found
mkdir -p sources
make -C ../image image-linux
make[1]: Entering directory '/home/paddy-hack/docker-ce-packaging/image'
cp Dockerfile.engine docker-ce/components/engine/Dockerfile.engine
cp: cannot create regular file 'docker-ce/components/engine/Dockerfile.engine': No such file or directory
Makefile:52: recipe for target 'docker-ce/components/engine/Dockerfile.engine' failed
make[1]: *** [docker-ce/components/engine/Dockerfile.engine] Error 1
make[1]: Leaving directory '/home/paddy-hack/docker-ce-packaging/image'
Makefile:130: recipe for target 'sources/engine-image' failed
make: *** [sources/engine-image] Error 2
Oops! Sorry for the noise, looks like I just need to have containerd
running ... hmm, smells like a :chicken:-and-:egg: problem.
Make sure to check the 18.09 branch in that repository as well; I know there was a "todo" to bring master up to date with the 18.09 branch, and I'm not sure if that was all done
Thanks for the tip. I've got a build running now using master
. I just needed to start dockerd
on my system, which starts containerd
, and fix up the *_DIR
variables to point to the right place :sweat_smile:
Trudging through the docker-ce-packaging
commit logs, it seems the "damage" was done in https://github.com/docker/docker-ce-packaging/commit/662e248f680eb49a9951a8b34125506b8f82dfed. I'm working on undoing that on 18.09
(but the changes should apply equally cleanly on master
). It will only fix docker-ce
. The containerd.io
packaging lives elsewhere, it seems.
I've pushed a branch that "Works For Me" when upgrading to and downgrading from the official 5:18.09.0~3-0~debian-stretch
versions of docker-ce
and docker-ce-cli
.
I haven't tested up- and downgrading from versions before the package split and/or before the sysvinit
support was dropped. Also haven't looked at using the packages on Debian yet.
BTW, I'm only looking at ASCII/Stretch.
I have not tested Upstart support (and don't plan to).
For easier comparing of changes; here's the diff between that branch and 18.09; https://github.com/docker/docker-ce-packaging/compare/18.09...paddy-hack:restore-sysvinit-and-upstart-support
@paddy-hack feel free to open it as a pul request, so that further discussion on the implementation can take place on the pull request.
To be clear; I'm not sure if the patch would be accepted (given that it's not targeting a platform/configuration that the packages are published for, and as such won't be a tested configuration), but having a pull request at least makes discussing the implementation easier 🤗
Seeing that my PR got merged on 18.09
and cherry-picked on master
of docker-ce-packaging
, my issue has been taken care of.
Although I'm glad to see my changes in 5:18.09.1~3-0~debian-stretch
, the apt upgrade
on my Devuan systems didn't go quite as smoothly as I had hoped for. It's fairly easily worked around but I'm wondering if this needs a bit more work.
See my post to the Devuan ML for details.
Expected behavior
Upgrading from
18.06.1~ce~3-0~debian
to5:18.09.0~3-0~debian-stretch
should restart a Docker daemon that was running before the upgrade.Actual behavior
Performing this upgrade on Devuan (ascii) stopped the daemon but did not restart it.
Steps to reproduce the behavior
Perform the upgrade on Devuan (ascii).
Output of
docker version
:Before running
sudo invoke-rc.d docker restart
after the upgradeAfter running
sudo invoke-rc.d docker restart
following the upgradeOutput of
docker info
:Before running
sudo invoke-rc.d docker restart
after the upgradeAfter running
sudo invoke-rc.d docker restart
following the upgradeAdditional environment details
I have been running
docker-ce
on Devuan (jessie as well as ascii) since around 17.10 and upgrades have so far always restarted the daemon. This upgrade is the first that doesn't.I have checked the new version's
postinst
andprerm
scripts and noticed that thesysvinit
support has been removed :unamused: Putting back the support that was there until18.06.1~ce~3-0~debian
should fix the problem as far as I understand.While I understand the package targets Debian, please be advised that there are derivatives/forks that do not use
systemd
, Devuan being one of them. It would be nice if your Debian packages could be used as is on (at least some of) these as well. I do not understand why thesysvinit
support had to be removed.On a related note, the
containerd.io
package does not supportsysvinit
based systems either. Perhaps it should.