"I can't kill my containers" or "restart=always-unless-stopped should be renamed to restart=always. And restart=always should be killed with fire"

[TZubiri]

Description

I use docker compose to orchestrate 2 images, I use restart=always on both, killing the container doesn't even change the container id or increase the restart count (?!) , I would expect another container to be ran. docker-compose stop doesn't remove the containers from docker ps either. I can't exec into the containers because they are in a stopped state (why are they shown in docker ps?). I can't update the restart policy because they are in a stopped state as well:

ubuntu@ubuntu:~/jcimagenes/build$ sudo docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
f575065bbb9b        build_orthanc-god     "python3 /src/main.p…"   4 weeks ago         Up 6 days                                                            build_orthanc-god_1
47710a7e6d3a        build_orthanc-proxy   "Orthanc /orthanc.js…"   4 weeks ago         Up 6 days           0.0.0.0:4242->4242/tcp, 0.0.0.0:8042->8042/tcp   build_orthanc-proxy_1
ubuntu@ubuntu:~/jcimagenes/build$ sudo docker exec -it 4771 /bin/bash
cannot exec in a stopped state: unknown
ubuntu@ubuntu:~/jcimagenes/build$ sudo docker kill 4771
4771
ubuntu@ubuntu:~/jcimagenes/build$ sudo docker rm 4771
Error response from daemon: You cannot remove a running container 47710a7e6d3ab706b9c2c28ca156db2a83e9f20fd0b69c222de6d7a2dd541538. Stop the container before attempting removal or force remove
ubuntu@ubuntu:~/jcimagenes/build$ sudo docker exec -it 4771 /bin/bash
cannot exec in a stopped state: unknown
ubuntu@ubuntu:~/jcimagenes/build$ sudo docker update 47710a --restart=no
Error response from daemon: Cannot update container 47710a7e6d3ab706b9c2c28ca156db2a83e9f20fd0b69c222de6d7a2dd541538: cannot update a stopped container: unknown

Steps to reproduce the issue: Sorry, can't help you here.

Describe the results you received: Uninstalling docker is the only reasonable way forward.

Describe the results you expected: I expect docker kill to kill my containers. --restart=always-unless-stopped is ok, but could we change its name to --restart=always?

Output of docker version: Client: Version: 18.09.7 API version: 1.39 Go version: go1.10.1 Git commit: 2d0083d Built: Wed Jul 3 12:14:38 2019 OS/Arch: linux/arm64 Experimental: false

Server: Engine: Version: 18.09.7 API version: 1.39 (minimum version 1.12) Go version: go1.10.1 Git commit: 2d0083d Built: Mon Jul 1 19:31:12 2019 OS/Arch: linux/arm64 Experimental: false

Output of docker info: Containers: 7 Running: 2 Paused: 0 Stopped: 5 Images: 28 Server Version: 18.09.7 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: runc version: N/A init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662) Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-1038-raspi2 Operating System: Ubuntu 18.04.2 LTS OSType: linux Architecture: aarch64 CPUs: 4 Total Memory: 912.1MiB Name: ubuntu ID: BAHV:NAAN:YTFL:MRV3:7BM3:FJRQ:6KYV:3WX6:K67Q:5GQT:SPRU:T7G6 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: No memory limit support WARNING: No swap limit support WARNING: No kernel memory limit support WARNING: No oom kill disable support

Here is a list of similar problems other people have encountered with restart=always

https://github.com/moby/moby/issues/38755 https://stackoverflow.com/questions/37599128/docker-how-do-you-disable-auto-restart-on-a-container https://stackoverflow.com/questions/39072357/how-to-stop-a-docker-container-which-started-with-restart-always https://stackoverflow.com/questions/53003404/stop-a-failing-container-with-restart-always https://stackoverflow.com/questions/40513545/how-to-prevent-docker-from-starting-a-container-automatically-on-system-startup

Is there a situation were restart=always provides any benefit over restart=unless-stopped? Is it worth it to provide this marginal benefit to users when weighed against the significant problems offered to mainstream users?

Please deprecate the kill resilient always restart policy, thank you.

P.S: Other than this incident, Docker's architecture has been enlightening, good job, keep it up.

[cpuguy83]

This sounds like a regression. Normally docker would not restart the container if it was manually stopped/killed (at least not until the daemon is restarted).

[TZubiri]

That makes sense, this is the arm build, which I surmise is less stable than x86.

I managed to solve the problem by using apt remove docker.io followed by apt purge docker.io, unfortunately, this deleted all of my docker history, so I don't have deep debugging information.

If someone is willing to try to replicate this issue by simulating my environment and using my docker files and docker-compose.ymls , you can reach out to me at the email listed in my github profile and I can provide you access to a redacted repo.

[cpuguy83]

Oh it's even deeper than the restart policy because it seems like there's a disconnect in the container state.

[TZubiri]

What a shame that I purged the docker data. In the future, what docker files or objects should I save for debugging purposes?