docker-init not found on Docker CE 19.03.1 with /sbin mounted

aaclarker commented 5 years ago

[x] This is a bug report
[ ] This is a feature request
[x] I searched existing issues before opening this one

Expected behavior

The Amazon Web Services ECS Container Agent, in some modes, requires the --init flag as well as mounting of /sbin.

The ECS Agent container example given in their documentation should start and does so on Docker CE 18

Actual behavior

Attempting to run the container with both --init and --volume=/sbin:/sbin options immediately fails with the error:

Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/sbin/docker-init\": stat /sbin/docker-init: no such file or directory"

Steps to reproduce the behavior

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:21:07 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 4
 Server Version: 19.03.1
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: splunk
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
  userns
 Kernel Version: 3.10.0-957.21.3.el7.x86_64
 Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.466GiB
 Name: ****
 ID: ****
 Docker Root Dir: /var/lib/docker/808080.808080
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.)

Tested on RHEL 7.6 in AWS using the latest Docker CE stable from yum as of 7/25/19.

Start the ECS Agent container using most flags from the example:

$ docker run --name ecs-agent \
    --init \
    --volume=/var/run:/var/run \
    --volume=/var/log/ecs/:/log:Z \
    --volume=/var/lib/ecs/data:/data:Z \
    --volume=/etc/ecs:/etc/ecs \
    --volume=/lib:/lib \
    --volume=/sbin:/sbin \
    --volume=/lib64:/lib64 \
    --volume=/usr/lib:/usr/lib \
    --volume=/proc:/host/proc \
    --volume=/sys/fs/cgroup:/sys/fs/cgroup \
    --volume=/var/lib/ecs/dhclient:/var/lib/dhclient \
    --net=host \
    --env-file=/etc/ecs/ecs.config \
    --userns=host \
    amazon/amazon-ecs-agent:latest

docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/sbin/docker-init\": stat /sbin/docker-init: no such file or directory": unknown.
ERRO[0001] error waiting for container: context canceled

$ which docker-init

/bin/docker-init

cpuguy83 commented 5 years ago

Work around would be to put docker-init into /sbin/docker-init.

The root cause seems to be that we bind-mount in docker-init to /sbin but the mount for that is done before user specified bind mounts.

ericjohnlucas commented 5 years ago

I started getting exactly the same issue just within the past week. Yes, copying /usr/bin/docker-init to /sbin/docker-init does solve the problem, but the question is: why did this issue just start happening now when I have had the same config for months?

cpuguy83 commented 5 years ago

It's because of this change: https://github.com/moby/moby/commit/bcacbf523b35b6cf22bd84ac33e4425784c5a0a2

aaclarker commented 5 years ago

Thanks for the update @cpuguy83! That's exactly the issue. I realize a couple of workarounds exist, but is there any chance this will be considered a bug and fixed in an upcoming release?

cpuguy83 commented 5 years ago

@kolyshkin WDYT, re: fixing this in a patch release?

I think we could add the mount for docker-init to the oci spec after all the bind mounts and volumes are added. Do you see any possible breakages here?

ngocketit commented 5 years ago

@cpuguy83's workaround did the trick.

docker / for-linux