search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
754 stars 85 forks source link

CRITICAL!!! docker remove /dev/* on the host #272

Closed softcombiz closed 6 years ago

softcombiz commented 6 years ago

DO NOT TRY TO REPRODUCE THIS ON A PHYSICAL HOST!!!

IT WILL BREAK OS!!!

[root@ceph1 ~]# yum update docker-ce
Loaded plugins: fastestmirror, remove-with-leaves
Loading mirror speeds from cached hostfile
 * base: mirror.freethought-internet.co.uk
 * elrepo: mirrors.coreix.net
 * epel: mirrors.coreix.net
 * extras: mirror.freethought-internet.co.uk
 * nux-dextop: mirror.li.nux.ro
 * remi-safe: rpms.remirepo.net
 * updates: mirror.sax.uk.as61049.net
Resolving Dependencies
--> Running transaction check
---> Package docker-ce.x86_64 0:17.12.1.ce-1.el7.centos will be updated
---> Package docker-ce.x86_64 0:18.03.0.ce-1.el7.centos will be an update
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================
 Package                    Arch                    Version                                   Repository                         Size
======================================================================================================================================
Updating:
 docker-ce                  x86_64                  18.03.0.ce-1.el7.centos                   docker-ce-stable                   35 M

Transaction Summary
======================================================================================================================================
Upgrade  1 Package

Total download size: 35 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
docker-ce-18.03.0.ce-1.el7.centos.x86_64.rpm                                                                   |  35 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : docker-ce-18.03.0.ce-1.el7.centos.x86_64                                                                           1/2 
  Cleanup    : docker-ce-17.12.1.ce-1.el7.centos.x86_64                                                                           2/2 
  Verifying  : docker-ce-18.03.0.ce-1.el7.centos.x86_64                                                                           1/2 
  Verifying  : docker-ce-17.12.1.ce-1.el7.centos.x86_64                                                                           2/2 

Updated:
  docker-ce.x86_64 0:18.03.0.ce-1.el7.centos                                                                                          

Complete!
[root@ceph1 ~]# docker version
Client:
 Version:   18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:09:15 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[root@ceph1 ~]# ls /dev/
autofs           fb0               mqueue  net                 sdb1      tty    tty23  tty39  tty54  ttyS3    vcsa4
block            fd                nbd0    network_latency     sdb2      tty0   tty24  tty4   tty55  uhid     vcsa5
bsg              full              nbd1    network_throughput  sdb3      tty1   tty25  tty40  tty56  uinput   vcsa6
btrfs-control    fuse              nbd10   null                sdb4      tty10  tty26  tty41  tty57  urandom  vfio
bus              hpet              nbd11   nvram               sdc       tty11  tty27  tty42  tty58  usbmon0  vga_arbiter
cdrom            hugepages         nbd12   port                sdc1      tty12  tty28  tty43  tty59  usbmon1  vhci
char             hwrng             nbd13   ppp                 sg0       tty13  tty29  tty44  tty6   vcs      vhost-net
cl               initctl           nbd14   ptmx                sg1       tty14  tty3   tty45  tty60  vcs1     virtio-ports
console          input             nbd15   pts                 sg2       tty15  tty30  tty46  tty61  vcs2     vport2p1
core             kmsg              nbd2    random              sg3       tty16  tty31  tty47  tty62  vcs3     zero
cpu              lightnvm          nbd3    raw                 shm       tty17  tty32  tty48  tty63  vcs4
cpu_dma_latency  log               nbd4    rtc                 snapshot  tty18  tty33  tty49  tty7   vcs5
cuse             loop-control      nbd5    rtc0                snd       tty19  tty34  tty5   tty8   vcs6
disk             mapper            nbd6    sda                 sr0       tty2   tty35  tty50  tty9   vcsa
dm-0             mcelog            nbd7    sda1                stderr    tty20  tty36  tty51  ttyS0  vcsa1
dm-1             mem               nbd8    sda2                stdin     tty21  tty37  tty52  ttyS1  vcsa2
dri              memory_bandwidth  nbd9    sdb                 stdout    tty22  tty38  tty53  ttyS2  vcsa3
[root@ceph1 ~]# systemctl start docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
[root@ceph1 ~]# ls /dev
mqueue  pts  shm
[root@ceph1 ~]#

The problem occurs when the plugin gyrotec/rbd-nbd:luminous-0.1 is installed

[root@ceph1 ~]# docker version
Client:
 Version:   18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:09:15 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:    Wed Mar 21 23:13:03 2018
  OS/Arch:  linux/amd64
  Experimental: false
[root@ceph1 ~]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.121-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953GiB
Name: ceph1
ID: 4RTD:D2QP:QJ5J:53MD:AD6B:7FZ5:CSVW:NVS4:D5CZ:JQGO:ANP5:ASAP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-ip6tables is disabled
[root@ceph1 ~]# ls /dev/
autofs           fb0               mqueue  net                 sdb1      tty    tty23  tty39  tty54  ttyS3    vcsa4
block            fd                nbd0    network_latency     sdb2      tty0   tty24  tty4   tty55  uhid     vcsa5
bsg              full              nbd1    network_throughput  sdb3      tty1   tty25  tty40  tty56  uinput   vcsa6
btrfs-control    fuse              nbd10   null                sdb4      tty10  tty26  tty41  tty57  urandom  vfio
bus              hpet              nbd11   nvram               sdc       tty11  tty27  tty42  tty58  usbmon0  vga_arbiter
cdrom            hugepages         nbd12   port                sdc1      tty12  tty28  tty43  tty59  usbmon1  vhci
char             hwrng             nbd13   ppp                 sg0       tty13  tty29  tty44  tty6   vcs      vhost-net
cl               initctl           nbd14   ptmx                sg1       tty14  tty3   tty45  tty60  vcs1     virtio-ports
console          input             nbd15   pts                 sg2       tty15  tty30  tty46  tty61  vcs2     vport2p1
core             kmsg              nbd2    random              sg3       tty16  tty31  tty47  tty62  vcs3     zero
cpu              lightnvm          nbd3    raw                 shm       tty17  tty32  tty48  tty63  vcs4
cpu_dma_latency  log               nbd4    rtc                 snapshot  tty18  tty33  tty49  tty7   vcs5
cuse             loop-control      nbd5    rtc0                snd       tty19  tty34  tty5   tty8   vcs6
disk             mapper            nbd6    sda                 sr0       tty2   tty35  tty50  tty9   vcsa
dm-0             mcelog            nbd7    sda1                stderr    tty20  tty36  tty51  ttyS0  vcsa1
dm-1             mem               nbd8    sda2                stdin     tty21  tty37  tty52  ttyS1  vcsa2
dri              memory_bandwidth  nbd9    sdb                 stdout    tty22  tty38  tty53  ttyS2  vcsa3
[root@ceph1 ~]# docker plugin install gyrotec/rbd-nbd:luminous-0.1
Plugin "gyrotec/rbd-nbd:luminous-0.1" is requesting the following privileges:
 - network: [host]
 - mount: [/dev]
 - mount: [/etc/ceph]
 - mount: [/var/log]
 - allow-all-devices: [true]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
luminous-0.1: Pulling from gyrotec/rbd-nbd
c936ac607fcf: Download complete 
Digest: sha256:36005f30dd17eac5a1cdd5137af23df5c8a18957a6ed601453b942aafcaca9b2
Status: Downloaded newer image for gyrotec/rbd-nbd:luminous-0.1
Error response from daemon: dial unix /run/docker/plugins/fb052a31b90616e41fc745cbd9e9d92b456d1241f526fd934736892dd8b95c33/rbd.sock: connect: no such file or directory
[root@ceph1 ~]# ls /dev/
mqueue  pts  shm
[root@ceph1 ~]# journalctl -u docker -o cat --no-pager
Starting Docker Application Container Engine...
time="2018-04-04T14:20:40.642155987-04:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2018-04-04T14:20:40.642852827-04:00" level=info msg="libcontainerd: started new docker-containerd process" pid=3301
time="2018-04-04T14:20:40-04:00" level=info msg="starting containerd" module=containerd revision=cfd04396dc68220d1cecbe686a6cc3aa5ce3667c version=v1.0.2
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." module=containerd type=io.containerd.content.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1
time="2018-04-04T14:20:40-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." module=containerd type=io.containerd.snapshotter.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." module=containerd type=io.containerd.metadata.v1
time="2018-04-04T14:20:40-04:00" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." module=containerd type=io.containerd.differ.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." module=containerd type=io.containerd.gc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." module=containerd type=io.containerd.monitor.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." module=containerd type=io.containerd.runtime.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." module=containerd type=io.containerd.grpc.v1
time="2018-04-04T14:20:40-04:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
time="2018-04-04T14:20:40-04:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
time="2018-04-04T14:20:40-04:00" level=info msg="containerd successfully booted in 0.010884s" module=containerd
time="2018-04-04T14:20:40.676332741-04:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2018-04-04T14:20:40.676900200-04:00" level=info msg="Loading containers: start."
time="2018-04-04T14:20:40.734891782-04:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2018-04-04T14:20:40.758990226-04:00" level=info msg="Loading containers: done."
time="2018-04-04T14:20:40.773882980-04:00" level=info msg="Docker daemon" commit=0520e24 graphdriver(s)=overlay2 version=18.03.0-ce
time="2018-04-04T14:20:40.774020253-04:00" level=info msg="Daemon has completed initialization"
time="2018-04-04T14:20:40.780368506-04:00" level=info msg="API listen on [::]:2375"
time="2018-04-04T14:20:40.780400136-04:00" level=info msg="API listen on /var/run/docker.sock"
Started Docker Application Container Engine.
time="2018-04-04T14:22:25-04:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/plugins.moby/fb052a31b90616e41fc745cbd9e9d92b456d1241f526fd934736892dd8b95c33/shim.sock" debug=false module="containerd/tasks" pid=3456
time="2018-04-04T14:22:25-04:00" level=info msg="shim reaped" id=fb052a31b90616e41fc745cbd9e9d92b456d1241f526fd934736892dd8b95c33 module="containerd/tasks"
time="2018-04-04T14:22:37.723164294-04:00" level=error msg="Sending SIGTERM to plugin failed with error: container is not running"
time="2018-04-04T14:22:37.723243481-04:00" level=error msg="Handler for POST /v1.37/plugins/gyrotec/rbd-nbd:luminous-0.1/enable returned error: dial unix /run/docker/plugins/fb052a31b90616e41fc745cbd9e9d92b456d1241f526fd934736892dd8b95c33/rbd.sock: connect: no such file or directory"

In log of plugins/gyrotec/rbd-nbd:luminous-0.1:

2018/04/04 18:15:36 main.go:96: INFO: starting rbd-docker-plugin version 1.6.1
2018/04/04 18:15:36 main.go:97: INFO: canCreateVolumes=%!q(bool=true), removeAction="rename"
2018/04/04 18:15:36 main.go:98: INFO: Setting up Ceph Driver for PluginID=rbd, cluster=ceph, user=admin, pool=rbd, mount=/var/lib/docker-volumes, 
                        config=/etc/ceph/ceph.conf, go-ceph=%!s(bool=false), useNbd=%!s(bool=true)
2018/04/04 18:15:36 driver.go:124: INFO: newCephRBDVolumeDriver: setting base mount dir=/var/lib/docker-volumes/rbd
2018/04/04 18:15:36 main.go:134: INFO: Creating Docker VolumeDriver Handler
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:188: Entering go-plugins-helpers getPath
2018/04/04 18:15:36 driver.go:605: INFO: pool rbd, name test
2018/04/04 18:15:36 driver.go:629: INFO: Get request(test) => /var/lib/docker-volumes/rbd/rbd/test
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:188: Entering go-plugins-helpers getPath
2018/04/04 18:15:36 driver.go:605: INFO: pool rbd, name 
2018/04/04 18:15:36 driver.go:629: INFO: Get request() => /var/lib/docker-volumes/rbd/rbd
2018/04/04 18:15:36 api.go:188: Entering go-plugins-helpers getPath
2018/04/04 18:15:36 driver.go:605: INFO: pool rbd, name portainer
2018/04/04 18:15:36 driver.go:629: INFO: Get request(portainer) => /var/lib/docker-volumes/rbd/rbd/portainer
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:188: Entering go-plugins-helpers getPath
2018/04/04 18:15:36 driver.go:605: INFO: pool rbd, name test
2018/04/04 18:15:36 driver.go:629: INFO: Get request(test) => /var/lib/docker-volumes/rbd/rbd/test
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:188: Entering go-plugins-helpers getPath
2018/04/04 18:15:36 driver.go:605: INFO: pool rbd, name test
2018/04/04 18:15:36 driver.go:629: INFO: Get request(test) => /var/lib/docker-volumes/rbd/rbd/test
2018/04/04 18:15:36 api.go:226: Entering go-plugins-helpers capabilitiesPath
2018/04/04 18:15:36 api.go:188: Entering go-plugins-helpers getPath
2018/04/04 18:15:36 driver.go:605: INFO: pool rbd, name test
2018/04/04 18:15:36 driver.go:629: INFO: Get request(test) => /var/lib/docker-volumes/rbd/rbd/test
2018/04/04 18:19:41 main.go:146: INFO: received TERM or KILL signal: terminated
2018/04/04 18:19:41 main.go:201: INFO: closing log file

Problem in version 18.03.0-ce 17.12.1 and below works well

same if install rexray/rbd

[root@ceph1 ~]# ls /dev
autofs           fb0               mqueue  net                 sdb1      tty    tty23  tty39  tty54  ttyS3    vcsa4
block            fd                nbd0    network_latency     sdb2      tty0   tty24  tty4   tty55  uhid     vcsa5
bsg              full              nbd1    network_throughput  sdb3      tty1   tty25  tty40  tty56  uinput   vcsa6
btrfs-control    fuse              nbd10   null                sdb4      tty10  tty26  tty41  tty57  urandom  vfio
bus              hpet              nbd11   nvram               sdc       tty11  tty27  tty42  tty58  usbmon0  vga_arbiter
cdrom            hugepages         nbd12   port                sdc1      tty12  tty28  tty43  tty59  usbmon1  vhci
char             hwrng             nbd13   ppp                 sg0       tty13  tty29  tty44  tty6   vcs      vhost-net
cl               initctl           nbd14   ptmx                sg1       tty14  tty3   tty45  tty60  vcs1     virtio-ports
console          input             nbd15   pts                 sg2       tty15  tty30  tty46  tty61  vcs2     vport2p1
core             kmsg              nbd2    random              sg3       tty16  tty31  tty47  tty62  vcs3     zero
cpu              lightnvm          nbd3    raw                 shm       tty17  tty32  tty48  tty63  vcs4
cpu_dma_latency  log               nbd4    rtc                 snapshot  tty18  tty33  tty49  tty7   vcs5
cuse             loop-control      nbd5    rtc0                snd       tty19  tty34  tty5   tty8   vcs6
disk             mapper            nbd6    sda                 sr0       tty2   tty35  tty50  tty9   vcsa
dm-0             mcelog            nbd7    sda1                stderr    tty20  tty36  tty51  ttyS0  vcsa1
dm-1             mem               nbd8    sda2                stdin     tty21  tty37  tty52  ttyS1  vcsa2
dri              memory_bandwidth  nbd9    sdb                 stdout    tty22  tty38  tty53  ttyS2  vcsa3
[root@ceph1 ~]# docker plugin install rexray/rbd
Plugin "rexray/rbd" is requesting the following privileges:
 - network: [host]
 - mount: [/dev]
 - mount: [/etc/ceph]
 - allow-all-devices: [true]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from rexray/rbd
20d5e3d9f105: Download complete 
Digest: sha256:4ea5ae92c7bfec0d7f7b227deddf6520019b9fb764663fd24782ba56bc9d4052
Status: Downloaded newer image for rexray/rbd:latest
Error response from daemon: dial unix /run/docker/plugins/cee19a1824c7b68b35ce36142d03a7f8c43a20b5cddc96aad82c44eb7cffe2bf/rexray.sock: connect: no such file or directory
[root@ceph1 ~]# ls /dev/
mqueue  null  pts  shm
[root@ceph1 ~]# docker version
Client:
 Version:   18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:09:15 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:    Wed Mar 21 23:13:03 2018
  OS/Arch:  linux/amd64
  Experimental: false
[root@ceph1 ~]#
cpuguy83 commented 6 years ago

Thanks, this is related to a regression with plugins, specifically plugins that mount the host /dev. This is fixed by https://github.com/moby/moby/pull/36711 and being backported to 18.03 in https://github.com/docker/docker-ce/pull/490