Replace syscall.Unlink with os.Remove so that the directory(e.g. /run…

[keloyang]

Docker start failed If docker's unix socket path exist，and it is a dir.

we can reproduce it like thks.

1. make sure live-restore is false
2. start 20 containers with -v /run/docker.sock:/run/docker.sock and --restart always

   
   #!/bin/bash

i=0 while [ ${i} -lt 20 ]; do docker run --name test-${i} -tid --restart always -v /run/docker.sock:/run/docker.sock busybox sleep 16881688 ((i++)) done

3.  kill container process and restart dockerd

[root@centos1 workspace]# killall -9 sleep;usleep 100;systemctl restart docker Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details. [root@centos1 workspace]# ls /run/docker docker/ dockershim.sock docker.sock/

we can see the dir /run/docker.sock/  is created and It causes docker to start failed.

use os.Remove can fix this issue.

[keloyang]

ping @calavera @offby1 @jamtur01 @vdemeester

[cpuguy83]

Unlink does remove the file if it's not being used I don't think this is correct and should be reverted.

[thaJeztah]

@cpuguy83 you mean that the os.Remove would remove it in the non-directory situation?

[cpuguy83]

The short term solution is to use --mount instead of "-v", because "--mount" will not attempt to create the directory.

[cpuguy83]

Yes.

[thaJeztah]

True; that would mostly prevent the situation. Once you're in there, things are more tricky (and something like this PR would help);

Should we check if it's a directory and in that case do a os.Remove() or syscall.Rmdir() ?

[cpuguy83]

I think it is difficult to properly remediate it from here. The Docker API should fail to start in this case.

The correct fix would likely be in Dockerd. Couple of options I can think of:

1. Have some restricted paths (like a list of unix sockets) that the volume manager can error out if it would have otherwise tried to create the path
2. Block container startup on daemon init until after the API is ready

1 seems fairly problematic 2 may have some other consequences... may be worth adding in there to block API requests until after the daemon is ready (so block daemon until API is at a point where it can serve, but don't allow it to serve until the daemon is ready).

[cpuguy83]

Or just the tried and true "don't do that" approach.

[thaJeztah]

Opened a PR to revert the change https://github.com/docker/go-connections/pull/73, and an alternative (if we want to consider that; https://github.com/docker/go-connections/pull/74)

[keloyang]

@cpuguy83 Thanks for your review.

Unlink does remove the file if it's not being used

Does this mean that Unlink does't remove the file if it's being used?

I use -v and mount /run/docker.sock to a container，then I write some code to use Unlink and try delete /run/docker.sock. At last /run/docker.sock is deleted successfully.

I want to know if I misunderstood something, PTAL, thanks very much.

the follownig is the code

package main

import "syscall"

func main() {
    syscall.Unlink("/run/docker.sock")
}

And the details

[root@VM_0_16_centos keloyang]# docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:25:41 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:24:18 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
[root@VM_0_16_centos keloyang]# ps -eaf|grep dockerd
root     27615     1  0 09:46 ?        00:00:00 /usr/bin/dockerd --containerd=/run/containerd/containerd.sock
root     27853 22441  0 09:46 pts/0    00:00:00 grep --color=auto dockerd
[root@VM_0_16_centos keloyang]# docker run --name sock -tid -v /run/docker.sock:/run/docker.sock busybox ash
2ac196c14d2af47178595c6d0eef44fdddce605bb6bf531a5b6c5eaaa247c7a9
[root@VM_0_16_centos keloyang]# lsof|grep docker.sock
systemd       1                 root   26u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615                 root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615                 root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27616           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27616           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27617           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27617           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27618           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27618           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27619           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27619           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27620           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27620           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27622           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27622           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27623           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27623           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27624           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27624           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27625           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27625           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27626           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27626           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
[root@VM_0_16_centos keloyang]# cat test.go 
package main

import "syscall"

func main() {
    syscall.Unlink("/run/docker.sock")
}
[root@VM_0_16_centos keloyang]# go build test.go
[root@VM_0_16_centos keloyang]# ls /run/docker.*
/run/docker.pid  /run/docker.sock
[root@VM_0_16_centos keloyang]# ./test 
[root@VM_0_16_centos keloyang]# ls /run/docker.*
/run/docker.pid
[root@VM_0_16_centos keloyang]# lsof |grep docker.sock
systemd       1                 root   26u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615                 root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615                 root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27616           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27616           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27617           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27617           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27618           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27618           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27619           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27619           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27620           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27620           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27622           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27622           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27623           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27623           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27624           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27624           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27625           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27625           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
dockerd   27615 27626           root    3u     unix 0xffff978dfc49e400       0t0   10958867 /var/run/docker.sock
dockerd   27615 27626           root    6u     unix 0xffff978dfcc1dc00       0t0   10959050 /var/run/docker.sock
[root@VM_0_16_centos keloyang]# docker info
Client:
 Debug Mode: false

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info