docker exec -it d2fecc07a43a bash rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:75: starting setns process caused "fork/exec /proc/self/exe: no such file or directory""

[cmays20]

Description One of our system engineers was going to debug a container, and noticed he couldn't exec into it. We then tried to exec into any container on the machine, and we found that none of them could be exec'd into. We use Mesosphere DC/OS for container orchestration. We have seen this issue in the past but have always had to resort to just bouncing the daemon. Let me know what other info is needed.

Steps to reproduce the issue: We don't know how to reproduce as it seems like it just started to happen.

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally): We have seen this issue happen in the past without any clear pattern. We do know that restarting the docker daemon does resolve this, however in a production environment where we have 50 containers running we don't want to have to resort to that.

Output of docker version:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:25 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:25 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 50
 Running: 49
 Paused: 0
 Stopped: 1
Images: 58
Server Version: 17.05.0-ce
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 122GB
 Data Space Total: 153GB
 Data Space Available: 31GB
 Metadata Space Used: 25.87MB
 Metadata Space Total: 1.606GB
 Metadata Space Available: 1.581GB
 Thin Pool Minimum Free Space: 15.3GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 503.5GiB
Name: dchicmdocker07.cme.in.here.com
ID: FFTR:3EPU:6DVD:MNXO:HLCT:LUPA:BFQV:GGVW:FI3E:IXJL:5OS6:U7XK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.): The machines are bare metal in our datacenter

[thaJeztah]

Docker 17.05 has reached end of life in June; are you still seeing this issue on a current version of Docker?

[manishppatel345]

Due to some compatibility with one of our docker orchestration tool that we are using to manage containers ,so we can not go with latest version. We are planning to downgrade to docker 1.13 ,so is this issue ever appeared in docker 1.13 ?.

[thaJeztah]

Do you know what the incompatibilities are? Note that older API versions are supported by every release, so (e.g) the current docker release has full compatibility for every older API version (up until Docker 1.0)

[thaJeztah]

Also, because of that, it's possible to run an up-to-date Docker Engine (dockerd), but use an older version of the CLI; the CLI will perform API-version negotiation, so a docker 1.13 or docker 17.03 CLI can work with a Docker 17.09 daemon

[KevinTHU]

@cubfan20 have you used docker cp ? we have the same problem with using docker cp to copy file into container, and after we do not use docker cp, problem ok!

[grodrigo]

In my case with Debian, I couldn't even stop a container, always I get this message, and restart of docker service didn't work. In my /etc/docker/daemon.json I have added the live-restore option, I delete it and restat docker service and everithing went ok again. So, check daemon configuration. Hope this helps :)

[thaJeztah]

closing, as this went stale