Open timstclair opened 8 years ago
This is happening very frequently, and if I ssh into the host and issue that command, I can reproduce it pretty reliably (roughly 1 of every 2 or 3 tries fails):
stclair@e2e-cadvisor-coreos-beta ~ $ sudo docker run -d --cpu-shares 2048 --cpuset-cpus 0 --memory 1073741824 --env TEST_VAR=FOO --label bar=baz kubernetes/pause
baef8231b979c04e03bb44d21bb85a5a85369771de97eb57034239f88f3a5524
docker: Error response from daemon: Container command not found or does not exist..
I observed that removing the --memory 1073741824
flag seems to prevent the problem (I couldn't reproduce without that flag).
Setting the memory limit to the minimum reliably gives this error:
stclair@e2e-cadvisor-coreos-beta ~ $ sudo docker run -d --cpu-shares 2048 --cpuset-cpus 0 --memory 4194304 --env TEST_VAR=FOO --label bar=baz kubernetes/pause
6d65e0c7d078990e78440b93944a7c668ac2f2fca649e043624ae19da50fd61a
docker: Error response from daemon: Cannot start container 6d65e0c7d078990e78440b93944a7c668ac2f2fca649e043624ae19da50fd61a: [9] System error: write parent: broken pipe.
And adding 1 reproduces the Container command not found
:
stclair@e2e-cadvisor-coreos-beta ~ $ sudo docker run -d --cpu-shares 2048 --cpuset-cpus 0 --memory 4194305 --env TEST_VAR=FOO --label bar=baz kubernetes/pause
ae7897e3f5ef100582abf512f08c2b7992ab66f7da5c4c399b2c89899f34055e
docker: Error response from daemon: Container command not found or does not exist..
This looks like a bug in the docker version running.
stclair@e2e-cadvisor-coreos-beta ~ $ sudo docker version
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 8acee1b
Built:
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 8acee1b
Built:
OS/Arch: linux/amd64
@Random-Liu did you see anything like this when you were testing Docker v1.10?
/cc @yifan-gu @sjpotter
More details from the docker logs:
$ sudo journalctl -u docker --no-pager
...
May 23 18:56:15 e2e-cadvisor-coreos-beta.c.kubernetes-jenkins.internal dockerd[1335]: time="2016-05-23T18:56:15.323147623Z" level=warning msg="signal: killed"
May 23 18:56:15 e2e-cadvisor-coreos-beta.c.kubernetes-jenkins.internal dockerd[1335]: time="2016-05-23T18:56:15.459091218Z" level=error msg="error locating sandbox id f8ce9a4850614d5bdec61b7d9ba824d52a90df94ea3178d4e7ed72f51f38c23f: sandbox f8ce9a4850614d5bdec61b7d9ba824d52a90df94ea3178d4e7ed72f51f38c23f not found"
May 23 18:56:15 e2e-cadvisor-coreos-beta.c.kubernetes-jenkins.internal dockerd[1335]: time="2016-05-23T18:56:15.459677653Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/9c2855c6d168e5662682d9d2eb858944d1d366010d1ac9667c86ceb8a7209b7b/shm: invalid argument"
May 23 18:56:15 e2e-cadvisor-coreos-beta.c.kubernetes-jenkins.internal dockerd[1335]: time="2016-05-23T18:56:15.459971658Z" level=error msg="Error unmounting container 9c2855c6d168e5662682d9d2eb858944d1d366010d1ac9667c86ceb8a7209b7b: not mounted"
May 23 18:56:15 e2e-cadvisor-coreos-beta.c.kubernetes-jenkins.internal dockerd[1335]: time="2016-05-23T18:56:15.460397447Z" level=error msg="Handler for POST /v1.22/containers/9c2855c6d168e5662682d9d2eb858944d1d366010d1ac9667c86ceb8a7209b7b/start returned error: Container command not found or does not exist."
Sample builds: