search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
754 stars 85 forks source link

No status provided on response: unknown #813

Closed vietthang207 closed 4 years ago

vietthang207 commented 5 years ago

Expected behavior

Actual behavior

docker: Error response from daemon: no status provided on response: unknown.
ERRO[0000] error waiting for container: context canceled

Steps to reproduce the behavior

sudo docker run --rm hello-world

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2
 Built:             Tue Oct  8 00:59:54 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2
  Built:            Tue Oct  8 00:58:28 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.5
  GitCommit:        bb71b10fd8f58240ca47fbb579b9d1028eea7c84
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 19.03.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-1046-gcp
 Operating System: Ubuntu 16.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.853GiB
 Name: dev-core
 ID: ZVTZ:2OZX:VHFQ:OFY7:R7NR:RN32:SO2L:QHAX:5H5R:27FJ:5L2B:C5Z7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.) AWS Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
vietthang207 commented 5 years ago

I’ve managed to fix this by reinstalling another version of runc and then reinstalling docker-ce

apt-get install runc=1.0.0~rc7+git20190403.029124da-0ubuntu1~16.04.4
apt-get install docker-ce

My docker version changed to:

Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2
 Built:             Tue Oct  8 00:59:54 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2
  Built:            Tue Oct  8 00:58:28 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

I think only the containerd version is different.

pmontrasio commented 5 years ago

Same error here after this morning automatic upgrade of Ubuntu

$ dpkg -l "*docker*"|grep ii
ii  docker-ce                  5:19.03.3~3-0~ubuntu-xenial amd64        Docker: the open-source application container engine
ii  docker-ce-cli              5:19.03.3~3-0~ubuntu-xenial amd64        Docker CLI: the open-source application container engine

Any docker run command fails

$ docker run -it --rm jekyll/jekyll jekyll serve --watch --incremental
docker: Error response from daemon: no status provided on response: unknown.

docker ps works. Today's log

$ journalctl -u docker|grep "Oct 10"
Oct 10 09:08:03 mylaptop systemd[1]: Stopping Docker Application Container Engine...
Oct 10 09:08:03 mylaptop dockerd[7640]: time="2019-10-10T09:08:03.826250545+02:00" level=info msg="Processing signal 'terminated'"
Oct 10 09:08:13 mylaptop dockerd[7640]: time="2019-10-10T09:08:13.957022897+02:00" level=info msg="Container f2ad4fc9189efb8945c5cc06e67ac4325f1285b2afe66fa50a87689e54587ea1 failed to exit within 10 seconds of signal 15 - using the force"
Oct 10 09:08:14 mylaptop dockerd[7640]: time="2019-10-10T09:08:14.165366954+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Oct 10 09:08:14 mylaptop dockerd[7640]: time="2019-10-10T09:08:14.357962940+02:00" level=warning msg="f2ad4fc9189efb8945c5cc06e67ac4325f1285b2afe66fa50a87689e54587ea1 cleanup: failed to unmount IPC: umount /var/lib/docker/containers/f2ad4fc9189efb8945c5cc06e67ac4325f1285b2afe66fa50a87689e54587ea1/mounts/shm, flags: 0x2: no such file or directory"
Oct 10 09:08:14 mylaptop dockerd[7640]: time="2019-10-10T09:08:14.511527912+02:00" level=warning error="no such file or directory" method="Cleanup()" storage-driver=aufs
Oct 10 09:08:14 mylaptop dockerd[7640]: time="2019-10-10T09:08:14.512692977+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Oct 10 09:08:14 mylaptop dockerd[7640]: time="2019-10-10T09:08:14.513065354+02:00" level=info msg="Daemon shutdown complete"
Oct 10 09:08:14 mylaptop dockerd[7640]: time="2019-10-10T09:08:14.513093396+02:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Oct 10 09:08:14 mylaptop systemd[1]: Stopped Docker Application Container Engine.
Oct 10 09:09:22 mylaptop systemd[1]: Starting Docker Application Container Engine...
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.521493852+02:00" level=info msg="Starting up"
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.522815400+02:00" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: file exists. Deleting synchronously"
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.523881840+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.523901967+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.523923782+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.523940359+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.543561205+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.543595978+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.543625744+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.543651558+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.577433689+02:00" level=info msg="[graphdriver] using prior storage driver: aufs"
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.577467270+02:00" level=warning msg="[graphdriver] WARNING: the aufs storage-driver is deprecated, and will be removed in a future release"
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.927265195+02:00" level=warning msg="Your kernel does not support swap memory limit"
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.927296673+02:00" level=warning msg="Your kernel does not support cgroup rt period"
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.927305108+02:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Oct 10 09:09:22 mylaptop dockerd[10112]: time="2019-10-10T09:09:22.927523924+02:00" level=info msg="Loading containers: start."
Oct 10 09:09:23 mylaptop dockerd[10112]: time="2019-10-10T09:09:23.131504529+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Oct 10 09:09:23 mylaptop dockerd[10112]: time="2019-10-10T09:09:23.165222672+02:00" level=info msg="Loading containers: done."
Oct 10 09:09:23 mylaptop dockerd[10112]: time="2019-10-10T09:09:23.351703467+02:00" level=info msg="Docker daemon" commit=a872fc2 graphdriver(s)=aufs version=19.03.3
Oct 10 09:09:23 mylaptop dockerd[10112]: time="2019-10-10T09:09:23.351960108+02:00" level=info msg="Daemon has completed initialization"
Oct 10 09:09:23 mylaptop dockerd[10112]: time="2019-10-10T09:09:23.393504949+02:00" level=info msg="API listen on /var/run/docker.sock"
Oct 10 09:09:23 mylaptop systemd[1]: Started Docker Application Container Engine.
Oct 10 10:37:32 mylaptop dockerd[10112]: time="2019-10-10T10:37:32.443359122+02:00" level=error msg="stream copy error: read /proc/self/fd/23: file already closed"
Oct 10 10:37:32 mylaptop dockerd[10112]: time="2019-10-10T10:37:32.657393693+02:00" level=warning msg="28ab96f1acfcca5010b6dc479ed4d34b53486383b368409c5a92c5710c144b7c cleanup: failed to unmount IPC: umount /var/lib/docker/containers/28ab96f1acfcca5010b6dc479ed4d34b53486383b368409c5a92c5710c144b7c/mounts/shm, flags: 0x2: no such file or directory"
Oct 10 10:37:32 mylaptop dockerd[10112]: time="2019-10-10T10:37:32.688689557+02:00" level=error msg="28ab96f1acfcca5010b6dc479ed4d34b53486383b368409c5a92c5710c144b7c cleanup: failed to delete container from containerd: no such container"
Oct 10 10:37:32 mylaptop dockerd[10112]: time="2019-10-10T10:37:32.709273445+02:00" level=error msg="Handler for POST /v1.40/containers/28ab96f1acfcca5010b6dc479ed4d34b53486383b368409c5a92c5710c144b7c/start returned error: no status provided on response: unknown"
Oct 10 10:38:03 mylaptop dockerd[10112]: time="2019-10-10T10:38:03.801997468+02:00" level=error msg="stream copy error: read /proc/self/fd/23: file already closed"
Oct 10 10:38:04 mylaptop dockerd[10112]: time="2019-10-10T10:38:04.059492370+02:00" level=warning msg="20f6f139d97263d46f8db5321243062b1250d35d21860b91cd2c309fbffefe1d cleanup: failed to unmount IPC: umount /var/lib/docker/containers/20f6f139d97263d46f8db5321243062b1250d35d21860b91cd2c309fbffefe1d/mounts/shm, flags: 0x2: no such file or directory"
Oct 10 10:38:04 mylaptop dockerd[10112]: time="2019-10-10T10:38:04.088204241+02:00" level=error msg="20f6f139d97263d46f8db5321243062b1250d35d21860b91cd2c309fbffefe1d cleanup: failed to delete container from containerd: no such container"
Oct 10 10:38:04 mylaptop dockerd[10112]: time="2019-10-10T10:38:04.108013831+02:00" level=error msg="Handler for POST /v1.40/containers/20f6f139d97263d46f8db5321243062b1250d35d21860b91cd2c309fbffefe1d/start returned error: no status provided on response: unknown"
Oct 10 10:40:50 mylaptop systemd[1]: Stopping Docker Application Container Engine...
Oct 10 10:40:50 mylaptop dockerd[10112]: time="2019-10-10T10:40:50.711698685+02:00" level=info msg="Processing signal 'terminated'"
Oct 10 10:40:50 mylaptop dockerd[10112]: time="2019-10-10T10:40:50.742151602+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Oct 10 10:40:50 mylaptop dockerd[10112]: time="2019-10-10T10:40:50.742602840+02:00" level=info msg="Daemon shutdown complete"
Oct 10 10:40:50 mylaptop systemd[1]: Stopped Docker Application Container Engine.
Oct 10 10:40:50 mylaptop systemd[1]: Starting Docker Application Container Engine...
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.810399831+02:00" level=info msg="Starting up"
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.812154741+02:00" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: file exists. Deleting synchronously"
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.817780196+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.818213725+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.818280493+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.818313498+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.821523707+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.821568357+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.821598794+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.821615705+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.862279985+02:00" level=info msg="[graphdriver] using prior storage driver: aufs"
Oct 10 10:40:50 mylaptop dockerd[25515]: time="2019-10-10T10:40:50.862318484+02:00" level=warning msg="[graphdriver] WARNING: the aufs storage-driver is deprecated, and will be removed in a future release"
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.229617689+02:00" level=warning msg="Your kernel does not support swap memory limit"
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.229664262+02:00" level=warning msg="Your kernel does not support cgroup rt period"
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.229677979+02:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.229900898+02:00" level=info msg="Loading containers: start."
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.431784072+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.591615954+02:00" level=info msg="Loading containers: done."
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.654749922+02:00" level=info msg="Docker daemon" commit=a872fc2 graphdriver(s)=aufs version=19.03.3
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.654847498+02:00" level=info msg="Daemon has completed initialization"
Oct 10 10:40:51 mylaptop dockerd[25515]: time="2019-10-10T10:40:51.690258074+02:00" level=info msg="API listen on /var/run/docker.sock"
Oct 10 10:40:51 mylaptop systemd[1]: Started Docker Application Container Engine.
Oct 10 10:40:58 mylaptop dockerd[25515]: time="2019-10-10T10:40:58.938480091+02:00" level=error msg="stream copy error: read /proc/self/fd/23: file already closed"
Oct 10 10:40:59 mylaptop dockerd[25515]: time="2019-10-10T10:40:59.161372218+02:00" level=warning msg="aa112274a7f5e96781014eedda25dd329f5458fe14329118df6a909ed87953cc cleanup: failed to unmount IPC: umount /var/lib/docker/containers/aa112274a7f5e96781014eedda25dd329f5458fe14329118df6a909ed87953cc/mounts/shm, flags: 0x2: no such file or directory"
Oct 10 10:40:59 mylaptop dockerd[25515]: time="2019-10-10T10:40:59.176384407+02:00" level=error msg="aa112274a7f5e96781014eedda25dd329f5458fe14329118df6a909ed87953cc cleanup: failed to delete container from containerd: no such container"
Oct 10 10:40:59 mylaptop dockerd[25515]: time="2019-10-10T10:40:59.190458037+02:00" level=error msg="Handler for POST /v1.40/containers/aa112274a7f5e96781014eedda25dd329f5458fe14329118df6a909ed87953cc/start returned error: no status provided on response: unknown"
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:    16.04
Codename:   xenial

$ uname -a
Linux mylaptop 4.15.0-62-generic #69~16.04.1-Ubuntu SMP Fri Sep 6 02:43:35 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
edorivai commented 5 years ago

@vietthang207 thanks for the quickfix. To clarify a bit, I had to:

  1. Add a deb repo: 
    sudo add-apt-repository 'deb http://security.ubuntu.com/ubuntu xenial-security main universe'
  2. Install runc (note the ~s, github messed them up in @vietthang207 's original post): 
    sudo apt-get install runc=1.0.0~rc7+git20190403.029124da-0ubuntu1~16.04.4
  3. Installing runc causes docker-ce to be removed, hence reinstall it: 
    sudo apt-get install docker-ce
vietthang207 commented 5 years ago
  1. Install runc (note the ~s, github messed them up in @vietthang207 's original post):

I've fixed this, thanks.

pmontrasio commented 4 years ago

It worked for me too, thanks.

This means that docker is broken on every autoupdating Ubuntu 16.04 server and desktop today, unless this workaround is applied?

gglentini commented 4 years ago

The fix doesn't work for me on Ubuntu 18.04.

I see from apt-get logs, the re-installation of docker-ce removes again runs and installs again containerd.io. So when trying again docker run commands, it fails again.

Does anyone know if there's a precise version (as the one proposed by @vietthang207 for 16.04) of runc to be installed on 18.04?

pmontrasio commented 4 years ago

Run this command to list the available versions of runc

$ apt-cache madison runc

The output I got on a 18.04 server is

runc | 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 | http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
runc | 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 | http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
runc | 1.0.0~rc4+dfsg1-6 | http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
runc | 1.0.0~rc4+dfsg1-6 | http://nova.clouds.archive.ubuntu.com/ubuntu bionic/universe Sources
runc | 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 | http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/universe Sources
runc | 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 | http://security.ubuntu.com/ubuntu bionic-security/universe Sources

One of them should be the right one for you. Apparently that server didn't have any problem today.

pypt commented 4 years ago

What worked for me on 16.04 was installing older docker-ce + containerd.io combination:

# apt-cache madison docker-ce | grep 18.09.2
 docker-ce | 5:18.09.2~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages

# apt-cache madison containerd.io | grep 1.2.6
containerd.io |    1.2.6-3 | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages

# apt-get install docker-ce=5:18.09.2~3-0~ubuntu-xenial docker-ce-cli=5:18.09.2~3-0~ubuntu-xenial containerd.io=1.2.6-3
<...>

# $ docker run hello-world
<...>
Hello from Docker!
<...>

(You might wanna go with newer version combinations if they work for you.)

Note to self: in the future, pin not only docker-ce's version but also its container engine too:

# apt-mark hold docker-ce
# apt-mark hold docker-ce-cli
# apt-mark hold containerd.io

Disabling unattended upgrades is not the worst idea either.

petiepooo commented 4 years ago

Installing the prior version of containerd using:

sudo apt-get install containerd.io=1.2.6-3

worked for me. No need to reinstall docker-ce that way.

gitathrun commented 4 years ago

@ventsip I think it is just solved part of some problem, with your re-installation, my docker-cli runs ok now, but it broken the nvidia-docker cmd, now if I run nvidia-docker, it returns

sudo: nvidia-docker: command not found

Any suggestions?

UPDATE:

I think it is the upgrade with nvidia-docker-cli as well, if I changed the cmdline from

sudo nvidia-docker run xxxxx

to

sudo docker run --gpus all xxxxx

It works as expected.

romainbou commented 4 years ago

I had this error today while building a container.

Building xxx
Step 1/10 : FROM node:alpine
...
Step 5/10 : RUN apk add git
 ---> Running in 1a18966ab87a
ERROR: Service 'xxx' failed to build: no status provided on response: unknown

It was just after upgrading my system on Manjaro (Arch). I probably got a docker update. My current version is Docker version 19.03.3-ce, build a872fc2f86

Restarting the docker service fixed the issue:

sudo systemctl restart docker
matpil commented 4 years ago

Installing the prior version of containerd using:

sudo apt-get install containerd.io=1.2.6-3

worked for me. No need to reinstall docker-ce that way.

it works also for me. No docker reinstall needed. thanks

mprasil commented 4 years ago

For me just restarting the docker service solved the problem.

matpil commented 4 years ago

For me just restarting the docker service solved the problem.

lucky guy :) I'try to restart docker.service (and also reload daemon) but nothing

abu-yusuf-dev commented 4 years ago

https://github.com/docker/for-linux/issues/813#issuecomment-540467330 This helped me indeed :)

antoniou79 commented 4 years ago

I also got this issue, after latest update on Ubuntu 16.04.6 LTS The fix that worked for me was downgrading the containerd.io and restarting docker service: sudo apt-get install containerd.io=1.2.6-3 sudo systemctl restart docker

kmjones1979 commented 4 years ago

Installing the prior version of containerd using:

sudo apt-get install containerd.io=1.2.6-3

worked for me. No need to reinstall docker-ce that way.

Thanks for this! Fixed it for me. :)

YxSkywalker commented 4 years ago

sudo apt-get install containerd.io=1.2.6-3

also work for me. After update the containerd, need to restart the docker daemon. systemctl restart docker

extra info: I face this issue when used docker-compose bring up the container. after update the containerd and restart, I have also remove the previous bring up failure containers, and docker-compose up -d again to get rid of " b'failed to listen to abstract unix socket "/containerd-shim/moby.../shim.sock: bind: address already in use: unknown" error

ventsip commented 4 years ago

Hi folks,

Are you sure you intended to tag me to this message?

Thanks, Ventsi

On Oct 11, 2019, at 7:30 PM, Teng Fu notifications@github.com wrote:

@ventsip https://github.com/ventsip I think it is just solved part of some problem, with your re-installation, my docker-cli runs ok now, but it broken the nvidia-docker cmd, now if I run nvidia-docker, it returns

sudo: nvidia-docker: command not found — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/docker/for-linux/issues/813?email_source=notifications&email_token=AFBTRHTF5HW337CYWDF5GVTQOCS2TA5CNFSM4I7JS37KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBAQY4Y#issuecomment-541133939, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBTRHSJ7K3PP37QF2F6FETQOCS2TANCNFSM4I7JS37A.

besnik commented 4 years ago

As per petiepooo comment downgrading to prior version of containerd.io and restarting docker service worked for me (Ubuntu 16.04.6 LTS)

sudo apt-get install containerd.io=1.2.6-3
sudo systemctl restart docker
gitathrun commented 4 years ago

@ventsip I think it is just solved part of some problem, with your re-installation, my docker-cli runs ok now, but it broken the nvidia-docker cmd, now if I run nvidia-docker, it returns

sudo: nvidia-docker: command not found

Any suggestions?

UPDATE:

I think it is the upgrade with nvidia-docker-cli as well, if I changed the cmdline from

sudo nvidia-docker run xxxxx

to

sudo docker run --gpus all xxxxx

It works as expected.

@ventsip Sorry, I think I got it wrong, I wish to @ vietthang207 , GitHub suggested you but I forgot to check.

ericblade commented 4 years ago

... i just followed the instructions here https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04

which updated my docker to 19.03.03 from 19.03.01 or some such, and now it's working at least for hello-world, and my docker containers are starting to build, without failing mysteriously after the FROM command.

fauno commented 4 years ago

on archlinux, with docker 19.03.2 downgrading containerd from 1.3.0 to 1.2.8 worked again

aMozejko1 commented 4 years ago

the following just worked for me after upgrading to docker 19.03.3. ub18.04

apt-get install runc=1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2
apt-get install docker-ce
MilindaLaknath commented 4 years ago

on archlinux, with docker 19.03.2 downgrading containerd from 1.3.0 to 1.2.8 worked again

How you did it? I faced the same issue in Manjaro.

MilindaLaknath commented 4 years ago

on archlinux, with docker 19.03.2 downgrading containerd from 1.3.0 to 1.2.8 worked again

Thanks. It worked. sudo pacman -U /var/cache/pacman/pkg/containerd-1.2.8-1-x86_64.pkg.tar.xz and Restarted the machine.

peter279k commented 4 years ago

I also have the same issue on my Ubuntu 16.04.

After I reinstall the docker-ce package and reboot my machine, it's worked successfully now :).

cGuille commented 4 years ago

I have the same issue on Kubuntu 16.04. I downgraded the containerd.io package and then I put it on hold so that it does not upgrade automatically:

# Downgrade to a working version:
sudo apt-get install containerd.io=1.2.6-3

# Restart the docker service:
sudo service docker restart

# Hold the package so that I can upgrade my packages without upgrading containerd.io to the broken version:
sudo apt-mark hold containerd.io

You can show which packages are on hold and unhold them like this:

apt-mark showhold
sudo apt-mark unhold containerd.io
rlipscombe commented 4 years ago

On Linux Mint 18.3 (i.e. Ubuntu 16.04), I had to do the following:

sudo apt-get install containerd.io=1.2.6-3 \
        docker-ce=5:19.03.1~3-0~ubuntu-xenial \
        docker-ce-cli=5:19.03.1~3-0~ubuntu-xenial

...and then reboot

cleeland commented 4 years ago

apt-get install containerd.io=1.2.6-3 worked for me, not even requiring a daemon restart.

fauno commented 4 years ago

Milinda Laknath Arambawela notifications@github.com writes:

on archlinux, with docker 19.03.2 downgrading containerd from 1.3.0 to 1.2.8 worked again

Thanks. It worked. sudo pacman -U /var/cache/pacman/pkg/containerd-1.2.8-1-x86_64.pkg.tar.xz and Restarted the machine.

if you happen to have cleaned the cache you can use https://archive.archlinux.org/ like i did :)

-- :O

naanlizard commented 4 years ago

Cleeland's downgrade suggestion for containerd.io worked for me on ubuntu 18.04

Honestly the only reason I use ubuntu is so broken stuff doesn't make it to the repos, what's the point if bugs make it through?

cleeland commented 4 years ago

Cleeland's downgrade suggestion

Glad it worked, but cannot take credit; swiped that suggestion from @petiepooo in https://github.com/docker/for-linux/issues/813#issuecomment-541126811

cGuille commented 4 years ago

Honestly the only reason I use ubuntu is so broken stuff doesn't make it to the repos, what's the point if bugs make it through?

This repo is not maintained by Ubuntu.

naanlizard commented 4 years ago

Good point, I'm an idiot. That's what I get for commenting before my coffee

yingw787 commented 4 years ago

This fix

I’ve managed to fix this by reinstalling another version of runc and then reinstalling docker-ce

apt-get install runc=1.0.0~rc7+git20190403.029124da-0ubuntu1~16.04.4
apt-get install docker-ce

My docker version changed to:

Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2
 Built:             Tue Oct  8 00:59:54 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2
  Built:            Tue Oct  8 00:58:28 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

I think only the containerd version is different.

This fixed it for me! Thanks very much :)

rlipscombe commented 4 years ago

apt-get install containerd.io=1.2.6-3 worked for me, not even requiring a daemon restart.

That's nice. It didn't work for me, including restarting the daemon a whole bunch of times.

trasba commented 4 years ago

Same issue here.

apt-get install containerd.io=1.2.6-3 worked for me, not even requiring a daemon restart.

Fixed it, the daemon was stopped after the downgrade though.

vvidovic commented 4 years ago

Same issue here, fixed through synaptic package manager... Ubuntu and kernel versions: 4.15.0-64-generic #73~16.04.1-Ubuntu In Synaptic it looked like containerd.io 1.2.10-2 is installed but docker was showing containerd 1.2.6 (containerd version 1.2.6-0ubuntu1~16.04.3 wasn't looking like it was installed in Synaptic) In Synaptic I did following:

MaesterZ commented 4 years ago

docker build triggered the daemon errors:

level=error msg="stream copy error: read /proc/self/fd/22: file already closed"
level=error msg="stream copy error: read /proc/self/fd/23: file already closed"
level=warning msg="c37d84784eead6ac3f798bb8397edfa789831069247da6b8a2b304de81665cfd cleanup: failed to unmount IPC: umount /var/lib/docker/containers/1a25aa73be4d7f4aecd87ad65ccfdb73fa71c77b59ee5f806d50bc841eba5434/mounts/shm, flags: 0x2: no such file or directory"
level=error msg="c37d84784eead6ac3f798bb8397edfa789831069247da6b8a2b304de81665cfd cleanup: failed to delete container from  containerd: no such container"

Edit: runc package is not installed on my system

thaJeztah commented 4 years ago

We're trying to reproduce the issue, but so far have not managed to; doing an install on ubuntu 16.04.6 (including containerd.io 1.2.10, which comes with runc 1.0.0-rc8-dev) seems to work correctly;

I see some people mention installing/downgrading the runc package. That package is not shipped by Docker. runc has not yet reached a stable release (it's still pre-v1.0.0), and for that reason, the version of runc is bundled in the containerd.io package, because different combinations can be incompatible. Once runc reaches 1.0.0, docker will likely ship it as a separate package.

Currently, installing the runc package will install the package as shipped by Ubuntu, and may cause a conflict with the containerd.io package.

From an install on Ubuntu 16.04;

Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2
 Built:             Tue Oct  8 00:59:54 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2
  Built:            Tue Oct  8 00:58:28 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.4.0-165-generic
 Operating System: Ubuntu 16.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.796GiB
 Name: ubuntu-s-4vcpu-8gb-ams3-01
 ID: AAC4:4YAW:QWTR:NHDI:3E54:N2IO:Q6GI:VVJG:WD5S:ZSI4:NJGO:A2JF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Note that this is on a 4.4.0 kernel. Kernel 5.2 is not a default kernel for Ubuntu 16.04, and is not a tested / supported configuration.

I do see people reporting with kernel 4.15; I'll see if I can find a machine with that kernel version, in case it's an issue with that kernel.

If people that are having this issue (and didn't include it in their comment yet) post their kernel version, to help finding if it's related to that, that would be helpful (uname -a should do the trick)

thaJeztah commented 4 years ago

Still not able to reproduce after installing the 4.15 kernel (https://wiki.ubuntu.com/Kernel/RollingLTSEnablementStack)

apt-get install -y --install-recommends linux-generic-hwe-16.04

reboot

uname -a
Linux ubuntu-s-4vcpu-8gb-ams3-01 4.15.0-65-generic #74~16.04.1-Ubuntu SMP Wed Sep 18 09:51:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
thaJeztah commented 4 years ago

/cc @seemethere @crosbymichael PTAL. Perhaps you have any ideas?

cleeland commented 4 years ago

I see some people mention installing/downgrading the runc package. That package is not shipped by Docker.

FWIW, I do not have runc installed as a separate package.

I am running:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

Relevant packages:

> dpkg -l | egrep '(docker|container)'
ii  containerd.io                                               1.2.6-3                                                 amd64        An open and reliable container runtime
ii  docker-ce                                                   5:19.03.3~3-0~ubuntu-xenial                             amd64        Docker: the open-source application container engine
ii  docker-ce-cli                                               5:19.03.3~3-0~ubuntu-xenial                             amd64        Docker CLI: the open-source application container engine
Berbe commented 4 years ago

Had the same problem on Devuan Beowulf, using the docker package from Debian Buster.

After having attempted to stop a container, it ended up having exited with status code 128.

Attempting to restart it yielded: failed to listen to abstract unix socket "/containerd-shim/moby/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/shim.sock": listen unix /containerd-shim/moby/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/shim.sock: bind: address already in use: unknown

After having deleted it, trying to recreate it: no status provided on response: unknown

From the daemon logs:

time="2019-10-16T16:41:18.072541633+02:00" level=info msg="Container 3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30 failed to exit within 10 seconds of signal 15 - using the force"
time="2019-10-16T16:41:19.107664777+02:00" level=warning msg="unmount task rootfs" error="no such file or directory" id=3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30 path="/var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30"
time="2019-10-16T16:41:19.141863286+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-10-16T16:41:19.141943668+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-10-16T16:41:20.175667036+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/shim.sock" debug=false pid=11182
time="2019-10-16T16:41:22.267373586+02:00" level=error msg="failed to kill shim" error="no status provided on response: unknown"
time="2019-10-16T16:41:22.268000833+02:00" level=error msg="stream copy error: read /proc/self/fd/11: file already closed"
time="2019-10-16T16:41:22.780019450+02:00" level=error msg="3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30 cleanup: failed to delete container from containerd: no such container"
time="2019-10-16T16:41:22.809681274+02:00" level=error msg="Handler for POST /v1.35/containers/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/restart returned error: Cannot restart container 3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30: no status provided on response: unknown"
time="2019-10-16T16:42:25.454902309+02:00" level=error msg="stream copy error: reading from a closed fifo"
time="2019-10-16T16:42:26.495264603+02:00" level=error msg="3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30 cleanup: failed to delete container from containerd: no such container"
time="2019-10-16T16:42:26.495343100+02:00" level=error msg="Handler for POST /v1.35/containers/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/start returned error: failed to listen to abstract unix socket \"/containerd-shim/moby/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/shim.sock\": listen unix \x00/containerd-shim/moby/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/shim.sock: bind: address already in use: unknown"
time="2019-10-16T16:45:56.486969391+02:00" level=error msg="stream copy error: reading from a closed fifo"
time="2019-10-16T16:45:57.197774820+02:00" level=error msg="3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30 cleanup: failed to delete container from containerd: no such container"
time="2019-10-16T16:45:57.197834448+02:00" level=error msg="Handler for POST /v1.35/containers/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/start returned error: failed to listen to abstract unix socket \"/containerd-shim/moby/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/shim.sock\": listen unix \x00/containerd-shim/moby/3cfd9b89b27a73028abeb8f3bcf394b2ade1c8a8e5c6389dd1e538ad4ab4ec30/shim.sock: bind: address already in use: unknown"
time="2019-10-16T16:47:05.693850822+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/2dea922699fb57c665cbe3c5a1ee339beb236e64ae0bd0a703bcd2accca781fb/shim.sock" debug=false pid=13450
time="2019-10-16T16:47:06.095904892+02:00" level=error msg="failed to kill shim" error="no status provided on response: unknown"
time="2019-10-16T16:47:06.096342159+02:00" level=error msg="stream copy error: read /proc/self/fd/68: file already closed"
time="2019-10-16T16:47:06.515996138+02:00" level=error msg="2dea922699fb57c665cbe3c5a1ee339beb236e64ae0bd0a703bcd2accca781fb cleanup: failed to delete container from containerd: no such container"
time="2019-10-16T16:47:06.516061221+02:00" level=error msg="Handler for POST /v1.35/containers/2dea922699fb57c665cbe3c5a1ee339beb236e64ae0bd0a703bcd2accca781fb/start returned error: no status provided on response: unknown"

Reinstalling the docker-ce package solved the problem. Version info is now:

Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2f86
 Built:             Tue Oct  8 00:58:05 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2f86
  Built:            Tue Oct  8 00:56:39 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Note I never installed the "corrupted" package from #812 while the package with bad tags was offered, hence problems do not come from there. I did install containerd v1.2.10 when properly released in the immediate aftermath.

naanlizard commented 4 years ago

FWIW Berbe I fixed that by just doing docker-compose down and then docker-compose up -d

and everything worked fine (after I downgraded the package)

Berbe commented 4 years ago

I did not downgrade anything, but container restart & recreation was not working, as shown in the output/logs offered.

Package reinstall did the trick, but on the contrary of @vietthang207, I did not take a sample of docker version output beforehand.

thaJeztah commented 4 years ago

I may be a lousy answer, but if you're running into this issue, be sure to also try to reboot the machine ("have you tried turning it off and on again?"); we've had cases where upgraded packages did not take effect until a reboot, and docker was not working correctly in the intermediate state.

That said, there could definitely be an issue somewhere, perhaps due to a combination of packages. Having a way to consistently reproduce it would be the first step to solve that.

ericblade commented 4 years ago

I picked up this problem simply by doing an apt-get upgrade a couple of days ago. No reboots were performed, this machine only ever gets rebooted when it's network hardware stops functioning out of the blue a couple times a year maybe. Until this problem started, so I went searching for a different docker setup, to make sure i had the latest, I had no alternative repos installed except for Vivaldi.

tiborvass commented 4 years ago

Thanks folks, we're looking into this.