Error response from daemon: i/o timeout

aliaugmenta commented 2 years ago

[x] I have tried with the latest version of Docker Desktop
[x] I have tried disabling enabled experimental features
[x] I have uploaded Diagnostics
Diagnostics ID: 6B1FD769-5648-4796-85BD-C7F5506BD940/20220819032124

Actual behavior

docker stats returns Error response from daemon: i/o timeout and most of the other docker operation in my code is much slower than before upgrading to 4.11. Sometimes, it seems that the docker daemon is paused and none of the container progress.

Expected behavior

docker stats should work!

Information

Is it reproducible? yes, on the same machine and also on multiple different machines.
Is the problem new? yes
Did the problem appear with an update? yes , after updating to the most recent version 4.11.1
A reproducible case if this is a bug, Dockerfiles with reproduction inside is best.

Windows Version: 10
Docker Desktop Version: 14.11.1
WSL2 or Hyper-V backend? WSL2
Are you running inside a virtualized Windows e.g. on a cloud server or a VM: No

Output of `& "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" check`

[2022-08-19T03:16:20.648396600Z][com.docker.diagnose.exe][I] set path configuration to OnHost
Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[FAIL] DD0031: does the Docker API work? Get "http://%2F%2F.%2Fpipe%2Fdocker_engine_linux/v1.24/containers/json?limit=0": context deadline exceeded
[FAIL] DD0004: is the Docker engine running? Get "http://ipc/docker": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:16:25.597444200Z][com.docker.diagnose.exe][I] ipc.NewClient: 864dca3d-com.docker.diagnose -> \\.\pipe\dockerLifecycleServer VMDockerdAPI
[linuxkit/pkg/desktop-host-tools/pkg/client.NewClientForPath(...)
[       linuxkit/pkg/desktop-host-tools/pkg/client/client.go:59
[linuxkit/pkg/desktop-host-tools/pkg/client.NewClient({0x1595167, 0x13})
[       linuxkit/pkg/desktop-host-tools/pkg/client/client.go:53 +0x99
[common/pkg/diagkit/gather/diagnose.isDockerEngineRunning()
[       common/pkg/diagkit/gather/diagnose/dockerd.go:21 +0x29
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x1b1cde0)
[       common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0x1b1cde0)
[       common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x13f2d77?, 0x1b1cde0)
[       common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x2, 0x1b1cde0, 0xc0002df728)
[       common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x1b1ce60?, 0xc0002df728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0xcb?, 0xc0002df728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x14d3a40?, 0xc0002df890)
[       common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0x1b1d5e0, 0xd913495f00000010?, {0xc0002dfb20, 0x1, 0x1})
[       common/pkg/diagkit/gather/diagnose/run.go:16 +0x1ca
[main.checkCmd({0xc00007a3d0?, 0xc00007a3d0?, 0x4?}, {0x0, 0x0})
[       common/cmd/com.docker.diagnose/main.go:133 +0x105
[main.main()
[       common/cmd/com.docker.diagnose/main.go:99 +0x288
[2022-08-19T03:16:25.600113000Z][com.docker.diagnose.exe][I] (349c4c45) 864dca3d-com.docker.diagnose C->S VMDockerdAPI GET /docker
[2022-08-19T03:16:35.606049700Z][com.docker.diagnose.exe][W] (349c4c45) 864dca3d-com.docker.diagnose C<-S NoResponse GET /docker (10.0059367s): Get "http://ipc/docker": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:16:35.606640100Z][com.docker.diagnose.exe][I] (349c4c45-1) 864dca3d-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2022-08-19T03:16:36.614088900Z][com.docker.diagnose.exe][W] (349c4c45-1) 864dca3d-com.docker.diagnose C<-S NoResponse GET /ping (1.0074488s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:16:37.625823000Z][com.docker.diagnose.exe][I] (349c4c45-2) 864dca3d-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2022-08-19T03:16:38.629242600Z][com.docker.diagnose.exe][W] (349c4c45-2) 864dca3d-com.docker.diagnose C<-S NoResponse GET /ping (1.0034196s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:16:39.641270800Z][com.docker.diagnose.exe][I] (349c4c45-3) 864dca3d-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2022-08-19T03:16:40.659864600Z][com.docker.diagnose.exe][W] (349c4c45-3) 864dca3d-com.docker.diagnose C<-S NoResponse GET /ping (1.0185938s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:16:41.663592800Z][com.docker.diagnose.exe][I] (349c4c45-4) 864dca3d-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2022-08-19T03:16:42.681348700Z][com.docker.diagnose.exe][W] (349c4c45-4) 864dca3d-com.docker.diagnose C<-S NoResponse GET /ping (1.0176958s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[FAIL] DD0011: are the LinuxKit services running? failed to ping VM diagnosticsd with error: Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:16:42.694731200Z][com.docker.diagnose.exe][I] ipc.NewClient: 24bfc533-diagnose -> \\.\pipe\dockerDiagnosticd diagnosticsd
[common/pkg/diagkit/gather/diagnose.glob..func14()
[       common/pkg/diagkit/gather/diagnose/linuxkit.go:18 +0x8b
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x1b1cd60)
[       common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0x1b1cd60)
[       common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x13f2d77?, 0x1b1cd60)
[       common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x3, 0x1b1cd60, 0xc00061f728)
[       common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x2, 0x1b1cde0?, 0xc00061f728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x1b1ce60?, 0xc00061f728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0xcb?, 0xc00061f728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x14d3a40?, 0xc0002df890)
[       common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0x1b1d5e0, 0xd913495f00000010?, {0xc0002dfb20, 0x1, 0x1})
[       common/pkg/diagkit/gather/diagnose/run.go:16 +0x1ca
[main.checkCmd({0xc00007a3d0?, 0xc00007a3d0?, 0x4?}, {0x0, 0x0})
[       common/cmd/com.docker.diagnose/main.go:133 +0x105
[main.main()
[       common/cmd/com.docker.diagnose/main.go:99 +0x288
[2022-08-19T03:16:42.695216500Z][com.docker.diagnose.exe][I] (7799b48d) 24bfc533-diagnose C->S diagnosticsd GET /ping
[2022-08-19T03:16:43.698768200Z][com.docker.diagnose.exe][W] (7799b48d) 24bfc533-diagnose C<-S NoResponse GET /ping (1.0035517s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0001: is the application running?
[SKIP] DD0018: does the host support virtualization?
[PASS] DD0002: does the bootloader have virtualization enabled?
[PASS] DD0017: can a VM be started?
[PASS] DD0024: is WSL installed?
[PASS] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0015: are the binary symlinks installed?
Error response from daemon: i/o timeout
[FAIL] DD0003: is the Docker CLI working? exit status 1
[PASS] DD0013: is the $PATH ok?
[PASS] DD0005: is the user in the docker-users group?
[PASS] DD0007: is the backend responding?
[FAIL] DD0014: are the backend processes running? 1 error occurred:
        * vpnkit.exe is not running

[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[PASS] DD0006: is the Docker Desktop Service responding?
[FAIL] DD0012: is the VM networking working? network checks failed: Post "http://ipc/check-network-connectivity": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:17:00.034723700Z][com.docker.diagnose.exe][I] ipc.NewClient: d9a8be01-diagnose-network -> \\.\pipe\dockerDiagnosticd diagnosticsd
[common/pkg/diagkit/gather/diagnose.runIsVMNetworkingOK()
[       common/pkg/diagkit/gather/diagnose/network.go:34 +0xd9
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x1b1cfe0)
[       common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0x1b1cfe0)
[       common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x2?, 0x1b1cfe0)
[       common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x1b1cfe0, 0xc0004e3728)
[       common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0xcb?, 0xc0004e3728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x14d3a40?, 0xc0002df890)
[       common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0x1b1d5e0, 0xd913495f00000010?, {0xc0002dfb20, 0x1, 0x1})
[       common/pkg/diagkit/gather/diagnose/run.go:16 +0x1ca
[main.checkCmd({0xc00007a3d0?, 0xc00007a3d0?, 0x4?}, {0x0, 0x0})
[       common/cmd/com.docker.diagnose/main.go:133 +0x105
[main.main()
[       common/cmd/com.docker.diagnose/main.go:99 +0x288
[2022-08-19T03:17:00.034723700Z][com.docker.diagnose.exe][I] (a6c1f3e5) d9a8be01-diagnose-network C->S diagnosticsd POST /check-network-connectivity: {"ips":["192.168.0.135","172.20.224.1","172.24.224.1"]}
[2022-08-19T03:17:05.035390200Z][com.docker.diagnose.exe][W] (a6c1f3e5) d9a8be01-diagnose-network C<-S NoResponse POST /check-network-connectivity (5.0006665s): Post "http://ipc/check-network-connectivity": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:17:05.035390200Z][com.docker.diagnose.exe][I] (a6c1f3e5-1) d9a8be01-diagnose-network C->S diagnosticsd GET /ping
[2022-08-19T03:17:06.045051600Z][com.docker.diagnose.exe][W] (a6c1f3e5-1) d9a8be01-diagnose-network C<-S NoResponse GET /ping (1.0096614s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:17:07.058384500Z][com.docker.diagnose.exe][I] (a6c1f3e5-2) d9a8be01-diagnose-network C->S diagnosticsd GET /ping
[2022-08-19T03:17:08.073496400Z][com.docker.diagnose.exe][W] (a6c1f3e5-2) d9a8be01-diagnose-network C<-S NoResponse GET /ping (1.015138s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:17:09.084203100Z][com.docker.diagnose.exe][I] (a6c1f3e5-3) d9a8be01-diagnose-network C->S diagnosticsd GET /ping
[2022-08-19T03:17:10.094560500Z][com.docker.diagnose.exe][W] (a6c1f3e5-3) d9a8be01-diagnose-network C<-S NoResponse GET /ping (1.0103574s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-19T03:17:11.106130300Z][com.docker.diagnose.exe][I] (a6c1f3e5-4) d9a8be01-diagnose-network C->S diagnosticsd GET /ping
[2022-08-19T03:17:12.117699500Z][com.docker.diagnose.exe][W] (a6c1f3e5-4) d9a8be01-diagnose-network C<-S NoResponse GET /ping (1.0115692s): Get "http://ipc/ping": i/o timeout (Client.Timeout exceeded while awaiting headers)

[FAIL] DD0032: do Docker networks overlap with host IPs? Get "http://%2F%2F.%2Fpipe%2Fdocker_engine_linux/v1.24/networks": context deadline exceeded
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?

Please investigate the following 3 issues:

1 : The test: are the LinuxKit services running?
    Failed with: failed to ping VM diagnosticsd with error: Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The Docker engine runs inside a Linux VM as a service. Therefore the services must have started.

2 : The test: are the backend processes running?
    Failed with: 1 error occurred:
        * vpnkit.exe is not running

Not all of the backend processes are running.

3 : The test: is the VM networking working?
    Failed with: network checks failed: Post "http://ipc/check-network-connectivity": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

VM seems to have a network connectivity issue. Please check your host firewall and anti-virus settings in case they are blocking the VM.

Steps to reproduce the behavior

I just run a few containers with some load running in them.

FlowsenAusMonotown commented 2 years ago

Seems that we have nearly the same issue with Docker Desktop 4.11. We start a new container, which mostly do docker commands like (docker inspect, docker load, docker create container, docker remove container). In 9/10 cases docker stops in total and we get the "Error response from daemon: i/o timeout" error. Only restarting the whole docker deamon helps to get out of this situation.

jazdw commented 2 years ago

I'm also encountering this. Sometimes Docker just sporadically locks up while I am building some container images, sometimes my build completes just fine. I have to restart Docker for it to work again (for a while).

Seems like the only solution is to downgrade.

This is my diagnostics ID: F97CE7C4-D8FC-41F6-B91C-4C3C8E9D6CAB/20220823205904 Diagnose output:

"C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" check
[2022-08-23T21:07:09.496567700Z][com.docker.diagnose.exe][I] set path configuration to OnHost
Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[FAIL] DD0031: does the Docker API work? Get "http://%2F%2F.%2Fpipe%2Fdocker_engine_linux/v1.24/containers/json?limit=0": context deadline exceeded
[FAIL] DD0004: is the Docker engine running? Get "http://ipc/docker": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-23T21:07:13.988380500Z][com.docker.diagnose.exe][I] ipc.NewClient: 69e5b754-com.docker.diagnose -> \\.\pipe\dockerLifecycleServer VMDockerdAPI
[linuxkit/pkg/desktop-host-tools/pkg/client.NewClientForPath(...)
[       linuxkit/pkg/desktop-host-tools/pkg/client/client.go:59
[linuxkit/pkg/desktop-host-tools/pkg/client.NewClient({0x12c5167, 0x13})
[       linuxkit/pkg/desktop-host-tools/pkg/client/client.go:53 +0x99
[common/pkg/diagkit/gather/diagnose.isDockerEngineRunning()
[       common/pkg/diagkit/gather/diagnose/dockerd.go:21 +0x29
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x184cde0)
[       common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0x184cde0)
[       common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x1122d77?, 0x184cde0)
[       common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x2, 0x184cde0, 0xc00035f728)
[       common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x184ce60?, 0xc00035f728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0xcb?, 0xc00035f728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x1203a40?, 0xc00035f890)
[       common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0x184d5e0, 0xfdf0fd0300000010?, {0xc00035fb20, 0x1, 0x1})
[       common/pkg/diagkit/gather/diagnose/run.go:16 +0x1ca
[main.checkCmd({0xc00007a3d0?, 0xc00007a3d0?, 0x4?}, {0x0, 0x0})
[       common/cmd/com.docker.diagnose/main.go:133 +0x105
[main.main()
[       common/cmd/com.docker.diagnose/main.go:99 +0x288
[2022-08-23T21:07:13.989983800Z][com.docker.diagnose.exe][I] (c7b22946) 69e5b754-com.docker.diagnose C->S VMDockerdAPI GET /docker
[2022-08-23T21:07:24.000901000Z][com.docker.diagnose.exe][W] (c7b22946) 69e5b754-com.docker.diagnose C<-S NoResponse GET /docker (10.0109172s): Get "http://ipc/docker": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-23T21:07:24.000901000Z][com.docker.diagnose.exe][I] (c7b22946-1) 69e5b754-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2022-08-23T21:07:25.013590200Z][com.docker.diagnose.exe][W] (c7b22946-1) 69e5b754-com.docker.diagnose C<-S NoResponse GET /ping (1.0126892s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[2022-08-23T21:07:26.025749600Z][com.docker.diagnose.exe][I] (c7b22946-2) 69e5b754-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2022-08-23T21:07:27.033802100Z][com.docker.diagnose.exe][W] (c7b22946-2) 69e5b754-com.docker.diagnose C<-S NoResponse GET /ping (1.0080525s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[2022-08-23T21:07:28.048334900Z][com.docker.diagnose.exe][I] (c7b22946-3) 69e5b754-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2022-08-23T21:07:29.059543700Z][com.docker.diagnose.exe][W] (c7b22946-3) 69e5b754-com.docker.diagnose C<-S NoResponse GET /ping (1.0112088s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[2022-08-23T21:07:30.067713300Z][com.docker.diagnose.exe][I] (c7b22946-4) 69e5b754-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2022-08-23T21:07:31.074870300Z][com.docker.diagnose.exe][W] (c7b22946-4) 69e5b754-com.docker.diagnose C<-S NoResponse GET /ping (1.007157s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[FAIL] DD0011: are the LinuxKit services running? failed to ping VM diagnosticsd with error: Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-23T21:07:31.078174100Z][com.docker.diagnose.exe][I] ipc.NewClient: f36436b0-diagnose -> \\.\pipe\dockerDiagnosticd diagnosticsd
[common/pkg/diagkit/gather/diagnose.glob..func14()
[       common/pkg/diagkit/gather/diagnose/linuxkit.go:18 +0x8b
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x184cd60)
[       common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0x184cd60)
[       common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x1122d77?, 0x184cd60)
[       common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x3, 0x184cd60, 0xc00059f728)
[       common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x2, 0x184cde0?, 0xc00059f728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x184ce60?, 0xc00059f728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0xcb?, 0xc00059f728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x1203a40?, 0xc00035f890)
[       common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0x184d5e0, 0xfdf0fd0300000010?, {0xc00035fb20, 0x1, 0x1})
[       common/pkg/diagkit/gather/diagnose/run.go:16 +0x1ca
[main.checkCmd({0xc00007a3d0?, 0xc00007a3d0?, 0x4?}, {0x0, 0x0})
[       common/cmd/com.docker.diagnose/main.go:133 +0x105
[main.main()
[       common/cmd/com.docker.diagnose/main.go:99 +0x288
[2022-08-23T21:07:31.078680600Z][com.docker.diagnose.exe][I] (853d77cd) f36436b0-diagnose C->S diagnosticsd GET /ping
[2022-08-23T21:07:32.084880000Z][com.docker.diagnose.exe][W] (853d77cd) f36436b0-diagnose C<-S NoResponse GET /ping (1.0061994s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0001: is the application running?
[SKIP] DD0018: does the host support virtualization?
[PASS] DD0002: does the bootloader have virtualization enabled?
[PASS] DD0017: can a VM be started?
[PASS] DD0024: is WSL installed?
[PASS] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0015: are the binary symlinks installed?
Error response from daemon: i/o timeout
[FAIL] DD0003: is the Docker CLI working? exit status 1
[PASS] DD0013: is the $PATH ok?
[PASS] DD0005: is the user in the docker-users group?
[PASS] DD0007: is the backend responding?
[FAIL] DD0014: are the backend processes running? 1 error occurred:
        * vpnkit.exe is not running

[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[PASS] DD0006: is the Docker Desktop Service responding?
[FAIL] DD0012: is the VM networking working? network checks failed: Post "http://ipc/check-network-connectivity": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-23T21:07:43.617672000Z][com.docker.diagnose.exe][I] ipc.NewClient: 681d1332-diagnose-network -> \\.\pipe\dockerDiagnosticd diagnosticsd
[common/pkg/diagkit/gather/diagnose.runIsVMNetworkingOK()
[       common/pkg/diagkit/gather/diagnose/network.go:34 +0xd9
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x184cfe0)
[       common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0x184cfe0)
[       common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x2?, 0x184cfe0)
[       common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x184cfe0, 0xc000487728)
[       common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0xcb?, 0xc000487728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x1203a40?, 0xc00035f890)
[       common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0x184d5e0, 0xfdf0fd0300000010?, {0xc00035fb20, 0x1, 0x1})
[       common/pkg/diagkit/gather/diagnose/run.go:16 +0x1ca
[main.checkCmd({0xc00007a3d0?, 0xc00007a3d0?, 0x4?}, {0x0, 0x0})
[       common/cmd/com.docker.diagnose/main.go:133 +0x105
[main.main()
[       common/cmd/com.docker.diagnose/main.go:99 +0x288
[2022-08-23T21:07:43.618289200Z][com.docker.diagnose.exe][I] (933ab570) 681d1332-diagnose-network C->S diagnosticsd POST /check-network-connectivity: {"ips":["192.168.56.1","192.168.86.41","172.21.192.1"]}
[2022-08-23T21:07:48.632659800Z][com.docker.diagnose.exe][W] (933ab570) 681d1332-diagnose-network C<-S NoResponse POST /check-network-connectivity (5.0149878s): Post "http://ipc/check-network-connectivity": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-23T21:07:48.632918000Z][com.docker.diagnose.exe][I] (933ab570-1) 681d1332-diagnose-network C->S diagnosticsd GET /ping
[2022-08-23T21:07:49.637645000Z][com.docker.diagnose.exe][W] (933ab570-1) 681d1332-diagnose-network C<-S NoResponse GET /ping (1.004727s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-23T21:07:50.651370100Z][com.docker.diagnose.exe][I] (933ab570-2) 681d1332-diagnose-network C->S diagnosticsd GET /ping
[2022-08-23T21:07:51.660987500Z][com.docker.diagnose.exe][W] (933ab570-2) 681d1332-diagnose-network C<-S NoResponse GET /ping (1.0098298s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-23T21:07:52.676113700Z][com.docker.diagnose.exe][I] (933ab570-3) 681d1332-diagnose-network C->S diagnosticsd GET /ping
[2022-08-23T21:07:53.685391700Z][com.docker.diagnose.exe][W] (933ab570-3) 681d1332-diagnose-network C<-S NoResponse GET /ping (1.009278s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-08-23T21:07:54.695130200Z][com.docker.diagnose.exe][I] (933ab570-4) 681d1332-diagnose-network C->S diagnosticsd GET /ping
[2022-08-23T21:07:55.704875200Z][com.docker.diagnose.exe][W] (933ab570-4) 681d1332-diagnose-network C<-S NoResponse GET /ping (1.0096755s): Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[FAIL] DD0032: do Docker networks overlap with host IPs? Get "http://%2F%2F.%2Fpipe%2Fdocker_engine_linux/v1.24/networks": context deadline exceeded
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?

Please investigate the following 3 issues:

1 : The test: are the LinuxKit services running?
    Failed with: failed to ping VM diagnosticsd with error: Get "http://ipc/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The Docker engine runs inside a Linux VM as a service. Therefore the services must have started.

2 : The test: are the backend processes running?
    Failed with: 1 error occurred:
        * vpnkit.exe is not running

Not all of the backend processes are running.

3 : The test: is the VM networking working?
    Failed with: network checks failed: Post "http://ipc/check-network-connectivity": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

VM seems to have a network connectivity issue. Please check your host firewall and anti-virus settings in case they are blocking the VM

jazdw commented 2 years ago

Downgraded to Docker 4.10.1 and I am still seeing this.

aliaugmenta commented 2 years ago

@jazdw, How did you downgrade? I got this error when I tried to use different older installers.

jazdw commented 2 years ago

@jazdw, How did you downgrade? I got this error when I tried to use different older installers.

Deleted all my data, uninstalled then installed the old version. 4.10.1 still seemed broken to me though, I haven't tried an older version yet.

FlowsenAusMonotown commented 2 years ago

With version 4.10.1 the issue doesn't exist from my side. So only 4.11.x isn't working for me.

jazdw commented 2 years ago

With version 4.10.1 the issue doesn't exist from my side. So only 4.11.x isn't working for me.

Yeah I have only encountered one lock up on 4.10.1 so it might be something else.

aliaugmenta commented 2 years ago

I got the same error on 4.12.0 too. It seems that it happens at the same time of some IO (volume) activities.

Guluis commented 2 years ago

I have the same problem. For me the issue has occurred since 4.10.x but probably even before that. Also, this only happens on my windows 11 machine that runs Linux containers. I have another that runs Windows containers and that work without issues.

Another thing I can add here is that if I restart docker and then run docker system prune it delays the issue. This is quite consistent, without running prune it can stop working within minutes/hours of restart and with daily system prune it almost doesn't happen.

So right now this is my workaround, running prune daily but even that doesn't guarantee it will work all day.

aliaugmenta commented 2 years ago

This is happening again. To make it worse, restating docker in such situation is a pain! Often it needs a computer restart (or killing all docker services and processes manually) This time, after hitting docker restart from docker desktop I got this exception:

Timeout waiting for Docker API
   at Docker.Engines.DockerDaemonChecker.<WaitAsync>d__6.MoveNext() in C:\workspaces\4.12.x\src\github.com\docker\pinata\win\src\Docker.Engines\DockerDaemonChecker.cs:line 65
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.Engines.LinuxkitDaemonStartup.<WaitAsync>d__5.MoveNext() in C:\workspaces\4.12.x\src\github.com\docker\pinata\win\src\Docker.Engines\LinuxkitDaemonStartup.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.Engines.WSL2.LinuxWSL2Engine.<DoStartAsync>d__26.MoveNext() in C:\workspaces\4.12.x\src\github.com\docker\pinata\win\src\Docker.Engines\WSL2\LinuxWSL2Engine.cs:line 170
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.ApiServices.StateMachines.TaskExtensions.<WrapAsyncInCancellationException>d__0.MoveNext() in C:\workspaces\4.12.x\src\github.com\docker\pinata\win\src\Docker.ApiServices\StateMachines\TaskExtensions.cs:line 29
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.ApiServices.StateMachines.StartTransition.<DoRunAsync>d__5.MoveNext() in C:\workspaces\4.12.x\src\github.com\docker\pinata\win\src\Docker.ApiServices\StateMachines\StartTransition.cs:line 67
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Docker.ApiServices.StateMachines.StartTransition.<DoRunAsync>d__5.MoveNext() in C:\workspaces\4.12.x\src\github.com\docker\pinata\win\src\Docker.ApiServices\StateMachines\StartTransition.cs:line 92
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.ApiServices.StateMachines.EngineStateMachine.<StartAsync>d__14.MoveNext() in C:\workspaces\4.12.x\src\github.com\docker\pinata\win\src\Docker.ApiServices\StateMachines\EngineStateMachine.cs:line 69
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.Engines.Engines.<StartAsync>d__22.MoveNext() in C:\workspaces\4.12.x\src\github.com\docker\pinata\win\src\Docker.Engines\Engines.cs:line 107

jazdw commented 2 years ago

@aliaugmenta @Guluis what are you doing when you see this issue? Personally I see this when building a Gradle project that builds about five Spring Boot images. Setting Gradle to build them sequentially sometimes seems to alleviate the issue, but if I run them in parallel it locks up almost immediately every time.

aliaugmenta commented 2 years ago

@jazdw building some apps and sometime just running a few applications in containers. It seems that issue happens when more than one container is actively running.

Guluis commented 2 years ago

@aliaugmenta @Guluis what are you doing when you see this issue? Personally I see this when building a Gradle project that builds about five Spring Boot images. Setting Gradle to build them sequentially sometimes seems to alleviate the issue, but if I run them in parallel it locks up almost immediately every time.

I run up to 6 concurrent builds (using Jenkins and Python scripts). Each build creates either one or two dockers. In the case of two dockers, it also creates a network between them. I also believe that reducing the number of threads reduces the occurrences of the issue but it's very hard to prove. In fact, I reduced the number of concurrent runs to 6 in order to alleviate it. I tried running multiple concurrent two-docker scripts but that didn't cause the issue to occur more often.

Guluis commented 2 years ago

Is everybody here using python docker library? I had some idea at some point that it may be related to that. Is anybody seeing this not using python?

jazdw commented 2 years ago

Is everybody here using python docker library? I had some idea at some point that it may be related to that. Is anybody seeing this not using python?

I'm not using Python. It seems like a concurrency issue to me.

aliaugmenta commented 1 year ago

Is everybody here using python docker library? I had some idea at some point that it may be related to that. Is anybody seeing this not using python?

I'm not using Python. It seems like a concurrency issue to me.

I agree. Every time it happened, I was running more than one docker operation simultaneously.

jazdw commented 1 year ago

Well I updated to Windows 11 and installed the latest version of WSL2 from the Windows Store and still no dice on Docker v4.12.0 Diagnostics ID F97CE7C4-D8FC-41F6-B91C-4C3C8E9D6CAB/20221006171130

hlovdal commented 1 year ago

One of our build agents was recently updated to Docker Desktop 4.12.0, and now many of the builds fail with errors like

...
Step 22/22 : ENTRYPOINT [ ... ]
 ---> Running in b25f310e5733
Removing intermediate container b25f310e5733
 ---> 5c4fc85fe75a
Successfully built 5c4fc85fe75a
Successfully tagged ...
Successfully tagged ...
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Building ...
Error response from daemon: i/o timeout
An error occurred when executing task 'dockerbuild'.
Error: Docker: Process returned an error (exit code 1).
##[error]PowerShell exited with code '1'.

The other build agents are deliberately not updated because of these problems.

Windows Version: Server 2019 Standard.
Docker Desktop Version: 4.12.0
WSL2 or Hyper-V backend? Hyper-V
Are you running inside a virtualized Windows e.g. on a cloud server or a VM: Yes

Since problem is observed both for WSL2 and Hyper-V it strongly indicates to be independent of backend type.

hlovdal commented 1 year ago

One very interesting observation

I was cleaning up old, unused docker images by running

docker image ls | awk '/somebranchname/{ print $1 ":" $2}' | xargs docker image rm

and while wating for the command to finish, the output from the command suddenly slowed down and then started failing all with "Error response from daemon: i/o timeout". I quickly opned the task manager to check if maybe a new build job had started. I thought so at first, but now later I think maybe the cpuload is caused by just docker alone. In any case the machine was 100% busy:

docker_io_error

So this in my opinion strongly indicates that this problem is corrolated with high CPU load.

Continouing to run the cleanup command line spikes the cpu to full load and it runs successful for some time but then eventually fails the same way again:

docker_io_error2

and again:

docker_io_error3

and again:

docker_io_error4

before it finally ran successfully till completion.

auriseu commented 1 year ago

Downgrading to docker desktop 3.5.2 fixed issue for me

Guluis commented 1 year ago

This definitely seems like some threading/load issue for docker. Our testing used a lot of docker exec commands (over 95% of hundreds of calls per test), other docker commands and also a bunch of python docker SDK calls. By removing the docker exec commands (found another solution), we significantly reduced the occurrence of this timeout. So much so that we had to system prune daily (and it would still occur sometimes) and now we don't and it occurs every so often. From my testing however, running multiple threads that call various docker commands doesn't recreate the issue. I believe it must have something to do with using both docker commands and some external access method (such as python SDK for us). Is there anyone here who is using purely docker command line and still seeing this issue?

hlovdal commented 1 year ago

Possibly a duplicate of #4413.

rg9400 commented 1 year ago

I have this issue pretty consistently. It mainly happens when one of my containers is doing major file activities, specifically on files mounted from outside WSL (CIFS share). I know file performance for non-WSL data is very slow in WSL2, so I am wondering if that is related.

I used to get this issue intermittently in much older builds, but it has become far worse on more recent builds of Docker.

Docker 4.13.0 just flat out fails to work with these files, often giving a Resource Temporarily Unavailable error.

hlovdal commented 1 year ago

Our build agents are not using WSL, so this is not directly related to that.

jazdw commented 1 year ago

Just upgraded to 4.13.1, same issue.

jekatarina commented 1 year ago

Same issue

Windows Version: 10 Docker Desktop Version: 4.13.0 Docker Version: Docker version 20.10.20, build 9fdeb9c WSL2 or Hyper-V backend? WSL2

PS C:\localhost> docker compose up -d Error response from daemon: i/o timeout

Ark-kun commented 1 year ago

I get these errors when I do even a little bit of parallel Docker calls. This error can easily be triggered with the following Python code:

import docker
from concurrent import futures
executor = futures.ThreadPoolExecutor()

len(list(executor.map(lambda x: docker.from_env(), range(1000))))

jukkahyv commented 1 year ago

Don't know if it matters, but I have two laptops, one with Core i9 (14 cores) and other with Core i7 (6 cores). The one with more cores is getting these I/O errors much more often.

jazdw commented 1 year ago

Anyone braved 4.14.x yet?

jazdw commented 1 year ago

Well I updated to 4.15.0 today and again banged my head against the wall trying to get it to work. The bug still exists but I found one thing that makes it work - limiting the processors available to WSL2 to 1 processor. In .wslconfig you can set:

[wsl2]
processors=1

See https://learn.microsoft.com/en-us/windows/wsl/wsl-config#configuration-setting-for-wslconfig

This sets the global max number of processors for all WSL2 distros so its not ideal. My laptop has 8 physical cores and 16 logical (with hyper threading), even using processors=2 does not work.

edit: Interestingly, if I let WSL2 use all my processors, but limit the affinity of vmmemWSL to a single processor it still locks up.

jazdw commented 1 year ago

I'm officially done with Docker Desktop.. I realized you can just run Docker inside WSL2 without the UI and it works far better and its faster.

I installed Docker CE inside Ubunutu on WSL2, made it listen on a TCP port so I can connect to it directly from Windows using Gradle etc. IntelliJ can also connect to Docker inside WSL2 without using the TCP port, and IntelliJ has its own UI for Docker. Works flawlessly.

Two helpful articles - https://dev.to/bowmanjd/install-docker-on-windows-wsl-without-docker-desktop-34m9 https://nickjanetakis.com/blog/install-docker-in-wsl-2-without-docker-desktop Note that WSL2 now supports using systemd, I recommend enabling that and using systemd to start Docker. I also didn't need to modify anything to do with iptables that must have been configured automatically.

docker-robott commented 1 year ago

There hasn't been any activity on this issue for a long time. If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment. If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale

hlovdal commented 1 year ago

/remove-lifecycle stale

docker-robot[bot] commented 1 year ago

There hasn't been any activity on this issue for a long time. If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment. If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale

Zaffer commented 1 year ago

I'm officially done with Docker Desktop.. I realized you can just run Docker inside WSL2 without the UI and it works far better and its faster.

I installed Docker CE inside Ubunutu on WSL2, made it listen on a TCP port so I can connect to it directly from Windows using Gradle etc. IntelliJ can also connect to Docker inside WSL2 without using the TCP port, and IntelliJ has its own UI for Docker. Works flawlessly.

Two helpful articles - https://dev.to/bowmanjd/install-docker-on-windows-wsl-without-docker-desktop-34m9 https://nickjanetakis.com/blog/install-docker-in-wsl-2-without-docker-desktop Note that WSL2 now supports using systemd, I recommend enabling that and using systemd to start Docker. I also didn't need to modify anything to do with iptables that must have been configured automatically.

be careful because this can cause your disk space to run out, we had an issue causing this. Docker Desktop for windows with Linux backend manages the size of the virtual disk in later versions, not if it runs inside linux.

hlovdal commented 1 year ago

/remove-lifecycle stale

CodeTectonics commented 10 months ago

I had a similar issue and fixed it by setting resource limits on my containers.

For example:

my-service:
    container_name: my-container
    image: my-image:latest
    deploy:
      resources:
        limits:
          memory: 600M

After reading a lot of posts about this, it seems to mostly happen when running containers on a virtualised Linux inside of a Windows host.

In my case, I had a container that was holding on to an increasing amount of memory, eventually maxing out the memory available to the VM. In situations like this, Docker would usually detect an Out Of Memory error and kill the container to protect its own components (the daemon, engine, etc). However, in this case, Docker did not kill the container, so the daemon ran out of resources and crashed.

Setting resource limits prevented the container from maxing out the VM's available resources, so the container stayed healthy and the daemon stopped crashing.

docker / for-win