docker / for-win

Bug reports for Docker Desktop for Windows
https://www.docker.com/products/docker#/windows
1.83k stars 279 forks source link

Timeouts when interacting with Docker #14054

Open Falco20019 opened 3 weeks ago

Falco20019 commented 3 weeks ago

Description

Since updating to 4.29.0 lots of networking issues appear. This affects all operations like pulling but also even login. This problem also appears randomly to some other colleagues.

Reproduce

  1. Update to 4.29.0
  2. Run docker login
  3. See net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Expected behavior

Docker completing the requests correctly and on time

docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:56 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 8
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.76GiB
 Name: docker-desktop
 ID: 20f9627d-8012-41bb-aef3-9f6e6c329936
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

7E71F66E-7F92-4493-A5F0-C66EB27E8EC7/20240506080323

Additional Info

Uploading the diagnostics is stuck with [2024-05-06T08:16:43.857526300Z][com.docker.diagnose.exe.public][I] using HTTP proxy inside com.docker.backend, so not possible to upload...

C:\Program Files\Docker\Docker\resources>com.docker.diagnose.exe upload -id 7E71F66E-7F92-4493-A5F0-C66EB27E8EC7/20240506080323
[2024-05-06T08:16:43.681212100Z][com.docker.diagnose.exe.settingsloader][I] loaded settings
[2024-05-06T08:16:43.852297700Z][com.docker.diagnose.exe][I] host will use proxy:  http_proxy=none https_proxy=none tcp_proxy=none noproxy=
[2024-05-06T08:16:43.853398800Z][com.docker.diagnose.exe][I] Linux will use proxy:  http_proxy=none https_proxy=none tcp_proxy=none noproxy=
[2024-05-06T08:16:43.857027200Z][com.docker.diagnose.exe][I] host will use proxy: automatic system http://pac.zscloud.net/YhwnNSrR3J9K/ProxyPAC
[2024-05-06T08:16:43.857027200Z][com.docker.diagnose.exe][I] Linux will use proxy: automatic system http://pac.zscloud.net/YhwnNSrR3J9K/ProxyPAC
[2024-05-06T08:16:43.857526300Z][com.docker.diagnose.exe.public][I] using HTTP proxy inside com.docker.backend
Error uploading: uploading file "C:\\Users\\dejhbk0l\\AppData\\Local\\Temp\\7E71F66E-7F92-4493-A5F0-C66EB27E8EC7\\20240506080323.zip" with key "incoming/3/7E71F66E-7F92-4493-A5F0-C66EB27E8EC7/20240506080323.zip": RequestError: send request failed
caused by: Put "https://docker-pinata-support.s3.amazonaws.com/incoming/3/7E71F66E-7F92-4493-A5F0-C66EB27E8EC7/20240506080323.zip": context deadline exceeded
[2024-05-06T08:20:44.327007100Z][com.docker.diagnose.exe][I] If your computer needs an HTTPS proxy, please set the HTTPS_PROXY environment variable and try again.

Docker check also seems to behave strange, but I saw similar outputs in other recent tickets.

C:\Program Files\Docker\Docker\resources>com.docker.diagnose.exe check
Starting diagnostics

[2024-05-06T07:56:16.882514700Z][com.docker.diagnose.exe.ipc][I] 3efd463f-diagnose -> \\.\pipe\dockerBackendApiServer BackendAPI
[2024-05-06T07:56:17.475902400Z][com.docker.diagnose.exe.ipc][I] (9ac4aff3) 3efd463f-diagnose C->S BackendAPI POST /idle/make-busy
[2024-05-06T07:56:17.478030200Z][com.docker.diagnose.exe.ipc][I] (9ac4aff3) 3efd463f-diagnose C<-S a0f27882-BackendAPI POST /idle/make-busy (2.1278ms): 0xc00046e010
[2024-05-06T07:56:18.490540000Z][com.docker.diagnose.exe.ipc][I] (d34e97e3) 3efd463f-diagnose C->S BackendAPI GET /idle
[2024-05-06T07:56:18.491544600Z][com.docker.diagnose.exe.ipc][I] (d34e97e3) 3efd463f-diagnose C<-S a0f27882-BackendAPI GET /idle (1.0046ms): {"apisInFlight":{},"booted":true,"busyReason":["timed activities: map[/idle/make-busy:28.986729s]"],"containers":0,"idle":"bool","kubernetesEnabled":false,"reduced":false,"services":0,"timedActivities":{"/idle/make-busy":"float64"},"vmPaused":false,"vmRunning":true,"vmStopped":false,"windowsContainers":false}
[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[SKIP] DD0002: does the bootloader have virtualization enabled?
[SKIP] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[SKIP] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[SKIP] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0024: is WSL installed?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[FAIL] DD0004: is the Docker engine running? unable to create docker client: protocol not available
[PASS] DD0015: are the binary symlinks installed?
[FAIL] DD0031: does the Docker API work? prereq failed: is the Docker engine running?
[PASS] DD0013: is the $PATH ok?
[FAIL] DD0003: is the Docker CLI working? prereq failed: is the Docker engine running?
[PASS] DD0005: is the user in the docker-users group?
[FAIL] DD0038: is the connection to Docker working? prereq failed: is the Docker engine running?
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[SKIP] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[SKIP] DD0006: is the Docker Desktop Service responding?
[FAIL] DD0030: is the image access management authorized? prereq failed: is the Docker engine running?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0002: does the bootloader have virtualization enabled?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[PASS] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0024: is WSL installed?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[WARN] DD0004: is the Docker engine running? unable to create docker client: protocol not available
[PASS] DD0015: are the binary symlinks installed?
[WARN] DD0031: does the Docker API work? prereq failed: is the Docker engine running?
[WARN] DD0032: do Docker networks overlap with host IPs? prereq failed: does the Docker API work?

Please note the following 3 warnings:

1 : The check: is the Docker engine running?
    Produced the following warning: unable to create docker client: protocol not available

The Docker engine manages all containers and images on the host. Check the dockerd.log to see why it failed to start.

2 : The check: does the Docker API work?
    Produced the following warning: prereq failed: is the Docker engine running?

If the Docker API is not available from the host then Docker Desktop will not work correctly.

3 : The check: do Docker networks overlap with host IPs?
    Produced the following warning: prereq failed: does the Docker API work?

If the subnet used by a Docker network overlaps with an IP used by the host, then containers
won't be able to contact the overlapping IP addresses.

Try configuring the IP address range used by networks: in your docker-compose.yml.
See https://docs.docker.com/compose/compose-file/compose-file-v2/#ipv4_address-ipv6_address

Please investigate the following 1 issue:

1 : The test: is the Docker engine running?
    Failed with: unable to create docker client: protocol not available

The Docker engine manages all containers and images on the host. Check the dockerd.log to see why it failed to start.
Falco20019 commented 3 weeks ago

After restarting the machine, it usually works for some time. I was able to backup the diagnostics and upload it after a restart. Updated the diagnostics ID accordingly.

gmeans commented 3 weeks ago

I'm here troubleshooting the same issue. It happens somewhat random, but docker operations begin failing with the message:

net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I typically see it with docker pull against our primary internal registry which is unauthenticated.

Docker Info

Client:
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 15
  Running: 2
  Paused: 0
  Stopped: 13
 Images: 11
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.47GiB
 Name: docker-desktop
 ID: f5a7dbcd-ebae-4c23-bd80-050deb34f636
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 68
  Goroutines: 80
  System Time: 2024-05-09T11:39:47.555552377Z
  EventsListeners: 12
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Docker Diagnose

Interestingly, I've rebooted and confirmed that docker is running but the diagnose command reports that the docker engine may not be running.

& "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" check

Starting diagnostics

[2024-05-09T11:52:52.739471000Z][com.docker.diagnose.exe.ipc][I] 49524e10-diagnose -> \\.\pipe\dockerBackendApiServer BackendAPI
[2024-05-09T11:52:53.305817900Z][com.docker.diagnose.exe.ipc][I] (4cb94040) 49524e10-diagnose C->S BackendAPI POST /idle/make-busy
[2024-05-09T11:52:53.306861000Z][com.docker.diagnose.exe.ipc][I] (4cb94040) 49524e10-diagnose C<-S 634fe691-BackendAPI POST /idle/make-busy (1.0431ms): 0xc000639840
[2024-05-09T11:52:54.320526100Z][com.docker.diagnose.exe.ipc][I] (20fe5824) 49524e10-diagnose C->S BackendAPI GET /idle
[2024-05-09T11:52:54.322177100Z][com.docker.diagnose.exe.ipc][I] (20fe5824) 49524e10-diagnose C<-S 634fe691-BackendAPI GET /idle (1.0876ms): {"apisInFlight":{},"booted":true,"busyReason":["2 container(s)","timed activities: map[/idle/make-busy:28.9857676s]"],"containers":2,"idle":"bool","kubernetesEnabled":false,"reduced":false,"services":0,"timedActivities":{"/idle/make-busy":"float64"},"vmPaused":false,"vmRunning":true,"vmStopped":false,"windowsContainers":false}
[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[SKIP] DD0002: does the bootloader have virtualization enabled?
[SKIP] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[SKIP] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[SKIP] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0024: is WSL installed?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[FAIL] DD0004: is the Docker engine running? unable to create docker client: protocol not available
[PASS] DD0015: are the binary symlinks installed?
[FAIL] DD0031: does the Docker API work? prereq failed: is the Docker engine running?
[PASS] DD0013: is the $PATH ok?
[FAIL] DD0003: is the Docker CLI working? prereq failed: is the Docker engine running?
[PASS] DD0005: is the user in the docker-users group?
[FAIL] DD0038: is the connection to Docker working? prereq failed: is the Docker engine running?
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[SKIP] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[SKIP] DD0006: is the Docker Desktop Service responding?
[FAIL] DD0030: is the image access management authorized? prereq failed: is the Docker engine running?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0002: does the bootloader have virtualization enabled?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[PASS] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0024: is WSL installed?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[WARN] DD0004: is the Docker engine running? unable to create docker client: protocol not available
[PASS] DD0015: are the binary symlinks installed?
[WARN] DD0031: does the Docker API work? prereq failed: is the Docker engine running?
[WARN] DD0032: do Docker networks overlap with host IPs? prereq failed: does the Docker API work?

Please note the following 3 warnings:

1 : The check: is the Docker engine running?
    Produced the following warning: unable to create docker client: protocol not available

The Docker engine manages all containers and images on the host. Check the dockerd.log to see why it failed to start.

2 : The check: does the Docker API work?
    Produced the following warning: prereq failed: is the Docker engine running?

If the Docker API is not available from the host then Docker Desktop will not work correctly.

3 : The check: do Docker networks overlap with host IPs?
    Produced the following warning: prereq failed: does the Docker API work?

If the subnet used by a Docker network overlaps with an IP used by the host, then containers
won't be able to contact the overlapping IP addresses.

Try configuring the IP address range used by networks: in your docker-compose.yml.
See https://docs.docker.com/compose/compose-file/compose-file-v2/#ipv4_address-ipv6_address

Please investigate the following 1 issue:

1 : The test: is the Docker engine running?
    Failed with: unable to create docker client: protocol not available

The Docker engine manages all containers and images on the host. Check the dockerd.log to see why it failed to start.