search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
757 stars 85 forks source link

Docker gets stuck while pulling large images #356

Open derinbay opened 6 years ago

derinbay commented 6 years ago

Expected behavior

Finish pull process successfully

Actual behavior

While downloading packages, it stucks after a while for a package. The package is not the same when retrying. This is not happening on small images, I experienced this on images more than 500 mb. When I checked the http traffic with wireshark, I see that there are resets, and some packages are not coming from your server.

administrator@testubuntu01:~/docker$ docker pull elgalu/selenium:3.12.0-p10
3.12.0-p10: Pulling from elgalu/selenium
b234f539f7a1: Already exists
55172d420b43: Already exists
5ba5bbeb6b91: Already exists
43ae2841ad7a: Already exists
f6c9c6de4190: Already exists
91785af236da: Pull complete
b27ff01c1bf1: Pull complete
fad231cd4dba: Pull complete
e6fc9451a4d6: Pull complete
fc24f700ab5f: Pull complete
a754e9830ba1: Pull complete
e666725ffedb: Pull complete
d2b70934f462: Pull complete
c91b09598f6c: Pull complete
b0c50a780037: Pull complete
2a081dc353e9: Pull complete
bf4e272f7214: Pull complete
a4cb4aafa654: Pull complete
e8baa080c953: Pull complete
f040bb666d12: Pull complete
832221fa85dc: Downloading [================================================>  ] 25.81MB/26.44MB
8c66cb482ef2: Download complete
bb4729b16cd1: Download complete
4091871cdcb8: Download complete
4873b47788e0: Download complete
04ed0d7d2e63: Download complete
83910e0c2c28: Download complete
4ede1dd6b7e6: Download complete
8a8f91317d48: Download complete

Wireshark output after stuck:

1531126031.074773 IP 104.18.121.25.443 > 172.20.0.223.37256: Flags [R], seq 1750338835, win 0, length 0
1531126036.535144 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [S], seq 983764761, win 29200, options [mss 1460,sackOK,TS val 4251401848 ecr 0,nop,wscale 7], length 0
1531126036.535466 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [S.], seq 2271912881, ack 983764762, win 14480, options [mss 1460,sackOK,TS val 420820279 ecr 4251401848,nop,wscale 7], length 0
1531126036.535492 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [.], ack 1, win 229, options [nop,nop,TS val 4251401849 ecr 420820279], length 0
1531126036.535563 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [P.], seq 1:168, ack 1, win 229, options [nop,nop,TS val 4251401849 ecr 420820279], length 167
1531126036.535801 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [.], ack 168, win 122, options [nop,nop,TS val 420820279 ecr 4251401849], length 0
1531126036.543247 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [P.], seq 1:1461, ack 168, win 122, options [nop,nop,TS val 420820280 ecr 4251401849], length 1460
1531126036.543259 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [.], ack 1461, win 251, options [nop,nop,TS val 4251401856 ecr 420820280], length 0
1531126036.543334 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [P.], seq 1461:2921, ack 168, win 122, options [nop,nop,TS val 420820280 ecr 4251401849], length 1460
1531126036.543343 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [.], ack 2921, win 274, options [nop,nop,TS val 4251401856 ecr 420820280], length 0
1531126036.543538 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [P.], seq 2921:3997, ack 168, win 122, options [nop,nop,TS val 420820280 ecr 4251401849], length 1076
1531126036.543549 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [.], ack 3997, win 297, options [nop,nop,TS val 4251401857 ecr 420820280], length 0
1531126036.568248 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [P.], seq 168:261, ack 3997, win 297, options [nop,nop,TS val 4251401881 ecr 420820280], length 93
1531126036.572119 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [P.], seq 3997:4048, ack 261, win 122, options [nop,nop,TS val 420820283 ecr 4251401881], length 51
1531126036.572285 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [P.], seq 261:854, ack 4048, win 297, options [nop,nop,TS val 4251401885 ecr 420820283], length 593
1531126036.607799 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [.], ack 854, win 131, options [nop,nop,TS val 420820287 ecr 4251401885], length 0
1531126036.626493 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [P.], seq 4048:5293, ack 854, win 131, options [nop,nop,TS val 420820288 ecr 4251401885], length 1245
1531126036.626511 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [F.], seq 5293, ack 854, win 131, options [nop,nop,TS val 420820288 ecr 4251401885], length 0
1531126036.626668 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [P.], seq 854:885, ack 5294, win 319, options [nop,nop,TS val 4251401940 ecr 420820288], length 31
1531126036.626693 IP 172.20.0.223.37276 > 104.18.121.25.443: Flags [F.], seq 885, ack 5294, win 319, options [nop,nop,TS val 4251401940 ecr 420820288], length 0
1531126036.626827 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [R], seq 2271918175, win 0, length 0
1531126036.626845 IP 104.18.121.25.443 > 172.20.0.223.37276: Flags [R], seq 2271918175, win 0, length 0
1531126052.545461 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251417859 ecr 420818808], length 0
1531126052.545574 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420821880 ecr 4249943158], length 0
1531126083.265471 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251448579 ecr 420821880], length 0
1531126083.265592 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420824952 ecr 4249943158], length 0
1531126113.985513 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251479299 ecr 420824952], length 0
1531126113.985653 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420828024 ecr 4249943158], length 0
1531126144.705481 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251510018 ecr 420828024], length 0
1531126144.705598 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420831096 ecr 4249943158], length 0
1531126175.425474 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251540738 ecr 420831096], length 0
1531126175.425597 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420834168 ecr 4249943158], length 0
1531126206.145479 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251571458 ecr 420834168], length 0
1531126206.145599 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420837240 ecr 4249943158], length 0
1531126236.865495 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251602178 ecr 420837240], length 0
1531126236.865611 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420840312 ecr 4249943158], length 0
1531126267.585487 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251632898 ecr 420840312], length 0
1531126267.585620 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420843383 ecr 4249943158], length 0
1531126298.305491 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251663618 ecr 420843383], length 0
1531126298.305625 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420846455 ecr 4249943158], length 0
1531126329.025476 IP 172.20.0.223.36946 > 104.18.121.25.443: Flags [.], ack 1, win 5397, options [nop,nop,TS val 4251694338 ecr 420846455], length 0
1531126329.025589 IP 104.18.121.25.443 > 172.20.0.223.36946: Flags [.], ack 1, win 131, options [nop,nop,TS val 420849527 ecr 4249943158], length 0

Output of docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Wed Jun 20 21:43:51 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Wed Jun 20 21:42:00 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 6
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-23-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 70.79GiB
Name: testubuntu01
ID: PPE7:IBBN:2C5N:UMMQ:VSCP:4ESD:J6VC:B633:VYD4:LP3A:I7JU:7WOR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

The host machine is in a VMware platform

PS: No, docker restart is not fixing the issue.

kevinimbrechts commented 6 years ago

I have the same error... Did you find a solution? I am on VirtualBox with Ubuntu 18.04.

derinbay commented 6 years ago

@kevinimbrechts I limit the parallel download of layers to 1 as a workaround. After that, pulling the images are working fine, but slower of course.

sudo dockerd --max-concurrent-download 1
kevinimbrechts commented 6 years ago

@derinbay Thank you but it's not working for me. I added "max-concurrent-downloads":1 in my daemon.json file and docker pull try to pulling 1 layer but it's stuck yet.

docker pull buildpack-deps:sid-scm
sid-scm: Pulling from library/buildpack-deps
26254fbb7f1e: Downloading [===>                                               ]  2.948MB/48.65MB
bb95abe62cf6: Waiting 
b1dbb6315c68: Waiting 
4e01fadb9041: Waiting
kevinimbrechts commented 6 years ago

Ok, I had selected "Nat network" network access mode in VirtualBox but I have to select "NAT".