jpetazzo
commented
10 years ago Which version of tar are you using? (I wonder if this might be an issue related to CoreOS?)
Closed tgummerer closed 10 years ago
jpetazzo
commented
10 years ago Which version of tar are you using? (I wonder if this might be an issue related to CoreOS?)
tgummerer
commented
10 years ago core@localhost ~ $ tar --version
tar (GNU tar) 1.26
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by John Gilmore and Jay Fenlason.Oh and I forgot to mention, it starts downloading stuff, and then the error pops up. Mostly it stops when only a few megabytes (1-5) were downloaded, but other times (less common) it manages to download 50-70 MB.
jpetazzo
commented
10 years ago Ah, in that case, I'm willing to place on bet on odd network conditions :-) How is the machine connected to the network? Since you mentioned CoreOS under vagrant, I suppose that this is running on your local machine. Can you connect to a different network and see if it behaves the same? Are you able to download big-ish files? (e.g. an Ubuntu ISO image)
On Wed, Oct 30, 2013 at 8:29 AM, Thomas Gummerer notifications@github.comwrote:
core@localhost ~ $ tar --version tar (GNU tar) 1.26 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Written by John Gilmore and Jay Fenlason.
Oh and I forgot to mention, it starts downloading stuff, and then the error pops up. Mostly it stops when only a few megabytes (1-5) were downloaded, but other times (less common) it manages to download 50-70 MB.
— Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/2461#issuecomment-27399522 .
tgummerer
commented
10 years ago Yeah you're right, the problem was more the wifi card than the actual network it seems, but that comes down to the same thing ;-) Sorry for the noise.
Downloading big files is possible, but only because wget handles read error and continues to download afterwards. Might that be something worth implementing in docker?
tavisrudd
commented
10 years ago I've run into this on multiple machines/networks when downloading tianon/gentoo. It fails the first time then succeeds on second run. Most recently was on Joyent, who have a fairly reliable network. "1c8fbfa2ee2d: Error pulling image (stage3-20130516) from tianon/gentoo, exit status 2: tar: Unexpected EOF in archive"
jpetazzo
commented
10 years ago I'm reopening.
@samalba do you know if there is an easy way to bypass the CDN (to check if that might be the CDN's fault)?
tavisrudd
commented
10 years ago Another data point: I ran into this again the first time I tried to pull from a digitalocean vm. Re-running it immediately resolved the issue as it had previously.
gmbuell
commented
10 years ago I'm seeing the same problem on a vagrant virtualbox image. Fails the first time, succeeds the second.
jvermillard
commented
10 years ago Same problem here, 3 tries before success:
core@localhost ~ $ docker pull centos
Pulling repository centos
539c0211cd76: Error pulling image (latest) from centos, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
539c0211cd76: Error pulling image (latest) from centos, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2013/11/21 10:33:33 Server error: 404 trying to fetch remote history for centos
core@localhost ~ $ docker pull centos
Pulling repository centos
539c0211cd76: Error pulling image (latest) from centos, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
539c0211cd76: Error pulling image (latest) from centos, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2013/11/21 11:06:38 Server error: 404 trying to fetch remote history for centos
core@localhost ~ $ docker pull centos
Pulling repository centos
539c0211cd76: Download complete
samalba
commented
10 years ago Thanks for the data. Could you give some details on how long does it take when it fails? Does it fail right away or is there some kind of timeout?
jvermillard
commented
10 years ago dunno exactly but looks like some kind of timeout when the download is stuck. For example I have a lot of trouble to download this image: kimh/oracledb (~1.3GB)
rogaha
commented
10 years ago Currently the cookie session lasts only one hour. If the downloading process takes more that one hour you should get the error 404 after that period of time.
@samalba, do you think that we could increase the expiration period?
tianon
commented
10 years ago Is there a particular reason why we can't just use a session cookie?
samalba
commented
10 years ago @tianon do you mean a cookie without expires? If yes, I we can.
tianon
commented
10 years ago Yes, that's exactly what I mean. Then we can take as long as we need to. :)
justcookin
commented
10 years ago Kept on getting this on several pull requests:
$sudo docker pull inokappa/squeeze-basic
Pulling repository inokappa/squeeze-basic
050462b16e32: Error pulling image (latest) from inokappa/squeeze-basic, endpoint: https://cdn-registry- 1.docker.io/v1/, exit status 2:
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
050462b16e32: Error pulling image (latest) from inokappa/squeeze-basic, exit status 2: Error is not recoverable: exiting now
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2013/12/06 15:23:02 Server error: 404 trying to fetch remote history for inokappa/squeeze-basicHad to run it 2 times for this particular repository, second time it worked fine. Running docker on an Amazon EC2 ubuntu instance.
samalba
commented
10 years ago We pushed a fix candidate on prod this week on the public Registry. Can anyone update the status on this? The problem is supposed to be solved.
Reference: https://github.com/dotcloud/docker-registry/commit/e5d96a1178ea45b7923d08a05a86be7c5d716085
rogaha
commented
10 years ago I was trying to replicate that problem and I couldn't do it any more. Hopefully it should be fixed with that fix candidate.
albertogg
commented
10 years ago I know this has been discussed several times... I'm always getting stuck while downloading the "ubuntu" image. My problem is that it keeps throwing the Unexpected EOF in archive error while been at 3.342 MB/94.86 MB 52m1s this has happened every single time for the past 6 tries. I'm running Docker within Vagrant within VirtualBox, with a bridged network.
$ docker pull ubuntu
Pulling repository ubuntu
8dbd9e392a96: Error pulling image (precise) from ubuntu, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
xz: (stdin): Unexpected end of input ] 3.342 MB/94.86 MB 52m1s
8dbd9e392a96: Error pulling image (precise) from ubuntu, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'ed EOF in archive
xz: (stdin): Unexpected end of inputng now
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting nowThe software versions.
$ docker version
Client version: 0.7.1
Go version (client): go1.2
Git commit (client): 88df052
Server version: 0.7.1
Git commit (server): 88df052
Go version (server): go1.2
Last stable version: 0.7.1
$ tar --version
tar (GNU tar) 1.26
$ vagrant -v
Vagrant 1.4.0
VirtualBox 4.3.4 with its guest additions.Its really weird that it always breaks in the same part... also I'm not getting the 404 error, I have to interrupt the execution and and kill the docker process to be availed to retry the docker pull ubuntu.
Update: now I'm getting this.
$ docker pull ubuntu
Pulling repository ubuntu
8dbd9e392a96: Downloading [====> ] 5.014 MB/58.34 MB 29m12s
b750fe79269d: Error pulling image (quantal) from ubuntu, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2:
gzip: stdin: unexpected end of fileendent layers
tar: Unexpected EOF in archive
b750fe79269d: Error pulling image (quantal) from ubuntu, exit status 2:
gzip: stdin: unexpected end of fileing now
8dbd9e392a96: Error pulling image (precise) from ubuntu, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
xz: (stdin): Unexpected end of input
8dbd9e392a96: Error pulling image (precise) from ubuntu, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'ed EOF in archive
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
xz: (stdin): Unexpected end of input
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2013/12/13 19:49:33 Could not find repository on any of the indexed registries.
mait
commented
10 years ago ➜ ~ sudo docker pull coreos/etcd
Pulling repository coreos/etcd
ff181fe39317: Error pulling image (latest) from coreos/etcd, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'ading dependent layers
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
xz: (stdin): Unexpected end of input
tar: Unexpected EOF in archive
ff181fe39317: Error pulling image (latest) from coreos/etcd, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
tar: Ignoring unknown extended header keyword `SCHILY.fflags'
xz: (stdin): Unexpected end of input
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2013/12/25 17:34:43 Could not find repository on any of the indexed registries.samsaffron/discourse, coreos/etcd failed.
busybox OK.
Now I'm trying ubuntu. It looks like working. In progress.
Uubuntu 13.10
Client version: 0.7.2
Go version (client): go1.2
Git commit (client): 28b162e
Server version: 0.7.2
Git commit (server): 28b162e
Go version (server): go1.2
Last stable version: 0.7.2
tar (GNU tar) 1.26
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by John Gilmore and Jay Fenlason.docker pull ubuntu started at 18:00. Now 22:50. Still 60MB/94MB in progress.
South Korea banned from CDN? I'm wondering why this is so slow.
albertogg
commented
10 years ago @mait it's stuck and it will not continue, that happens to me also... when it doesn't break with the tar: Unexpected EOF in archive
dinogun
commented
10 years ago $ sudo docker pull ubuntu 2014/01/13 12:47:43 POST /v1.8/images/create?fromImage=ubuntu&tag= Pulling repository ubuntu 8dbd9e392a96: Downloading [==========> ] 11.7 MB/58.34 MB 12m24s b750fe79269d: Error pulling image (quantal) from ubuntu, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now b750fe79269d: Error pulling image (quantal) from ubuntu, exit status 2: tar: Ignoring unknown extended header keyword `SCHILY.fflags'
gzip: stdin: unexpected end of file tar: Unexpected EOF in archive 8dbd9e392a96: Download complete tar: Error is not recoverable: exiting now
2014/01/13 13:03:39 Could not find repository on any of the indexed registries.
This is from Bangalore, India
porterjamesj
commented
10 years ago I am also seeing this from inside of a private cloud:
$ docker pull chapmanb/bcbio-nextgen-vm
Pulling repository chapmanb/bcbio-nextgen-vm
2014/01/15 22:49:37 HTTP code: 404
ubuntu@docker-test:/usr/local/Cellar/samtools/0.1.19/share/man/man1$ sudo docker pull chapmanb/bcbio-nextgen-devel
Pulling repository chapmanb/bcbio-nextgen-devel
d6c738103f22: Error pulling image (latest) from chapmanb/bcbio-nextgen-devel, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2:
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archivee
tar: Unexpected EOF in archivee
tar: Error is not recoverable: exiting nowlayers
d6c738103f22: Error pulling image (latest) from chapmanb/bcbio-nextgen-devel, exit status 2:
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2014/01/15 22:54:48 Could not find repository on any of the indexed registries.and as others have reported; it works fine on the second try.
narkisr
commented
10 years ago I can confirm this happening also
samalba
commented
10 years ago I was able to reproduce a couple of times. It looks like while Docker is fetching layers from CloudFlare, the connection gets interrupted sometimes.
We're currently implementing bytes-range support on the Registry, so we'll be able to transparently retry on fail (and not have to restart from scratch).
I am assigning this to me, to coordinate.
tavisrudd
commented
10 years ago I'm still seeing this but I'm now seeing a different failure upon retrying:
# docker run -i -t -rm tianon/gentoo /bin/bash
Unable to find image 'tianon/gentoo' (tag: latest) locally
Pulling repository tianon/gentoo
17340a975f03: Download complete
1c8fbfa2ee2d: Download complete
7b4d1e067a9d: Error pulling image (stage3-20130627) from tianon/gentoo, Server error: Status 524 while fetching image layer (7b4d1e067a9de13d585df70a9696c7800cde0ad2f0afa37f28112f8211c8f2f4) 5df70a9696c7800cde0ad2f0afa37f28112f8211c8f2f4)
cca6e0292bd0: Download complete
2014/01/18 10:44:34 Could not find repository on any of the indexed registries.Retry again and third time I'm lucky. This happens to me every time, regardless of which network I'm on, so I don't think it's anything related to my local network.
blue5tar
commented
10 years ago i have the same problem, i use vagrant, i pull busybox ok!
kevinw
commented
10 years ago I'm seeing this problem as well, with docker-osx (which is using docker 0.7.5, also the VM has tar 1.26).
zen0wu
commented
10 years ago Same problem here, it's obviously due to bad network connection. The pull stops after downloaded part of the data and says
Pulling repository daprlabs/archlinux-jdk
2573c8e8bd78: Error pulling image (latest) from daprlabs/archlinux-jdk, endpoint: https://cdn-registry-1.docker.io/v1/, exit status 2:
gzip: stdin: unexpected end of fileendent layers
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2573c8e8bd78: Error pulling image (latest) from daprlabs/archlinux-jdk, exit status 2:
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2014/02/02 04:34:20 Could not find repository on any of the indexed registries.I think the ultimate solution to this is to allow download to resume from the last fail point rather than start over. As long as it can download some data, it eventually will succeed to download all of them, as long as every time it starts from where it's left over.
samalba
commented
10 years ago Correct, I added the bytes range support in docker-registry https://github.com/dotcloud/docker-registry/pull/221 especially for this resume.
Now we need to implement on the docker side (only if the registry advertises that it supports "Range"). If "Range" is not supported, docker would retry the request from scratch.
Any volunteer?
jvermillard
commented
10 years ago just took a look at the code, it's quite a tricky feature to write, the code is using ReadCloser everywhere. I think a good idea is to wrap the Http response stream and try a resume if the wrapped stream is producing timeout or closing unexpectedly.
Another way is to try to download the file in /tmp, try resuming until it's complete and return a ReadCloser on the downloaded file (which delete the file when closed) .
WDYT?
amaline
commented
10 years ago I am getting a similar problem between two virtualbox VM's on the same host using a private registry. Pushing to the registry was not a problem, but pulling gets between 4 and 50meg through the download and errors with exist status 2:
gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now 2014/02/05 13:20:56 Could not find repository on any of the indexed registries.
This is on a Fedora VM (3.12.9-301.fc20.x86_64) Docker version 0.7.2, build 28b162e/0.7.2 tar 1.26 gzip 1.6
UPDATE: this problem went away when I moved the VM's from a private network to a public network. So maybe like it has something to do with going through two NATS (vbox and docker)?
kentlangley
commented
10 years ago Not sure if it's helpful to you but in my case, when I ran into a lot of these same issues this evening it was because I was using the docker 0.8.0 on OSX via boot2docker / virtualbox and the init script had failed to create the 40GB drive so I quickly ran out of disk space on my virtualbox VM. Once I fixed that issue all was well... In other words, my host OS ran out of disk space and on the surface it looked like the Docker.IO CDN was wigging out..
merriam
commented
10 years ago I ran into this today as well, using 0.8 on OS X.
$ boot2docker up worked fine to recover.The VirtualBox partition lists 92% full, but the ~/.boot2docker/boot2docker.vmdk is under 2GB of its 40GB default size.
Here is a (longish) raw transcript, as I can never tell what matters in an unreproducible bug.
unclejack
vieux
sanemat
sidorykp
michaelneale
bryanlarsen
jcelliott
Hi,
I've started to try out docker, but can't pull large images.
docker pull busyboxfor example works fine, whiledocker pull centosordocker pull ubuntugive me an error similar to the following:The daemon gives the followinng output when debug is turned on, in case it's helpful:
I'm running CoreOS under vagrant.