search

probonopd / zsync-curl

Partial/differential file download client over HTTP(S)
110 stars 23 forks source link

Cannot download from download.opensuse.org MirrorBrain #22

Closed probonopd closed 7 years ago

probonopd commented 7 years ago 
CURLOPT_VERBOSE=1 ./zsync_curl http://download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-latest-x86_64.AppImage.zsync

*   Trying 2001:67c:2178:8::13...
* TCP_NODELAY set
* Connected to download.opensuse.org (2001:67c:2178:8::13) port 80 (#0)
> GET /repositories/home:/probono/AppImage/QtQuickApp-latest-x86_64.AppImage.zsync HTTP/1.1
Host: download.opensuse.org
Accept: */*

< HTTP/1.1 302 Found
< Date: Sun, 28 May 2017 08:18:02 GMT
< Server: Apache/2.2.12 (Linux/SUSE)
< X-MirrorBrain-Mirror: anorien.csc.warwick.ac.uk
< X-MirrorBrain-Realm: region
< Link: <http://download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-latest-x86_64.AppImage.zsync.meta4>; rel=describedby; type="application/metalink4+xml"
< Link: <http://anorien.csc.warwick.ac.uk/mirrors/download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync>; rel=duplicate; pri=1; geo=gb
< Link: <http://ftp1.nluug.nl/os/Linux/distr/opensuse/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync>; rel=duplicate; pri=2; geo=nl
< Link: <http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync>; rel=duplicate; pri=3; geo=nl
< Digest: MD5=L3Ho46G1PRJnz0UYYDCwNQ==
< Digest: SHA=56R/1LlBZWmBM/r1uWHvJJllJ0o=
< Digest: SHA-256=9WIl0VtswH7vyEaHqcDhkyhY/ks3XtcXl34BnDkg4ag=
< Location: http://anorien.csc.warwick.ac.uk/mirrors/download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync
< Content-Length: 436
< Content-Type: text/html; charset=iso-8859-1
< 
* Ignoring the response-body
* Connection #0 to host download.opensuse.org left intact
* Issue another request to this URL: 'http://anorien.csc.warwick.ac.uk/mirrors/download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync'
*   Trying 2001:470:1f09:1027:137:205:124:72...
* TCP_NODELAY set
* Connected to anorien.csc.warwick.ac.uk (2001:470:1f09:1027:137:205:124:72) port 80 (#1)
> GET /mirrors/download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync HTTP/1.1
Host: anorien.csc.warwick.ac.uk
Accept: */*

< HTTP/1.1 200 OK
< Date: Sun, 28 May 2017 08:18:12 GMT
< Server: Apache/2.2.12 (Linux/SUSE)
< Last-Modified: Sat, 27 May 2017 10:26:57 GMT
< ETag: "b0f4-5507ee4b93e40"
< Accept-Ranges: bytes
< Content-Length: 45300
< X-Clacks-Overhead: GNU Terry Pratchett
< Content-Type: text/plain
< 
* Connection #1 to host anorien.csc.warwick.ac.uk left intact

### Leaving referer as http://download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-latest-x86_64.AppImage.zsync

### Setting redirected to http://download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-latest-x86_64.AppImage.zsync
reading seed file QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.part: *************
Read QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.part. Target 1.0% complete.      

### fetch from QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage ### USE THE REDIRECTED URL FROM NOW ON
)
### make_url_absolute(http://download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-latest-x86_64.AppImage.zsync, QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage)
### Redirected payload URL: http://ftp1.nluug.nl/os/Linux/distr/opensuse/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage

### make_url_absolute(http://anorien.csc.warwick.ac.uk/mirrors/download.opensuse.org/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync, http://ftp1.nluug.nl/os/Linux/distr/opensuse/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage)
downloading from http://ftp1.nluug.nl/os/Linux/distr/opensuse/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage:
*   Trying 2001:67c:6ec:221:145:220:21:40...
* TCP_NODELAY set
* Connected to ftp1.nluug.nl (2001:67c:6ec:221:145:220:21:40) port 80 (#0)
> GET /os/Linux/distr/opensuse/repositories/home:/probono/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage HTTP/1.1
Host: ftp1.nluug.nl
Range: bytes=126976-13166591
Accept: */*

< HTTP/1.1 206 Partial Content
< Date: Sun, 28 May 2017 08:18:13 GMT
< Server: Apache/2.2.15 (CentOS)
< Last-Modified: Sat, 27 May 2017 18:27:58 GMT
< ETag: "480a08222-c8f498-550859cfc27db"
< Accept-Ranges: bytes
< Content-Length: 13039616
< Content-Range: bytes 126976-13166591/13169816
< Content-Type: text/plain; charset=UTF-8
< 
* Closing connection 0
failed to retrieve from QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage 1
Aborting, download available in QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.part

Why is it Closing connection 0?

TheAssassin commented 7 years ago

I am not sure. At a fist glance everything looks fine. I'll have to reproduce the problem and step in with a debugger.

TheAssassin commented 7 years ago

Might be a problem of one of the mirrors not being up to date. Both requests are redirected to two different locations, one being a UK and one an NL mirror. Have to investigate that.

probonopd commented 7 years ago

Also cannot download when trying to download directly from the dereferenced URL of the mirror:

CURLOPT_VERBOSE=1 ./zsync_curl "http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync"
*   Trying 2001:67c:6ec:221:145:220:21:40...
* TCP_NODELAY set
* Connected to ftp2.nluug.nl (2001:67c:6ec:221:145:220:21:40) port 80 (#0)
> GET /os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync HTTP/1.1
Host: ftp2.nluug.nl
Accept: */*

< HTTP/1.1 200 OK
< Date: Fri, 02 Jun 2017 17:40:57 GMT
< Server: Apache/2.2.15 (CentOS)
< Last-Modified: Thu, 01 Jun 2017 21:27:25 GMT
< ETag: "d3bb801a-adf2-550ecb3f16940"
< Accept-Ranges: bytes
< Content-Length: 44530
< Content-Type: text/plain; charset=UTF-8
< 
* Connection #0 to host ftp2.nluug.nl left intact

### Leaving referer as http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync

### Setting redirected to http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync
Target QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage
No relevent local data found - I will be downloading the whole file. If that's not what you want, CTRL-C out. You should specify the local file is the old version of the file to download with -i (you might have to decompress it with gzip -d first). Or perhaps you just have no data that helps download the file

### fetch from QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage ### USE THE REDIRECTED URL FROM NOW ON
)
### make_url_absolute(http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync, QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage)
### Redirected payload URL: http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage

### make_url_absolute(http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.zsync, http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage)
downloading from http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage:
-------------------- 0.0%*   Trying 2001:67c:6ec:221:145:220:21:40...
* TCP_NODELAY set
* Connected to ftp2.nluug.nl (2001:67c:6ec:221:145:220:21:40) port 80 (#0)
> GET /os/Linux/distr/opensuse/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage HTTP/1.1
Host: ftp2.nluug.nl
Range: bytes=0-12945407
Accept: */*

< HTTP/1.1 206 Partial Content
< Date: Fri, 02 Jun 2017 17:40:57 GMT
< Server: Apache/2.2.15 (CentOS)
< Last-Modified: Thu, 01 Jun 2017 21:27:26 GMT
< ETag: "d3bb800f-c58498-550ecb40652e0"
< Accept-Ranges: bytes
< Content-Length: 12944536
< Content-Range: bytes 0-12944535/12944536
< Content-Type: text/plain; charset=UTF-8
< 
-------------------- 0.3% 21.8 kBps aborted    

* Closing connection 0
failed to retrieve from QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage 1
Aborting, download available in QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage.part
verifying download...
probonopd commented 7 years ago

It is caused by a sha1sum mismatch:

wget -c https://download.opensuse.org/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-latest-x86_64.AppImage.zsync
wget -c https://download.opensuse.org/repositories/home:/adrianSuSE:/AppImage/AppImage/QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage

me@host:~$ sha1sum QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage 
fde5bdf0d0d97ea6279ea618a1381ae234720f57  QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage

me@host:~$ head QtQuickApp-latest-x86_64.AppImage.zsync -n 8
zsync: 0.6.2
Filename: QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage
MTime: Wed, 31 May 2017 04:38:08 +0000
Blocksize: 2048
Length: 12944536
Hash-Lengths: 2,2,5
URL: QtQuickApp-1495317991.fce72f3.glibc2.2.5-x86_64.AppImage
SHA-1: 2d05488e0881e661e961b370fde992ae11f72ccd

@adrianschroeter do you have any idea what might be causing this?

TheAssassin commented 7 years ago

This might be that "sign after zsyncmake" bug which caused similar errors last time. @adrianschroeter might have to update AppImageKit on OBS.

probonopd commented 7 years ago

Right, it was fixed in https://github.com/probonopd/AppImageKit/commit/88853e9e80ac4badc2c62e42c2c1d8a2f9f275ae

I can currently find on OBS:

Not sure what is going on.

probonopd commented 7 years ago

According to @adrianschroeter, OBS does not use appimagetool -s because at the time when appimagetool runs, the signing key is not available for security reasons. Instead, the AppImage is signed after the fact using obs-sign, which of course changes the checksum of the AppImage, which causes the mismatch with the zsync file. One option around this would be to (re-)generate the zsync file after the AppImage gets signed.

So not a zsync-curl issue, hence opened https://github.com/openSUSE/open-build-service/issues/3203 and closing here.

probonopd commented 7 years ago

Will be handled by regenerating zsync files in https://github.com/openSUSE/open-build-service/blob/master/src/backend/bs_signer

probonopd commented 7 years ago

I can confirm that it is working now with zsync files on OBS generated from now on. Thanks @adrianschroeter.