search

probonopd / zsync-curl

Partial/differential file download client over HTTP(S)
110 stars 23 forks source link

Amazon S3 download fails #17

Closed probonopd closed 7 years ago

probonopd commented 7 years ago

Running AppImageUpdate with increased verbosity

CURLOPT_VERBOSE=1 ./AppImageUpdate-391.gccf4598-x86_64.AppImage /home/me/Downloads/kmail-git20170213-16_27_45-x86_64.AppImage

results in

### make_url_absolute(https://s3-eu-central-1.amazonaws.com/ds9-apps/kmail-master-appimage/kmail_latestversion.zsync, https://s3-eu-central-1.amazonaws.com/ds9-apps/kmail-master-appimage/kmail-git20170213-18:53:49-x86_64.AppImage)
downloading from https://s3-eu-central-1.amazonaws.com/ds9-apps/kmail-master-appimage/kmail-git20170213-18:53:49-x86_64.AppImage:
Target kmail-git20170213-18:53:49-x86_64.AppImage

###################- 97.8%*   Trying 54.231.192.45...
* TCP_NODELAY set
* Connected to s3-eu-central-1.amazonaws.com (54.231.192.45) port 443 (#0)
* SSL connected
> GET /ds9-apps/kmail-master-appimage/kmail-git20170213-18:53:49-x86_64.AppImage HTTP/1.1
Host: s3-eu-central-1.amazonaws.com
Range: bytes=106496-114687
Accept: */*

* Curl_http_done: called premature == 0
* Empty reply from server
* Connection #0 to host s3-eu-central-1.amazonaws.com left intact
Other error? -1
-1 returned

Empty reply from server is bad here. So let's retrieve the same range of bytes using curl manually like this:

me@host:~$ curl -v --header "Range: bytes=106496-114687" "https://s3-eu-central-1.amazonaws.com/ds9-apps/kmail-master-appimage/kmail-git20170213-18:53:49-x86_64.AppImage" | hexdump -C
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0*   Trying 54.231.192.53...
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0* Connected to s3-eu-central-1.amazonaws.com (54.231.192.53) port 443 (#0)
* found 187 certificates in /etc/ssl/certs/ca-certificates.crt
* found 758 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.s3.eu-central-1.amazonaws.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=Washington,L=Seattle,O=Amazon.com Inc.,CN=*.s3.eu-central-1.amazonaws.com
*        start date: Mon, 18 Jul 2016 00:00:00 GMT
*        expire date: Fri, 06 Oct 2017 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert Baltimore CA-2 G2
*        compression: NULL
* ALPN, server did not agree to a protocol
> GET /ds9-apps/kmail-master-appimage/kmail-git20170213-18:53:49-x86_64.AppImage HTTP/1.1
> Host: s3-eu-central-1.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> Range: bytes=106496-114687
> 
< HTTP/1.1 206 Partial Content
< x-amz-id-2: EQsT6RkoyPv6xNCttLKKoEAq++F/Qe1AyoFMkS4gwrBRAYzW695NFg8c45gkWUzr+TtKB5QMURA=
< x-amz-request-id: D2E6FF0F455DAB26
< Date: Mon, 13 Feb 2017 19:50:26 GMT
< Last-Modified: Mon, 13 Feb 2017 18:54:24 GMT
< ETag: "e300456af33db744380591cf46ff4673-37"
< Accept-Ranges: bytes
< Content-Range: bytes 106496-114687/188893208
< Content-Type: application/octet-stream
< Content-Length: 8192
< Server: AmazonS3
< 
{ [8192 bytes data]
00000000  01 4e 0e 60 02 c1 0e 68  51 0e 70 4a 0e 78 41 0e  |.N.`...hQ.pJ.xA.|
00000010  80 01 53 0e 60 00 00 00  64 00 00 00 f4 32 00 00  |..S.`...d....2..|
00000020  d0 71 ff ff 82 01 00 00  00 42 0e 10 8f 02 42 0e  |.q.......B....B.|

Looks like this works, which means that there must be a bug in zsync_curl which prevents this range of bytes from being downloaded, whereas manually downloding the same range of bytes succeeds. But why?

Is the SSL connected without any frurther information suspicious? Should we try AppImageUpdate or zsync_curl without WolfSSL statically linked and with OpenSSL instead to make sure that isn't the cause?

probonopd commented 7 years ago

Turns out it is actually a SSH problem, because zsync_curl build with openssl seems to be able to download from that URL just fine:

sudo apt-get -y install git devscripts autotools-dev libcurl4-openssl-dev
git clone https://github.com/probonopd/zsync-curl.git
cd zsync-curl/
./zsync-curl/build.sh
ldd /usr/local/bin/zsync_curl # Uses the system's OpenSSL
wget -c https://s3-eu-central-1.amazonaws.com/ds9-apps/kmail-master-appimage/kmail-git20170213-18:53:49-x86_64.AppImage
# I pressed Ctrl-C mid-download, to see whether zsync_curl can pick up from there
zsync_curl https://s3-eu-central-1.amazonaws.com/ds9-apps/kmail-master-appimage/kmail_latestversion.zsync
# Works

So we need to either make zsync_curl that gets bundled into AppImageUpdate use the system's OpenSSL (how? Given that there seem to be different versions floating around), or bundle OpenSSL into the AppImage, or fix whatever is the issue with the statically linked WolfSSL we have been using so far.

Possibly this is also the root cause of

To be confirmed.

probonopd commented 7 years ago

Or we could use dlopen:

We've used dlopen() for loading the methods from the OpenSSL library. When MailerQ starts, it tries to locate a version of the OpenSSL library that is known to be sufficient for MailerQ, and it uses the functions from that library. By using dlopen() in such a way we've managed to create a single executable file that can be installed on many different Linux distributions, and that uses the version of OpenSSL that is available.

http://www.bruijntjes.net/blog/post/8c504e2f4a79ff837221d77183c88d00

probonopd commented 7 years ago

AppImageUpdate version 443 and later should be able to download from S3 now, because they are using the semistatic build of zsync-curl (that links to libcurl statically but to OpenSSL dynamically).

Please reports how it works for you.

probonopd commented 7 years ago

Just fixed a bug in appimagetool that would result in the SHA-1 of the zsync file to be wrong when the AppImage was signed (since the zsync file had been generated prior to signing the AppImage). https://github.com/probonopd/AppImageKit/commit/88853e9e80ac4badc2c62e42c2c1d8a2f9f275ae

To verify that this is not the case with your AppImage, run

wget https://github.com/probonopd/AppImageKit/releases/download/continuous/appimagetool-x86_64.AppImage
wget https://github.com/probonopd/AppImageKit/releases/download/continuous/appimagetool-x86_64.AppImage.zsync

strings *zsync |  grep SHA-1 | cut -d " " -f 2
sha1sum *.AppImage

The signatures must match. If they don't, update your appimagetool.

With this in place, I am now able to download from GitHub Releases with zsync-curl successfully. Hence closing. Feel free to comment if you think that this needs to be reopened.