search

probonopd / zsync-curl

Partial/differential file download client over HTTP(S)
109 stars 23 forks source link

Download fails on files larger than 2147483647 bytes #19

Closed andrewwipf closed 7 years ago

andrewwipf commented 7 years ago

Symptom

I received this error trying to download a large (+5GB) ISO file:

downloading from <redacted>/<largefilename>:
-------------------- 0.0%bad status code 416
Other error? -1
-1 returned
-------------------- 0.0% 0.0 kBps aborted

failed to retrieve from <largefilename> -1

Steps to reproduce

Create two test files and corresponding .zsync files from /dev/random, one slightly smaller and one slightly larger than 2147483647 bytes. Here are mine:

large_file.2.0 downloads via zsync_curl without issue. Here is debug output when downloading large_file.2.2:

downloading from https://<redacted>/large_file.2.2:

<cert info redacted>

> GET <redacted>/large_file.2.2 HTTP/1.1
Range: bytes=0-18446744071700615167
Host: <redacted>
Accept: */*

< HTTP/1.1 416 Requested Range Not Satisfiable
< Date: Mon, 15 May 2017 19:03:57 GMT
< Server: Artifactory/<redacted>
< X-Artifactory-Id: <redacted>
< X-Artifactory-Node-Id: <redacted>
< Content-Range: bytes */2286026765
< Content-Length: 0
< Content-Type: application/x-troff-man
<
* Connection #2 to host <redacted> left intact
bad status code 416
Other error? -1
-1 returned
-------------------- 0.0% 0.0 kBps aborted

failed to retrieve from large_file.2.2 -1
Aborting, download available in large_file.2.2.part
verifying download...

Exit code: 2

Comments

I do not know C, but it seems that somewhere an INT for file size is being exceeded, a mathematical mystery occurs, then zsync_curl requests the first 18 Quintillion (and change) bytes of the file. My webserver responds with '416 Requested Range Not Satisfiable' because that's a lot of bytes to ask for.

Environment

Tested on latest commit to Master: 314ab4935c1cee6adec81e47d64ba1534e9c9555 Confirmed behavior on:

probonopd commented 7 years ago

Thank you very much for reporting this @andrewwipf, excellent catch. Unfortunately I don't know how to fix this yet. Any help from anyone appreciated.

probonopd commented 7 years ago

Fixed in https://github.com/probonopd/zsync-curl/commit/107ffd6776483b21af0bb3ecf8833ec60e92c526