Implement resumable downloads (ref. #2)

max-mapper / nugget

minimalist wget clone written in node. HTTP GET files and downloads them into the current directory

BSD 3-Clause "New" or "Revised" License

172 stars 30 forks source link

Implement resumable downloads (ref. #2) #3

Closed grncdr closed 10 years ago

grncdr commented 10 years ago

This implements resumable downloads by attempting to use range requests when the target file already exists.

A target file is considered incomplete if it's size is less than that of the remote file.
The remote files size is determined by the content-length returned by a HEAD request to the remote URL.
If the target files size is greater than or equal to the remote files size, nothing will be done.
If the remote server does not accept range requests, the existing file will be overwritten.
Otherwise, the rest of the files contents will be downloaded and appended to the target file by using a Range header.

grncdr commented 10 years ago

fair warning: I haven't run any of the tests yet. Just seeing how it looks. If the design isn't too horrible (it nearly doubles the code size :frowning:), I will see about adding another test for it.

max-mapper commented 10 years ago

@grncdr I read through the diffs, this looks really good so far!

grncdr commented 10 years ago

Cool, I added a test and figured out I was doing something slightly wrong. I was assuming that content-length should refer to the length of the entire entity, but it turns out that it should always reflect just the payload for the current request. The test is correct afaict, but I should probably do some manual testing against an nginx or apache server just to make sure my test server isn't doing something wrong.

grncdr commented 10 years ago

Ok, I've tested this against a local nginx by killing a download partway through and then resuming it and everything works (the md5 matches for original file and downloaded files).