Downloading Text file gives an issue while in resume mode.

cavaliergopher / grab

A download manager package for Go

BSD 3-Clause "New" or "Revised" License

1.38k stars 151 forks source link

Downloading Text file gives an issue while in resume mode. #18

Closed jagtapajittheneo closed 6 years ago

jagtapajittheneo commented 6 years ago

I am trying to to download text file form a URL.

Grab downloads it perfectly first time.

But if I make some small change in source text file and try to download again; Grab append that change to end of a file. now downloaded text file is of a no use.

Can you please help in this regards?

cavaliercoder commented 6 years ago

Interesting problem! At this stage, grab is stateless; it doesn't store file progress or checksums anywhere. This could be solved by the parent program using E-Tags or Last-Modified headers.

I may also be able to account for this issue in grab by comparing local and remote timestamps, but this is prone to error when either party has incorrect config.

I'll think on this some more. Thanks for raising the issue.

cavaliercoder commented 6 years ago

There's another complication to this issue: Incomplete downloads will always appear to be more recent than a remote file, as the local timestamp is only set once the download is complete. This means incomplete downloads will always assume the local data is for the most recent version of the file and will therefore never be overwritten.

Example:

remote file is dated c.2000
grab downloads 50% of file - local timestamp is current system time
remote file is modified at origin and now dated c.2010
grab compares local time to remote time - local file appears to be up to date
grab resumes download from 50% and corrupts the local file

cavaliercoder commented 6 years ago

I've created PR #19 to track development for this issue.

cavaliercoder commented 6 years ago

Please see notes on #21. For now, grab does not offer any correctness guarantees above those in net/http. The parent application can take advantage of checksums or implement their own state storage to overcome this issue.