melbahja / got

Got: Simple golang package and CLI tool to download large files faster 🏃 than cURL and Wget!
MIT License
709 stars 45 forks source link

just 2 issue while using got #29

Open brownchow opened 3 years ago

brownchow commented 3 years ago

Hi melbahja! I got 2 issue while using got, maybe we can improve

  1. save disk space I use got to download a file from http server support range feature. the file size is about 8GB, so it take a moment to complete. while downloading, I find in C:\Users\{username}\AppData\Local\Temp\GotChunks{some digits} directory, there are 40 chunk files, chunk-0 ~ chunk-39. they are temp files to assemble the origin file, this means, my computer should have 16GB free disk space to complete the download progress, is there any way to optimize? I mean, since got remove these chunk-i files after merge them together, but, if I don't have 16GB free disk space, the 8GB file can't be download. what if the the file is 200GB? I should first have 400GB free disk space, that's unreasonable.

  2. merge chunk files take a long time while all chunk files are downloaded, got didn't quit immediately, it take a while to merge these chunk files, after all chunk files merged, got quit, and we got the origin file, all chunk files removed also. if the origin 8GB file, it take a momnet to merge, what if a 200GB file, how long will it take to merge chunk files? I don't know if wget or curl will suspended a moment while download progress bar reach 100%

thanks anyway!

melbahja commented 3 years ago

Hi,

  1. What I can do now is to delete the every chunk after the merge immediately, and download the first chunk into the dest file directly. (temporary fix until I figure out a real solution).

  2. if got downloaded chunk-1, chunk-3, chunk-4, we can't merge chunk-3 until chunk-2 is finished, I think the solution to this depends on issue 1.

thanks :)

melbahja commented 3 years ago

the next major got version is optimized to fix this issue.

brownchow commented 3 years ago

thanks a lot!