MGThePro
commented
3 years ago I am also looking to use a library for downloading files and multiple connection seems to be the only major piece that is missing for my needs
Open daveuk100 opened 3 years ago
MGThePro
commented
3 years ago I am also looking to use a library for downloading files and multiple connection seems to be the only major piece that is missing for my needs
daveuk100
commented
3 years ago I am also looking to use a library for downloading files and multiple connection seems to be the only major piece that is missing for my needs
As there was not a reply here, I went looking and adopted another project “GetParty” which has been fantastic:
https://github.com/vbauerster/getparty
It supports multi-part chunked downloading and is amazing at retrying the parts if they fail. Absolutely perfect for my needs and any issues I raise are resolved within 24 hours as the maintainer is very active.
I think that if Grab is still being maintained (not sure) it would be great for them to adopt the same approach and/or work with GetParty to create the best overall combination of the two projects. One go download library to rule them all!
justinfx
commented
1 year ago I'm using Grab pretty deeply integrated into my current solution, with the callback hooks, and now I also want to support Range chunk downloading. I was looking at the current implementation to see if there is a decent hook in to apply it in my own code. The only spot I see is a custom HTTPClient interface to intercept and handle the http Request. But it doesn't seem like it plays well with the rest of the design of the state machine. GetParty, mentioned in the earlier comment, doesn't appear to be a library; only a CLI. I've had a play with this option, which is both a CLI and library: https://github.com/pgollangi/fastget I do need this support, so I may end up trying to implement the fastget logic into Grab and see where I get. If it works out, I can submit a PR.
justinfx
commented
1 year ago I've got a PR up with support for parallel "Range" requests, if anyone wants to have a look? https://github.com/cavaliergopher/grab/pull/102
grab works very well and I see that you are already making use of Accept-Ranges response to support resuming of downloads, but do you have any supports or plans to support multipart downloading? (i.e. allowing us to download a file in chunks using several connections).
Scenario: I have a large file to download (several GB) and I find that when I use other go packages that support multipart I am able to download the file in a fraction of the time when compared to grab's single connection.