shawwn / llama-dl

High-speed download of LLaMA, Facebook's 65B parameter GPT model
GNU General Public License v3.0
4.17k stars 419 forks source link

Prefer using aria2 instead of wget #14

Open arguiot opened 1 year ago

arguiot commented 1 year ago

Aria2 is a much faster downloading utility. https://github.com/aria2/aria2 You can easily fix your script by replacing all the wget occurrences with aria2c and lower case the -O

nelsonjchen commented 1 year ago

I guess much faster way would be to use aria2's multi-connection download capability. From hosts with very high capacity bandwidth like 300Mb/s +, this would help noticeably. 16, split, simultaneously? Very fast.

https://stackoverflow.com/questions/55166245/aria2c-parallel-download-parameters

My suggestion would be to opportunistically use aria2c if it's available with fast parameters set.

nelsonjchen commented 1 year ago

But curl and wget must support multi-connection from different replicas and ips on DNS already...

Do you have any examples? I cannot find any documentation on how this can be accomplished. In fact, I find the opposite as all documentation seems to point to curl/wget being only able to handle a single connection at a time for a single file.

BrunoFerreira95 commented 1 year ago

Reading this GitHub, I realize something about making a download split by number and use Node Streams solution to make multiple download of one file split ready the files and groups all in the end