New Features: Recursive and multiple downloads

[kylepmanuel]

New feature requests:

• -r | --recursive - Recursively download folder and contents without zipping
• -i | --input FILE_NAME - Use a text file as input for download. One FILE_ID per line

Recursively download a folder and it's contents without zipping:

kaput files download -r FILE_ID

Download all files/folders listed by id from an input file. This would still zip folders:

kaput files download -i downloads.txt

Download all files/folders listed by id from an input file recursively. No zipping of folders:

kaput files download -ir downloads.txt

Possibly allow piping as input which would effectively mean this command downloads everything recursively without zipping:

kaput files list | perl -ne '/^\|\s(\d{8,10})\s+\|/ && print "$1\n";' | kaput files download -r

or if you are in the mood for formatting output something like this:

kaput files list -o '{id}\n' | kaput files download -r

where {field_name} is one of the available fields printed from kaput files list headers

[davidchalifoux]

Can you explain the use case for this? Why doesn't a ZIP suffice?

[kylepmanuel]

Example 1:

I have 100Gb free space on a drive. I want to download a folder whose contents totals 60Gb. I don't have enough room to download the zip and then extract it.

Example 2:

I am setting up local on-prem backup. This would be an initial "download all" without any extra processes.

Example 3:

Honestly, I'm lazy and don't want an extra step.

[jklausa]

I wanted to +1 the -r flag.

I am often downloading files directly to my media HDD, and unzipping often takes longer than the download itself.

(I know having 2Gbps downlink and slow spinning rust HDDs is not exactly a super common combination, but it would be nice to be able to wait less on things :) )

[davidchalifoux]

I wanted to +1 the -r flag.

I am often downloading files directly to my media HDD, and unzipping often takes longer than the download itself.

(I know having 2Gbps downlink and slow spinning rust HDDs is not exactly a super common combination, but it would be nice to be able to wait less on things :) )

Alright, I’ll try to get to this when I can.

[davidchalifoux]

Got an initial version of recursive downloads working. If you would like to test it, try running the branch: https://github.com/davidchalifoux/kaput-cli/tree/feat/recursive-downloads

Notes:

• If you pass the -r flag when downloading, it will recursively download all files and also create all subfolders including the parent. I still need to test on Windows to make sure the file paths aren't broken there.
• Because I'm just calling curl, it will also skip files already downloaded and pick up downloads that didn't finish 😄
• With this, I'll also add a flag for specifying a download path.

Give it a try and let me know if you have feedback. I'm hoping to have it in a release this weekend.

[davidchalifoux]

Recursive downloads are now available in v2.4.0. I'll work on adding multiple downloads and potentially input files as well.

Let me know if I broke anything. I did a lot of refactoring.

[kylepmanuel]

I have tried this with 3 different folders with varying structures. Everything appears to have downloaded correctly. I have not retried many other features, but this does work.

[jklausa]

I also checked it out recently, and worked perfectly — appreciate your work!