Download all files from a Catalog at once

[ebuchlin]

Downloading files from a Catalog currently involves downloading each file separately or adding each file to a Downloader before downloading them.

from parfive import Downloader
downloader = Downloader()
result.iloc[:10].apply(
   lambda row: FileMetadata(row).download_file(
      "/tmp/spice-files",  # base directory
      release="2.0",
      downloader=downloader
   ),
   axis=1
)
downloader.download()

This could be a method of the Catalog object.

[safimuhammad]

hello, I'd like to work on this issue, would you explain the requirements and details of it?

[ebuchlin]

Yes, if you would like to.

The logic should be exactly the same as in FileMetadata.download_file(), but it should work on all files of a Catalog.

• The core of the function can be the pandas.DataFrame.apply() given in the description of this issue
• base_dir (the directory where the files should be written), base_url (the base URL where the files come from) and release (the release the files come from), and keep_tree (keep directory tree structure) can be just passed down to FileMetadata.download_file()
• The parfive.Downloader object passed as argument downloader should be used (and passed down to FileMetadata.download_file()), or created if it is None.
• If a downloader is provided as argument, the files are only enqueued in it, and not effectively downloaded within the function (the user will do it later). Otherwise, they are downloaded, and the resulting parfive.Result object is returned.

The function could be named Catalog.download_files() (plural) if that's not too confusing.

[safimuhammad]

thanks for the details will start working on it, I'll let you know if I need more info.

[safimuhammad]

one more thing, should the method download the entire catalog or should it ask for a range of files to download like in earlier examples?

p.s: my current implementation downloads the entire Catalog which I think is not what you intended also downloading the entire Catalog takes a lot of time due to num of files.

[ebuchlin]

It is true that downloading the full catalog of a data release (currently 1.5TB for data release 4.0) would take an enormous amount of time (and resources).

But I think the logic should still be that most of the time all the data corresponding to a full Catalog object should be downloaded, as this is simpler. The expectation is that the user does this only after downselecting the catalog to a much smaller catalog than the full data release.

However there could be a max_download parameter with a sufficiently large default (I would say at least 1000 files) that can be overridden by the user; with a warning if we hit the limit.

[safimuhammad]

Makes sense, implemented it now opening up pull request.