search

cernopendata / cernopendata-client

CERN Open Data command-line client
http://cernopendata-client.readthedocs.io/
GNU General Public License v3.0
10 stars 9 forks source link

download-files: expose HTTP downloader choice (requests vs pycurl) #116

Closed tiborsimko closed 3 years ago

tiborsimko commented 3 years ago

Current behaviour

Currently, when a user downloads a file over HTTP, for example:

$ cernopendata-client download-files --recid 5500

then the downloader uses either "requests" library or "pycurl" library, depending on which one is installed.

The user cannot easily influence choosing one or another if both are installed, such as in the Docker container cernopendata/cernopendata-client:0.2.0. When both are available, "pycurl" is automatically selected by the client package.

This would be OK if both "requests" and "pycurl" downloaders provide the same functionality. However, recently, as part of the work on the resume download functionality #99, one downloader technique offers richer user experience than the other. It would be therefore good to expose the downloader choice up to the user.

Expected behaviour

Let us introduce a new command-line option for the download-files command that would expose the choice of HTTP downloader to use, i.e. "requests" or "pycurl", up to the end user. Example:

$ cernopendata-client download-files --recid 5500 --downloader requests
$ cernopendata-client download-files --recid 5500 --downloader pycurl

Note that some combinations are illegal and will have to be checked for, such as:

$ cernopendata-client download-files --recid 5500 --protocol xrootd --downloader requests

However the new --downloader option can come handy for XRootD protocol too, for example when we shall support both "xrootdpyfs" (current) and vanilla "xrootd" (future) downloaders for the XRootD protocol as well.

We can therefore implement the option for the HTTP downloader, keep the XRootD downloader hard-coded to xrootdpyfs, introduce a CLI option validator to make sure only good combinations are accepted, and introduce a test case for it.

tiborsimko commented 3 years ago

Closed by #118. The option name is --download-engine.