download-files: xrootd protocol

tiborsimko commented 4 years ago

In addition to http protocol implemented in #22, we should add support for xrootd protocol which provides more bandwidth.

Check whether client system contains xrootd commands (xrdcp) or Python libraries (xrootd).

If yes, enable support for --protocol root.

If not, report help string what people has to do do install them.

This is not an MVP functionality for the first public release; it can be added later.

tiborsimko commented 3 years ago

Regarding Python libraries, we could use upstream xrootd or even our xrootdpyfs.

ParthS007 commented 3 years ago

Both options look good to me. 👍

How to choose b/w these two 😅 ?

https://github.com/google/s2geometry/issues/87#issuecomment-652605768

tiborsimko commented 3 years ago

This will need to amend also Dockerfile to have xrootd support there. There are two options:

a) Currently we are based on python:3.8-slim which is basically a stripped-down Debian distribution. Either we compile xrootd there in the usual way, or look forxrootd packages, e.g. for Ubuntu 18.04 LTS one can use:

$ echo 'deb http://storage-ci.web.cern.ch/storage-ci/debian/xrootd/ bionic master' >> /etc/apt/sources.list
$ curl -sL http://storage-ci.web.cern.ch/storage-ci/storageci.key | apt-key add -
$ apt-get update
$ apt-get install xrootd-client

b) We can switch our base image to CentOS7, which is what CERN batch farms typically use, where xrootd can be installed via EPEL:

$ yum install epel-release
$ yum update
$ yum install xrootd-client

Note that we have some prior art in opendata.cern.ch repository (see its Dockerfile) or in reana-env-root6 (this is for ROOT, but analogously one can do xrootd).

@ParthS007 please let me know if you shall look into adding xrootd to our Docker image yourself or whether I should.

cernopendata / cernopendata-client

download-files: xrootd protocol #24