LAW support of HDFS targets

riga / law

Build large-scale task workflows: luigi + job submission + remote targets + environment sandboxing using Docker/Singularity

http://law.readthedocs.io

BSD 3-Clause "New" or "Revised" License

98 stars 41 forks source link

LAW support of HDFS targets #174

Open BlancoFS opened 6 months ago

BlancoFS commented 6 months ago

The Pull Request (PR) introduces the possibility of handling targets in HDFS (Hadoop Distributed File System). The structure of the new classes mimics the GFALFileInterface and WLCGFileSystem ones.

New classes:

HDFSFileInterface
HDFSFileSystem
HDFSTarget
HDFSFileTarget
HDFSDirectoryTarget