riga / law

Build large-scale task workflows: luigi + job submission + remote targets + environment sandboxing using Docker/Singularity
http://law.readthedocs.io
BSD 3-Clause "New" or "Revised" License
98 stars 41 forks source link

Enable copying of directories (directory targets) #109

Closed Nollde closed 2 years ago

Nollde commented 3 years ago

The target module implements the copying of files (file targets) 1. It would be nice to also enable the copying of whole directories (directory targets). A possible implementation of a file-directory-capable copying function could look like this 2:

import shutil, errno

def copyanything(src, dst):
    try:
        shutil.copytree(src, dst)
    except OSError as exc: # python >2.5
        if exc.errno == errno.ENOTDIR:
            shutil.copy(src, dst)
        else: raise
riga commented 3 years ago

Hi @Nollde , thanks for opening. Yes, this has been on my list for quite a while. The local implementation is straight forward, unlike the remote one which requires some time. However, it should actually be possible with the updated remote file interface.

riga commented 2 years ago

Added in 44e0f0ea93e8226b1457d28305b70723a7e85b76.