insitro / redun

Yet another redundant workflow engine
https://insitro.github.io/redun/
Apache License 2.0
510 stars 43 forks source link

How to isolate tasks in their own directory? #87

Open Asutu opened 11 months ago

Asutu commented 11 months ago

I'm trying to isolate a task in its own directory because it can generate a large number of files. I saw tempdir and thought that it could serve this purpose, but when I run the task with tempdir=True it returns an empty list, whereas by omitting this switch it does return all files.

The task:

@task(tempdir=True)
def splitter(data: File) -> List[File]:
    return script(
        f'''
        split -d -l 2 mydata.txt part.
        ''',
        inputs = [data.stage('mydata.txt')],
        outputs = [File(f).stage(str(f)) for f in glob_file('part.*')],
    )

Maybe tempdir=True has a different purpose? In any case, what is the recommended approach to isolate tasks in their own directory. (I'm running redun locally in my own laptop)

Thanks.