I'm trying to isolate a task in its own directory because it can generate a large number of files. I saw tempdir and thought that it could serve this purpose, but when I run the task with tempdir=True it returns an empty list, whereas by omitting this switch it does return all files.
The task:
@task(tempdir=True)
def splitter(data: File) -> List[File]:
return script(
f'''
split -d -l 2 mydata.txt part.
''',
inputs = [data.stage('mydata.txt')],
outputs = [File(f).stage(str(f)) for f in glob_file('part.*')],
)
Maybe tempdir=True has a different purpose? In any case, what is the recommended approach to isolate tasks in their own directory. (I'm running redun locally in my own laptop)
I'm trying to isolate a task in its own directory because it can generate a large number of files. I saw
tempdir
and thought that it could serve this purpose, but when I run the task withtempdir=True
it returns an empty list, whereas by omitting this switch it does return all files.The task:
Maybe
tempdir=True
has a different purpose? In any case, what is the recommended approach to isolate tasks in their own directory. (I'm running redun locally in my own laptop)Thanks.