Parsl supports caching of function output when properly configured, but this seems to be limited to computed output of a function. It may not immediately apply to our use case of files as output.
First we should investigate to determine if the above statement is true. Perhaps it does support caching of file output.
Second, we could implement a quick check in each function (though this should be pushed up to a parent class) that can look for the presence of a file on disk and short circuit the expensive work if the file already exists. Or, if a force=True parameter is passed, then redo the work to produce the output file.
Parsl supports caching of function output when properly configured, but this seems to be limited to computed output of a function. It may not immediately apply to our use case of files as output.
First we should investigate to determine if the above statement is true. Perhaps it does support caching of file output.
Second, we could implement a quick check in each function (though this should be pushed up to a parent class) that can look for the presence of a file on disk and short circuit the expensive work if the file already exists. Or, if a force=True parameter is passed, then redo the work to produce the output file.