search

rwth-i6 / sisyphus

A Workflow Manager in Python
Mozilla Public License 2.0
45 stars 24 forks source link

extend file caching #79

Open christophmluscher opened 2 years ago

christophmluscher commented 2 years ago

I was thinking about the extension of the file caching in Sisyphus. I would like to make an interface available so that the user can read and write using local file caching.

The idea would be to extend Sisyphus to support three user-defined functions. One is already available via file_caching. The other two would support copying between remote and local paths.

The Sisyphus AbstractPath Object would need additional functions to then facilitate remote/local copy operations.

This would leave the actual usage of file caching in the hands of the user because this would need to be implemented inside the jobs. This gives the user more direct control over the file caching.

@curufinwe @critias Do you think this is a good approach? Or do you think it would be better to abstract everything away? So hide inside Sisyphus? I have extended the API to support the above idea but wanted to discuss the general approach before making a pull request.

critias commented 2 years ago

I'm not quite sure what you mean. Are you talking about something like this: some_path.copy_from_server() do_stuff(some_path) some_path.copy_back_to_server()

Overall I'm not really happy with the current state of file_caching, so I'm very open for suggestions how to improve it.