It4innovations / hyperqueue

Scheduler for sub-node tasks for HPC systems with batch scheduling
https://it4innovations.github.io/hyperqueue
MIT License
272 stars 21 forks source link

Add automatic file cleanup after task success #613

Closed Kobzol closed 1 year ago

Kobzol commented 1 year ago

If a task succeeds, often we do not care about its stdout/stderr output. This PR adds a "file closing mode" to stdout/stderr, which makes HQ automatically remove the files from disk if their task succeeds.

Documentation and changelog isn't done yet, I first want to bakeshed the various names and flags :)

Kobzol commented 1 year ago

I do not like inconsistency between "cleanup" and "delete-if-success". How you want to name keyword for removing empty files, as it will be also kind of "cleanup"?

I agree! But what name should be used on the CLI? Isn't delete-if-finished a bit too long? Maybe it's fine…

spirali commented 1 year ago

Brainstorming:

spirali commented 1 year ago

Looking at my examples, I'm starting to slightly prefer the last one.

Kobzol commented 1 year ago

Renamed to rm-if-finished and added documentation and changelog.