search

holgerBerger / hpc-workspace

Automatically exported from code.google.com/p/hpc-workspace
GNU General Public License v3.0
18 stars 13 forks source link

[feature_request] add force deletion to ws_release #96

Open riedel opened 2 years ago

riedel commented 2 years ago

Background: ws_release does not recover quota immediately, which is a design decision. However, manually triggering deletion particularly when dealing with a lot of files requires a lot of care (e.g. using lfs find to not trigger a lot of metadata reads).

It would be great if ws_release could provide an optional flag for immediate deletion.

Ideally this could be done not as part of the running main job (as this waste a lot of resources). However, it would be also OK if it is blocking because it at least caring about things (as another problem I concretely have is that my enroot containers creates files that are undeletable and need a change of permissions first)

Edit: As clarification: My use case might be a bit atypical: I need the workspace as ephemeral storage during a longer job. The job result is much smaller, why I can release the workspace at the end of the job automatically.

holgerBerger commented 1 year ago

yes, that is something that was considered a few times. I will have a look again.

cniethammer commented 1 year ago

I also got several request for an easy way to "delete" a workspace - not just releasing it - especially on systems where quotas are applied.

I just created a ws_delete script, which performs file deletion and ws_release in one command. The actual method used to delete the files and directories in the workspace might get tailored to the used filesystem, later, e.g., to apply a deletion rate limit. For the first go I used the rsync method for fastest deletion.

See #98

holgerBerger commented 1 year ago

ws_release --delete-data it has a warning and a 5 second delay to interrupt. if it gets interrupted, the workspace is released and recoverable. If it is not interrupted, data is deleted and the workspace is gone and not recoverable, DB entry is wiped as well.

cniethammer commented 1 year ago

For reference: Commit that first implemented this functionality: ac912be6243526539e1e0a5b883ced21e96386d2

I guess it's open for testing and feedback ;)