Cleanup Workflows directory in bee_workdir

pagrubel commented 8 months ago

Once a workflow has been completed and it is archived, the directory in bee_workdir/workflows should be removed. The following is the current example where two workflows completed.

$ ls -l ~/.beeflow/archives
total 56
-rw-rw-r-- 1 pagrubel pagrubel 28362 Oct 30 13:25 49a68c1836cf40b8a3a74f4bc5064e43.tgz
-rw-rw-r-- 1 pagrubel pagrubel 28156 Oct 30 13:34 8297889ce5624d70bc08872eb563875c.tgz
$ ls -l ~/.beeflow/workflows
total 0
drwxrwxr-x 2 pagrubel pagrubel 4096 Oct 30 13:25 49a68c1836cf40b8a3a74f4bc5064e43
drwxrwxr-x 2 pagrubel pagrubel 4096 Oct 30 13:34 8297889ce5624d70bc08872eb563875c

[ ] For archived this behavior is probably fine, but we'll definitely to keep the workflow directory around if it's Archived/Failed
[ ] Add config option to toggle this behavior

pagrubel commented 5 days ago

@rstyd @kabir-vats I don't see any reason to keep the workflow in the workflows directory at all. You can always untar the archive

kabir-vats commented 5 days ago

I will implement for all then

pagrubel commented 5 days ago

@rstyd Do you have any objection to not keeping FAILED/ARCHIVED workflow directories since the tarball can be untarred if you need to look at the failure?

lanl / BEE

Cleanup Workflows directory in bee_workdir #735