Delete orphaned files from MongoDB

TUM-DAML / seml

SEML: Slurm Experiment Management Library

Other

165 stars 29 forks source link

Delete orphaned files from MongoDB #97

Closed danielzuegner closed 2 years ago

danielzuegner commented 2 years ago

improve workflow for cleaning all unreferenced files from the database

Reference issue

Addresses Issue #95.

What does this implement/fix?

Delete source files uploaded by Sacred when an experiment is deleted
Improve seml clean-db workflow for removing orphaned files in the MongoDB.

Additional information

heborras commented 2 years ago

This is a really nice extension to seml. Thanks a lot for taking the time! From looking at the code it seems that clean_unreferenced_artifacts now properly deletes the orphaned artifacts. And delete_experiments now appears to properly delete the source files added by sacred. However, as far as I can tell the artifacts associated with an experiment are not deleted by delete_experiments. Did I miss something in the code or could this functionality maybe be added? That way one wouldn't need to run clean_unreferenced_artifacts as often.

danielzuegner commented 2 years ago

This is a really nice extension to seml. Thanks a lot for taking the time! From looking at the code it seems that clean_unreferenced_artifacts now properly deletes the orphaned artifacts. And delete_experiments now appears to properly delete the source files added by sacred. However, as far as I can tell the artifacts associated with an experiment are not deleted by delete_experiments. Did I miss something in the code or could this functionality maybe be added? That way one wouldn't need to run clean_unreferenced_artifacts as often.

Good point! I've added that in the latest commit.

heborras commented 2 years ago

Do you think it would be possible to create a new release on pip and conda, when the PR is merged? This would be a really nice feature to have available in the published package.