search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 655 forks source link

`StorageManager` enhancements #1171

Open artium-rsipvision opened 11 months ago

artium-rsipvision commented 11 months ago

I have recently found out that StorageManager can be used on its own and since then I have been using it together with ClearML's fileserver to store some files that are not related to a particular experiment or dataset. Also, being able to access the file by URL, without setting up your own file server is very useful.

I would like to propose the following enhancements:

  1. In the docs, elaborate on what is the expected behavior when you upload a file that has already been uploaded.
  2. Add a delete_file() method.
  3. Make it easier for the user to manage the remote URLs by providing a method for sanitizing them (currently I'm using pathlib.Path.as_uri() to get this job done, but you should already have this code somewhere anyway for the artifact and debug samples uploading, just need to expose it to the user).
ainoam commented 11 months ago

Glad you're finding the project useful @artium-rsipvision, and thanks for the suggestions :)

These sound like excellent enhancements to have - We'll add them to our list and update as we get around to adding.