allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.72k stars 657 forks source link

delete datasets after call get_local_copy #1273

Open lolpa1n opened 6 months ago

lolpa1n commented 6 months ago

Hello, I started to understand datasets In clearml and would like to know, after I created the datasets, I call get_local_copy, train the model, for example, and after that I need to delete this local copy, how do I do this? rm -rf /root/.clearml/cache/storage_manager/datasets/* ? or is there some other correct way

jkhenning commented 4 months ago

Hi @lolpa1n, since the get_local_copy() call returns the file, you can always use a Python call to delete it (i.e. os.remove(file)). However, please note these files are downloaded to the cache, which is limited (by default, a maximum of 100 files), so the ClearML SDK will automatically delete older files when new files are being downloaded there)