search

allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
381 stars 131 forks source link

The default dataset path #153

Open jis478 opened 2 years ago

jis478 commented 2 years ago

Hi,

Thanks for the awesome project! I'm wondering if I can change the default path for datasets? Currently they are residing in '/opt/clearml/fileserver/{myproject}/.datasets' but it is a root storage so it has been used by ClearML causing all the experiments to halt. I'd like to switch it to one of storage folders which is big enough to hold all of my datasets.

ainoam commented 2 years ago

@jis478 To override the default dataset storage location use the output_uri parameter (available in either CLI or SDK)

jis478 commented 2 years ago

@ainoam thanks a lot! By the way, I've uploaded my files to the clearML server using output_uri but another use of the project is not able to download (find) it using Dataset.get API.

It seems like she cannot access the output_uri path in my docker container (because we are using two different containers!)

Do you have the best practice in this situation?

image

jkhenning commented 2 years ago

Hi @jis478 , what output_uri did you use? is it a mounted dir?

jkhenning commented 2 years ago

@jis478 did you manage to solve this?