search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 655 forks source link

Modified helper files to download files from wasabi and updated the c… #955

Closed Shyam-AI closed 1 year ago

Shyam-AI commented 1 year ago

…onfig file

Patch Description

endpoint url modified in helper.py of storage manager (If not already mentioned in the original conversation)

Testing Instructions

The modifications can download files from wasabi (similar to aws)

Other Information

jkhenning commented 1 year ago

Hi @Shyam-AI,

The ClearML AWS S3 helper (which also handles S3-like servers like minio and wasabi) already padded endpoint_url when initializing boto3 - that's automatically done when the storage path contains a port number, so for wasabi and minio you should use a storage URL such as s3://<server-address>:433/... (if using a secure connection, for example)

jkhenning commented 1 year ago

@Shyam-AI can you confirm my previous comment works for you?

Shyam-AI commented 1 year ago

Hi @jkhenning was tied up with a different work. I am unable to use wasabi with the storage manager. I might be missing out on somethimg. Could you possibly provide with an example as to how to use storage manager with wasabi?

Thanks

jkhenning commented 1 year ago

Hi, As I said before:

for wasabi and minio you should use a storage URL such as s3://my-server-address:433/... (if using a secure connection, for example)

jkhenning commented 1 year ago

@Shyam-AI did you get it to work with my suggested configuration?

jkhenning commented 1 year ago

Closing this as stale, please reopen if required.