Is the storage documentation oudated? (S3)

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

https://clear.ml/docs

Apache License 2.0

5.61k stars 651 forks source link

Is the storage documentation oudated? (S3) #1097

Closed d13g0 closed 1 year ago

d13g0 commented 1 year ago

Describe the bug

A clear and concise description of what the bug is.

The documentation says that you should do: pip install clearml[s3]

to use cloud storage (AWS), however I get:

no matches found clearml[s3]

Is the documentation oudated?

To reproduce

Exact steps to reproduce the bug. Provide example code if possible. pip install clearml[s3]

Expected behaviour

What is the expected behaviour? What should've happened but didn't?

Install a library?

Environment

Server type: N/A
ClearML SDK Version: 1.12.0 ? (I ran pip show clearml)
ClearML Server Version (Only for self hosted). N/A
Python Version: 3.10.12
OS (Windows \ Linux \ Macos): MacOS Ventura

ainoam commented 1 year ago

@d13g0 Looks like you had a typo in installing clearm[s3] instead of clearml[s3]?

d13g0 commented 1 year ago

thanks @ainoam I have updated the ticket

jkhenning commented 1 year ago

@d13g0 I think this is a zsh issue parsing the command (like discussed here and in other places, it's probably detecting [...] and is trying to find something)

d13g0 commented 1 year ago

@jkhenning interesting, thanks for sharing. I wonder what the solution should be for this issue since zsh is the new default for mac. Also, it seems so far that clearml 1.12.1 is able to upload s3 artifacts (no need for pip install clearml[s3] ...?)

jkhenning commented 1 year ago

Hi @d13g0

I wonder what the solution should be for this issue since zsh is the new default for mac.

This is a pip issue - the standard way to install extra packages is using the [] notation...

Also, it seems so far that clearml 1.12.1 is able to upload s3 artifacts (no need for pip install clearml[s3] ...?)

I think this is a coincidence - [s3] basically pulls the boto3 package, and I assume you simply already had it (or used one of the ClearML examples which has it in its requirements

d13g0 commented 1 year ago

Hi @jkhenning that makes sense as I already had the boto3 package.

I just found this recommendation: when trying to install packages that include square brackets [] use simple quotes:

pip install 'clearml[s3]'

This seems to work!