search

allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
386 stars 134 forks source link

How to write artifacts to S3 from server side? #226

Closed kryvokhyzha closed 10 months ago

kryvokhyzha commented 10 months ago

I have deployed ClearML on AWS EKS. EKS pods can be configured to access S3 bucket and perform get, put and delete operations.

Can I configure a local ClearML client to write artifacts to S3 bucket from API Server or File Server instead of uploading them directly to S3 from local? Thus, server must act as a proxy for accessing or uploading remote artifacts to the storage

jkhenning commented 10 months ago

Hi @kryvokhyzha, the ClearML client (SDK) can use data from a separate object storage (e.g. S3) or from the ClearML Fileserver. You cannot proxy data through the server. If your S3 bucket can be mounted locally as a filesystem folder, you can theoretically make the fileserver use that for files storage.