search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.67k stars 653 forks source link

Support uv package manager #1339

Open falkaer opened 2 weeks ago

falkaer commented 2 weeks ago

Proposal Summary

It would be nice to have the option of using uv to manage dependencies instead of pip/conda/poetry.

The simplest support would be to just use uv pip as a drop-in replacement for pip when installing from a task's requirements in the clearml agent.

Motivation

Much faster environment build times on slow file systems.

On my university cluster all storage on most GPU nodes is network-attached with high throughput but very slow random access, and building my environment using pip hits a massive performance cliff and can easily take over an hour to complete even with all downloads already cached, compared to uv which can do this in about a minute.

I've resorted to handrolling my own clearml agent replacement for now because environment building takes so long with pip (and environments are rebuilt from scratch with every change)

Related Discussion

If this continues a slack thread, please provide a link to the original slack thread.

ainoam commented 1 week ago

Thanks for suggesting @falkaer - Do note we're already tracking this in https://github.com/allegroai/clearml-agent/issues/214