search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
34k stars 5.78k forks source link

[core] Implement runtime plugins for additional package managers (mamba, micromamba, pixi, etc.) #45572

Open cynddl opened 5 months ago

cynddl commented 5 months ago

Description

Opening this issue to see if there's interest to add other package managers for runtime environments? The conda plugin is fairly big since it needs to create new environments—but I'd love to have a very simple plugin to activate a virtual environment from e.g. mamba, micromamba, or pixi.

Despite conda now using libmamba (as mentioned in #33596), many people use other package managers. Not having to play with fire (I had to set ‘alias conda=mamba’) to use ray would be much appreciated. This seems like a shame since using any environment, from any package manager, is a fairly simple task with a small runtime plugin with _modifycontext.

I'd be happy to contribute a small plugin for pixi (https://github.com/prefix-dev/pixi).

Use case

There has been much progress in the development of package managers since (ana/mini)conda. Some of them are much smaller and easier to install on HPC (micromamba), others allow the centralised development of multiple conda+pip environments (pixi). Having the option to support running workers in these environments would maximise usability of ray outside conda.

cynddl commented 5 months ago

Just wanted to kindly bump this issue. @anyscalesam @jjyao, is there anything I can do to help here? Would this contribution be of interest?