Allow mamba to be used to install the conda environment

[deepio-oc]

Description

Allow mamba to be used to install the conda environment.

mamba is a reimplementation of the conda package manager in C++.

• parallel downloading of repository data and package files using multi-threading
• libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE
• core parts of mamba are implemented in C++ for maximum efficiency

• https://github.com/mamba-org/mamba

Use case

Conda can be very slow to create runtime environment. While the same environment is created much faster using mamba. It would be good if ray can allow using mamba to install conda environments in place of conda.

In our test, conda hangs forever in solving dependencies so we are not able to run ray scripts. We have tried a workaround of symlink conda to mamba and that works. If would be better if this was an option in ray runtime env.

Can submit a PR if team agrees that this would be a good way to solve the conda related slow startups.

[jjyao]

How mature or popular of mamba? We can definitely support it and your contribution is very welcomed!

[deepio-oc]

Mamba's first release(0.0.1) was 4 years ago. Today it is 1.4.2 so I think it has matured over 22 releases. It has 4.5K GitHub stars so it is gaining popularity.

For us, conda didnt even work when using an environment file but by using mamba, ray script was able to install the environment in few minutes.

[antoniomdk]

What about using libmamba solver? Although it has some differences with the classic solver, it won't probably require any change in Ray core. If users want to opt-in for the mamba solver, they can just install it and set it as default in their dockerfiles.

[rupertcw]

Hi - we would also like to use mamba to install our dependencies. What's the current status?

[antoniomdk]

Mamba is now the default resolver for conda.

https://github.com/conda/conda/releases/tag/23.10.0 https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community

So, if your Ray cluster is using one of the latest versions of conda, you should expect a similar performance as Mamba.