Add support for pip/virtualenv environments

[akx]

The main meat of this PR is PipInstall, a sibling of CondaInstall that runs venv (or uv venv) and pip install (or uv pip install) to install requirements from requirements.txt files.

The base image needs to have a python3 on PATH for this to work at all, of course.

As alluded to above, this also includes support for the uv package installer as a faster alternative to pip; this is enabled by the currently-undocumented COTAINR_USE_UV environment variable being set to a non-empty value.

All tests, old and new, pass locally (using a dockerized singularity since I'm on a Mac).

[eskech]

Just a short comment. @Chroxvi is on vacation and is back in two weeks. I really want him to do the review. Sorry for letting you wait until he is back.

[akx]

Gentle nudge (as I too am back from vacation) :)

[akx]

Rebased post #56.

[Chroxvi]

Admittedly, adding support for additional conda/pip package managers in cotainr is pretty far down the line on our roadmap since:

• We should already be able to serve all conda/pip use cases via our current conda/mamba setup.
• The alternative pip/conda package managers of interest (uv and pixi, mostly, I think) are still in the process of maturing, so we don't know exactly what we are trying to support long term.
• We are a fairly small development team so:
  1. We are a bit concerned about committing to a larger maintenance burden.
  2. We would generally like to prioritize enabling other use cases (e.g. R package management) over implementing alternatives way to handle conda/pip packages.

That being said, we are still open to implementing support for uv, if it enables some significant use cases - after all the cotainr development is driven by requests for support for various use cases.

@akx, picking up on your comment in https://github.com/DeiC-HPC/cotainr/pull/68#discussion_r1789131905 stating that you would be using cotainr on LUMI if this PR #55 was merged, what is your main motivation for wanting uv over the current conda/mamba implementation in cotainr that allows you to install pip packages by adding them to your conda_env.yml file as show in e.g. the cotainr LUMI PyTorch example? (You can even directly include requirements.txt in the conda_env.yml.) Are you looking for a faster environment installer? Or a better dependency solver than pip? Or something else?

I think it is likely that conda/mamba at some point is going to be replaced by pixi (which uses rattler for installing conda packages and uv for installing pip packages) or uv (if at some point it implements support for conda packages which I believe is essential to the Scientific Python community). Even though it appears that pixi can be used as a direct replacement for conda/pip, it still has a slightly different design philosophy around managing projects and not just environments making it somewhat odd to integrate into cotainr. @akx Do you have any opinion about pros/cons of integrating cotainr with pixi and using it's uv integration over directly interfacing with uv?

[akx]

Admittedly, adding support for additional conda/pip package managers in cotainr is pretty far down the line on our roadmap

To be clear, the primary thing this PR adds support for is vanilla standard Python virtualenvs, i.e. not using Conda/Mamba distributions.

uv support here is an experimental option, and it's really only just replacing python -m venv with uv venv and python -m pip with uv pip.

What is your main motivation for wanting uv over the current conda/mamba implementation in cotainr that allows you to install pip packages by adding them to your conda_env.yml file [...]

The reason I wrote this was that I don't know (or care, to be honest) much about Conda-based distributions (and I've had my share of headaches troubleshooting customer Conda images over at $dayjob (@valohai)). I wanted a simpler option, when you have e.g. a standard pyproject.toml based project (e.g. by way of -e . in a requirements file) or a requirements.txt that you just want to use without having to write a YAML file or, well, deal with Conda/Mamba. As a side effect, you won't be required to accept the Miniforge license terms with this method...

Are you looking for a faster environment installer? Or a better dependency solver than pip? Or something else?

Again, uv support for making installation faster and resolution more stable is a side effect in this PR. This does not use uv's project management features that'd use uv.lock, etc.

Do you have any opinion about pros/cons of integrating cotainr with pixi and using it's uv integration over directly interfacing with uv?

I hadn't heard of Pixi before, again because I don't keep tabs on the Conda/Mamba world. No opinion.