prefix-dev / pixi

Package management made easy
https://pixi.sh
BSD 3-Clause "New" or "Revised" License
3.2k stars 175 forks source link

Allow for the creation of a seperate pypi build environment #1110

Open ruben-arts opened 6 months ago

ruben-arts commented 6 months ago

Problem description

Currently we solve and install all environments that require a pypi solve as you need the correct python version to do the pypi solving. This also requires the rest of that environment to install as you might need some of the tools in order to build some sdists. This is a cool feature until you don't need all of the complete environments and you are downloading them for no reason.

Imagine this project:

[project]
name = "bla"
platforms = ["linux-64"]
channels = ["conda-forge"]

[dependencies]
tensorflow = "*"

[pypi-dependencies]
ruff = "*"

[feature.py39.dependencies]
python = "3.9.*"

[feature.py310.dependencies]
python = "3.10.*"

[feature.py311.dependencies]
python = "3.11.*"

[environments]
py39 = ["py39"]
py310 = ["py310"]
py311 = ["py311"]

All of these environments would be installed. That means you need to download 3 different version of tensorflow to be able to solve for ruff. Which on a slow internet connection could take minutes.

The ability to define a separate pypi build env would solve the slow installation problem here.

baszalmstra commented 6 months ago

Would also be useful for: https://github.com/prefix-dev/pixi/issues/1051

tdejager commented 6 months ago

But when using different Python interpreters for different sdists this might change the build dependencies. Which in that case you need more than 1 Python build environment 😀

Not sure how often this happens in practice.

ruben-arts commented 6 months ago

I guess the build env still needs to be specific to an normal env but just a slimmed down version of it. Maybe reusing the build-dependencies would be a good idea, that you initialize the environment for the solve only with those dependencies. e.g.

[dependencies]
pytorch = "*"

[build-dependencies]
python = "3.9.*"

[feature.py310.build-dependencies]
python = "3.10.*"

[environments]
py310 = ["py310"]

resulting in only python in the solve environment.