A sub-environment without solving

[gshiba]

Problem description

I'd like to slice out a subset of dependencies from a larger (default) env.

1. Start with a pixi.toml and pixi.lock with only the default env

2. (wait a few hours or day for some of the packages to get updates)

3. Append the following to pixi.toml

[environments.newenv]
features = ["newenv"]
no-default-feature = true
solve-group = "default"

[feature.newenv.dependencies]
pandas = "*"

4. run pixi list -e new

Desired: the set of packages listed in environments.newenv.packages.linux-64 is a strict subset of environments.default.packages.linux-64

Actual: while the pandas version in newenv env matches exactly the default env, other transient dependencies are newer. For example:

https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-13.2.0-hc0a3c3a_7.conda  # default env
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-13.2.0-hc0a3c3a_13.conda  # newenv env

or

https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.8.2-pyhd8ed1ab_0.tar.bz2. # default env
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.9.0-pyhd8ed1ab_0.conda. # newenv env

Use case: I'm in a large python repo and would like to build a docker image that can run a specific feature in the repo (e.g. foo.bar.main). I know foo.bar.main only needs pandas so I want to pixi to add a new env to pixi.lock that is a strict subset. This lock file will be used in the docker build process. (if this sound like a poor man's version of a monorepo build system like bazel to you, you are correct)

(pixi 0.23.0)

[baszalmstra]

Sounds like a bug! 🐛 Thanks for filing!

[baszalmstra]

I have been trying to reproduce this but have so far not been able to. Would you be able to share your complete pixi.toml?

[gshiba]

Starting with this pixi.toml:

[project]
name = "tmp"
version = "0.1.0"
channels = ["conda-forge", "bioconda"]
platforms = ["osx-64", "linux-64"]

[dependencies]
samtools = "1.20"
python = "3.10"
pandas = "2.2.2"

... pixi generated a lock file. The pair of files can be found in the original revision of this gist. This lock file was created a few weeks ago.

Today, I appended the following to pixi.toml:

[feature.pandas-only.dependencies]
pandas = "2.2.2"

[environments]
pandas-only = {features=["pandas-only"], solve-group="default"}

and ran:

$ pixi list --frozen --environment pandas-only
✘ No packages found.  # hmm a better error message is desired?

$ pixi list --environment pandas-only
Environment: pandas-only
Package          Version       Build               Size       Kind   Source
bzip2            1.0.8         hfdf4475_7          131 KiB    conda  bzip2-1.0.8-hfdf4475_7.conda
c-ares           1.32.3        h51dda26_0          156.5 KiB  conda  c-ares-1.32.3-h51dda26_0.conda
ca-certificates  2024.7.4      h8857fd0_0          150.9 KiB  conda  ca-certificates-2024.7.4-h8857fd0_0.conda
htslib           1.20          hec81eee_2          2.7 MiB    conda  htslib-1.20-hec81eee_2.tar.bz2
krb5             1.21.3        h37d8d59_0          1.1 MiB    conda  krb5-1.21.3-h37d8d59_0.conda
libblas          3.9.0         22_osx64_openblas   14.4 KiB   conda  libblas-3.9.0-22_osx64_openblas.conda
libcblas         3.9.0         22_osx64_openblas   14.3 KiB   conda  libcblas-3.9.0-22_osx64_openblas.conda
libcurl          8.9.0         hfcf2730_0          387.1 KiB  conda  libcurl-8.9.0-hfcf2730_0.conda
libcxx           18.1.8        hef8daea_0          1.3 MiB    conda  libcxx-18.1.8-hef8daea_0.conda
libdeflate       1.20          h49d49c5_0          68.7 KiB   conda  libdeflate-1.20-h49d49c5_0.conda
libedit          3.1.20191231  h0678c8f_2          102.9 KiB  conda  libedit-3.1.20191231-h0678c8f_2.tar.bz2
libev            4.33          h10d778d_2          104.2 KiB  conda  libev-4.33-h10d778d_2.conda
libffi           3.4.2         h0d85af4_5          50.1 KiB   conda  libffi-3.4.2-h0d85af4_5.tar.bz2
libgfortran      5.0.0         13_2_0_h97931a8_3   107.5 KiB  conda  libgfortran-5.0.0-13_2_0_h97931a8_3.conda
libgfortran5     13.2.0        h2873a65_3          1.5 MiB    conda  libgfortran5-13.2.0-h2873a65_3.conda
liblapack        3.9.0         22_osx64_openblas   14.3 KiB   conda  liblapack-3.9.0-22_osx64_openblas.conda
libnghttp2       1.58.0        h64cf6d3_1          585.7 KiB  conda  libnghttp2-1.58.0-h64cf6d3_1.conda
libopenblas      0.3.27        openmp_h8869122_1   5.8 MiB    conda  libopenblas-0.3.27-openmp_h8869122_1.conda
libsqlite        3.46.0        h1b8f9f3_0          887.3 KiB  conda  libsqlite-3.46.0-h1b8f9f3_0.conda
libssh2          1.11.0        hd019ec5_0          253.5 KiB  conda  libssh2-1.11.0-hd019ec5_0.conda
libzlib          1.3.1         h87427d6_1          56 KiB     conda  libzlib-1.3.1-h87427d6_1.conda
llvm-openmp      18.1.8        h15ab845_0          293.6 KiB  conda  llvm-openmp-18.1.8-h15ab845_0.conda
ncurses          6.5           h5846eda_0          804.3 KiB  conda  ncurses-6.5-h5846eda_0.conda
numpy            2.0.1         py310he367959_0     6.6 MiB    conda  numpy-2.0.1-py310he367959_0.conda
openssl          3.3.1         h87427d6_2          2.4 MiB    conda  openssl-3.3.1-h87427d6_2.conda
pandas           2.2.2         py310hbf2a7f0_1     11.6 MiB   conda  pandas-2.2.2-py310hbf2a7f0_1.conda
python           3.10.0        h38b4d05_3_cpython  12.9 MiB   conda  python-3.10.0-h38b4d05_3_cpython.tar.bz2
python-dateutil  2.9.0         pyhd8ed1ab_0        217.5 KiB  conda  python-dateutil-2.9.0-pyhd8ed1ab_0.conda
python-tzdata    2024.1        pyhd8ed1ab_0        140.6 KiB  conda  python-tzdata-2024.1-pyhd8ed1ab_0.conda
python_abi       3.10          4_cp310             6.3 KiB    conda  python_abi-3.10-4_cp310.conda
pytz             2024.1        pyhd8ed1ab_0        184.1 KiB  conda  pytz-2024.1-pyhd8ed1ab_0.conda
readline         8.2           h9e318b2_1          249.9 KiB  conda  readline-8.2-h9e318b2_1.conda
samtools         1.20          h94387ee_1          453.2 KiB  conda  samtools-1.20-h94387ee_1.tar.bz2
six              1.16.0        pyh6c4a22f_0        13.9 KiB   conda  six-1.16.0-pyh6c4a22f_0.tar.bz2
sqlite           3.46.0        h28673e1_0          891 KiB    conda  sqlite-3.46.0-h28673e1_0.conda
tk               8.6.13        h1abcd95_1          3.1 MiB    conda  tk-8.6.13-h1abcd95_1.conda
tzdata           2024a         h0c530f3_0          117 KiB    conda  tzdata-2024a-h0c530f3_0.conda
xz               5.2.6         h775f41a_0          232.5 KiB  conda  xz-5.2.6-h775f41a_0.tar.bz2
zstd             1.5.6         h915ae27_0          487.2 KiB  conda  zstd-1.5.6-h915ae27_0.conda

The resulting diff to pixi.toml and pixi.lock can be found in the first revision of the same gist (linked above).

(warning: the diff view on github is a bit misleading, so probably better to view locally)

Actual: Among the common transient dependencies between the two environments (default and pandas-only), some of the transient dependencies are different. For example, numpy is 2.0.0 in the default environment, and it is 2.0.1 in the pandas-only environment. (2.0.1 was released on july 21)

Desired: There is a way to create a lock for the new pandas-only environment that is a strict subset of the original default environment.

Use case: I'd like to create a new pixi environment by 'slicing out' a set of dependencies from a bigger existing pixi environment. This is useful when I have one large (default) environment where I add any and all dependencies I need in my project (a "monorepo"), and later slice out a specific feature (say, projects/foo/main.py and only its dependencies) and package it into a minimal docker image. Philosophically, this is the opposite approach to the implemented feature, where one can specify features and build up an environment; in a monorepo context, I don't necessarily know ahead of time what environments I may need in the future, let alone how to divvy up the dependencies into features that can be composed into environments.

[baszalmstra]

Mm Im having a hard time recreating this from your gist because all the revisions that I could find already included a pandas-only environment. The first revision of the gist contains a pixi.toml without the environment but the lock file does still contain the environment information.

Removing the data for that environment from the lock-file yields the expected result.

[gshiba]

Original revision. There is one numpy in this lock file (2.0.0).

Updated revision with feature.pandas-only. There are two numpys in this lock file (2.0.0 and 2.0.1).

[baszalmstra]

Thanks I will take a closer look!