search

conda / conda-build

Commands and tools for building conda packages
https://docs.conda.io/projects/conda-build/
Other
381 stars 423 forks source link

noarch generic doesn't work as expected when python in host in multi-output #5245

Open JeanChristopheMorinPerso opened 7 months ago

JeanChristopheMorinPerso commented 7 months ago

Checklist

What happened?

When using noarch: generic in a recipe that has multiple outputs and that depends on python, if python is found in added to host, then conda-build will no generate a generic noarch variant.

For example, let's take this recipe:

package:
  name: test
  version: 1.0.0

outputs:
  - name: test-subpackage1
    build:
      noarch: generic
    requirements:
      host:
        - python
      run:
        - python

If I run conda build <feedstock> --output, it will print

/Users/jcmorin/miniconda/conda-bld/noarch/test-subpackage1-1.0.0-py310hd3eb1b0_0.tar.bz2
/Users/jcmorin/miniconda/conda-bld/noarch/test-subpackage1-1.0.0-py311hd3eb1b0_0.tar.bz2
/Users/jcmorin/miniconda/conda-bld/noarch/test-subpackage1-1.0.0-py312hd3eb1b0_0.tar.bz2
/Users/jcmorin/miniconda/conda-bld/noarch/test-subpackage1-1.0.0-py38hd3eb1b0_0.tar.bz2
/Users/jcmorin/miniconda/conda-bld/noarch/test-subpackage1-1.0.0-py39hd3eb1b0_0.tar.bz2

(notice how packages would go in noarch but are still python version specific).

Now, if we take the same recipe and remove python from host:

package:
  name: test
  version: 1.0.0

outputs:
  - name: test-subpackage1
    build:
      noarch: generic
    requirements:
      run:
        - python

and run conda build --output again, I get:

/Users/jcmorin/miniconda/conda-bld/noarch/test-subpackage1-1.0.0-hd3eb1b0_0.tar.bz2

I was expecting the first case to only generate one variant (a fully generic variant).

Conda Info

active environment : base
    active env location : /Users/jcmorin/miniconda
            shell level : 1
       user config file : /Users/jcmorin/.condarc
 populated config files : /Users/jcmorin/.condarc
          conda version : 24.3.0
    conda-build version : 24.3.0
         python version : 3.12.2.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=m2
                          __conda=24.3.0=0
                          __osx=14.4=0
                          __unix=0=0
       base environment : /Users/jcmorin/miniconda  (writable)
      conda av data dir : /Users/jcmorin/miniconda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/jcmorin/miniconda/pkgs
                          /Users/jcmorin/.conda/pkgs
       envs directories : /Users/jcmorin/miniconda/envs
                          /Users/jcmorin/.conda/envs
               platform : osx-arm64
             user-agent : conda/24.3.0 requests/2.31.0 CPython/3.12.2 Darwin/23.4.0 OSX/14.4 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.3
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Conda Config

==> /Users/jcmorin/.condarc <==
report_errors: False

Conda list

# packages in environment at /Users/jcmorin/miniconda:
#
# Name                    Version                   Build  Channel
archspec                  0.2.3              pyhd3eb1b0_0    defaults
attrs                     23.1.0          py312hca03da5_0    defaults
backports                 1.1                pyhd3eb1b0_0    defaults
backports.functools_lru_cache 1.6.4              pyhd3eb1b0_0    defaults
backports.tempfile        1.0                pyhd3eb1b0_1    defaults
backports.weakref         1.0.post1                  py_1    defaults
beautifulsoup4            4.12.2          py312hca03da5_0    defaults
boltons                   23.0.0          py312hca03da5_0    defaults
brotli-python             1.0.9           py312h313beb8_7    defaults
bzip2                     1.0.8                h80987f9_5    defaults
c-ares                    1.19.1               h80987f9_0    defaults
ca-certificates           2024.3.11            hca03da5_0    defaults
cctools                   949.0.1             hc179dcd_25    defaults
cctools_osx-arm64         949.0.1             h332cad3_25    defaults
certifi                   2024.2.2        py312hca03da5_0    defaults
cffi                      1.16.0          py312h80987f9_0    defaults
chardet                   4.0.0           py312hca03da5_1003    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.1.7           py312hca03da5_0    defaults
conda                     24.3.0          py312hca03da5_0    defaults
conda-build               24.3.0          py312hca03da5_0    defaults
conda-content-trust       0.2.0           py312hca03da5_0    defaults
conda-index               0.4.0              pyhd3eb1b0_0    defaults
conda-libmamba-solver     23.12.0            pyhd3eb1b0_1    defaults
conda-package-handling    2.2.0           py312hca03da5_0    defaults
conda-package-streaming   0.9.0           py312hca03da5_0    defaults
conda-verify              3.4.2                      py_1    defaults
cryptography              42.0.5          py312hd4332d6_0    defaults
distro                    1.8.0           py312hca03da5_0    defaults
expat                     2.5.0                h313beb8_0    defaults
filelock                  3.13.1          py312hca03da5_0    defaults
fmt                       9.1.0                h48ca7d4_0    defaults
future                    0.18.3          py312hca03da5_0    defaults
glob2                     0.7                pyhd3eb1b0_0    defaults
icu                       73.1                 h313beb8_0    defaults
idna                      3.4             py312hca03da5_0    defaults
jinja2                    3.1.3           py312hca03da5_0    defaults
jsonpatch                 1.33            py312hca03da5_0    defaults
jsonpointer               2.1                pyhd3eb1b0_0    defaults
jsonschema                4.19.2          py312hca03da5_0    defaults
jsonschema-specifications 2023.7.1        py312hca03da5_0    defaults
krb5                      1.20.1               hf3e1bf2_1    defaults
ld64                      530                 hb29bf3f_25    defaults
ld64_osx-arm64            530                 h001ce53_25    defaults
ldid                      2.1.5                h20b2a84_3    defaults
libarchive                3.6.2                h62fee54_2    defaults
libcurl                   8.5.0                h3e2b118_0    defaults
libcxx                    14.0.6               h848a8c0_0    defaults
libedit                   3.1.20230828         h80987f9_0    defaults
libev                     4.33                 h1a28f6b_1    defaults
libffi                    3.4.4                hca03da5_0    defaults
libiconv                  1.16                 h1a28f6b_2    defaults
liblief                   0.12.3               h313beb8_0    defaults
libllvm14                 14.0.6               h7ec7a93_3    defaults
libmamba                  1.5.3                h15e39b3_0    defaults
libmambapy                1.5.3           py312h1c5506f_0    defaults
libnghttp2                1.57.0               h62f6fdd_0    defaults
libsolv                   0.7.24               h514c7bf_0    defaults
libssh2                   1.10.0               h02f6b3c_2    defaults
libxml2                   2.10.4               h0dcf63f_1    defaults
lz4-c                     1.9.4                h313beb8_0    defaults
markupsafe                2.1.3           py312h80987f9_0    defaults
menuinst                  2.0.2           py312hca03da5_0    defaults
more-itertools            10.1.0          py312hca03da5_0    defaults
ncurses                   6.4                  h313beb8_0    defaults
openssl                   3.0.13               h1a28f6b_0    defaults
packaging                 23.2            py312hca03da5_0    defaults
patch                     2.7.6             h1a28f6b_1001    defaults
pcre2                     10.42                hb066dcc_0    defaults
pip                       23.3.1          py312hca03da5_0    defaults
pkginfo                   1.9.6           py312hca03da5_0    defaults
platformdirs              3.10.0          py312hca03da5_0    defaults
pluggy                    1.0.0           py312hca03da5_1    defaults
psutil                    5.9.0           py312h80987f9_0    defaults
py-lief                   0.12.3          py312h313beb8_0    defaults
pybind11-abi              4                    hd3eb1b0_1    defaults
pycosat                   0.6.6           py312h80987f9_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 24.0.0          py312hca03da5_0    defaults
pysocks                   1.7.1           py312hca03da5_0    defaults
python                    3.12.2               h99e199e_0    defaults
python-libarchive-c       2.9                pyhd3eb1b0_1    defaults
python.app                3               py312h80987f9_0    defaults
pytz                      2023.3.post1    py312hca03da5_0    defaults
pyyaml                    6.0.1           py312h80987f9_0    defaults
readline                  8.2                  h1a28f6b_0    defaults
referencing               0.30.2          py312hca03da5_0    defaults
reproc                    14.2.4               hc377ac9_1    defaults
reproc-cpp                14.2.4               hc377ac9_1    defaults
requests                  2.31.0          py312hca03da5_1    defaults
rpds-py                   0.10.6          py312hf0e4da2_0    defaults
ruamel.yaml               0.17.21         py312h80987f9_0    defaults
setuptools                68.2.2          py312hca03da5_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
soupsieve                 2.5             py312hca03da5_0    defaults
sqlite                    3.41.2               h80987f9_0    defaults
tapi                      1100.0.11            h8754e6a_1    defaults
tk                        8.6.12               hb8d0fd4_0    defaults
tomli                     2.0.1           py312hca03da5_1    defaults
toolz                     0.12.0          py312hca03da5_0    defaults
tqdm                      4.65.0          py312h989b03a_0    defaults
truststore                0.8.0           py312hca03da5_0    defaults
tzdata                    2024a                h04d1e81_0    defaults
urllib3                   2.1.0           py312hca03da5_1    defaults
wheel                     0.41.2          py312hca03da5_0    defaults
xz                        5.4.6                h80987f9_0    defaults
yaml                      0.2.5                h1a28f6b_0    defaults
yaml-cpp                  0.8.0                h313beb8_0    defaults
zlib                      1.2.13               h5a0b063_0    defaults
zstandard                 0.19.0          py312h80987f9_0    defaults
zstd                      1.5.5                hd90d995_0    defaults

Additional Context

No response

mbargull commented 7 months ago

AFAICT, this is "working as intended" in that the normal "entry in requirements/host induces variant differentiation". Note that noarch: generic should mostly behave like the non-noarch build. Meaning, noarch: python really is the odd one out since it causes a bunch of implicit special handling to happen.

Changing noarch: generic to not do any variant handling would be a breaking change (and python should be handled like any other dependency in that case [yeah, the old build string customization still happens, unfortunately...]).

Since conda-build is a bit peculiar with its implicit "variant handling happens if it's a bare build dependency and also has entries in conda_build_config.yaml", you'd have to workaround this if you want to suppress that variant handling by adding some version/build specification instead of just the bare package name. E.g., in your case, you'd want something like - python >="sensible lower bound" instead of just - python in the requirements.