search

conda / conda-build

Commands and tools for building conda packages
https://docs.conda.io/projects/conda-build/
Other
379 stars 420 forks source link

Versions `>=24.7.0` skip installing requirements for some multi-output recipes #5444

Open iisakkirotko opened 1 month ago

iisakkirotko commented 1 month ago

Checklist

What happened?

Upon an upgrade to conda-build=24.7.1, the build process for our conda-forge multi-output package solara broke. Looking at the build logs, I see that for the other "sub-packages" that are to be output (solara-ui and solara-server), the output folder is reloaded, and a new environment is created (see attached excerpt from the logs) before the build script is executed. However, for solara this step is skipped, and the build script fails with a ModuleNotFoundError for hatchling.

Log excerpt for `solara-server` ``` Packaging solara-server Reloading output folder (local): ...working... done Solving environment (_h_env): ...working... done ## Package Plan ## environment location: /home/conda/feedstock_root/build_artifacts/solara_1722651845590/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac The following NEW packages will be INSTALLED: _libgcc_mutex: 0.1-conda_forge conda-forge _openmp_mutex: 4.5-2_gnu conda-forge bzip2: 1.0.8-h4bc722e_7 conda-forge ca-certificates: 2024.7.4-hbcca054_0 conda-forge editables: 0.5-pyhd8ed1ab_0 conda-forge hatchling: 1.25.0-pyhd8ed1ab_0 conda-forge importlib-metadata: 8.2.0-pyha770c72_0 conda-forge ld_impl_linux-64: 2.40-hf3520f5_7 conda-forge libexpat: 2.6.2-h59595ed_0 conda-forge libffi: 3.4.2-h7f98852_5 conda-forge libgcc-ng: 14.1.0-h77fa898_0 conda-forge libgomp: 14.1.0-h77fa898_0 conda-forge libnsl: 2.0.1-hd590300_0 conda-forge libsqlite: 3.46.0-hde9e2c9_0 conda-forge libuuid: 2.38.1-h0b41bf4_0 conda-forge libxcrypt: 4.4.36-hd590300_1 conda-forge libzlib: 1.3.1-h4ab18f5_1 conda-forge ncurses: 6.5-h59595ed_0 conda-forge openssl: 3.3.1-h4bc722e_2 conda-forge packaging: 24.1-pyhd8ed1ab_0 conda-forge pathspec: 0.12.1-pyhd8ed1ab_0 conda-forge pip: 24.2-pyhd8ed1ab_0 conda-forge pluggy: 1.5.0-pyhd8ed1ab_0 conda-forge python: 3.12.4-h194c7f8_0_cpython conda-forge readline: 8.2-h8228510_1 conda-forge setuptools: 72.1.0-pyhd8ed1ab_0 conda-forge tk: 8.6.13-noxft_h4845f30_101 conda-forge tomli: 2.0.1-pyhd8ed1ab_0 conda-forge trove-classifiers: 2024.7.2-pyhd8ed1ab_0 conda-forge tzdata: 2024a-h0c530f3_0 conda-forge wheel: 0.43.0-pyhd8ed1ab_1 conda-forge xz: 5.2.6-h166bdaf_0 conda-forge zipp: 3.19.2-pyhd8ed1ab_0 conda-forge Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done + cd solara_server + python -m pip install -vv --no-deps . ... ```
Log excerpt for `solara` ``` Packaging solara + cd solara + python -m pip install -vv --no-deps . Using pip 24.2 from /opt/conda/lib/python3.10/site-packages/pip (python 3.10) Non-user install because user site-packages disabled Ignoring indexes: https://pypi.org/simple Created temporary directory: /tmp/pip-build-tracker-ofbmes6n Initialized build tracking at /tmp/pip-build-tracker-ofbmes6n Created build tracker: /tmp/pip-build-tracker-ofbmes6n Entered build tracker: /tmp/pip-build-tracker-ofbmes6n Created temporary directory: /tmp/pip-install-s66an4dl Created temporary directory: /tmp/pip-ephem-wheel-cache-hbeg1kh_ Processing /home/conda/feedstock_root/build_artifacts/solara_1722651845590/work/solara Added file:///home/conda/feedstock_root/build_artifacts/solara_1722651845590/work/solara to build tracker '/tmp/pip-build-tracker-ofbmes6n' Created temporary directory: /tmp/pip-modern-metadata-l71i1z1u Preparing metadata (pyproject.toml): started Running command Preparing metadata (pyproject.toml) Preparing metadata (pyproject.toml): finished with status 'done' ERROR: Exception: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper status = _inner_run() File "/opt/conda/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run return self.run(options, args) File "/opt/conda/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper return func(self, options, args) File "/opt/conda/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 379, in run requirement_set = resolver.resolve( File "/opt/conda/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve collected = self.factory.collect_root_requirements(root_reqs) File "/opt/conda/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 539, in collect_root_requirements reqs = list( File "/opt/conda/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 495, in _make_requirements_from_install_req cand = self._make_base_candidate_from_link( File "/opt/conda/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 232, in _make_base_candidate_from_link self._link_candidate_cache[link] = LinkCandidate( File "/opt/conda/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 303, in __init__ super().__init__( File "/opt/conda/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__ self.dist = self._prepare() File "/opt/conda/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 235, in _prepare dist = self._prepare_distribution() File "/opt/conda/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 314, in _prepare_distribution return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True) File "/opt/conda/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement return self._prepare_linked_requirement(req, parallel_builds) File "/opt/conda/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 642, in _prepare_linked_requirement dist = _get_prepared_distribution( File "/opt/conda/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 72, in _get_prepared_distribution abstract_dist.prepare_distribution_metadata( File "/opt/conda/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 69, in prepare_distribution_metadata self.req.prepare_metadata() File "/opt/conda/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 575, in prepare_metadata self.metadata_directory = generate_metadata( File "/opt/conda/lib/python3.10/site-packages/pip/_internal/operations/build/metadata.py", line 35, in generate_metadata distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir) File "/opt/conda/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 727, in prepare_metadata_for_build_wheel return super().prepare_metadata_for_build_wheel( File "/opt/conda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 186, in prepare_metadata_for_build_wheel return self._call_hook('prepare_metadata_for_build_wheel', { File "/opt/conda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook raise BackendUnavailable(data.get('traceback', '')) pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend obj = import_module(mod_path) File "/opt/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 992, in _find_and_load_unlocked File "", line 241, in _call_with_frames_removed File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1004, in _find_and_load_unlocked ModuleNotFoundError: No module named 'hatchling' ```

Limiting conda-build<24.7.1 (which installs 24.5.1) fixes this issue.

Conda Info

active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/conda/.condarc
 populated config files : /opt/conda/.condarc
                          /home/conda/.condarc
          conda version : 24.7.1
    conda-build version : 24.7.1
         python version : 3.10.14.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=x86_64
                          __conda=24.7.1=0
                          __glibc=2.17=0
                          __linux=6.5.11=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/conda/feedstock_root/build_artifacts/pkg_cache
                          /opt/conda/pkgs
       envs directories : /opt/conda/envs
                          /home/conda/.conda/envs
               platform : linux-64
             user-agent : conda/24.7.1 requests/2.32.3 CPython/3.10.14 Linux/6.5.11-linuxkit centos/7.9.2009 glibc/2.17 solver/libmamba conda-libmamba-solver/24.7.0 libmambapy/1.5.8
                UID:GID : 501:32767
             netrc file : None
           offline mode : False

Conda Config

==> /opt/conda/.condarc <==
add_pip_as_python_dependency: False
auto_update_conda: False
aggressive_update_packages:
  - ca-certificates
  - certifi
channel_priority: strict
channels:
  - conda-forge
show_channel_urls: True
conda_build:
  error_overlinking: True
  pkg_format: 2
  zstd_compression_level: 19

==> /home/conda/.condarc <==
pkgs_dirs:
  - /home/conda/feedstock_root/build_artifacts/pkg_cache
  - /opt/conda/pkgs
solver: libmamba
conda-build:
  root-dir: /home/conda/feedstock_root/build_artifacts

==> envvars <==
allow_softlinks: False
bld_path: /home/conda/feedstock_root/build_artifacts

Conda list

No response

Additional Context

cc: @dhirschfeld, @mariobuikhuizen, @hmaarrfk

h-vetinari commented 1 month ago

I think you're running into a combination of two issues here. The first is that 24.7 started applying setting environment variables like PIP_NO_BUILD_ISOLATION also for outputs. That in itself should be fine because you're listing hatchling as a requirement.

However, it turns into a problem because you have named your package: the same as an output:, which is a case that conda-build unfortunately does not warn on, because it leads to very confusing behaviour. In this case, everything specified under

outputs:
  - name: solara
    [...]

is 100% ignored (c.f. https://github.com/conda/conda-build/issues/4172), and so it turns out that the "global" configuration section (which is mapped to solara because of the matching package name) is applied, which is almost completely empty in your case (i.e. does not contain requirements:, much less hatchling).

The easiest solution is

 package:
-  name: solara
+  name: solara-split

 source:

and 


 extra:
+  feedstock-name: solara
   recipe-maintainers:
     - maartenbreddels

Though of course it would be good for conda-build (or at least the conda-smithy linter) to warn on cases where the configuration of an output gets ignored.

iisakkirotko commented 1 month ago

Thanks a lot @h-vetinari, that fixes our issue!

I agree that it might be nice to get a warning from somewhere in this case. These name collisions resulting in issues could also be mentioned in the docs, for example at debugging conda recipes.

Should this issue to stay open for future reference, like https://github.com/conda/conda-build/issues/4172?