BUG: noarch package in multi-output recipe changes CPU architecture mid-build

h-vetinari commented 1 month ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

We recently split up the pyarrow package into more bits and pieces (e.g. distinguish minimal variant for small footprint from full-featured "give me everything"), and now we have a sandwich where the first and last package in a telescopic chain of dependencies are per python, whereas there are two empty metapackages in the middle that could easily be noarch:

pyarrow-core    # depends on libarrow*;     per python version resp. CUDA/non-CUDA
pyarrow         # depends on pyarrow-core;  could be noarch
pyarrow-all     # depends on pyarrow;       could be noarch
pyarrow-tests   # depends on pyarrow-all;   per python version

I've tried doing that in https://github.com/conda-forge/pyarrow-feedstock/pull/119, and it works fine in native builds, but fails with non-sensical errors on aarch/ppc:

TEST END: /home/conda/feedstock_root/build_artifacts/linux-aarch64/pyarrow-core-16.0.0-py39ha95f412_1_cpu.conda
TEST START: /home/conda/feedstock_root/build_artifacts/noarch/pyarrow-16.0.0-hd8ed1ab_1.conda
WARNING: Multiple meta files found. The meta.yaml file in the base directory (/tmp/tmpnher07k4/info/recipe) will be used.
Reloading output folder (local): ...working... done
Solving environment (_test_env): ...working... failed
WARNING: failed to get package records, retrying.  exception was: Unsatisfiable dependencies for platform linux-64: {MatchSpec("16.0.0=*_1_"), MatchSpec("pyarrow==16.0.0=hd8ed1ab_1"), MatchSpec("pyarrow-core=16.0.0[build=*_1_*]")}
Encountered problems while solving:
  - nothing provides requested pyarrow-core 16.0.0.* *_1_*
  - nothing provides pyarrow-core 16.0.0 *_1_* needed by pyarrow-16.0.0-hd8ed1ab_1

When I say non-sensical, I mean that that the TEST END: at the top just confirmed the local existence of an artefact that should match the requested pattern. Looking closer, what's happening is

TEST END: /home/conda/feedstock_root/build_artifacts/linux-aarch64/pyarrow-core-16.0.0-py39ha95f412_1_cpu.conda
[...]                                                ^^^^^^^^^^^^^
Unsatisfiable dependencies for platform linux-64: ...
                                        ~~~~~~~~

i.e. the build switches architecture when getting to the noarch output.

Note that all jobs in that PR want to build:

noarch/pyarrow-16.0.0-hd8ed1ab_1.conda
noarch/pyarrow-all-16.0.0-hd8ed1ab_1.conda

which is the goal - we need to build it in each job in order to successfully build the per-python pyarrow-tests on top, but only one build would be uploaded in the end across all jobs, due to the (intentional!) hash collision.

Conda Info

No response

Conda Config

No response

Conda list

No response

Additional Context

No response

h-vetinari commented 1 month ago

The old logs are lost (so harder to analyze what changed), but it seems that this now works...? 🤔

jaimergp commented 1 month ago

Intriguing. I would have assumed that https://github.com/conda/conda-build/pull/5350 was responsible for the fix, but that's not released yet :/

h-vetinari commented 1 month ago

aaaaand, it broke again (only difference was inclusion of sparse or not as a test dependence). I'll recheck again after #5350 is released.

h-vetinari commented 2 weeks ago

Can confirm that with conda-build installed from main, this now passes 🥳

conda / conda-build