Publish and install build dependencies to/from alternative package index

[ryanking13]

Proposal

I propose hosting multiple versions of packages used as a build dependency (== numpy) in an alternative package index, and let pyodide-build use it when installing build dependencies.

Why?

When building packages out-of-tree, each package may require a different version of Numpy. For now, if a package requires numpy as a build dependency, we just replace it with the numpy that is included in the Pyodide distribution, and hope it work.

For example, after we update the Numpy version in Pyodide distribution >= 2.0, some packages that use Numpy < 2.0 in the build step will fail to build because of the version mismatch.

How?

I've been publishing packages built in pyodide/pyodide-recipes to Anaconda package registry: Anaconda Pyodide org. For now, these packages cannot be installed in the Pyodide runtime (CORS issue), but we can still use it during build time.

pyodide-build will point to this package index when installing build dependencies.

Issues / Concerns

• How should we handle the in-tree cases?
• After unvendoring recipes, what should change?

Overview

As-is:

sequenceDiagram
    Note over pyodide-build: Start building a package
    PyPI->>pyodide-build: Install native build dependency
    xbuildenv->>pyodide-build: Replace cross-build files
    Note over pyodide-build: Continue building package

To-be:

sequenceDiagram
    Note over pyodide-build: Start building a package
    Pyodide Package Index->>pyodide-build: Install cross-compiled build dependency
    Note over pyodide-build: Continue building package

[hoodmane]

Nice diagrams. Sounds like a reasonable idea.

[ryanking13]

To implement this issue, I started looking at the behavior of pypabuild and pip. The tricky part is that pip treats multiple index urls equally (https://github.com/pypa/pip/issues/8606)

So, say we have a package that have build dependencies: ["numpy", "setuptools", "setuptools-scm"]. The behavior we want is:

1. Install numpy from Pyodide-hosted package index (wasm32 wheel)
2. Install setuptools and setuptools-scm from PyPI (manylinux wheel or py3-none-any wheel)

but this is not achievable since numpy exists both in the Pyodide-hosted package index and PyPI, and pip will choose the latest version, which might not be the Pyodide-hosted one.

So, I think we have two options:

1. Host all build dependencies, such as setuptools, poetry, hatch, Cython, ... in the alternative package index as well.
2. Maintain a list of packages that need to be installed from Pyodide package index, and only these packages are installed from the Pyodide package index.

I think option 1 is the better long-term solution, but I'd be interested in implementing option 2 in the short term. It's logically equivalent to how we currently specify cross-build-env in meta.yaml.

Pseudo-code for Option 2 would be:

build_deps = ["numpy", "setuptools", "setuptools-scm"]

cross_build_deps, host_build_deps = split_build_deps(build_deps)
# cross_build_deps = ["numpy"]
# host_build_deps = ["setuptools", "setuptools-scm"]

build_env.install_reqs(cross_build_deps, env={"PIP_INDEX_URL": PYODIDE_PACKAGE_INDEX})
build_env.install_reqs(host_build_deps, env={"PIP_INDEX_URL": PYPI})

[ryanking13]

These are the list of build dependencies for packages in the Pyodide distribution as of 2024/11/26. It is not as much than I thought, and I believe most of them are pure Python packages except for a few packages. So maybe we can just host them (not included in the Pyodide distribution, but only for build time dependency)

build
cffi
cmake
colorama
cppy
cysignals
cython
ewah-bool-utils
expandvars
extension-helpers
flit_core
flit_scm
gmpy2
hatch-fancy-pypi-readme
hatch-vcs
hatchling
jinja2
lazy_loader
maturin
meson-python
meson
mypy_extensions
nanobind
ninja
numpy
oldest-supported-numpy
packaging
pip
pkgconfig
poetry-core
py-cpuinfo
pybind11
pythran
rebound
scikit-build-core
scikit-build
scipy
setuptools
setuptools-git-versioning
setuptools-rust
setuptools-scm
sphinx
tomli;
types-psutil
types-setuptools
typing-extensions
versioneer[toml]
wheel