pip check fails due to dependency on ninja

[awvwgk]

Running pip check inside a conda environment currently fails because meson-python depends on the PyPI version of ninja and cannot detect the actually installed ninja version from conda-forge.

❯ mamba list
# packages in environment at /home/awvwgk/software/opt/conda/envs/mesonpep517:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
auditwheel                5.1.2            py39hf3d152e_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
cffi                      1.15.0           py39h4bc2ebd_0    conda-forge
dftd3                     0.5.1                    pypi_0    pypi
importlib-metadata        4.11.3           py39hf3d152e_1    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libblas                   3.9.0           14_linux64_openblas    conda-forge
libcblas                  3.9.0           14_linux64_openblas    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_16    conda-forge
libgfortran-ng            11.2.0              h69a702a_16    conda-forge
libgfortran5              11.2.0              h5c6108e_16    conda-forge
libgomp                   11.2.0              h1d223b6_16    conda-forge
liblapack                 3.9.0           14_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
meson                     0.62.0             pyhd8ed1ab_0    conda-forge
meson-python              0.3.0                    pypi_0    pypi
mesonpep517               0.2                pyhd8ed1ab_0    conda-forge
minpack                   2.0.0                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.10.2               h4bd325d_1    conda-forge
numpy                     1.22.3           py39h18676bf_2    conda-forge
openssl                   3.0.3                h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pep517                    0.12.0           py39hf3d152e_2    conda-forge
pint                      0.19.1             pyhd8ed1ab_0    conda-forge
pip                       22.0.4             pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0            py39hf3d152e_3    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.9.0            py39hb9d737c_1    conda-forge
pyelftools                0.28             py39hf3d152e_2    conda-forge
pyparsing                 3.0.8              pyhd8ed1ab_0    conda-forge
pytest                    7.1.2            py39hf3d152e_0    conda-forge
python                    3.9.12          h2660328_1_cpython    conda-forge
python-build              0.7.0              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
qcelemental               0.24.0             pyhd8ed1ab_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.1.0           py39hf3d152e_0    conda-forge
sqlite                    3.38.2               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
typing-extensions         4.2.0                hd8ed1ab_1    conda-forge
typing_extensions         4.2.0              pyha770c72_1    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
❯ pip check
meson-python 0.3.0 requires ninja, which is not installed.

[rgommers]

That seems simply expected to me, given thatpip cannot know about conda-installed packages.

This will resolve itself as soon as we package meson-python for conda-forge (and defaults will follow later). Till then, either:

• don't worry about it (I've never found a reason to use pip check, since it cannot know about my most important dependencies anyway)
• install ninja and meson from PyPI if you really want pip check to pass

I propose to close this, since this is normal and not a problem.

[awvwgk]

This will resolve itself as soon as we package meson-python for conda-forge

That's why I'm asking actually, see https://github.com/conda-forge/staged-recipes/pull/19033.

[rgommers]

That's why I'm asking actually, see conda-forge/staged-recipes#19033.

Thanks for doing that packaging work. I commented on your PR there.

Running pip check inside a conda environment currently fails because meson-python depends on the PyPI version of ninja and cannot detect the actually installed ninja version from conda-forge.

Can you post the actual traceback you're seeing, and/or how to reproduce the problem?

[awvwgk]

This should more or less match the used check in the recipe

❯ mamba create -n mesonpy colorama meson pep621 tomli --override-channels --channel conda-forge
... snip ...
❯ mamba activate mesonpy
❯ pip install meson-python==0.4.0 --no-deps
Collecting meson-python==0.4.0
  Using cached meson_python-0.4.0-py3-none-any.whl (14 kB)
Installing collected packages: meson-python
Successfully installed meson-python-0.4.0
❯ mamba list
# packages in environment at /home/awvwgk/software/opt/conda/envs/mesonpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
meson                     0.62.1             pyhd8ed1ab_0    conda-forge
meson-python              0.4.0                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
openssl                   3.0.3                h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pep621                    0.4.0              pyhd8ed1ab_0    conda-forge
pip                       22.1.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
python                    3.10.4          h2660328_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.3.2          py310hff52083_0    conda-forge
sqlite                    3.38.5               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
❯ pip check
meson-python 0.4.0 requires ninja, which is not installed.
❯ echo $?
1

[rgommers]

That all makes sense. It may be the case that the pip shipped by conda-forge is patched so that it can recognize Python packages installed by conda/mamba, but that patches is failing to pick up a non-Python package on PyPI.

How about just asking on the conda-forge PR how to bypass pip check?

[awvwgk]

How about just asking on the conda-forge PR how to bypass pip check?

I can simply remove the pip check test from the recipe, still I would prefer to pass this check as a safe-guard should dependencies change with a future update.

[rgommers]

Then maybe a patch is needed for pip itself. I don't think it can be fixed in meson-python.

[awvwgk]

It may be the case that the pip shipped by conda-forge is patched so that it can recognize Python packages installed by conda/mamba

Python packages installed by conda / mamba are not fundamentally different from other Python packages installed by system package managers or pip. In fact most Python package are packaged via pip install on conda-forge.

The issue is that meson-python expects a non-Python package (ninja) to be installed and findable via pip, which is generally not true. The Kitware distribution of ninja on PyPI is to my knowledge the only ninja distribution which provides Python metadata, however depending on this can be dangerous because it cannot be installed in all environments, especially on Windows this dependency is not 100% reliable.

[rgommers]

The issue is that meson-python expects a non-Python package (ninja) to be installed and findable via pip, which is generally not true.

I'm not sure why you say that meson-python expects ninja to be "findable" via pip, and what that means? I think for an isolated build (with pip or build), ninja will be installed in the isolated venv, and for a build without build isolation, ninja from conda-forge works perfectly fine. Can you elaborate?

[awvwgk]

Installing meson-python with pip will always pull in ninja from PyPI, even if the environment actually provides a version, but not the metadata required for pip to recognize the installation. This leads to an inevitable additional installation of ninja as demonstrated here:

❯ mamba activate mesonpy
❯ mamba list
# packages in environment at /home/awvwgk/software/opt/conda/envs/mesonpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
meson                     0.62.1             pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
openssl                   3.0.3                h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pep621                    0.4.0              pyhd8ed1ab_0    conda-forge
pip                       22.1.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
python                    3.10.4          h2660328_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.3.2          py310hff52083_0    conda-forge
sqlite                    3.38.5               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
❯ pip install .
Processing /home/awvwgk/projects/src/git/meson-python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: meson>=0.60.0 in /home/awvwgk/software/opt/conda/envs/mesonpy/lib/python3.10/site-packages (from meson-python==0.4.0) (0.62.1)
Collecting ninja
  Using cached ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
Requirement already satisfied: tomli>=1.0.0 in /home/awvwgk/software/opt/conda/envs/mesonpy/lib/python3.10/site-packages (from meson-python==0.4.0) (2.0.1)
Building wheels for collected packages: meson-python
  Building wheel for meson-python (pyproject.toml) ... done
  Created wheel for meson-python: filename=meson_python-0.4.0-py3-none-any.whl size=14560 sha256=566f3387e324164dc31d4658f5da20cdc6fd56668db02b6971c2228cc8af6f6e
  Stored in directory: /home/awvwgk/.cache/pip/wheels/1e/e7/8d/f91cbe34dbc546a667d1e8b03e26c415a829c511e6d1e22f6e
Successfully built meson-python
Installing collected packages: ninja, meson-python
Successfully installed meson-python-0.4.0 ninja-1.10.2.3
❯ mamba list
# packages in environment at /home/awvwgk/software/opt/conda/envs/mesonpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
meson                     0.62.1             pyhd8ed1ab_0    conda-forge
meson-python              0.4.0                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.10.2.3                 pypi_0    pypi
openssl                   3.0.3                h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pep621                    0.4.0              pyhd8ed1ab_0    conda-forge
pip                       22.1.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
python                    3.10.4          h2660328_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.3.2          py310hff52083_0    conda-forge
sqlite                    3.38.5               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge

If I can see this correctly the meson package on PyPI does not depend on Kitware's ninja distribution, therefore I'm wondering why meson-python is adding it as dependency. Keep in mind that ninja is a C++ program and not a Python package, pip is not supposed to manage it.

[rgommers]

even if the environment actually provides a version, but not the metadata required for pip to recognize the installation. This leads to an inevitable additional installation of ninja as demonstrated here:

So I think that is https://github.com/conda-forge/ninja-feedstock/issues/26. If the ninja conda-forge package would declare it provides ninja, things would work as expected here.

The dependency is there, and saying it's not is kinda backwards. The problem with PyPI is not that ninja is available there, it's that other (native) dependencies are not available.

Installing meson-python with pip

This is never necessary anymore once meson-python is packaged for conda.

[eli-schwartz]

If I can see this correctly the meson package on PyPI does not depend on Kitware's ninja distribution, therefore I'm wondering why meson-python is adding it as dependency.

Meson does have an optional extra, pip install 'meson[ninja]' will also guarantee that ninja support is resolved via PyPI rather than being assumed to exist via the OS repositories. And it is frequently used by linux distros which do not install PyPI-repackaged versions of system C++ command-line tools, so it cannot be a hard dependency.

meson-python is obviously making the tradeoff that pip install foo where foo is a package that utilizes this build backend, cannot work reliably if it cannot install ninja, and it cannot do that via pacman, emerge, dnf, apt, xbps... so better to install an extra copy than to break pip install. Obviously, this will mean distribution via channels that install ninja outside of pip, will need to patch meson-python to not depend on ninja.

[rgommers]

So I think that is conda-forge/ninja-feedstock#26. If the ninja conda-forge package would declare it provides ninja, things would work as expected here.

After some more digging in that repo, the correct fix is to create a new python-ninja package that installs the equivalent of the Python part of the ninja PyPI package. And then the meson-python conda-forge package should depend on python-ninja rather than on ninja, that will make everything work.

[rgommers]

And it is frequently used by linux distros which do not install PyPI-repackaged versions of system C++ command-line tools, so it cannot be a hard dependency.

I think this is the difference between meson and meson-python; the former is very Linux-oriented, while the latter is by nature cross-platform. On macOS and Windows users will typically not have ninja installed, so a hard dependency in meson-python seems like the right choice to me.

[rgommers]

Update: one idea @FFY00 longer term came up with is to write a small wrapper which checks if ninja is already installed in an environment (e.g., run it in a subprocess) and if so, don't add it to the required runtime dependencies.

For now, the status quo should be okay though.

[wolfv]

Maybe we could add a --ignore flag to pip check? And we could have an environment variable that adds some common ignore packages, such as ninja, cmake under an env var like PIP_CHECK_IGNORE=ninja,cmake.... I think that could be quite helpful for conda-forge.

[rgommers]

That does seem reasonable to me @wolfv, and perhaps better than every redistributor having to patch pyproject.toml files. Do you want to propose that on the Pip issue tracker?

[wolfv]

@rgommers I've opened: https://github.com/pypa/pip/issues/11157 I think the implementation would be quite straightforward for someone with pip experience as the function that is called in pip check already supports an ignore argument:

https://github.com/pypa/pip/blob/b91dbde21f9274c4df8c76be750401cda3e0fc3b/src/pip/_internal/operations/check.py#L51-L53

https://github.com/pypa/pip/blob/b91dbde21f9274c4df8c76be750401cda3e0fc3b/src/pip/_internal/commands/check.py#L25

[FFY00]

Thank you all for your suggestions on how to handle this. This is the old issue of the Python packaging ecosystem not integrating well with external dependencies.

As Ralf mentioned, one possible solution would be to try to detect if ninja is available, by looking for a ninja executable. Even though this would be a suitable solution, I think it is a bit unsafe, and so I am a reluctant to do it.

[mgorny]

As Ralf mentioned, one possible solution would be to try to detect if ninja is available, by looking for a ninja executable. Even though this would be a suitable solution, I think it is a bit unsafe, and so I am a reluctant to do it.

Yes, I don't think this is a good idea since it would mean that the package would produce a different wheel depending on what's installed on the system.

One option that could make sense is to give an option for vendors to easily control the presence of the dep, something like -Dsystem-ninja=... that would skip the dep if true (i.e. "system provides ninja") and add it if it's false (i.e. "we need to provide ninja").

[eli-schwartz]

Why would it produce a different wheel?

EDIT: Wait. I had assumed this would be adding it as an additional get_requires_for_build_wheel return value, not a run dependency of meson-python itself. That's in fact the exact approach being taken in #175

[mgorny]

Yeah, I was thinking of the current "run dependency" approach. Indeed having it returned via get_requires_for_build_wheel() should avoid the problem I was thinking about.

[FFY00]

@eli-schwartz that is what is happening, the PR is just keeping the dependency for bootstrap.

@mgorny we do not pin the dependency, so this is the same having a different version of the PyPI dependency. If you care about reproducibility you should be controlling the environment anyway.

The issue here is that we depend on a native dependency, and the PyPI ninja package is nothing but a hack, to workaround the fact that these kind of dependencies are not handled by Python packaging tooling. From a purity standpoint, one could argue that the ninja PyPI dependency shouldn't even be there.

Maybe one day we will have a way to better handle native dependencies, and then we should reconsider this situation, but for now let's settle for something that works well for the current status quo.

[FFY00]

If this causes issue anyway, we can also come back and reconsider.

[dnicolodi]

meson-python does not hard depend on the ninja PyPI package anymore. ninja is detected at runtime and added to the build backend dependencies only if necessary.