search

rgommers / pixi-dev-scipystack

Experimental out-of-tree dev setup for NumPy, SciPy and related projects with Pixi
Other
8 stars 1 forks source link

Missing `libxcrypt` dependency #12

Open dhirschfeld opened 1 month ago

dhirschfeld commented 1 month ago

pixi run wheel builds a Python 3.12 wheel no problems. I need a Python 3.10 wheel though. How can I specify the Python version to build for?

I naively added a constraint in the pixi.toml:

[dependencies]
python = "3.10"

...but that gave an error about a missing header file:

❯ pixi run wheel
✨ Pixi task (wheel in default): python -m build -wnx -Cbuild-dir=build-whl && cp dist/*.whl ../../wheelhouse/
* Building wheel...
+ meson setup --reconfigure ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/build-whl -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/build-whl/meson-python-native-file.ini
The Meson build system
Version: 1.5.1
Source dir: ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy
Build dir: ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/build-whl
Build type: native build
Project name: scipy
Project version: 1.15.0.dev0+git20241011.1adc8cb
C compiler for the host machine: ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/x86_64-conda-linux-gnu-cc (gcc 12.4.0 "x86_64-conda-linux-gnu-cc (conda-forge gcc 12.4.0-0) 12.4.0")
C linker for the host machine: ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.40
C++ compiler for the host machine: ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/x86_64-conda-linux-gnu-c++ (gcc 12.4.0 "x86_64-conda-linux-gnu-c++ (conda-forge gcc 12.4.0-0) 12.4.0")
C++ linker for the host machine: ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.40
Cython compiler for the host machine: cython (cython 3.0.11)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/python)
Program cython found: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/cython)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C supports arguments -Wno-unused-function: YES (cached)
Compiler for C supports arguments -Wno-conversion: YES (cached)
Compiler for C supports arguments -Wno-misleading-indentation: YES (cached)
Library m found: YES
Fortran compiler for the host machine: ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/x86_64-conda-linux-gnu-gfortran (gcc 12.4.0 "GNU Fortran (conda-forge gcc 12.4.0-0) 12.4.0")
Fortran linker for the host machine: ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/x86_64-conda-linux-gnu-gfortran ld.bfd 2.40
Compiler for Fortran supports arguments -Wno-conversion: YES (cached)
Checking if "-Wl,--version-script" : links: YES (cached)
Program tools/generate_f2pymod.py found: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/python3.10 ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/tools/generate_f2pymod.py)
Program scipy/_build_utils/tempita.py found: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/python3.10 ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/scipy/_build_utils/tempita.py)
Program pythran found: YES 0.16.1 0.16.1 (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/pythran)
Found pkg-config: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/pkg-config) 0.29.2
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency xsimd found: NO (tried pkgconfig and cmake)
Dependency threads found: YES unknown (cached)
Library npymath found: YES
Dependency pybind11 found: YES 2.13.6 (cached)
Program f2py found: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/f2py)
Run-time dependency scipy-openblas found: NO (tried pkgconfig)
Dependency openblas found: YES 0.3.27 (cached)
Dependency openblas found: YES 0.3.27 (cached)
Compiler for C supports arguments -Wno-maybe-uninitialized: YES (cached)
Compiler for C supports arguments -Wno-discarded-qualifiers: YES (cached)
Compiler for C supports arguments -Wno-empty-body: YES (cached)
Compiler for C supports arguments -Wno-implicit-function-declaration: YES (cached)
Compiler for C supports arguments -Wno-parentheses: YES (cached)
Compiler for C supports arguments -Wno-switch: YES (cached)
Compiler for C supports arguments -Wno-unused-label: YES (cached)
Compiler for C supports arguments -Wno-unused-result: YES (cached)
Compiler for C supports arguments -Wno-unused-variable: YES (cached)
Compiler for C++ supports arguments -Wno-bitwise-instead-of-logical: NO (cached)
Compiler for C++ supports arguments -Wno-cpp: YES (cached)
Compiler for C++ supports arguments -Wno-class-memaccess: YES (cached)
Compiler for C++ supports arguments -Wno-deprecated-declarations: YES (cached)
Compiler for C++ supports arguments -Wno-deprecated-builtins: NO (cached)
Compiler for C++ supports arguments -Wno-format-truncation: YES (cached)
Compiler for C++ supports arguments -Wno-non-virtual-dtor: YES (cached)
Compiler for C++ supports arguments -Wno-sign-compare: YES (cached)
Compiler for C++ supports arguments -Wno-switch: YES (cached)
Compiler for C++ supports arguments -Wno-terminate: YES (cached)
Compiler for C++ supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C++ supports arguments -Wno-unused-function: YES (cached)
Compiler for C++ supports arguments -Wno-unused-local-typedefs: YES (cached)
Compiler for C++ supports arguments -Wno-unused-variable: YES (cached)
Compiler for C++ supports arguments -Wno-int-in-bool-context: YES (cached)
Compiler for Fortran supports arguments -Wno-argument-mismatch: YES (cached)
Compiler for Fortran supports arguments -Wno-conversion: YES (cached)
Compiler for Fortran supports arguments -Wno-intrinsic-shadow: YES (cached)
Compiler for Fortran supports arguments -Wno-maybe-uninitialized: YES (cached)
Compiler for Fortran supports arguments -Wno-surprising: YES (cached)
Compiler for Fortran supports arguments -Wno-uninitialized: YES (cached)
Compiler for Fortran supports arguments -Wno-unused-dummy-argument: YES (cached)
Compiler for Fortran supports arguments -Wno-unused-label: YES (cached)
Compiler for Fortran supports arguments -Wno-unused-variable: YES (cached)
Compiler for Fortran supports arguments -Wno-tabs: YES (cached)
Compiler for Fortran supports arguments -Wno-argument-mismatch: YES (cached)
Compiler for Fortran supports arguments -Wno-conversion: YES (cached)
Compiler for Fortran supports arguments -Wno-maybe-uninitialized: YES (cached)
Compiler for Fortran supports arguments -Wno-unused-dummy-argument: YES (cached)
Compiler for Fortran supports arguments -Wno-unused-label: YES (cached)
Compiler for Fortran supports arguments -Wno-unused-variable: YES (cached)
Compiler for Fortran supports arguments -Wno-tabs: YES (cached)
Checking if "Check atomic builtins without -latomic" : links: YES (cached)
Configuring __config__.py using configuration
Checking for function "open_memstream" : YES (cached)
Configuring messagestream_config.h using configuration
Program _generate_pyx.py found: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/python3.10 ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/scipy/special/_generate_pyx.py)
Program _generate_pyx.py found: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/python3.10 ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/scipy/linalg/_generate_pyx.py)
Program ../_generate_sparsetools.py found: YES (~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/python3.10 ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/scipy/sparse/sparsetools/../_generate_sparsetools.py)
Checking for size of "void*" : 8 (cached)
Compiler for Fortran supports arguments -w: YES (cached)
Build targets in project: 194

scipy 1.15.0.dev0+git20241011.1adc8cb

  User defined options
    Native files: ~/code/github/scipy/pixi-dev-scipystack/scipy/scipy/build-whl/meson-python-native-file.ini
    buildtype   : release
    b_ndebug    : if-release
    b_vscrt     : md

Found ninja-1.12.1 at ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/ninja
+ ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/ninja
[9/1259] Compiling C object scipy/lib_fortranobject.a.p/.._.._.pixi_envs_default_lib_python3.10_site-packages_numpy_f2py_src_fortranobject.c.o
FAILED: scipy/lib_fortranobject.a.p/.._.._.pixi_envs_default_lib_python3.10_site-packages_numpy_f2py_src_fortranobject.c.o 
~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/x86_64-conda-linux-gnu-cc -Iscipy/lib_fortranobject.a.p -Iscipy -I../scipy -I../../.pixi/envs/default/lib/python3.10/site-packages/numpy/_core/include -I../../.pixi/envs/default/lib/python3.10/site-packages/numpy/f2py/src -I~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include/python3.10 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c17 -O3 -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -MD -MQ scipy/lib_fortranobject.a.p/.._.._.pixi_envs_default_lib_python3.10_site-packages_numpy_f2py_src_fortranobject.c.o -MF scipy/lib_fortranobject.a.p/.._.._.pixi_envs_default_lib_python3.10_site-packages_numpy_f2py_src_fortranobject.c.o.d -o scipy/lib_fortranobject.a.p/.._.._.pixi_envs_default_lib_python3.10_site-packages_numpy_f2py_src_fortranobject.c.o -c ../../.pixi/envs/default/lib/python3.10/site-packages/numpy/f2py/src/fortranobject.c
In file included from ../../.pixi/envs/default/lib/python3.10/site-packages/numpy/f2py/src/fortranobject.h:7,
                 from ../../.pixi/envs/default/lib/python3.10/site-packages/numpy/f2py/src/fortranobject.c:2:
~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include/python3.10/Python.h:44:10: fatal error: crypt.h: No such file or directory
   44 | #include <crypt.h>
      |          ^~~~~~~~~
compilation terminated.
[10/1259] Compiling C object scipy/libdummy_g77_abi_wrappers.a.p/_build_utils_src_wrap_dummy_g77_abi.c.o
FAILED: scipy/libdummy_g77_abi_wrappers.a.p/_build_utils_src_wrap_dummy_g77_abi.c.o 
~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/bin/x86_64-conda-linux-gnu-cc -Iscipy/libdummy_g77_abi_wrappers.a.p -Iscipy -I../scipy -I../../.pixi/envs/default/lib/python3.10/site-packages/numpy/_core/include -I~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include/python3.10 -I~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c17 -O3 -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem ~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -MD -MQ scipy/libdummy_g77_abi_wrappers.a.p/_build_utils_src_wrap_dummy_g77_abi.c.o -MF scipy/libdummy_g77_abi_wrappers.a.p/_build_utils_src_wrap_dummy_g77_abi.c.o.d -o scipy/libdummy_g77_abi_wrappers.a.p/_build_utils_src_wrap_dummy_g77_abi.c.o -c ../scipy/_build_utils/src/wrap_dummy_g77_abi.c
In file included from ../../.pixi/envs/default/lib/python3.10/site-packages/numpy/_core/include/numpy/npy_common.h:5,
                 from ../scipy/_build_utils/src/npy_cblas.h:9,
                 from ../scipy/_build_utils/src/wrap_dummy_g77_abi.c:25:
~/code/github/scipy/pixi-dev-scipystack/scipy/.pixi/envs/default/include/python3.10/Python.h:44:10: fatal error: crypt.h: No such file or directory
   44 | #include <crypt.h>
      |          ^~~~~~~~~
compilation terminated.
[18/1259] Copying file scipy/special/_hyp0f1.pxd
ninja: build stopped: subcommand failed.

ERROR Backend subprocess exited when trying to invoke build_wheel

How can a build for a specific Python version?

rgommers commented 1 month ago

I naively added a constraint in the pixi.toml:

What you did should work, so that's probably an issue with a cached/non-empty build directory or some such thing.

More importantly though, the wheel produced by pixi run wheel aren't meant to be portable to other environments/machines, so it's unlikely that you actually need this kind of wheel. If that's what you want, it needs an auditwheel (or delocate or delvewheel) invocation in the command for the wheel pixi task.

dhirschfeld commented 1 month ago

What you did should work, so that's probably an issue with a cached/non-empty build directory or some such thing.

I did a git clean -xfd in both pixi-dev-scipystack and pixi-dev-scipystack/scipy/scipy and it still failed with the crypt.h error. I would've assumed a missing dependency. Does a 3.10 build work for you with pixi-dev-scipystack?

rgommers commented 1 month ago

Yes, that works fine. I've tested both 3.10.0 and 3.10.14, all good. I was thinking perhaps you got 3.10.0, and the correct constraint is one of 3.10.* or >=3.10,<3.11 - you can check with pixi ls python. Either way it all works for me.

dhirschfeld commented 1 month ago

Thanks - it's good to know that it should work! Now I just need to figure out what I'm doing wrong... 😬

dhirschfeld commented 3 weeks ago

Googling turns up multiple instances of the missing crypt.h error: https://www.google.com/search?q=Python.h%3A44%3A10+fatal+error+%22crypt.h%22+%22No+such+file+or+directory%22

Which seems to suggest a missing libxcrypt dependency. Specifying that package as a dep seemed to fix it for me:

rgommers commented 3 weeks ago

Ah, I can reproduce it now - only with Python 3.10.0 on Linux (only had macOS at hand before). This is a curious case, looks like a bug in pixi, but isn't. It seems to be related to https://github.com/conda-forge/linux-sysroot-feedstock/issues/52, but that should have only removed libcrypt as a dependency for Python <3.9.

libxcrypt is a dependency of Python 3.10 in conda-forge, more specifically 3.10.15. I checked in two ways, first by creating a new env with just python 3.10:

$ mamba create -n crypt python=3.10
$ mamba activate crypt
$ ls ~/mambaforge/envs/crypt/include/cryp*
/home/rgommers/mambaforge/envs/crypt/include/crypt.h
$ mamba repoquery whoneeds libxcrypt
...
 Name   Version Build              Depends            Channel    
──────────────────────────────────────────────────────────────────
 python 3.10.15 h4a871b0_2_cpython libxcrypt >=4.4.36 conda-forge

And then by looking at the metadata in https://prefix.dev/channels/conda-forge/packages/python:

image

I reproduced the bug by adding python = "3.10" to pixi.toml, which gives you 3.10.0 (a bit of a footgun) instead of 3.10.15. When you then later update to "python = 3.10.* you will still get 3.10.0 since the already-present 3.10.0 in the lock file satisfies that .*. If instead you had used the .* syntax from the start, you'd get 3.10.15 and that doesn't have this issue.

@dhirschfeld can you please ensure you get <3.10.15` installed and the problem goes away then?