search

scipy / scipy

SciPy library main repository
https://scipy.org
BSD 3-Clause "New" or "Revised" License
13.15k stars 5.21k forks source link

BLD: failure with py314t & Cython 3.1.0a1 #21970

Open clin1234 opened 3 days ago

clin1234 commented 3 days ago

Fix past goof with #21968 

@clin1234 ➜ /workspaces $ venv/bin/pip install -U --pre --no-build-isolation  git+https://github.com/scipy/scipy
Collecting git+https://github.com/scipy/scipy
  Cloning https://github.com/scipy/scipy to /tmp/pip-req-build-g5vptxkj
  Running command git clone --filter=blob:none --quiet https://github.com/scipy/scipy /tmp/pip-req-build-g5vptxkj
  Resolved https://github.com/scipy/scipy to commit aa4e5771d25e36ff31bc270a3e4d44b6ba240f1e
  Running command git submodule update --init --recursive -q
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [261 lines of output]
      + meson setup /tmp/pip-req-build-g5vptxkj /tmp/pip-req-build-g5vptxkj/.mesonpy-65hh9e9b -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-req-build-g5vptxkj/.mesonpy-65hh9e9b/meson-python-native-file.ini
      The Meson build system
      Version: 1.6.0
      Source dir: /tmp/pip-req-build-g5vptxkj
      Build dir: /tmp/pip-req-build-g5vptxkj/.mesonpy-65hh9e9b
      Build type: native build
      Project name: scipy
      Project version: 1.15.0.dev0+git20241128.aa4e577
      C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      C linker for the host machine: cc ld.bfd 2.38
      C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      C++ linker for the host machine: c++ ld.bfd 2.38
      Cython compiler for the host machine: cython (cython 3.1.0)
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      Program python found: YES (/workspaces/venv/bin/python3.14-nogil)
      Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
      Run-time dependency python found: YES 3.14
      Program cython found: YES (/workspaces/venv/bin/cython)
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES
      Compiler for C supports arguments -Wno-unused-function: YES
      Compiler for C supports arguments -Wno-conversion: YES
      Compiler for C supports arguments -Wno-misleading-indentation: YES
      Library m found: YES
      Fortran compiler for the host machine: gfortran (gcc 11.4.0 "GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      Fortran linker for the host machine: gfortran ld.bfd 2.38
      ../meson.build:86: WARNING: Consider using the built-in option for language standard version instead of using "-std=legacy".
      Compiler for Fortran supports arguments -Wno-conversion: YES
      Checking if "-Wl,--version-script" : links: YES
      Program tools/generate_f2pymod.py found: YES (/workspaces/venv/bin/python3.14-nogil /tmp/pip-req-build-g5vptxkj/tools/generate_f2pymod.py)
      Program scipy/_build_utils/tempita.py found: YES (/workspaces/venv/bin/python3.14-nogil /tmp/pip-req-build-g5vptxkj/scipy/_build_utils/tempita.py)
      Program pythran found: YES 0.17.0 0.17.0 (/workspaces/venv/bin/pythran)
      Found CMake: /usr/bin/cmake (3.22.1)
      WARNING: CMake Toolchain: Failed to determine CMake compilers state
      Run-time dependency xsimd found: NO (tried pkgconfig and cmake)
      Run-time dependency threads found: YES
      numpy-config found: YES (/workspaces/venv/bin/numpy-config) 2.2.0
      Run-time dependency numpy found: YES 2.2.0
      Library npymath found: YES
      pybind11-config found: YES (/workspaces/venv/bin/pybind11-config) 2.13.6
      Run-time dependency pybind11 found: YES 2.13.6
      Checking if "thread_local" compiles: NO
      Checking if "_Thread_local" compiles: YES
      Checking if "__thread" compiles: YES
      Configuring scipy_config.h using configuration
      Program f2py found: YES (/workspaces/venv/bin/f2py)
      Run-time dependency scipy-openblas found: NO (tried pkgconfig)
      Run-time dependency openblas found: YES 0.3.20
      Dependency openblas found: YES 0.3.20 (cached)
      Compiler for C supports arguments -Wno-maybe-uninitialized: YES
      Compiler for C supports arguments -Wno-discarded-qualifiers: YES
      Compiler for C supports arguments -Wno-empty-body: YES
      Compiler for C supports arguments -Wno-implicit-function-declaration: YES
      Compiler for C supports arguments -Wno-parentheses: YES
      Compiler for C supports arguments -Wno-switch: YES
      Compiler for C supports arguments -Wno-unused-label: YES
      Compiler for C supports arguments -Wno-unused-result: YES
      Compiler for C supports arguments -Wno-unused-variable: YES
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
      Compiler for C++ supports arguments -Wno-bitwise-instead-of-logical: NO
      Compiler for C++ supports arguments -Wno-cpp: YES
      Compiler for C++ supports arguments -Wno-class-memaccess: YES
      Compiler for C++ supports arguments -Wno-deprecated-declarations: YES
      Compiler for C++ supports arguments -Wno-deprecated-builtins: NO
      Compiler for C++ supports arguments -Wno-format-truncation: YES
      Compiler for C++ supports arguments -Wno-non-virtual-dtor: YES
      Compiler for C++ supports arguments -Wno-sign-compare: YES
      Compiler for C++ supports arguments -Wno-switch: YES
      Compiler for C++ supports arguments -Wno-terminate: YES
      Compiler for C++ supports arguments -Wno-unused-but-set-variable: YES
      Compiler for C++ supports arguments -Wno-unused-function: YES
      Compiler for C++ supports arguments -Wno-unused-local-typedefs: YES
      Compiler for C++ supports arguments -Wno-unused-variable: YES
      Compiler for C++ supports arguments -Wno-int-in-bool-context: YES
      Compiler for Fortran supports arguments -Wno-argument-mismatch: YES
      Compiler for Fortran supports arguments -Wno-conversion: YES (cached)
      Compiler for Fortran supports arguments -Wno-intrinsic-shadow: YES
      Compiler for Fortran supports arguments -Wno-maybe-uninitialized: YES
      Compiler for Fortran supports arguments -Wno-surprising: YES
      Compiler for Fortran supports arguments -Wno-uninitialized: YES
      Compiler for Fortran supports arguments -Wno-unused-dummy-argument: YES
      Compiler for Fortran supports arguments -Wno-unused-label: YES
      Compiler for Fortran supports arguments -Wno-unused-variable: YES
      Compiler for Fortran supports arguments -Wno-tabs: YES
      Compiler for Fortran supports arguments -Wno-argument-mismatch: YES (cached)
      Compiler for Fortran supports arguments -Wno-conversion: YES (cached)
      Compiler for Fortran supports arguments -Wno-maybe-uninitialized: YES (cached)
      Compiler for Fortran supports arguments -Wno-unused-dummy-argument: YES (cached)
      Compiler for Fortran supports arguments -Wno-unused-label: YES (cached)
      Compiler for Fortran supports arguments -Wno-unused-variable: YES (cached)
      Compiler for Fortran supports arguments -Wno-tabs: YES (cached)
      Checking if "Check atomic builtins without -latomic" : links: YES
      Configuring __config__.py using configuration
      Checking for function "open_memstream" : YES
      Configuring messagestream_config.h using configuration
      Program _generate_pyx.py found: YES (/workspaces/venv/bin/python3.14-nogil /tmp/pip-req-build-g5vptxkj/scipy/special/_generate_pyx.py)
      Program _generate_pyx.py found: YES (/workspaces/venv/bin/python3.14-nogil /tmp/pip-req-build-g5vptxkj/scipy/linalg/_generate_pyx.py)
      Program ../_generate_sparsetools.py found: YES (/workspaces/venv/bin/python3.14-nogil /tmp/pip-req-build-g5vptxkj/scipy/sparse/sparsetools/../_generate_sparsetools.py)
      Checking for size of "void*" : 8
      Compiler for Fortran supports arguments -w: YES
      Checking for size of "void*" : 8

      Executing subproject highs

      highs| Project name: highs
      highs| Project version: 1.8.0
      highs| C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      highs| C linker for the host machine: cc ld.bfd 2.38
      highs| C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      highs| C++ linker for the host machine: c++ ld.bfd 2.38
      highs| Compiler for C++ supports arguments -Wno-invalid-offsetof: YES
      highs| Compiler for C++ supports arguments -Wno-maybe-uninitialized: YES
      highs| Compiler for C++ supports arguments -Wno-reorder: YES
      highs| Compiler for C++ supports arguments -Wno-reorder-ctor: NO
      highs| Compiler for C++ supports arguments -Wno-sometimes-uninitialized: NO
      highs| Compiler for C++ supports arguments -Wno-unused-but-set-variable: YES (cached)
      highs| Compiler for C++ supports arguments -Wno-unused-variable: YES (cached)
      highs| Compiler for C++ supports arguments -Wno-use-after-free: NO
      highs| Compiler for C++ supports arguments -Wno-comment: YES
      highs| Compiler for C supports arguments -Wno-comment: YES
      highs| Compiler for C supports arguments -Wno-invalid-offsetof: NO
      highs| Compiler for C supports arguments -Wno-maybe-uninitialized: YES (cached)
      highs| Compiler for C supports arguments -Wno-sometimes-uninitialized: NO
      highs| Compiler for C supports arguments -Wno-unused-label: YES (cached)
      highs| Compiler for C supports arguments -Wno-use-after-free: NO
      highs| Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
      highs| Compiler for C supports arguments -Wno-unused-variable: YES (cached)
      highs| Compiler for C supports arguments -Wno-use-after-free: NO (cached)
      highs| Compiler for C++ supports arguments -mpopcnt: YES
      highs| Dependency threads found: YES unknown (cached)
      highs| Checking if "Check atomic builtins without -latomic" : links: YES (cached)
      highs| Dependency zlib skipped: feature use_zlib disabled
      highs| Checking if "mm_pause check" compiles: YES
      highs| Checking if "builtin_clz check" compiles: YES
      highs| Configuring HConfig.h.meson.interim using configuration
      highs| Found git repository at /tmp/pip-req-build-g5vptxkj
      highs| Build targets in project: 146
      highs| Subproject highs finished.

      Build targets in project: 195

      scipy 1.15.0.dev0+git20241128.aa4e577

        Subprojects
          highs       : YES

        User defined options
          Native files: /tmp/pip-req-build-g5vptxkj/.mesonpy-65hh9e9b/meson-python-native-file.ini
          b_ndebug    : if-release
          b_vscrt     : md
          buildtype   : release

      Found ninja-1.11.1.git.kitware.jobserver-1 at /workspaces/venv/bin/ninja

      Generating targets:   0%|          | 0/195 eta ?
      Generating targets:  25%|██▌       | 49/195 eta 00:00
      Generating targets:  49%|████▊     | 95/195 eta 00:00
      Generating targets:  67%|██████▋   | 131/195 eta 00:00
      Generating targets:  83%|████████▎ | 161/195 eta 00:00
      Generating targets: 100%|██████████| 195/195 eta 00:00

      Writing build.ninja:   0%|          | 0/1606 eta ?

      + /workspaces/venv/bin/ninja
      [1/1464] Generating scipy/generate-version with a custom command
      [2/1464] Compiling C object scipy/libdummy_g77_abi_wrappers.a.p/_build_utils_src_wrap_dummy_g77_abi.c.o
      [3/1464] Linking static target scipy/libdummy_g77_abi_wrappers.a
      [4/1464] Copying file scipy/__init__.py
      [5/1464] Copying file scipy/optimize.pxd
      [6/1464] Copying file scipy/linalg.pxd
      [7/1464] Copying file scipy/special.pxd
      [8/1464] Copying file scipy/_lib/_ccallback_c.pxd
      [9/1464] Copying file scipy/_lib/__init__.py
      [10/1464] Copying file scipy/_lib/ccallback.pxd
      [11/1464] Copying file scipy/_lib/messagestream.pxd
      [12/1464] Compiling C object scipy/lib_fortranobject.a.p/.._.._.._workspaces_venv_lib_python3.14t_site-packages_numpy_f2py_src_fortranobject.c.o
      ../../../workspaces/venv/lib/python3.14t/site-packages/numpy/f2py/src/fortranobject.c: In function ‘fortran_getattr’:
      ../../../workspaces/venv/lib/python3.14t/site-packages/numpy/f2py/src/fortranobject.c:366:9: warning: ‘_PyDict_GetItemStringWithError’ is deprecated [-Wdeprecated-declarations]
        366 |         PyObject *v = _PyDict_GetItemStringWithError(fp->dict, name);
            |         ^~~~~~~~
      In file included from /usr/include/python3.14t/dictobject.h:101,
                       from /usr/include/python3.14t/Python.h:91,
                       from ../../../workspaces/venv/lib/python3.14t/site-packages/numpy/f2py/src/fortranobject.h:7,
                       from ../../../workspaces/venv/lib/python3.14t/site-packages/numpy/f2py/src/fortranobject.c:2:
      /usr/include/python3.14t/cpython/dictobject.h:38:44: note: declared here
         38 | Py_DEPRECATED(3.14) PyAPI_FUNC(PyObject *) _PyDict_GetItemStringWithError(PyObject *, const char *);
            |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      [13/1464] Linking static target scipy/lib_fortranobject.a
      [14/1464] Compiling C object scipy/_lib/_test_ccallback.cpython-314t-x86_64-linux-gnu.so.p/src__test_ccallback.c.o
      [15/1464] Compiling C object scipy/_lib/_fpumode.cpython-314t-x86_64-linux-gnu.so.p/_fpumode.c.o
      [16/1464] Linking target scipy/_lib/_test_ccallback.cpython-314t-x86_64-linux-gnu.so
      [17/1464] Linking target scipy/_lib/_fpumode.cpython-314t-x86_64-linux-gnu.so
      [18/1464] Generating 'scipy/_lib/_test_deprecation_call.cpython-314t-x86_64-linux-gnu.so.p/_test_deprecation_call.c'
      [19/1464] Generating 'scipy/_lib/_test_deprecation_def.cpython-314t-x86_64-linux-gnu.so.p/_test_deprecation_def.c'
      [20/1464] Compiling C object scipy/_lib/_test_deprecation_call.cpython-314t-x86_64-linux-gnu.so.p/meson-generated__test_deprecation_call.c.o
      [21/1464] Compiling C object scipy/_lib/_test_deprecation_def.cpython-314t-x86_64-linux-gnu.so.p/meson-generated__test_deprecation_def.c.o
      [22/1464] Linking target scipy/_lib/_test_deprecation_def.cpython-314t-x86_64-linux-gnu.so
      [23/1464] Generating 'scipy/_lib/_ccallback_c.cpython-314t-x86_64-linux-gnu.so.p/_ccallback_c.c'
      [24/1464] Linking target scipy/_lib/_test_deprecation_call.cpython-314t-x86_64-linux-gnu.so
      [25/1464] Compiling C object scipy/_lib/_ccallback_c.cpython-314t-x86_64-linux-gnu.so.p/meson-generated__ccallback_c.c.o
      [26/1464] Linking target scipy/_lib/_ccallback_c.cpython-314t-x86_64-linux-gnu.so
      [27/1464] Compiling C++ object scipy/_lib/_uarray/_uarray.cpython-314t-x86_64-linux-gnu.so.p/vectorcall.cxx.o
      [28/1464] Copying file scipy/special/__init__.py
      [29/1464] Copying file scipy/special/_agm.pxd
      [30/1464] Generating 'scipy/_lib/messagestream.cpython-314t-x86_64-linux-gnu.so.p/messagestream.c'
      [31/1464] Copying file scipy/special/_boxcox.pxd
      [32/1464] Copying file scipy/special/_cdflib_wrappers.pxd
      [33/1464] Copying file scipy/special/_complexstuff.pxd
      [34/1464] Copying file scipy/special/_convex_analysis.pxd
      [35/1464] Copying file scipy/special/_cunity.pxd
      [36/1464] Compiling C++ object scipy/_lib/_uarray/_uarray.cpython-314t-x86_64-linux-gnu.so.p/_uarray_dispatch.cxx.o
      [37/1464] Copying file scipy/special/_ellip_harm.pxd
      [38/1464] Linking target scipy/_lib/_uarray/_uarray.cpython-314t-x86_64-linux-gnu.so
      [39/1464] Copying file scipy/special/_ellip_harm_2.pxd
      [40/1464] Copying file scipy/special/_factorial.pxd
      [41/1464] Copying file scipy/special/_ellipk.pxd
      [42/1464] Copying file scipy/special/_hyp0f1.pxd
      [43/1464] Copying file scipy/special/_hypergeometric.pxd
      [44/1464] Copying file scipy/special/_legacy.pxd
      [45/1464] Copying file scipy/special/_ndtri_exp.pxd
      [46/1464] Copying file scipy/special/_sici.pxd
      [47/1464] Copying file scipy/special/_spence.pxd
      [48/1464] Copying file scipy/special/_xlogy.pxd
      [49/1464] Copying file scipy/special/orthogonal_eval.pxd
      [50/1464] Copying file scipy/special/sf_error.pxd
      [51/1464] Compiling C object scipy/_lib/messagestream.cpython-314t-x86_64-linux-gnu.so.p/meson-generated_messagestream.c.o
      [52/1464] Copying file scipy/special/_ufuncs_extra_code.pxi
      [53/1464] Linking target scipy/_lib/messagestream.cpython-314t-x86_64-linux-gnu.so
      [54/1464] Copying file scipy/special/_ufuncs_extra_code_common.pxi
      [55/1464] Compiling C object scipy/special/libsf_error_state.so.p/sf_error_state.c.o
      [56/1464] Linking target scipy/special/libsf_error_state.so
      [57/1464] Generating symbol file scipy/special/libsf_error_state.so.p/libsf_error_state.so.symbols
      [58/1464] Compiling C++ object scipy/special/_special_ufuncs.cpython-314t-x86_64-linux-gnu.so.p/_special_ufuncs_docs.cpp.o
      [59/1464] Compiling C++ object scipy/special/_special_ufuncs.cpython-314t-x86_64-linux-gnu.so.p/sf_error.cc.o
      [60/1464] Compiling C object scipy/special/libcdflib.a.p/cdflib.c.o
      [61/1464] Linking static target scipy/special/libcdflib.a
      [62/1464] Compiling C++ object scipy/special/_gufuncs.cpython-314t-x86_64-linux-gnu.so.p/_gufuncs_docs.cpp.o
      [63/1464] Compiling C++ object scipy/special/_gufuncs.cpython-314t-x86_64-linux-gnu.so.p/sf_error.cc.o
      [64/1464] Generating 'scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/_specfun.cpp'
      [65/1464] Compiling C++ object scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/meson-generated__specfun.cpp.o
      FAILED: scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/meson-generated__specfun.cpp.o
      c++ -Iscipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p -Iscipy/special -I../scipy/special -I../../../workspaces/venv/lib/python3.14t/site-packages/numpy/_core/include -I/usr/include/python3.14t -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++17 -O3 -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -MD -MQ scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/meson-generated__specfun.cpp.o -MF scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/meson-generated__specfun.cpp.o.d -o scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/meson-generated__specfun.cpp.o -c scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/_specfun.cpp
      In file included from /usr/include/python3.14t/internal/pycore_frame.h:13,
                       from scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/_specfun.cpp:12385:
      /usr/include/python3.14t/internal/pycore_code.h: In function ‘_Py_CODEUNIT* _PyCode_GetTLBCFast(PyThreadState*, PyCodeObject*)’:
      /usr/include/python3.14t/internal/pycore_code.h:617:53: error: invalid conversion from ‘void*’ to ‘_PyCodeArray*’ [-fpermissive]
        617 |     _PyCodeArray *code = _Py_atomic_load_ptr_acquire(&co->co_tlbc);
            |                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
            |                                                     |
            |                                                     void*
      In file included from scipy/special/_specfun.cpython-314t-x86_64-linux-gnu.so.p/_specfun.cpp:12385:
      /usr/include/python3.14t/internal/pycore_frame.h: In function ‘_Py_CODEUNIT* _PyFrame_GetBytecode(_PyInterpreterFrame*)’:
      /usr/include/python3.14t/internal/pycore_frame.h:96:53: error: invalid conversion from ‘void*’ to ‘_PyCodeArray*’ [-fpermissive]
         96 |     _PyCodeArray *tlbc = _Py_atomic_load_ptr_acquire(&co->co_tlbc);
            |                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
            |                                                     |
            |                                                     void*
      [66/1464] Compiling C++ object scipy/special/_gufuncs.cpython-314t-x86_64-linux-gnu.so.p/_gufuncs.cpp.o
      [67/1464] Compiling C++ object scipy/special/_special_ufuncs.cpython-314t-x86_64-linux-gnu.so.p/_special_ufuncs.cpp.o
      ninja: build stopped: subcommand failed.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
clin1234 commented 3 days ago

Possibly a CPython issue as well: python/cpython#127411

rgommers commented 3 days ago

Okay, that's a real problem. At first sight, it looks like it's Cython digging into CPython private internals, which have probably just changed. That tends to break regularly, and is more a synchronization problem than a bug - Cython will not be up-to-date with changes to CPython internals at this point in the CPython release cycle.

I'll note that we're targeting the 3.13 free-threading interpreter for now, there is no CI and no active work to keep main branches of all projects working together. NumPy typically starts looking at this when CPython hits beta 1 around May.

If you want this to work now, you should probably be prepared to participate in the fun of fixing issues like these up as they arise.

da-woods commented 2 days ago

This access into the frame internals is really just to set a line number. That seem silly but it's we don't have another good way of doing it and it's pretty necessary to get useful tracebacks out of Cython code.

In a real emergency we could could resort to the limited API fallback code instead. I'd rather not because it makes exceptions really slow, but it's an option if nothing else works.

For a wild guess at using that as a short-term solution you can define the C macro CYTHON_LIMITED_API=1 (but not Py_LIMITED_API). I don't see anything in _specfunc.pyx to make me think that won't work, but I haven't tried it myself.