search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.39k stars 472 forks source link

Upgrade scipy to 1.9.x, add meson toolchain #34081

Closed mkoeppe closed 1 year ago

mkoeppe commented 2 years ago

This ticket is also the target ticket for https://github.com/FFY00/meson-python/blob/main/.github/workflows/ci-sage.yml

Previous update:

See also:

Depends on #34572 Depends on #32182

CC: @dimpase @antonio-rojas @kiwifb @orlitzky

Component: packages: standard

Author: Matthias Koeppe

Branch: 853a699

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/34081

mkoeppe commented 2 years ago

Dependencies: #32423

mkoeppe commented 2 years ago

Branch: u/mkoeppe/numpy_1_23_x__scipy_1_9_x

mkoeppe commented 2 years ago

Commit: d87212c

mkoeppe commented 2 years ago

Changed dependencies from #32423 to #32423, #33789

mkoeppe commented 2 years ago

Last 10 new commits:

e1998daMerge #32423
507fc2ebuild/pkgs/numpy: Update to 1.23.0
58d62f4build/bin/write-dockerfile.sh: ADD src/VERSION.txt
99f3617Merge #34017
3fab415build/pkgs/numpy: Switch to using .tar.gz
443b3d3build/pkgs/numpy/spkg-install.in: Fix 32bit build on x86_64
9cd5ea6Merge #34017
ee79c2ebuild/pkgs/numpy/spkg-install.in: Fix 32bit build on x86_64 (fixup)
2534f5abuild/pkgs/numpy/patches/21891.patch: New
d87212cMerge #32423
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

738ab26build/bin/sage-dist-helpers (sdh_pip_install): Change default to --build-isolation
79c6002build/bin/sage-dist-helpers (sdh_pip_install): Fall back to --no-build-isolation on error
f6ba526build/pkgs/tomli/spkg-install.in: Build it with --no-build-isolation
0b744beMerge #33789
10aaf61build/pkgs/meson: New
3ca53c9build/pkgs/pyproject_metadata: New
be09d5abuild/pkgs/ninja: New
3b92d3fbuild/pkgs/ninja_build/type: Change to standard
563ba19build/pkgs/pyproject_metadata/dependencies: Update
bf62474build/pkgs/meson_python: Update dependencies, spkg-install
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from d87212c to bf62474

mkoeppe commented 2 years ago

Changed dependencies from #32423, #33789 to #32423, #33789, #33866

mkoeppe commented 2 years ago
comment:6
[scipy-1.9.0rc1]       hook = backend.prepare_metadata_for_build_wheel
[scipy-1.9.0rc1]   AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1]   During handling of the above exception, another exception occurred:
[scipy-1.9.0rc1] 
[}}}
mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,8 @@
 - https://numpy.org/doc/stable/release/1.23.0-notes.html# (released)
 - https://scipy.github.io/devdocs/release.1.9.0.html (rc)
   - scipy.optimize.milp (HiGHS MIP solver)
+  - https://scipy.github.io/devdocs/release.1.9.0.html#scipy-switched-to-meson-as-its-build-system
+

 Previous update: 
 - #32423 Update numpy to 1.22.x, scipy 1.8.x, networkx 2.8.x
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from bf62474 to 22f91fd

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

5f29fb6build/pkgs/poetry_core: New (tomlkit build dep)
0804b62build/pkgs/poetry_core/patches/5547.diff_bin: New
0c4d0ebbuild/pkgs/hatchling/dependencies: Add pluggy, packaging
3e54474build/pkgs/pip: Update to 22.1.2
7e9722fbuild/pkgs/setuptools_scm: Update to 7.0.2, add typing_extensions as dep
6f72e8cbuild/pkgs/wheel: Update to 0.37.1
3e9d4a8Revert "build/pkgs/setuptools_scm: Update to 7.0.2, add typing_extensions as dep"
5d8111fbuild/pkgs/hatchling/SPKG.rst: Add license info
6607401src/sage/misc/package.py (pip_installed_packages): Make doctests more flexible
22f91fdMerge #33866
mkoeppe commented 2 years ago
comment:9

Opened https://github.com/scipy/scipy/issues/16536 (BUG: 1.9.0rc1 fails to build on macOS when CXX="g++ -std=gnu++11")

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 22f91fd to 02d9685

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

02d9685build/pkgs/scipy/spkg-install.in: Remove all old compiler/linker settings, add workaround for https://github.com/scipy/scipy/issues/16536
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 02d9685 to 48e6783

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

48e6783build/pkgs/scipy/spkg-install.in: Relax version pinning of numpy in pyproject.toml
mkoeppe commented 2 years ago
comment:12

Build goes almost through but ends with:

[scipy-1.9.0rc1] [1569/1573] Linking target scipy/spatial/_qhull.cpython-38-darwin.so
[scipy-1.9.0rc1] [1570/1573] Compiling C++ object scipy/interpolate/_rbfinterp_pythran.cpython-38-darwin.so.p/meson-generated_..__rbfinterp_pythran.cpp.o
[scipy-1.9.0rc1] [1571/1573] Linking target scipy/interpolate/_rbfinterp_pythran.cpython-38-darwin.so
[scipy-1.9.0rc1] [1572/1573] Compiling C++ object scipy/fft/_pocketfft/pypocketfft.cpython-38-darwin.so.p/pypocketfft.cxx.o
[scipy-1.9.0rc1] [1573/1573] Linking target scipy/fft/_pocketfft/pypocketfft.cpython-38-darwin.so
[scipy-1.9.0rc1] ninja: Entering directory `/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/build'
[scipy-1.9.0rc1] [1/2] Generating scipy/generate-version with a custom command
[scipy-1.9.0rc1] [2/2] Generating scipy/generate-config with a custom command
[scipy-1.9.0rc1] Installing scipy/__config__.py to /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install/usr/local/Cellar/python@3.8/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//scipy

<--- these directories are wrong

[scipy-1.9.0rc1] Installing scipy/version.py to /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install/usr/local/Cellar/python@3.8/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//scipy
[scipy-1.9.0rc1] Installing scipy/_lib/_ccallback_c.cpython-38-darwin.so to /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install/usr/local/Cellar/python@3.8/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scipy/_lib
[scipy-1.9.0rc1] Stripping target 'scipy/_lib/_ccallback_c.cpython-38-darwin.so'.
[scipy-1.9.0rc1] Could not strip file.
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] Stdout:
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] Stderr:
[scipy-1.9.0rc1] /Library/Developer/CommandLineTools/usr/bin/strip: error: symbols referenced by indirect symbol table entries that can't be stripped in: /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install/usr/local/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scipy/_lib/_ccallback_c.cpython-38-darwin.so
[scipy-1.9.0rc1] _PyBytes_AsStringAndSize
[scipy-1.9.0rc1] _PyBytes_FromString
[...]
[scipy-1.9.0rc1] __Py_NoneStruct
[scipy-1.9.0rc1] __Py_TrueStruct
[scipy-1.9.0rc1] ___stack_chk_guard
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] + meson setup --native-file=/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/usr/local/Cellar/python@3.8/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8 /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/build
[scipy-1.9.0rc1] + meson compile
[scipy-1.9.0rc1] + meson install --destdir /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install
[scipy-1.9.0rc1] Traceback (most recent call last):
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
[scipy-1.9.0rc1]     main()
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 345, in main
[scipy-1.9.0rc1]     json_out['return_val'] = hook(**hook_input['kwargs'])
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
[scipy-1.9.0rc1]     return _build_backend().build_wheel(wheel_directory, config_settings,
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 837, in build_wheel
[scipy-1.9.0rc1]     return project.wheel(out).name
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 786, in wheel
[scipy-1.9.0rc1]     wheel = _WheelBuilder(self).build(self._install_plan, self._copy_files, self._build_dir)
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 318, in build
[scipy-1.9.0rc1]     self._project.build()  # ensure project is built
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 482, in build
[scipy-1.9.0rc1]     self._meson('install', '--destdir', os.fspath(self._install_dir))
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 430, in _meson
[scipy-1.9.0rc1]     return self._proc('meson', *args)
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 426, in _proc
[scipy-1.9.0rc1]     subprocess.check_call(list(args))
[scipy-1.9.0rc1]   File "/usr/local/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
[scipy-1.9.0rc1]     raise CalledProcessError(retcode, cmd)
[scipy-1.9.0rc1] subprocess.CalledProcessError: Command '['meson', 'install', '--destdir', '/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install']' returned non-zero exit status 1.
[scipy-1.9.0rc1] * Getting dependencies for wheel...
[scipy-1.9.0rc1] * Building wheel...
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] ERROR Backend subprocess exited when trying to invoke build_wheel
mkoeppe commented 2 years ago
comment:13

That's https://github.com/scipy/scipy/issues/16446, https://github.com/FFY00/meson-python/pull/90

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

9b41154build/pkgs/meson_python/patches: Add https://github.com/FFY00/meson-python/pull/90
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 48e6783 to 9b41154

mkoeppe commented 2 years ago
comment:15

That fixes it, and now I get

[scipy-1.9.0rc1] Successfully built scipy-1.9.0rc1-cp38-cp38-macosx_12_x86_64.whl
[scipy-1.9.0rc1] WARNING: Skipping scipy as it is not installed.
[scipy-1.9.0rc1] Using pip 22.1.2 from /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pip (python 3.8)
[scipy-1.9.0rc1] Looking in links: /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/lib/sage/wheels
[scipy-1.9.0rc1] ERROR: scipy-1.9.0rc1-cp38-cp38-macosx_12_x86_64.whl is not a supported wheel on this platform.
mkoeppe commented 2 years ago
comment:16

After renaming it from ...macosx_12... to ...macosx_12_0..., the wheel can be installed

mkoeppe commented 2 years ago
comment:17

That's now https://github.com/FFY00/meson-python/issues/91

mkoeppe commented 2 years ago

Changed dependencies from #32423, #33789, #33866 to #32423, #33789, #33866, #34110

mkoeppe commented 2 years ago

Author: Matthias Koeppe

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,3 @@
-- https://numpy.org/doc/stable/release/1.23.0-notes.html# (released)
 - https://scipy.github.io/devdocs/release.1.9.0.html (rc)
   - scipy.optimize.milp (HiGHS MIP solver)
   - https://scipy.github.io/devdocs/release.1.9.0.html#scipy-switched-to-meson-as-its-build-system
@@ -6,4 +5,4 @@

 Previous update: 
 - #32423 Update numpy to 1.22.x, scipy 1.8.x, networkx 2.8.x
-
+- #34110 numpy 1.23.x
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a4b03ddbuild/pkgs/scipy/spkg-install.in: Back to sdh_pip_install
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 9b41154 to a4b03dd

mkoeppe commented 2 years ago

Changed dependencies from #32423, #33789, #33866, #34110 to #32423, #33789, #33866, #34110, #34119

mkoeppe commented 2 years ago
comment:21

Replying to @mkoeppe:

Opened https://github.com/scipy/scipy/issues/16536 (BUG: 1.9.0rc1 fails to build on macOS when CXX="g++ -std=gnu++11")

Opened #34119 (src/bin/sage-env: Export CXXCPP)

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a4b03dd to f83327a

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

4cce3efMerge tag '9.7.beta4' into t/34110/numpy_1_23
8f5860abuild/pkgs/numpy: Update to 1.23.1
5f6e21bMerge #34110
f83327abuild/pkgs/scipy: Use 1.9.0rc2
mkoeppe commented 2 years ago
comment:24

Replying to @mkoeppe:

That's now https://github.com/FFY00/meson-python/issues/91

Reported as resolved in meson-python 0.8.0

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from f83327a to eab320f

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

df36af4Merge tag '9.7.beta6' into t/34081/numpy_1_23_x__scipy_1_9_x
69045e3build/pkgs/meson_python: Update to 0.8.0
c51bdedbuild/pkgs/scipy: Update to 1.9.0rc3
eab320fbuild/pkgs/meson_python/patches/90.patch: Remove (upstreamed)
mkoeppe commented 2 years ago
comment:26

Indeed it creates a wheel with an acceptable filename now:

[scipy-1.9.0rc3]   Created wheel for scipy: filename=scipy-1.9.0rc3-cp39-cp39-macosx_12_0_x86_64.whl size=22657115 sha256=d2a44150f7bd42caf6610a06c9f4282430e4e8a119273fe42e9bea02d0aa0d10
mkoeppe commented 2 years ago

Reviewer: https://github.com/mkoeppe/sage/actions/runs/2741873215, https://github.com/mkoeppe/sage/actions/runs/2741873210

mkoeppe commented 2 years ago

Changed dependencies from #32423, #33789, #33866, #34110, #34119 to #34110

mkoeppe commented 2 years ago
comment:29

mesonpy introduces a hidden dependency on patchelf on Linux systems https://github.com/FFY00/meson-python/blob/865183e46a3ed434c0c347c1ca752aec1ce6fce9/mesonpy/__init__.py#L927

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from eab320f to 0223e55

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

ee16112build/pkgs/patchelf: New (scipy/meson_py dependency)
a9eaae5tox.ini, build/bin/write-dockerfile.sh: Add 'tox -e docker-...-incremental'
b9bfbf9tox.ini: Add comment
a0f0f2eMerge #34228
4e88370build/pkgs/meson_python/dependencies: fixup
a830dfbbuild/pkgs/distro: New (skbuild dep)
0223e55build/pkgs/scikit_build: New (patchelf build dependency)
mkoeppe commented 2 years ago

Changed dependencies from #34110 to #34110, #34228

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c59f6d8build/pkgs/patchelf: Switch to using non-Python package
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 0223e55 to c59f6d8

mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/2741873215, https://github.com/mkoeppe/sage/actions/runs/2741873210 to https://github.com/mkoeppe/sage/actions/runs/2743631300, https://github.com/mkoeppe/sage/actions/runs/2743631299

mkoeppe commented 2 years ago
comment:34

On debian-stretch-standard (https://github.com/mkoeppe/sage/runs/7532141718?check_suite_focus=true)

  [patchelf-0.15.0]   g++ -std=gnu++11 -DPACKAGE_NAME=\"patchelf\" -DPACKAGE_TARNAME=\"patchelf\" -DPACKAGE_VERSION=\"0.15.0\" -DPACKAGE_STRING=\"patchelf\ 0.15.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"patchelf\" -DVERSION=\"0.15.0\" -I.    -Wall -std=c++17 -D_FILE_OFFSET_BITS=64     -g -O2 -c -o patchelf.o patchelf.cc
  [patchelf-0.15.0]   patchelf.cc:29:20: fatal error: optional: No such file or directory
  [patchelf-0.15.0]    #include <optional>
  [patchelf-0.15.0]                       ^
mkoeppe commented 2 years ago
comment:35

This is a C++17 feature, https://stackoverflow.com/questions/38253971/std-optional-no-such-file-or-directory. On this platform, gcc is 6.3.0-18+deb9u1 and apparently does not have sufficient C++17 support

mkoeppe commented 2 years ago
comment:36

Opened https://github.com/NixOS/patchelf/issues/393

mkoeppe commented 2 years ago

Changed dependencies from #34110, #34228 to #34110, #34228, #34115