search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.39k stars 472 forks source link

Upgrade scipy to 1.9.x, add meson toolchain #34081

Closed mkoeppe closed 1 year ago

mkoeppe commented 2 years ago

This ticket is also the target ticket for https://github.com/FFY00/meson-python/blob/main/.github/workflows/ci-sage.yml

Previous update:

See also:

Depends on #34572 Depends on #32182

CC: @dimpase @antonio-rojas @kiwifb @orlitzky

Component: packages: standard

Author: Matthias Koeppe

Branch: 853a699

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/34081

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

71948ba.github/workflows/docker.yml: Add opensuse-15.4, centos-stream-9
5b5c76e.github/workflows/ci-linux.yml: Change name to CI Linux
fb80dceMerge #33791
7ddad62.github/workflows/docker.yml: Fix input type
0ee5b2e.github/workflows/docker.yml: Pass inputs.target* to tox
7d4dd69.github/workflows/docker.yml: Use upstream artifact only if: inputs.upstream_artifact
c161c5cMerge tag '9.7.beta5' into t/34115/tox_yml__refactor_using_reusable_workflows
c75c499Merge tag '9.7.beta6' into t/34115/tox_yml__refactor_using_reusable_workflows
5cb365a.github/workflows/docker.yml: Reduce NUMPROC to 4
b4e6615Merge #34115
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from c59f6d8 to b4e6615

mkoeppe commented 2 years ago
comment:39

On debian-stretch-standard (https://github.com/mkoeppe/sage/runs/7545914147?check_suite_focus=true)

Unpacking ninja-build (1.7.2-1) ...

 ...
 ERROR: Could not detect Ninja v1.8.2 or newer

  A full log can be found at /sage/local/var/lib/sage/venv-python3.10.5/var/tmp/sage/build/meson_python-0.8.0/src/.mesonpy-7_1ejlan/build/meson-logs/meson-log.txt
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

85797abbuild/pkgs/ninja_build: Update to 1.11.0
8f9135ebuild/pkgs/ninja_build/spkg-configure.m4: Set lower version bound to 1.8.2
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from b4e6615 to 8f9135e

mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/2743631300, https://github.com/mkoeppe/sage/actions/runs/2743631299 to https://github.com/mkoeppe/sage/actions/runs/2748565810, https://github.com/mkoeppe/sage/actions/runs/2743631299

antonio-rojas commented 2 years ago

Changed dependencies from #34110, #34228, #34115 to #34110, #34228, #34115, #34246

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-- https://scipy.github.io/devdocs/release.1.9.0.html (rc)
+- https://scipy.github.io/devdocs/release.1.9.0.html (released)
   - scipy.optimize.milp (HiGHS MIP solver)
   - https://scipy.github.io/devdocs/release.1.9.0.html#scipy-switched-to-meson-as-its-build-system
mkoeppe commented 2 years ago
comment:44

I'll rebase this away from the numpy upgrade (#34110)

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

fd0eb69build/pkgs/scipy: Use 1.9.0rc2
f517441build/pkgs/meson_python: Update to 0.8.0
ee75343build/pkgs/scipy: Update to 1.9.0rc3
bfd5ff7build/pkgs/meson_python/patches/90.patch: Remove (upstreamed)
30c2c79build/pkgs/patchelf: New (scipy/meson_py dependency)
66d989abuild/pkgs/meson_python/dependencies: fixup
b8e8ad7build/pkgs/patchelf: Switch to using non-Python package
270789ebuild/pkgs/patchelf: Downgrade to 0.13.1
1c53962build/pkgs/ninja_build: Update to 1.11.0
4fb3a63build/pkgs/ninja_build/spkg-configure.m4: Set lower version bound to 1.8.2
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 8f9135e to 4fb3a63

mkoeppe commented 2 years ago

Changed dependencies from #34110, #34228, #34115, #34246 to #34228, #34115, #34246

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -5,4 +5,6 @@

 Previous update: 
 - #32423 Update numpy to 1.22.x, scipy 1.8.x, networkx 2.8.x
+
+See also:
 - #34110 numpy 1.23.x
mkoeppe commented 2 years ago
comment:47

On 32bit, it builds a wheel that is named wrong

  [scipy-1.9.0rc3]   Using pip 22.1.2 from /sage/local/var/lib/sage/venv-python3.10.5/lib/python3.10/site-packages/pip (python 3.10)
  [scipy-1.9.0rc3]   Looking in links: /sage/local/var/lib/sage/venv-python3.10.5/var/lib/sage/wheels
  [scipy-1.9.0rc3]   ERROR: scipy-1.9.0rc3-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.

https://github.com/mkoeppe/sage/runs/7599750782?check_suite_focus=true

mkoeppe commented 2 years ago
comment:48

Opened https://github.com/FFY00/meson-python/issues/123

mkoeppe commented 2 years ago
comment:49

After renaming the wheel file to scipy-1.9.0rc3-cp310-cp310-linux_i686.whl, it can be installed with pip, and 

$ file venv/lib/python3.10/site-packages/scipy/**/*.so 
venv/lib/python3.10/site-packages/scipy/_lib/_ccallback_c.cpython-310-i386-linux-gnu.so:              ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=7ea4f88eb1db30a2264fa6414e073f9d42cb90c4, not stripped
venv/lib/python3.10/site-packages/scipy/_lib/_fpumode.cpython-310-i386-linux-gnu.so:                  ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=f9ae2041983e2b111529320c4b6b25d6fc1cd4d1, not stripped
...

confirms that this is a correct 32-bit build.

mkoeppe commented 2 years ago
comment:51

Replying to @mkoeppe:

mesonpy introduces a hidden dependency on patchelf on Linux systems https://github.com/FFY00/meson-python/blob/865183e46a3ed434c0c347c1ca752aec1ce6fce9/mesonpy/__init__.py#L927

see https://github.com/FFY00/meson-python/issues/125 (Only use and depend on patchelf if it is actually needed)

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 4fb3a63 to 90b07de

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

bcd9443Merge tag '9.7.beta7' into t/34081/numpy_1_23_x__scipy_1_9_x
a4c3821build/pkgs/meson_python/patches: Add https://github.com/FFY00/meson-python/pull/126
b354a38tox.ini, .github/workflows/docker.yml: Reimplement -incremental via SKIP_SYSTEM_PACKAGES
5decaddbuild/bin/write-dockerfile.sh: Do not use persistent env var PACKAGES (except on nix)
90b07deMerge #34228
mkoeppe commented 2 years ago
comment:53

Tested successfully with tox -e docker-ubuntu-bionic-i386-standard-incremental -- scipy

mkoeppe commented 2 years ago
comment:55

The build errors are all fixed, let's get this in

mkoeppe commented 2 years ago
comment:57

On archlinux-standard (https://github.com/mkoeppe/sage/runs/7659424080?check_suite_focus=true) I see 

 [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/csgraph/__init__.py", line 182, in <module>
  [sagemath_doc_html-none]   [plotting ]     from ._laplacian import laplacian
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/csgraph/_laplacian.py", line 7, in <module>
  [sagemath_doc_html-none]   [plotting ]     from scipy.sparse.linalg import LinearOperator
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/__init__.py", line 120, in <module>
  [sagemath_doc_html-none]   [plotting ]     from ._isolve import *
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/__init__.py", line 4, in <module>
  [sagemath_doc_html-none]   [plotting ]     from .iterative import *
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/iterative.py", line 9, in <module>
  [sagemath_doc_html-none]   [plotting ]     from . import _iterative
  [sagemath_doc_html-none]   [plotting ] ImportError: /sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-310-x86_64-linux-gnu.so: undefined symbol: slamch_
  [sagemath_doc_html-none]   [plotting ] /sage/src/sage/plot/plot.py:docstring of sage.plot.plot.multi_graphics:59: WARNING: Exception occurred in plotting plot-50
mkoeppe commented 2 years ago
comment:58

Also on gentoo-python3.9-standard (https://github.com/mkoeppe/sage/runs/7659423834?check_suite_focus=true) during the docbuild I see these strange errors:

  [sagemath_doc_html-none]   [plotting ] /sage/src/sage/plot/plot.py:docstring of sage.plot.plot.multi_graphics:59: WARNING: Exception occurred in plotting plot-50
  [sagemath_doc_html-none]   [plotting ]  from /sage/src/doc/en/reference/plotting/sage/plot/plot.rst:
  [sagemath_doc_html-none]   [plotting ] Traceback (most recent call last):
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/scipy/__init__.py", line 134, in <module>
  [sagemath_doc_html-none]   [plotting ]     from scipy.__config__ import show as show_config
  [sagemath_doc_html-none]   [plotting ] ModuleNotFoundError: No module named 'scipy.__config__'
  [sagemath_doc_html-none]   [plotting ] The above exception was the direct cause of the following exception:
  [sagemath_doc_html-none]   [plotting ] Traceback (most recent call last):
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 517, in _run_code
  [sagemath_doc_html-none]   [plotting ]     exec(code, ns)
  [sagemath_doc_html-none]   [plotting ]   File "<string>", line 2, in <module>
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 410, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **kwds)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 410, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **kwds)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 491, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **options)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/plot/matrix_plot.py", line 565, in matrix_plot
  [sagemath_doc_html-none]   [plotting ]     import scipy.sparse as scipysparse
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/scipy/__init__.py", line 139, in <module>
  [sagemath_doc_html-none]   [plotting ]     raise ImportError(msg) from e
  [sagemath_doc_html-none]   [plotting ] ImportError: Error importing SciPy: you cannot import SciPy while
  [sagemath_doc_html-none]   [plotting ]         being in scipy source directory; please exit the SciPy source
  [sagemath_doc_html-none]   [plotting ]         tree first and relaunch your Python interpreter.
mkoeppe commented 2 years ago
comment:59

I've opened https://github.com/FFY00/meson-python/issues/127 for this failure on gentoo

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 90b07de to 12698ea

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

12698eabuild/pkgs/scipy: Update to 1.9.0
antonio-rojas commented 2 years ago
comment:61

Replying to @mkoeppe:

[sagemath_doc_html-none] [plotting ] ImportError: /sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-310-x8664-linux-gnu.so: undefined symbol: slamch

I suppose this is caused by our openblas not including lapack symbols. Our distro package is fine since we haven't switched to meson (yet)

antonio-rojas commented 2 years ago
comment:62

Replying to @antonio-rojas:

I suppose this is caused by our openblas not including lapack symbols. Our distro package is fine since we haven't switched to meson (yet)

Indeed, building with -Dlapack=lapack fixes this when building the distro package with meson. Not sure how to fix this in sage.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

f896ae1build/pkgs/meson_python: Update to 0.8.1
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 12698ea to f896ae1

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from f896ae1 to 9e64ba4

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

9e64ba4build/pkgs/meson: Update to 0.63.0
kiwifb commented 2 years ago
comment:66

Replying to @mkoeppe:

Also on gentoo-python3.9-standard (https://github.com/mkoeppe/sage/runs/7659423834?check_suite_focus=true) during the docbuild I see these strange errors:

  [sagemath_doc_html-none]   [plotting ] /sage/src/sage/plot/plot.py:docstring of sage.plot.plot.multi_graphics:59: WARNING: Exception occurred in plotting plot-50
  [sagemath_doc_html-none]   [plotting ]  from /sage/src/doc/en/reference/plotting/sage/plot/plot.rst:
  [sagemath_doc_html-none]   [plotting ] Traceback (most recent call last):
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/scipy/__init__.py", line 134, in <module>
  [sagemath_doc_html-none]   [plotting ]     from scipy.__config__ import show as show_config
  [sagemath_doc_html-none]   [plotting ] ModuleNotFoundError: No module named 'scipy.__config__'
  [sagemath_doc_html-none]   [plotting ] The above exception was the direct cause of the following exception:
  [sagemath_doc_html-none]   [plotting ] Traceback (most recent call last):
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 517, in _run_code
  [sagemath_doc_html-none]   [plotting ]     exec(code, ns)
  [sagemath_doc_html-none]   [plotting ]   File "<string>", line 2, in <module>
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 410, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **kwds)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 410, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **kwds)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 491, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **options)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/plot/matrix_plot.py", line 565, in matrix_plot
  [sagemath_doc_html-none]   [plotting ]     import scipy.sparse as scipysparse
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/scipy/__init__.py", line 139, in <module>
  [sagemath_doc_html-none]   [plotting ]     raise ImportError(msg) from e
  [sagemath_doc_html-none]   [plotting ] ImportError: Error importing SciPy: you cannot import SciPy while
  [sagemath_doc_html-none]   [plotting ]         being in scipy source directory; please exit the SciPy source
  [sagemath_doc_html-none]   [plotting ]         tree first and relaunch your Python interpreter.

I just built sage-on-gentoo with scipy 1.9.0 without issue. It was with python 3.10 and ninja_build 1.10.2 though.

kiwifb commented 2 years ago
comment:67

Getting ninja to 1.11.0 didn't bring any issues in sage-on-gentoo.

mkoeppe commented 2 years ago

Changed dependencies from #34228, #34115, #34246 to #34228, #34115, #34246, #32992

mkoeppe commented 2 years ago

Changed dependencies from #34228, #34115, #34246, #32992 to #34228, #34115, #34246, #32992, #34449

mkoeppe commented 2 years ago
comment:71

New versions: scipy 1.9.1, meson 0.63.1, per update in https://github.com/FFY00/meson-python/issues/127#issuecomment-1235798227

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

2253e81Merge tag '9.7.rc0' into t/34081/numpy_1_23_x__scipy_1_9_x
66e1f10build/pkgs/scipy: Update to 1.9.1
b85c2d6build/pkgs/meson: Update to 0.63.1
adddf8eMerge tag '9.7.beta8' into t/34228/tox__e_docker_____incremental
a90608fbuild/bin/write-dockerfile.sh: Invoke sage-package directly, do not go through ./sage
85ce9a6Merge tag '9.7.rc0' into t/34228/tox__e_docker_____incremental
3d7229eMerge #34228
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 9e64ba4 to 3d7229e

mkoeppe commented 2 years ago
comment:73

The issue in comment:58 persists after these updates

mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/2748565810, https://github.com/mkoeppe/sage/actions/runs/2743631299 to none

mkoeppe commented 2 years ago

Changed dependencies from #34228, #34115, #34246, #32992, #34449 to #34228, #34115, #34246, #32992, #34449, #34344

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 3d7229e to aff3fca

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a2d8f2fbuild/pkgs/scipy/spkg-check.in: New
aff3fca.github/workflows/docker.yml: Lowercase for DOCKER_PUSH_REPOSITORY
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

a10bd29build/pkgs/scipy/spkg-check.in: Use scipy.test()
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from aff3fca to a10bd29

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a10bd29 to 0749844

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0749844build/pkgs/scipy/dependencies_check: New - need pytest
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 0749844 to df0fb15

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

df0fb15Merge tag '9.7.rc1' into t/34081/numpy_1_23_x__scipy_1_9_x