search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.39k stars 472 forks source link

Upgrade scipy to 1.9.x, add meson toolchain #34081

Closed mkoeppe closed 1 year ago

mkoeppe commented 2 years ago

This ticket is also the target ticket for https://github.com/FFY00/meson-python/blob/main/.github/workflows/ci-sage.yml

Previous update:

See also:

Depends on #34572 Depends on #32182

CC: @dimpase @antonio-rojas @kiwifb @orlitzky

Component: packages: standard

Author: Matthias Koeppe

Branch: 853a699

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/34081

mkoeppe commented 1 year ago
comment:113

Yes, there are known failures of the meson-python test suite on some platforms.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1. New commits:

79eccefbuild/make/Makefile.in: Only warn if meson_python testsuite fails
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from 3897adb to 79eccef

vbraun commented 1 year ago
comment:116

Doesn't seem to work, testsuite still fails with hard error

[meson_python-0.10.0] =========================== short test summary info ============================
[meson_python-0.10.0] FAILED tests/test_wheel.py::test_scipy_like - AssertionError: assert {'mypkg-...
[meson_python-0.10.0] ERROR tests/test_wheel.py::test_local_lib - subprocess.CalledProcessError: Co...
[meson_python-0.10.0] ERROR tests/test_wheel.py::test_detect_wheel_tag_module - subprocess.CalledPr...
[meson_python-0.10.0] ERROR tests/test_wheel.py::test_rpath - subprocess.CalledProcessError: Comman...
[meson_python-0.10.0] ERROR tests/test_wheel.py::test_uneeded_rpath - subprocess.CalledProcessError...
[meson_python-0.10.0] ============== 1 failed, 46 passed, 2 xfailed, 4 errors in 48.55s ==============
[meson_python-0.10.0] 
[meson_python-0.10.0] real  0m49.034s
[meson_python-0.10.0] user  0m36.548s
[meson_python-0.10.0] sys   0m3.388s
[meson_python-0.10.0] ************************************************************************
[meson_python-0.10.0] Error testing package meson_python-0.10.0
[meson_python-0.10.0] ************************************************************************
[meson_python-0.10.0] Please email sage-devel (http://groups.google.com/group/sage-devel)
[meson_python-0.10.0] explaining the problem and including the log files
[meson_python-0.10.0]   /var/lib/buildbot/slave/sage_git/build/logs/pkgs/meson_python-0.10.0.log
[meson_python-0.10.0] and
[meson_python-0.10.0]   /var/lib/buildbot/slave/sage_git/build/config.log
[meson_python-0.10.0] Describe your computer, operating system, etc.
[meson_python-0.10.0] If you want to try to fix the problem yourself, *don't* just cd to
[meson_python-0.10.0] /var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/var/tmp/sage/build/meson_python-0.10.0 and type 'make check' or whatever is appropriate.
[meson_python-0.10.0] Instead, the following commands setup all environment variables
[meson_python-0.10.0] correctly and load a subshell for you to debug the error:
[meson_python-0.10.0]   (cd '/var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/var/tmp/sage/build/meson_python-0.10.0' && '/var/lib/buildbot/slave/sage_git/build/sage' --buildsh)
[meson_python-0.10.0] When you are done debugging, you can type "exit" to leave the subshell.
[meson_python-0.10.0] ************************************************************************
Makefile:3079: recipe for target 'meson_python-SAGE_VENV-no-deps' failed
make[4]: *** [meson_python-SAGE_VENV-no-deps] Error 1
Makefile:3079: recipe for target '/var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/var/lib/sage/installed/meson_python-0.10.0' failed
make[3]: *** [/var/lib/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/var/lib/sage/installed/meson_python-0.10.0] Error 2
make[3]: Target 'all-sage' not remade because of errors.
Makefile:2735: recipe for target 'all-build' failed
make[2]: *** [all-build] Error 2
make[2]: Target 'build-start' not remade because of errors.
make[2]: Leaving directory '/var/lib/buildbot/slave/sage_git/build/build/make'
vbraun commented 1 year ago
comment:117

The buildbot has

SAGE_CHECK_PACKAGES=!gsl,!givaro,!python2,!python3,!yasm,!zeromq,!curl,!arb,!cython,!cvxopt,!cmake,!networkx,!r,!nose,!normaliz,!rpy2,!giac,!openssl,!flint
SAGE_CHECK=yes

so I'm guessing the SAGE_CHECK_PACKAGES_DEFAULT_yes aren't merged in but overwritten? Is that the desired behavior?

mkoeppe commented 1 year ago
comment:118

Yes, by setting this environment variable, you are overriding the defaults. This works as intended.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Changed commit from 79eccef to 853a699

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 1 year ago

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

853a699Merge tag '9.8.beta3' into t/34081/numpy_1_23_x__scipy_1_9_x
vbraun commented 1 year ago
comment:121

Install fails on OSX Catalina with 

  [1216/1224] scipy/io/arff/tests/data/test7.arff
  [1217/1224] scipy/io/arff/tests/data/test8.arff
  [1218/1224] scipy/io/arff/tests/data/test9.arff
  [1219/1224] scipy/io/_harwell_boeing/__init__.py
  [1220/1224] scipy/io/_harwell_boeing/_fortran_format_parser.py
  [1221/1224] scipy/io/_harwell_boeing/hb.py
  [1222/1224] scipy/io/_harwell_boeing/tests/__init__.py
  [1223/1224] scipy/io/_harwell_boeing/tests/test_fortran_format.py
  [1224/1224] scipy/io/_harwell_boeing/tests/test_hb.py
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: scipy
  Building wheel for scipy (pyproject.toml): started
  Running command Building wheel for scipy (pyproject.toml)
  Building wheel for scipy (pyproject.toml): finished with status 'done'
  Created wheel for scipy: filename=scipy-1.9.2-cp310-cp310-macosx_10_15_7_x86_64.whl size=23108774 sha256=879e2b49697cfab5f3ade64070cd1e4190bab808a13ddf592a2ad2019c1798a0
  Stored in directory: /private/tmp/pip-ephem-wheel-cache-xb5awkgq/wheels/48/64/0a/04a7b283865a80737d531abd1c9cf4c8b45e2c32c4e8771b91
Successfully built scipy
WARNING: Skipping scipy as it is not installed.
Using pip 22.3 from /Users/buildbot-sage/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/lib/python3.10/site-packages/pip (python 3.10)
Looking in links: /Users/buildbot-sage/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/var/lib/sage/wheels
ERROR: scipy-1.9.2-cp310-cp310-macosx_10_15_7_x86_64.whl is not a supported wheel on this platform.
Warning: installing with "python3 -m pip install --verbose --no-index --find-links=/Users/buildbot-sage/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/var/lib/sage/wheels --disable-pip-version-check --isolated --no-cache-dir" failed. Retrying, adding "--no-deps --ignore-installed --ignore-requires-python"
Using pip 22.3 from /Users/buildbot-sage/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/lib/python3.10/site-packages/pip (python 3.10)
Looking in links: /Users/buildbot-sage/slave/sage_git/build/local/var/lib/sage/venv-python3.10.8/var/lib/sage/wheels
ERROR: scipy-1.9.2-cp310-cp310-macosx_10_15_7_x86_64.whl is not a supported wheel on this platform.
Error: installing with pip  failed
********************************************************************************
Error installing scipy-1.9.2-cp310-cp310-macosx_10_15_7_x86_64.whl
********************************************************************************
mkoeppe commented 1 year ago
comment:122

I have reported it upstream at https://github.com/mesonbuild/meson-python/issues/160#issuecomment-1305912020 but at some point (maybe now?) we'll have to drop support for Catalina. It's no longer supported by Apple IIRC, and also scheduled for removal on GH Actions.

Also homebrew no longer supports it - https://github.com/sagemath/sage/actions/runs/3355189832/jobs/5559268353 

Warning: You are using macOS 10.15.
We (and Apple) do not provide support for this old version.
It is expected behaviour that some formulae will fail to build in this old version.
It is expected behaviour that Homebrew will be buggy and slow.
Do not create any issues about this on Homebrew's GitHub repositories.
Do not create any issues even if you think this message is unrelated.
Any opened issues will be immediately closed without response.
Do not ask for help from MacHomebrew on Twitter.
You may ask for help in Homebrew's discussions but are unlikely to receive a response.
Try to figure out the problem yourself and submit a fix as a pull request.
We will review it but may or may not accept it.
vbraun commented 1 year ago

Changed branch from u/mkoeppe/numpy_1_23_x__scipy_1_9_x to 853a699

mkoeppe commented 1 year ago

Changed commit from 853a699 to none

mkoeppe commented 1 year ago
comment:124

Replying to Antonio Rojas:

Replying to @antonio-rojas:

I suppose this is caused by our openblas not including lapack symbols. Our distro package is fine since we haven't switched to meson (yet)

Indeed, building with -Dlapack=lapack fixes this when building the distro package with meson. Not sure how to fix this in sage.

I've opened #34779 for this