Update numpy to 1.23.5, scipy 1.9.3, networkx 2.8.8, meson_python 0.11.0

[mkoeppe]

https://github.com/numpy/numpy/releases/tag/v1.23.4

https://docs.scipy.org/doc/scipy/release.1.9.3.html

https://meson-python.readthedocs.io/en/latest/changelog.html#changelog

Depends on #34081

Upstream: Fixed upstream, in a later stable release.

CC: @dimpase @dcoudert @kiwifb @orlitzky @jhpalmieri

Component: packages: standard

Author: Matthias Koeppe

Branch/Commit: d74528c

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/34658

[mkoeppe]

Branch: u/mkoeppe/update_numpy_to_1_23_4

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Commit: 0d18405

[mkoeppe]

New commits:

0d18405

build/pkgs/numpy: Update to 1.23.4

[mkoeppe]

Reviewer: https://github.com/mkoeppe/sage/actions/runs/3251926884

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,2 +1,3 @@
 https://github.com/numpy/numpy/releases/tag/v1.23.4

+https://docs.scipy.org/doc/scipy/release.1.9.3.html

[mkoeppe]

Dependencies: #34081

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 0d18405 to ba5c196

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

d36ed98	tox.ini (python3.*-minimal): Provide system liblzma so that system python can be accepted; not system xz
f62b625	Merge #32182
8990d53	.github/workflows/docker.yml: Add option free_disk_space
09244b0	tox.ini (python3.x): Do not clobber user-provided EXTRA_SAGE_PACKAGES
9fb4eac	.github/workflows/docker.yml: Add option extra_sage_packages
3897adb	tox.ini (manylinux-minimal): Provide system liblzma so that system python can be accepted; not system xz
4b7f679	Merge #34081
fcc7087	build/pkgs/scipy: Update to 1.9.3
a2b5da4	build/pkgs/networkx: Update to 2.8.7
ba5c196	build/pkgs/pythran: Update to 0.12.0

[dimpase]

comment:7

I get test errors in matrix2.pyx, line 6935 which calls scipy on these tasks:

A = matrix(RDF, [[1, -2], [3, 4]])
B = matrix(RDF, [[0, 7], [2, -3]])
D, P = A.eigenmatrix_left(B)
(P * A - D * P * B).norm() < 1e-14

(and a similar at line 7151 (for _right rather than _left).

And it's not just numerical noise:

sage: (P * A - D * P * B).norm()
12.951156314021555
sage: P * A - D * P * B
[ 1.1102230246251565e-16     -12.951156314021551]
[-2.7755575615628914e-17 -3.1086244689504383e-15]

[dimpase]

comment:8

the 1st of two gen. eigenvectors is wrong.

sage: A.eigenvectors_left(B)
[(-2.1075759441619084, [(-0.9905315991237724, 0.1372849268394109)], 1),
 (-0.3922809560592623, [(-0.9668186179746967, 0.2554638133620835)], 1)]

vs. the previous version (1.8.1), correct one:

sage: A.eigenvectors_left(B)
[(1.8208523846306908, [(-0.5400711566794881, -0.8416193591659354)], 1),
 (-0.39228095605926194, [(-0.9668186179746967, 0.2554638133620835)], 1)]

This can be traced down to scipy call:

sage:         v, eig = scipy.linalg.eig(A.numpy(),
....:                                   B.numpy(),
....:                                   right=False, left=True,
....:                                   homogeneous_eigvals=False)
sage: v
array([-2.10757594+0.j, -0.39228096+0.j])
sage: eig
array([[-0.9905316 , -1.        ],
       [ 0.13728493,  0.26423138]])

vs, correct, previous version:

sage: v
array([ 1.82085238+0.j, -0.39228096+0.j])
sage: eig
array([[-0.54007116, -1.        ],
       [-0.84161936,  0.26423138]])

[dimpase]

Upstream: Not yet reported upstream; Will do shortly.

[dimpase]

comment:9

I can reproduce this with "plain" scipy on Gentoo (installed as system Gentoo package)

[dimpase]

comment:10

This is !Gentoo/Sage-specific. If instead of system- (or Sage-)built scipy I use binary wheels from pypi, everything is fine.

The next test will be to try building Sage's scipy on a non-Gentoo system.

[dimpase]

comment:11

cc-ing Gentoo people. On Gentoo-built scipy 1.9.3, one gets incorrect generalised eigenvectors.

$ python3
Python 3.10.7 (main, Oct 10 2022, 13:00:40) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy, numpy
>>> A=numpy.array([[ 1., -2.],[ 3.,  4.]])
>>> B=numpy.array([[ 0.,  7.],[ 2., -3.]])
>>> scipy.linalg.eig(A,B,right=False, left=True, homogeneous_eigvals=False)
(array([-2.10757594+0.j, -0.39228096+0.j]), array([[-0.9905316 , -1.        ],
       [ 0.13728493,  0.26423138]]))

instead of correct

(array([ 1.82085238+0.j, -0.39228096+0.j]), array([[-0.54007116, -1.        ],
       [-0.84161936,  0.26423138]]))

pip-installing binary wheels for numpy ans scipy from PyPI (same versions) produces correct results.

[dimpase]

comment:12

On Fedora 34, everything is fine (with openblas 0.3.19, not sure whether it matters)

[dimpase]

comment:13

Building Sage's OpenBLAS (Gentoo provides an older OpenBLAS, 0.3.20) appears to fix the problem. Should we ban 0.3.20 ? Anyhow, Gentoo has to fix it.

[dimpase]

comment:14

Submitted https://bugs.gentoo.org/878883

[dimpase]

Changed upstream from Not yet reported upstream; Will do shortly. to Reported upstream. No feedback yet.

[dimpase]

Changed upstream from Reported upstream. No feedback yet. to Reported upstream. Developers acknowledge bug.

[dimpase]

comment:17

Further on arm64 (Apple M1) one gets

...
[scipy-1.9.3]   [1224/1224] scipy/io/_harwell_boeing/tests/test_hb.py
[scipy-1.9.3]   Preparing metadata (pyproject.toml): finished with status 'done'
[scipy-1.9.3] Building wheels for collected packages: scipy
[scipy-1.9.3]   Building wheel for scipy (pyproject.toml): started
[scipy-1.9.3]   Running command Building wheel for scipy (pyproject.toml)
[scipy-1.9.3]   Building wheel for scipy (pyproject.toml): finished with status 'done'
[scipy-1.9.3]   Created wheel for scipy: filename=scipy-1.9.3-cp310-cp310-macosx_13_arm64.whl size=21560240 sha256=7f146f5c01cacccb1f543dd03f7852c9fa69832ac83e3438a3054b07ddadf986
[scipy-1.9.3]   Stored in directory: /private/var/folders/81/vt06lnj17rj0cpc9gdshy4q00000gr/T/pip-ephem-wheel-cache-u972yyp0/wheels/07/68/a7/3970e07638c02f51cea93e74047bd18f05b865d87665dddaa5
[scipy-1.9.3] Successfully built scipy
[scipy-1.9.3] WARNING: Skipping scipy as it is not installed.
[scipy-1.9.3] Using pip 22.2.2 from /Volumes/dima/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/pip (python 3.10)
[scipy-1.9.3] Looking in links: /Volumes/dima/sage/local/var/lib/sage/venv-python3.10/var/lib/sage/wheels
[scipy-1.9.3] ERROR: scipy-1.9.3-cp310-cp310-macosx_13_arm64.whl is not a supported wheel on this platform.
[scipy-1.9.3] Warning: installing with "python3 -m pip install --verbose --no-index --find-links=/Volumes/dima/sage/local/var/lib/sage/venv-python3.10/var/lib/sage/wheels --disable-pip-version-check --isolated --no-cache-dir" failed. Retrying, adding "--no-deps --ignore-installed --ignore-requires-python"
[scipy-1.9.3] Using pip 22.2.2 from /Volumes/dima/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/pip (python 3.10)
[scipy-1.9.3] Looking in links: /Volumes/dima/sage/local/var/lib/sage/venv-python3.10/var/lib/sage/wheels
[scipy-1.9.3] ERROR: scipy-1.9.3-cp310-cp310-macosx_13_arm64.whl is not a supported wheel on this platform.
[scipy-1.9.3] Error: installing with pip  failed

[dimpase]

comment:18

upgrading to pip 22.3 does not help

[mkoeppe]

comment:19

Reported in https://github.com/mesonbuild/meson-python/issues/160#issuecomment-1297972659

[dimpase]

comment:20

on intel macOS 13 with Homebrew, I get meson complaining about perfectly OK gfortran.

---

update - in fact, Apple's command line tools didn't update themselves, and that was the reason for this error. Sorry for noise.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5fc95bd	Merge tag '9.8.beta3' into t/34658/update_numpy_to_1_23_4
313d0fd	build/pkgs/networkx: Update to 2.8.8
3706d5d	build/pkgs/meson_python/patches: Add https://github.com/mesonbuild/meson-python/pull/202

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from ba5c196 to 3706d5d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e9590f5	build/pkgs/meson_python: Update to 0.11.0
bbf5f95	build/pkgs/numpy: Update to 1.23.5

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3706d5d to bbf5f95

[mkoeppe]

New commits:

e9590f5	build/pkgs/meson_python: Update to 0.11.0
bbf5f95	build/pkgs/numpy: Update to 1.23.5

[mkoeppe]

Changed upstream from Reported upstream. Developers acknowledge bug. to Fixed upstream, in a later stable release.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,3 +1,6 @@
 https://github.com/numpy/numpy/releases/tag/v1.23.4

 https://docs.scipy.org/doc/scipy/release.1.9.3.html
+
+https://meson-python.readthedocs.io/en/latest/changelog.html#changelog
+

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

d74528c

Merge tag '9.8.beta4' into t/34658/update_numpy_to_1_23_4

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bbf5f95 to d74528c

[jhpalmieri]

comment:29

On OS X M2, scipy fails some tests in its test suite, but installation succeeds anyway, not sure why.

=========================== short test summary info ============================
FAILED linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: 
FAILED optimize/tests/test_linprog.py::TestLinprogIPSparse::test_bug_6139 - A...
FAILED optimize/tests/test_linprog.py::TestLinprogIPSparsePresolve::test_bug_6139
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_maxiter_worsening[gmres]
= 4 failed, 37207 passed, 3314 skipped, 12061 deselected, 136 xfailed, 12 xpassed, 326 warnings in 515.97s (0:08:35) =

[jhpalmieri]

comment:30

Test suite passes on OS X Intel.

[mkoeppe]

comment:31

One of them may be https://github.com/scipy/scipy/issues/16926

[jhpalmieri]

comment:32

I would prefer that a failing test suite meant a failed installation, but maybe python3 -c 'import scipy; scipy.test()' doesn't return a nonzero code if it fails.

[mkoeppe]

comment:33

I agree, we should fix this.

[jhpalmieri]

comment:34

Maybe we should merge this now and patch things up in follow-up tickets, since I (for one) can't build the most recent beta without this. Are there other issues that should be addressed on this ticket?

[jhpalmieri]

comment:35

(I took a quick glance at the scipy code, and it looks like it's trying to return different exit codes. It was only a very quick glance, though.)

[mkoeppe]

comment:36

Tests for Linux are still running at https://github.com/mkoeppe/sage/actions/runs/3519527763

[mkoeppe]

comment:38

There is a problem on archlinux-standard related to scipy; that is unchanged by this upgrade. That's #34779

[mkoeppe]

comment:39

The other Linux platforms look fine.

[jhpalmieri]

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/3251926884 to John Palmieri

[jhpalmieri]

comment:40

Okay, then let's proceed here.

[mkoeppe]

comment:41

Thanks!

[dimpase]

comment:42

blocker, as on a recent macOS one cannot build scipy (1.9.20.

[vbraun]

Changed branch from u/mkoeppe/update_numpy_to_1_23_4 to d74528c