TEST: 1.10.x + blas variants

h-vetinari commented 1 year ago

Following the same scheme as #223, #199, #172, #153 & #130. Should not be merged for the same reasons as #153.

conda-forge-webservices[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

h-vetinari commented 1 year ago

Scipy 1.10.0

TL;DR: Very similar to 1.9.3, but slight improvements

From 32 failures out of 96 runs for 1.9.3, we're now at 28 failures out of 96 CI runs

Notable

No PyPy builds (progressing in #208)
No Propack tests (though looking excellent in #200)
Emulation problems on PPC might finally be over?! at least it's looking like that: #225
Enforcing presence/absence of avx512 is a gamble based on which agent gets assigned; not all CI failures are actual failures

The good news:

the osx + accelerate failures are all very minor tolerance violations
linux + aarch64 failure gone

The bad news

persistent errors on win+openblas, unix+x64 and mkl+avx512

Restrictions

aarch64/ppc64le only run tests with label='fast', not the full test suite

the following test skips:

# these multithreading tests occasionally hang in emulation
{% set tests_to_skip = tests_to_skip + " or test_immediate_updating" %}         # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or test_mixed_threads_processes" %}    # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or test_multiprocess[fft]" %}          # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test__differential_evolution and test_parallel)" %}  # [aarch64 or ppc64le]
# on the slowest agents, these tests take more than 20min in emulation
{% set tests_to_skip = tests_to_skip + " or (test_lsq_linear and test_large_rank_deficient)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_iterative and test_precond_inverse[case1])" %}  # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_optimize and TestBrute and test_workers)" %}    # [aarch64 or ppc64le]
# tests that run into timeouts (set in scipy test suite) in emulation
{% set tests_to_skip = tests_to_skip + " or (test_propack and test_examples)" %}                  # [aarch64 or ppc64le]

Details

lib	for 1.9.3	now	updated version
`scipy`	`1.9.3`	`1.10.0`	X
`numpy`	`1.24.1`	`1.24.1`
`libblas`	`3.9.0-16`	`3.9.0-16`
`blis`	`0.9.0-0`	`0.9.0-0`
`openblas`	`0.3.21-pthreads-3`	`0.3.21-pthreads-3`
`mkl`	`2022.1.0-915`	`2022.1.0-915`
`netlib`	`3.9.0-5`	`3.9.0-5`
`pypy`	not used	not used
`pythran`	`0.12.1-0`	`0.12.1-0`
`qemu-user-static`	`7.2.0-1`	`7.2.0-1`

variant	1.9.3	1.10.0
linux + x64 + mkl + avx512	1 test failure; https://github.com/scipy/scipy/issues/15533	remains
linux + x64 + openblas + non-avx512	1 test failure https://github.com/scipy/scipy/issues/16927	remains
linux + aarch64 + openblas	1 test failure	passes 🥳
linux + ppc64le	test suite disabled due to emulation problems	looks like this might finally get solved 🥳
osx + x64 + accelerate	7 test failures	6 failures
osx + x64 + blis	1 test failure https://github.com/scipy/scipy/issues/16927	remains
win + mkl + avx512	1 test failure; https://github.com/scipy/scipy/issues/15533	remains
win + openblas	7 test failures https://github.com/scipy/scipy/issues/17869	same failures remain

variant	accelerate	blis	mkl w/ avx512	mkl w/o avx512	netlib	openblas w/ avx512	openblas w/o avx512	sum**
linux / x64	➖	✔️	❌	✔️	✔️	✔️	❌	8F
linux / aarch64	➖	➖	➖	➖	✔️	➖	✔️	-
linux / ppc64le	➖	➖	➖	➖	❔	➖	❔	-
osx / arm	❔	➖	➖	➖	❔	➖	❔	-
osx / x64	❌	❌	➖	✔️	✔️	➖	✔️	8F
win / x64	➖	✔️	❌	✔️	✔️	❌	❌	12F
sum**	4F	4F	8F	-	-	4F	8F	28F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 96 CI combinations being tested

Build logs: Azure

linux + x64 + mkl + avx512: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: Items are not equal: ACTUAL: -11 DESIRED: 0 = 1 failed, 50141 passed, 2295 skipped, 255 xfailed, 9 xpassed, 268 warnings in 960.42s (0:16:00) = ```

linux + x64 + openblas + non-avx512: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (,) were emitted. The list of emitted warnings is: []. = 1 failed, 50140 passed, 2296 skipped, 253 xfailed, 11 xpassed, 270 warnings in 1133.54s (0:18:53) = ```

osx + x64 + accelerate: 6 failures

``` =========================== short test summary info ============================ FAILED interpolate/tests/test_rbfinterp.py::TestRBFInterpolatorNeighbors20::test_pickleable - AssertionError: Arrays are not equal Mismatched elements: 1 / 50 (2%) Max absolute difference: 5.55111512e-17 Max relative difference: 2.02452314e-16 x: array([-0.238461, 0.257024, -0.811244, 0.295313, 0.132623, 0.2038 , -0.99615 , 0.29755 , -0.019889, 0.230791, -0.516801, 0.279867, 0.227767, 0.178071, -0.172756, 0.251327, -0.406818, 0.271919,... y: array([-0.238461, 0.257024, -0.811244, 0.295313, 0.132623, 0.2038 , -0.99615 , 0.29755 , -0.019889, 0.230791, -0.516801, 0.279867, 0.227767, 0.178071, -0.172756, 0.251327, -0.406818, 0.271919,... FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: Arrays are not equal item=0 Mismatched elements: 86 / 100 (86%) Max absolute difference: 2.22044605e-16 Max relative difference: 2.02120457e-15 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: Arrays are not equal item=0 Mismatched elements: 25 / 100 (25%) Max absolute difference: 5.55111512e-17 Max relative difference: 5.62192122e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: Arrays are not equal item=0 Mismatched elements: 38 / 100 (38%) Max absolute difference: 5.55111512e-17 Max relative difference: 5.62192122e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: Arrays are not equal item=0 Mismatched elements: 15 / 100 (15%) Max absolute difference: 2.77555756e-17 Max relative difference: 2.84357857e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... = 6 failed, 50122 passed, 2307 skipped, 255 xfailed, 11 xpassed, 269 warnings in 908.74s (0:15:08) = ```

osx + x64 + blis: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (,) were emitted. The list of emitted warnings is: []. = 1 failed, 50127 passed, 2307 skipped, 255 xfailed, 11 xpassed, 268 warnings in 907.17s (0:15:07) = ```

windows + mkl + avx512: 1 failure

``` =========================== short test summary info =========================== FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] = 1 failed, 50112 passed, 2324 skipped, 255 xfailed, 9 xpassed, 269 warnings in 1256.40s (0:20:56) = ```

windows + openblas: 7 failures

``` =========================== short test summary info =========================== FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE... FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128] = 7 failed, 50105 passed, 2325 skipped, 255 xfailed, 9 xpassed, 271 warnings in 1028.16s (0:17:08) = ```

h-vetinari commented 1 year ago

Paused the runs here until we figure out #232.

h-vetinari commented 1 year ago

Scipy 1.10.1

TL;DR: Now with PyPy & PPC, import error on linux+netlib, new failures

From 28 failures out of 96 runs for 1.10.0, we're now at 66 failures out of 138 CI runs

Notable

PyPy is back (see #208)
No Propack tests (though looking good in #200)
PPC testing enabled (see #225) 🥳
Enforcing presence/absence of avx512 is a gamble based on which agent gets assigned; not all CI failures are actual failures
Completely orthogonal failures on osx+MKL depending on CPython or PyPy!

The good news:

[as for 1.10.0] for osx + accelerate, 6/7 failures are very minor tolerance violations

The bad news

An import error due to a missing symbol for netlib on all linux arches; this has a big blas(t) radius because it causes an import error!
[as for 1.10.0] persistent errors on win+openblas, unix+x64 and mkl+avx512

Restrictions

test on aarch64/ppc64le resp. for PyPy only run tests with label='fast', not the full test suite

the following test skips:

# skip a test that fails with MKL + AVX512 (non-AVX512 passes), scipy/scipy#15533
{% set tests_to_skip = tests_to_skip + " or test_x0_equals_Mb[bicgstab]" %}
# scipy/scipy#16927
{% set tests_to_skip = tests_to_skip + " or test_failure_to_run_iterations" %}  # [linux]
# on the slowest agents, these tests take more than 20min in emulation
{% set tests_to_skip = tests_to_skip + " or (test_lsq_linear and test_large_rank_deficient)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_iterative and test_precond_inverse[case1])" %}  # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_optimize and TestBrute and test_workers)" %}    # [aarch64 or ppc64le]
# tests that run into timeouts (set in scipy test suite) in emulation
{% set tests_to_skip = tests_to_skip + " or (test_propack and test_examples)" %}                  # [aarch64 or ppc64le]
# occasionally runs out of memory on windows + pypy
{% set tests_to_skip = tests_to_skip + " or test_bootstrap_against_" %}                           # [(python_impl == "pypy") and win]
# very slow tests for pypy in emulation
[several]

Details

lib	for 1.10.0	now	updated version
`scipy`	`1.10.0`	`1.10.1`	X
`numpy`	`1.24.1`	`1.24.3`	X
`libblas`	`3.9.0-16`	`3.9.0-16`
`blis`	`0.9.0-0`	`0.9.0-0`
`openblas`	`0.3.21-pthreads-3`	`0.3.21-pthreads-3`
`mkl`	`2022.1.0-915`	`2022.2.1-16997` (linux) `2022.2.1-16952` (osx) `2022.1.0-874` (win)	X
`netlib`	`3.9.0-5`	`3.9.0-5`
`pypy`	`7.3.9-7` (for 1.8.1)	`7.3.11-0`	X
`pythran`	`0.12.1-0`	`0.12.1-0`
`qemu-user-static`	`7.2.0-1`	`7.2.0-1`

variant	1.10.0	now
linux + x64 + mkl + avx512	1 test failure; https://github.com/scipy/scipy/issues/15533	remains
linux + x64 + openblas + non-avx512	1 test failure https://github.com/scipy/scipy/issues/16927	remains
linux + netlib	-	missing symbol `cblas_cdotc_sub`; causes ImportError! ☢️
linux + ppc64le	test suite disabled due to emulation problems	runs through on openblas🥳 missing symbol on netlib
osx + x64 + accelerate	6 test failures	7 test failures
osx + x64 + blis	1 test failure https://github.com/scipy/scipy/issues/16927	remains
osx + x64 + mkl + CPython	-	13-14 test failures
osx + x64 + mkl + PyPy	-	2-3 test failures
win + mkl + avx512	1 test failure; https://github.com/scipy/scipy/issues/15533	remains
win + openblas	7 test failures https://github.com/scipy/scipy/issues/17869	same failures remain

variant	accelerate	blis	mkl w/ avx512	mkl w/o avx512	netlib	openblas w/ avx512	openblas w/o avx512	sum**
linux / x64	➖	✔️	❌	✔️	❌	✔️	❌	18F
linux / aarch64	➖	➖	➖	➖	❌	➖	✔️	6F
linux / ppc64le	➖	➖	➖	➖	❌	➖	✔️	6F
osx / arm	❔	➖	➖	➖	❔	➖	❔	-
osx / x64	❌	❌	➖	❌	✔️	➖	✔️	18F
win / x64	➖	✔️	❌	✔️	✔️	❌	❌	18F
sum**	6F	6F	12F	6F	18F	6F	12F	66F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 138 CI combinations being tested

Build logs: Azure

(linux or win) + x64 + mkl + avx512: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: Items are not equal: ACTUAL: -11 DESIRED: 0 = 1 failed, 50177 passed, 2294 skipped, 252 xfailed, 10 xpassed, 268 warnings in 758.88s (0:12:38) = ```

linux + netlib: ImportError! ☢️

``` import: 'scipy.cluster' Traceback (most recent call last): [...] from . import _iterative ImportError: $PREFIX/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-310-x86_64-linux-gnu.so: undefined symbol: cblas_cdotc_sub ```

linux + x64 + openblas + non-avx512: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (,) were emitted. The list of emitted warnings is: []. = 1 failed, 50177 passed, 2294 skipped, 250 xfailed, 12 xpassed, 268 warnings in 1219.20s (0:20:19) = ```

osx + x64 + accelerate: 7 failures

``` =========================== short test summary info ============================ FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: Arrays are not equal item=0 Mismatched elements: 5 / 100 (5%) Max absolute difference: 2.77555756e-17 Max relative difference: 3.0222348e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: Arrays are not equal item=0 Mismatched elements: 25 / 100 (25%) Max absolute difference: 4.16333634e-17 Max relative difference: 4.25664195e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: Arrays are not equal item=0 Mismatched elements: 70 / 100 (70%) Max absolute difference: 5.55111512e-17 Max relative difference: 5.8956122e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: Arrays are not equal item=0 Mismatched elements: 49 / 100 (49%) Max absolute difference: 4.16333634e-17 Max relative difference: 3.80861356e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state2] - AssertionError: Arrays are not equal item=0 Mismatched elements: 32 / 100 (32%) Max absolute difference: 5.55111512e-17 Max relative difference: 5.24318491e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state3] - AssertionError: Arrays are not equal item=0 Mismatched elements: 28 / 100 (28%) Max absolute difference: 4.16333634e-17 Max relative difference: 4.38230693e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... = 7 failed, 50158 passed, 2305 skipped, 252 xfailed, 12 xpassed, 265 warnings in 1150.50s (0:19:10) = ```

osx + x64 + blis: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (,) were emitted. The list of emitted warnings is: []. = 1 failed, 38011 passed, 2277 skipped, 139 xfailed, 11 xpassed, 67 warnings in 2058.33s (0:34:18) = ```

osx + x64 + mkl + CPython: 13-14 failures

``` =========================== short test summary info ============================ FAILED sparse/linalg/tests/test_propack.py::test_examples[False-complex64] FAILED sparse/linalg/tests/test_propack.py::test_examples[True-complex64] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-csc_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex64-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex64-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex128-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex128-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-csc_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex128-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex64-array] = 14 failed, 50135 passed, 2304 skipped, 253 xfailed, 11 xpassed, 265 warnings in 1244.19s (0:20:44) = ```

osx + x64 + mkl + PyPy: 2-3 failures

``` =========================== short test summary info ============================ FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-float32] - AssertionError: Not equal to tolerance rtol=0, atol=1.19209e-06 Mismatched elements: 132 / 1600 (8.25%) Max absolute difference: 1.2262092 Max relative difference: 2.0007246 x: array([[ 0.088203, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223086, ..., 0. , 0. ,... y: array([[ 0.088203, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048776, 0.223085, ..., 0. , 0. ,... FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: Not equal to tolerance rtol=0, atol=1.19209e-06 Mismatched elements: 228 / 1600 (14.2%) Max absolute difference: 1.2262108 Max relative difference: 2.0003054 x: array([[ 0.088202, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223086, ..., 0. , 0. ,... y: array([[ 0.088203, -0.328942, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223085, ..., 0. , 0. ,... FAILED linalg/tests/test_decomp_update.py::TestQRinsert_F::test_sqr_p_row = 3 failed, 38009 passed, 2277 skipped, 140 xfailed, 10 xpassed, 67 warnings in 2631.58s (0:43:51) = ```

windows + openblas: 7 failures

``` =========================== short test summary info =========================== FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE... FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128] = 7 failed, 50138 passed, 2327 skipped, 252 xfailed, 10 xpassed, 273 warnings in 1743.42s (0:29:03) = ```

h-vetinari commented 1 year ago

Ah well, turns out my rebase wasn't that great after all:

../scipy/sparse/linalg/_propack/meson.build:93:12: ERROR: Unknown variable "cargs_propack".

h-vetinari commented 1 year ago

Seems something about those patches isn't distutils- and/or windows-compatible; but we'll just exclude them on windows for main then...

%SRC_DIR%\scipy\_build_utils\src\npy_cblas.h(9): fatal error C1083: Cannot open include file: 'numpy/npy_common.h': No such file or directory

h-vetinari commented 1 year ago

Scipy 1.10.1 (distutils)

This already ran a few days ago to verify a suspicion that https://github.com/scipy/scipy/issues/18371 had something to do with the switch to meson. While looking at the results of testing PROPRACK in https://github.com/conda-forge/scipy-feedstock/pull/200, I noticed some familiar failures and ended up writing this up because it's relevant.

TL;DR: Switch to meson broke a bunch of things

From 66 failures out of 138 runs for 1.10.1 built with meson (on unix), we're now at 44 failures out of 138 CI runs

Notable

I had overlooked in the last report that the errors on OSX + MKL had completely orthogonal failures depending on CPython or PyPy -- updated there.
More importantly, the CPython portion of those failures is also related to the switch to meson, and not yet fixed by https://github.com/scipy/scipy/pull/18426 and the backport work from #233 (which is not visible from the CI run here yet, but the one in #200)

Restrictions & Library Versions

as for 1.10.1

variant	1.10.1	now
linux + x64 + mkl + avx512	1 test failure; https://github.com/scipy/scipy/issues/15533	remains
linux + x64 + openblas + non-avx512	1 test failure https://github.com/scipy/scipy/issues/16927	remains
linux + netlib	missing symbol `cblas_cdotc_sub`; causes ImportError! ☢️	gone! 🥳
osx + x64 + accelerate	7 test failures	remain
osx + x64 + blis	1 test failure https://github.com/scipy/scipy/issues/16927	remains
osx + x64 + mkl + CPython	13-14 test failures	gone! 🥳
osx + x64 + mkl + PyPy	2-3 test failures	remain
win + mkl + avx512	1 test failure; https://github.com/scipy/scipy/issues/15533	remains
win + openblas	7 test failures https://github.com/scipy/scipy/issues/17869	remain

variant	accelerate	blis	mkl w/ avx512	mkl w/o avx512	netlib	openblas w/ avx512	openblas w/o avx512	sum**
linux / x64	➖	✔️	❌	✔️	✔️	✔️	❌	12F
linux / aarch64	➖	➖	➖	➖	✔️	➖	✔️	-
linux / ppc64le	➖	➖	➖	➖	✔️	➖	✔️	-
osx / arm	❔	➖	➖	➖	❔	➖	❔	-
osx / x64	❌	❌	➖	✔️ (CPython) ❌ (PyPy)	✔️	➖	✔️	14F
win / x64	➖	✔️	❌	✔️	✔️	❌	❌	18F
sum**	6F	6F	12F	2F	-	6F	12F	44F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 138 CI combinations being tested

Build logs: Azure

(linux or win) + x64 + mkl + avx512: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: Items are not equal: ACTUAL: -11 DESIRED: 0 = 1 failed, 50177 passed, 2294 skipped, 252 xfailed, 10 xpassed, 268 warnings in 758.88s (0:12:38) = ```

linux + x64 + openblas + non-avx512: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (,) were emitted. The list of emitted warnings is: []. = 1 failed, 50177 passed, 2294 skipped, 250 xfailed, 12 xpassed, 268 warnings in 1219.20s (0:20:19) = ```

osx + x64 + accelerate: 7 failures

``` =========================== short test summary info ============================ FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: Arrays are not equal item=0 Mismatched elements: 5 / 100 (5%) Max absolute difference: 2.77555756e-17 Max relative difference: 3.0222348e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: Arrays are not equal item=0 Mismatched elements: 25 / 100 (25%) Max absolute difference: 4.16333634e-17 Max relative difference: 4.25664195e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: Arrays are not equal item=0 Mismatched elements: 70 / 100 (70%) Max absolute difference: 5.55111512e-17 Max relative difference: 5.8956122e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: Arrays are not equal item=0 Mismatched elements: 49 / 100 (49%) Max absolute difference: 4.16333634e-17 Max relative difference: 3.80861356e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state2] - AssertionError: Arrays are not equal item=0 Mismatched elements: 32 / 100 (32%) Max absolute difference: 5.55111512e-17 Max relative difference: 5.24318491e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state3] - AssertionError: Arrays are not equal item=0 Mismatched elements: 28 / 100 (28%) Max absolute difference: 4.16333634e-17 Max relative difference: 4.38230693e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... = 7 failed, 50158 passed, 2305 skipped, 252 xfailed, 12 xpassed, 265 warnings in 1150.50s (0:19:10) = ```

osx + x64 + blis: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (,) were emitted. The list of emitted warnings is: []. = 1 failed, 38011 passed, 2277 skipped, 139 xfailed, 11 xpassed, 67 warnings in 2058.33s (0:34:18) = ```

osx + x64 + mkl + pypy: 2-3 failures

``` =========================== short test summary info ============================ FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-float32] - AssertionError: Not equal to tolerance rtol=0, atol=1.19209e-06 Mismatched elements: 132 / 1600 (8.25%) Max absolute difference: 1.2262092 Max relative difference: 2.0007246 x: array([[ 0.088203, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223086, ..., 0. , 0. ,... y: array([[ 0.088203, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048776, 0.223085, ..., 0. , 0. ,... FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: Not equal to tolerance rtol=0, atol=1.19209e-06 Mismatched elements: 228 / 1600 (14.2%) Max absolute difference: 1.2262108 Max relative difference: 2.0003054 x: array([[ 0.088202, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223086, ..., 0. , 0. ,... y: array([[ 0.088203, -0.328942, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223085, ..., 0. , 0. ,... FAILED linalg/tests/test_decomp_update.py::TestQRinsert_F::test_sqr_p_row = 3 failed, 38009 passed, 2277 skipped, 140 xfailed, 10 xpassed, 67 warnings in 2631.58s (0:43:51) = ```

windows + openblas: 7 failures

``` =========================== short test summary info =========================== FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE... FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128] = 7 failed, 50138 passed, 2327 skipped, 252 xfailed, 10 xpassed, 273 warnings in 1743.42s (0:29:03) = ```

h-vetinari commented 1 year ago

Scipy 1.10.1 (meson, with patches)

Now again with meson after the backported patches from #233

TL;DR: OSX+MKL still broken on meson

From 66 failures out of 138 runs for 1.10.1 built with meson (on unix) and 44 failures when built with distutils, we're now at 48 failures out of 138 CI runs

Notable

OSX+MKL has the same errors as before the patching, but at least the failures remain stable across CPython python versions (in contrast to what happens when enabling PROPACK, see https://github.com/conda-forge/scipy-feedstock/pull/200; a fact that baffles me to no end)

Restrictions & Library Versions

as for 1.10.1

variant	1.10.1	now
linux + x64 + mkl + avx512	1 test failure; https://github.com/scipy/scipy/issues/15533	remains
linux + x64 + openblas + non-avx512	1 test failure https://github.com/scipy/scipy/issues/16927	remains
linux + netlib	missing symbol `cblas_cdotc_sub`; causes ImportError! ☢️	gone! 🥳
osx + x64 + accelerate	7 test failures	remain
osx + x64 + blis	1 test failure https://github.com/scipy/scipy/issues/16927	remains
osx + x64 + mkl + CPython	13-14 test failures	remains in contrast to distutils build
osx + x64 + mkl + PyPy	2-3 test failures	remain
win + mkl + avx512	1 test failure; https://github.com/scipy/scipy/issues/15533	remains
win + openblas	7 test failures https://github.com/scipy/scipy/issues/17869	remain

variant	accelerate	blis	mkl w/ avx512	mkl w/o avx512	netlib	openblas w/ avx512	openblas w/o avx512	sum**
linux / x64	➖	✔️	❌	✔️	✔️	✔️	❌	12F
linux / aarch64	➖	➖	➖	➖	✔️	➖	✔️	-
linux / ppc64le	➖	➖	➖	➖	✔️	➖	✔️	-
osx / arm	❔	➖	➖	➖	❔	➖	❔	-
osx / x64	❌	❌	➖	❌	✔️	➖	✔️	18F
win / x64	➖	✔️	❌	✔️	✔️	❌	❌	18F
sum**	6F	6F	12F	6F	-	6F	12F	48F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 138 CI combinations being tested

Build logs: Azure

(linux or win) + x64 + mkl + avx512: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: Items are not equal: ACTUAL: -11 DESIRED: 0 = 1 failed, 50177 passed, 2294 skipped, 252 xfailed, 10 xpassed, 268 warnings in 758.88s (0:12:38) = ```

linux + x64 + openblas + non-avx512: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (,) were emitted. The list of emitted warnings is: []. = 1 failed, 50177 passed, 2294 skipped, 250 xfailed, 12 xpassed, 268 warnings in 1219.20s (0:20:19) = ```

osx + x64 + accelerate: 7 failures

``` =========================== short test summary info ============================ FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: Arrays are not equal item=0 Mismatched elements: 5 / 100 (5%) Max absolute difference: 2.77555756e-17 Max relative difference: 3.0222348e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: Arrays are not equal item=0 Mismatched elements: 25 / 100 (25%) Max absolute difference: 4.16333634e-17 Max relative difference: 4.25664195e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: Arrays are not equal item=0 Mismatched elements: 70 / 100 (70%) Max absolute difference: 5.55111512e-17 Max relative difference: 5.8956122e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: Arrays are not equal item=0 Mismatched elements: 49 / 100 (49%) Max absolute difference: 4.16333634e-17 Max relative difference: 3.80861356e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state2] - AssertionError: Arrays are not equal item=0 Mismatched elements: 32 / 100 (32%) Max absolute difference: 5.55111512e-17 Max relative difference: 5.24318491e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state3] - AssertionError: Arrays are not equal item=0 Mismatched elements: 28 / 100 (28%) Max absolute difference: 4.16333634e-17 Max relative difference: 4.38230693e-16 x: array([[-0.109858], [-0.106207], [-0.108685],... y: array([[-0.109858], [-0.106207], [-0.108685],... = 7 failed, 50158 passed, 2305 skipped, 252 xfailed, 12 xpassed, 265 warnings in 1150.50s (0:19:10) = ```

osx + x64 + blis: 1 failure

``` =========================== short test summary info ============================ FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (,) were emitted. The list of emitted warnings is: []. = 1 failed, 38011 passed, 2277 skipped, 139 xfailed, 11 xpassed, 67 warnings in 2058.33s (0:34:18) = ```

osx + x64 + mkl + CPython: 13-14 failures

``` =========================== short test summary info ============================ FAILED sparse/linalg/tests/test_propack.py::test_examples[False-complex128] FAILED sparse/linalg/tests/test_propack.py::test_examples[False-complex64] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex64-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex64-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex128-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex128-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-csc_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-csr_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex64-array] FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex64-csc_matrix] FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex64-csr_matrix] = 14 failed, 50142 passed, 2304 skipped, 253 xfailed, 11 xpassed, 265 warnings in 1276.44s (0:21:16) = ```

osx + x64 + mkl + pypy: 2-3 failures

``` =========================== short test summary info ============================ FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-float32] - AssertionError: Not equal to tolerance rtol=0, atol=1.19209e-06 Mismatched elements: 132 / 1600 (8.25%) Max absolute difference: 1.2262092 Max relative difference: 2.0007246 x: array([[ 0.088203, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223086, ..., 0. , 0. ,... y: array([[ 0.088203, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048776, 0.223085, ..., 0. , 0. ,... FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: Not equal to tolerance rtol=0, atol=1.19209e-06 Mismatched elements: 228 / 1600 (14.2%) Max absolute difference: 1.2262108 Max relative difference: 2.0003054 x: array([[ 0.088202, -0.328941, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223086, ..., 0. , 0. ,... y: array([[ 0.088203, -0.328942, -0.031064, ..., 0. , 0. , 0. ], [ 0.233614, 0.048775, 0.223085, ..., 0. , 0. ,... FAILED linalg/tests/test_decomp_update.py::TestQRinsert_F::test_sqr_p_row = 3 failed, 38009 passed, 2277 skipped, 140 xfailed, 10 xpassed, 67 warnings in 2631.58s (0:43:51) = ```

windows + openblas: 7 failures

``` =========================== short test summary info =========================== FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE... FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64] FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128] = 7 failed, 50138 passed, 2327 skipped, 252 xfailed, 10 xpassed, 273 warnings in 1743.42s (0:29:03) = ```

rgommers commented 1 year ago

All of the entries with a single failure are from the same test, test_failure_to_run_iterations, which is already fixed in SciPy's main branch.

The precision issues in the macOS/Accelerate tests should be fixed by https://github.com/scipy/scipy/pull/18442. Then there is one left:

FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args

that is a crash, probably a faulty G77 ABI wrapper. If that happens again after the current round of -use-g77-abi fixes, I'll investigate more.

h-vetinari commented 7 months ago

Closing this as obsolete

conda-forge / scipy-feedstock