Unable to pip install scipy

[laserjim]

According to https://www.graalvm.org/python/compatibility/ indicates that I should be able to install scipy==1.10.1.

However, I've been trying with the latest version (24.0.2) of graalpy using:

graalpy -m venv .venv
source .venv/bin/activate
graalpy -m pip install scipy==1.10.1

I consistently get errors like this:

      [270/1524] Generating 'scipy/special/_ufuncs.graalpy240-310-native-x86_64-linux.so.p/_ufuncs.c'
      FAILED: scipy/special/_ufuncs.graalpy240-310-native-x86_64-linux.so.p/_ufuncs.c
      /tmp/pip-build-env-km3wo_vc/overlay/bin/cython -3 --fast-fail --output-file scipy/special/_ufuncs.graalpy240-310-native-x86_64-linux.so.p/_ufuncs.c --include-dir . scipy/special/_ufuncs.pyx
      WARNING: Overriding pythran description with argspec information for: numpy.random.bytes
      Killed
      [271/1524] Generating 'scipy/special/_comb.graalpy240-310-native-x86_64-linux.so.p/_comb.c'
      WARNING: Overriding pythran description with argspec information for: numpy.random.bytes
      [272/1524] Generating 'scipy/special/_test_internal.graalpy240-310-native-x86_64-linux.so.p/_test_internal.c'
      WARNING: Overriding pythran description with argspec information for: numpy.random.bytes
      [273/1524] Generating 'scipy/special/cython_special.graalpy240-310-native-x86_64-linux.so.p/cython_special.c'
      WARNING: Overriding pythran description with argspec information for: numpy.random.bytes
      ninja: build stopped: subcommand failed.
      [end of output]

I've tried on multiple versions of ubuntu, fedora, centos, etc. Nothing works. Has anyone gotten this package to install? What platform/configuration? Any tricks?

[msimacek]

The problem is that that version of scipy doesn't build against the current version of numpy that released recently. It should be fixed for the next release of graalpy. In the current one, you can work around it if you modify lib/graalpy24.0/patches/scipy/scipy-1.10.1.patch file and change line 167 from + "numpy", to + "numpy==1.23.5",.

[laserjim]

I updated the patch file, created a new venv, and ran pip install again. It looks like it got a little further, but then still crashed.

This is the new error:

[295/1524] Compiling Fortran object scipy/linalg/_interpolative.graalpy240-310-native-x86_64-linux.so.p/src_id_dist_src_dfft.f.o
      FAILED: scipy/linalg/_interpolative.graalpy240-310-native-x86_64-linux.so.p/src_id_dist_src_dfft.f.o
      gfortran -Iscipy/linalg/_interpolative.graalpy240-310-native-x86_64-linux.so.p -Iscipy/linalg -I../scipy/linalg -I/tmp/pip-build-env-6oqwfq6w/overlay/lib/python3.10/site-packages/numpy/core/include -I/tmp/pip-build-env-6oqwfq6w/overlay/lib/python3.10/site-packages/numpy/f2py/src -Iscipy/lib_fortranobject.a.p -I/usr/include/x86_64-linux-gnu/openblas-pthread/ -I/root/graalpy-24.0.2-linux-amd64/include/python3.10 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -O2 -Wno-conversion -fPIC -Wno-argument-mismatch -Wno-conversion -Wno-maybe-uninitialized -Wno-unused-dummy-argument -Wno-unused-label -Wno-unused-variable -Wno-tabs -w -Jscipy/linalg/_interpolative.graalpy240-310-native-x86_64-linux.so.p -o scipy/linalg/_interpolative.graalpy240-310-native-x86_64-linux.so.p/src_id_dist_src_dfft.f.o -c ../scipy/linalg/src/id_dist/src/dfft.f
      ../scipy/linalg/src/id_dist/src/dfft.f:2902:72:

       2902 |       CALL DFFTI1 (N,WSAVE(N+1),WSAVE(2*N+1))
            |                                                                        1
      Error: Type mismatch in argument ‘ifac’ at (1); passed REAL(8) to INTEGER(4)
      ../scipy/linalg/src/id_dist/src/dfft.f:2835:72:

       2835 |       CALL DFFTF1 (N,R,WSAVE,WSAVE(N+1),WSAVE(2*N+1))
            |                                                                        1
      Error: Type mismatch in argument ‘ifac’ at (1); passed REAL(8) to INTEGER(4)
      ../scipy/linalg/src/id_dist/src/dfft.f:2765:72:

       2765 |       CALL DFFTB1 (N,R,WSAVE,WSAVE(N+1),WSAVE(2*N+1))
            |                                                                        1
      Error: Type mismatch in argument ‘ifac’ at (1); passed REAL(8) to INTEGER(4)
      ../scipy/linalg/src/id_dist/src/dfft.f:939:72:

        939 |       CALL ZFFTB1 (N,C,WSAVE,WSAVE(IW1),WSAVE(IW2))
            |                                                                        1
      Error: Type mismatch in argument ‘ifac’ at (1); passed REAL(8) to INTEGER(4)
      ../scipy/linalg/src/id_dist/src/dfft.f:1013:72:

       1013 |       CALL ZFFTF1 (N,C,WSAVE,WSAVE(IW1),WSAVE(IW2))
            |                                                                        1
      Error: Type mismatch in argument ‘ifac’ at (1); passed REAL(8) to INTEGER(4)
      ../scipy/linalg/src/id_dist/src/dfft.f:1086:72:

       1086 |       CALL ZFFTI1 (N,WSAVE(IW1),WSAVE(IW2))
            |                                                                        1
      Error: Type mismatch in argument ‘ifac’ at (1); passed REAL(8) to INTEGER(4)
      ../scipy/linalg/src/id_dist/src/dfft.f:1365:72:

       1365 |       CALL DZFFT1 (N,WSAVE(2*N+1),WSAVE(3*N+1))
            |                                                                        1
      Error: Type mismatch in argument ‘ifac’ at (1); passed REAL(8) to INTEGER(4)
      ../scipy/linalg/src/id_dist/src/dfft.f:2997:72:

       2997 |       CALL DSINT1(N,X,WSAVE,WSAVE(IW1),WSAVE(IW2),WSAVE(IW3))
            |                                                                        1
      Error: Type mismatch in argument ‘ifac’ at (1); passed REAL(8) to INTEGER(4)
      [296/1524] Generating 'scipy/special/cython_special.graalpy240-310-native-x86_64-linux.so.p/cython_special.c'
      WARNING: Overriding pythran description with argspec information for: numpy.random.bytes
      [297/1524] Generating scipy/linalg/flapack_module with a custom command
      [298/1524] Generating 'scipy/special/_ufuncs.graalpy240-310-native-x86_64-linux.so.p/_ufuncs.c'
      WARNING: Overriding pythran description with argspec information for: numpy.random.bytes
      ninja: build stopped: subcommand failed.

Is there another change I need to make somewhere?

[msimacek]

Ah, it also doesn't like being compiled with current gcc. Can you try an early access build of the next graalpy version from https://github.com/graalvm/graal-languages-ea-builds/releases/tag/graalpy-24.1.0-ea.23?

[msimacek]

Alternatively, you could try to replace the patch file with https://github.com/oracle/graalpython/blob/master/graalpython/lib-graalpython/patches/scipy/scipy-1.10.1.patch

[laserjim]

Ok, good news! I was able to get scipy working in a couple different configurations. It worked on 24.0.2 after I downgraded my gfortran to version 9.5.0 (plus the patch above). It also worked without any downgrades or file patching when using the EA release (graalpy-24.1.0-ea.23) when built for native.

However, I probably wouldn't consider this fully resolved, because when I tried on the jvm release (graalpy-jvm-24.1.0-ea.23), it seems to get stuck on the Preparing metadata (pyproject.toml) step as shown in the log below:

Collecting scipy==1.10.1
  Downloading scipy-1.10.1.tar.gz (42.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.4/42.4 MB 12.5 MB/s eta 0:00:00
auto-patching C API usages in /tmp/pip-install-195mo8dl/scipy_5db74f21342f4efb91af3729fc01de55/scipy/_lib/boost/boost/mpi/python/skeleton_and_content.hpp
auto-patching C API usages in /tmp/pip-install-195mo8dl/scipy_5db74f21342f4efb91af3729fc01de55/scipy/_lib/boost/boost/mpi/python/serialize.hpp
auto-patching C API usages in /tmp/pip-install-195mo8dl/scipy_5db74f21342f4efb91af3729fc01de55/scipy/_lib/boost/boost/python/detail/wrap_python.hpp
auto-patching C API usages in /tmp/pip-install-195mo8dl/scipy_5db74f21342f4efb91af3729fc01de55/scipy/_lib/boost/boost/python/suite/indexing/detail/indexing_suite_detail.hpp
Looking for GraalPy patches for scipy
Patching package scipy using /root/graalpy-24.1.0-dev-linux-amd64/lib/graalpy24.1/patches/scipy/scipy-1.10.1.patch
patching file meson.build
patching file mesonpy_wrapper.py
patching file pyproject.toml
patching file scipy/_lib/_ccallback_c.pyx
patching file scipy/_lib/setup.py
patching file scipy/_lib/src/messagestream_config.h.in
patching file scipy/sparse/linalg/meson.build
patching file scipy/special/meson.build
patching file tools/cythonize.py
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... -

[msimacek]

The pip output is a bit misleading, it prints preparing metadata, but it's actually doing the build. The JVM release can be very slow in this case, because scipy build spawns many subproceses and the JVM version is slow at starting up/warming up. Can you please just try to give it more time?

[laserjim]

Just ran it overnight, and indeed you are right, it did finish successfully. Thank you!

[laserjim]

As a side note, it might be worth having more verbose logging by default, just to let the user see that forward progress is indeed still being made.