graalpython - m ginstall with custom index-url

[limanhei]

My dev environment is behind firewall and no proxy can be configured. Would like to see if ginstall can work for custom index-url which pull binaries from custom repository. I am using pip now but hitting issue when pip install scipy.

[msimacek]

Hi @limanhei, there are two environment variables that control the index URL, here they are with their defaults for PyPI:

GINSTALL_PACKAGE_PATTERN="https://pypi.org/pypi/%s/json"
GINSTALL_PACKAGE_VERSION_PATTERN="https://pypi.org/pypi/%s/%s/json"

So if you want to use a custom repository, replace pypi.org with your index and set both of them.

[limanhei]

when I tried to install, it shows "Package not found: 'scipy'", in my repo I only see wheel file and tar ball, there is no json.

[msimacek]

We don't support installing from local directories in ginstall, just real repostitory servers. What's the issue when installing with pip? The difference between using pip and ginstall is mostly that they pick different versions of packages. I think you should be able to install it with pip install numpy==1.16.4 scipy==1.3.1 (that's the versions that ginstall would use).

[paymok]

hi i work with @limanhei in the team and want to share some finding.

I have doubt is numpy 1.16.4 compatible with scipy 1.3.1 in non AIX platform. I checked in the scipy 1.3.1 requirements, in our case Ubuntu 20.04.4 LTS python 3.8, only numpy 1.14.5 is compatible with non AIX platform. Since the only numpy graalpython patch available is for 1.16.4, Does it mean there is no support for non AIX?

below is the pyproject.toml requirementi found in scipy 1.3.1 project.

    "numpy==1.13.3; python_version=='3.5' and platform_system!='AIX'",
    "numpy==1.13.3; python_version=='3.6' and platform_system!='AIX'",
    "numpy==1.14.5; python_version>='3.7' and platform_system!='AIX'",
    "numpy==1.16.0; python_version=='3.5' and platform_system=='AIX'",
    "numpy==1.16.0; python_version=='3.6' and platform_system=='AIX'",
    "numpy==1.16.0; python_version>='3.7' and platform_system=='AIX'", 

when I install scipy 1.3.1 with graalpython in ubuntu, pip ignored the numpy 1.16.4 I already installed in the venv, and go ahead installing numpy 1.14.5. evetually the pip install failed when compiling the numpy 1.14.5 due to no patch applied.

[msimacek]

We don't support AIX, I think that we just ignored the dependencies. You could make pip ignore them too with --no-dependencies.

I tried to dig more into ginstall and it seems you can make ginstall install from local files after all, the variables then would look like this:

export GINSTALL_PACKAGE_PATTERN=file:///path/to/your/repo/%s.zip
export GINSTALL_PACKAGE_VERSION_PATTERN=file:///path/to/your/repo/%s-%s.zip

You'll need the zip sources, not binary wheels

[paymok]

i sorted out the installation problem, although im hitting other exception when trying to use scipy.

It seems the problem is due to the scipy 1.3.1 is too old and not compatible with the latest gcc-11 and gfortran-11. From scipy issue page some suggested to use gfortran-9 that could solve the problem. And there are some extra packages required in order to make the scipy build success.

sudo apt install -y patch gcc-9 gfortran-9 curl automake g++ subversion python3-dev libopenblas-dev liblapack-dev 

Also i need to default gcc and gfortran to use v9 to keep the scipy compile happy.

#default to use gfortran 9 as default compiler, this setting is for install scipy 1.3.1
sudo ln -fs /usr/bin/gcc-9 /usr/bin/gcc
sudo ln -fs /usr/bin/gcc-ar-9 /usr/bin/g-ar
sudo ln -fs /usr/bin/gcc-nm-9 /usr/bin/g-nm
sudo ln -fs /usr/bin/gcc-ranlib-9 /usr/bin/g-ranlib
sudo ln -fs /usr/bin/gfortran-9 /usr/bin/gfortran

In the end i need to manually install scipy dependencies since i skip any requirement install from scipy. also need to add --no-build-isolation and --no-deps to avoid pip installing an older version of numpy which doesn't have patch from graalpython.

pip install wheel==0.34.0 Cython==0.29.13 numpy==1.16.4
pip install --no-build-isolation --no-deps scipy==1.3.1

Below is full list of command to setup the env

export GRAALPY_VERSION=22.2.0
export GRAALPY_WORKSPACE=/home/ubuntu/workspace/graealpython
export JAVA_HOME=$GRAALPY_WORKSPACE/tools/graalvm-ce-java11-$GRAALPY_VERSION
export PVENV_HOME=$GRAALPY_WORKSPACE/tools/pvenv
export PATH=$PVENV_HOME/bin:$JAVA_HOME/bin:$PATH
export PYTHON_EXECUTABLE=$PVENV_HOME/graalpython
export LLVM_TOOLCHAIN=$($JAVA_HOME/bin/lli --print-toolchain-path)

mkdir -p $GRAALPY_WORKSPACE/tools
sudo apt install -y patch gcc-9 gfortran-9 curl automake g++ subversion python3-dev libopenblas-dev liblapack-dev 
#force to use gfortran 9 as default compiler, this setting is for install scipy 1.3.1
sudo ln -fs /usr/bin/gcc-9 /usr/bin/gcc
sudo ln -fs /usr/bin/gcc-ar-9 /usr/bin/g-ar
sudo ln -fs /usr/bin/gcc-nm-9 /usr/bin/g-nm
sudo ln -fs /usr/bin/gcc-ranlib-9 /usr/bin/g-ranlib
sudo ln -fs /usr/bin/gfortran-9 /usr/bin/gfortran
sudo apt clean all
cd $GRAALPY_WORKSPACE
curl -SLO "https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-$GRAALPY_VERSION/graalvm-ce-java11-linux-amd64-$GRAALPY_VERSION.tar.gz"
curl -SLO "https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-$GRAALPY_VERSION/llvm-toolchain-installable-java11-linux-amd64-$GRAALPY_VERSION.jar"
curl -SLO "https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-$GRAALPY_VERSION/llvm-installable-svm-java11-linux-amd64-$GRAALPY_VERSION.jar"
curl -SLO "https://github.com/graalvm/graalpython/releases/download/vm-$GRAALPY_VERSION/python-installable-svm-java11-linux-amd64-$GRAALPY_VERSION.jar"
tar -xf "graalvm-ce-java11-linux-amd64-$GRAALPY_VERSION.tar.gz" -C $GRAALPY_WORKSPACE/tools
sudo chmod 777 $GRAALPY_WORKSPACE/tools/graalvm-ce-java11-$GRAALPY_VERSION/bin/gu
gu -L install "llvm-toolchain-installable-java11-linux-amd64-$GRAALPY_VERSION.jar"
gu -L install "llvm-installable-svm-java11-linux-amd64-$GRAALPY_VERSION.jar" 
gu -L install "python-installable-svm-java11-linux-amd64-$GRAALPY_VERSION.jar"
gu -L install "native-image-installable-svm-java11-linux-amd64-$GRAALPY_VERSION.jar"
graalpython -m venv tools/pvenv
cd $GRAALPY_WORKSPACE
source tools/pvenv/bin/activate
pip install wheel==0.34.0 Cython==0.29.13 numpy==1.16.4
pip install --no-build-isolation --no-deps scipy==1.3.1

About the scipy exception, after successfully install scipy through pip, i am getting LLVMFunction error. there is another open issue Scipy via ginstall fails due to fortran compiler errors seems having the same problem related to LLVM and fortran, im not familar in that space so i don't think i could contribute much on that front, but it would be better to state scipy is not supported in graalpython, or list out which graalpython version is compatible with scipy 1.3.1?

(pvenv) ubuntu@test-ubuntu:~/workspace/graealpython$ graalpython
Python 3.8.5 (Fri Apr 22 13:06:23 UTC 2022)
[Graal, GraalVM CE, Java 11.0.15] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy.special import exp10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/workspace/graealpython/tools/pvenv/lib/python3.8/site-packages/scipy/special/__init__.py", line 643, in <module '__init__.py'>
    from .basic import *
  File "/home/ubuntu/workspace/graealpython/tools/pvenv/lib/python3.8/site-packages/scipy/special/basic.py", line 18, in <module 'basic.py'>
    from . import specfun
ImportError: cannot load /home/ubuntu/workspace/graealpython/tools/pvenv/lib/python3.8/site-packages/scipy/special/specfun.graalpython-38-native-x86_64-linux.so: External LLVMFunction clqmn_ cannot be found.

[cosminbasca]

Hi @paymok !

Thanks for your interest in graalpy and in investigating how to get scipy to work with it. As you already observed, support for scipy is limited at this stage. We are working on providing support for installing a newer version of scipy, but there will still be missing native bits that are not yet supported or are simply missing from graalpy at the moment - we are working on filling those gaps! In particular support for fortran is somewhat limited at this stage and it may take a while for it to become comprehensive.

[msimacek]

In GraalPy 23.0, we no longer use LLVM for C extension execution by default, so there shouldn't be problems with compiling fortran anymore. We are still working on some missing parts and bugs, but in general, scipy should work. Please open a new issue if you run into specific problems.