NUMPY does not seem to work with JDK17 or JDK20

[manticore-projects]

Greetings.

Good news first: I got NUMPY installed with GraalPy JDK11.

However, it fails for JDK17 or JDK20 and I suspect the reason is the provided NUMPY version vs. the provided Python 3.10 environment.

Can please someone clarify the status and what we can expect? The Web Site and the releases don't give the impression that JDK17/20 are more experimental and provide less functionality than JDK11.

Problem is: NUMPY works with JDK11 and fails with all others. At the same time ScyPi fails with JDK 11:

/usr/bin/gfortran -Wall -g build/temp.linux-x86_64-3.8/build/src.linux-x86_64-3.8/scipy/fftpack/_fftpackmodule.o build/temp.linux-x86_64-3.8/scipy/fftpack/src/zfft.o build/temp.linux-x86_64-3.8/scipy/fftpack/src/drfft.o build/temp.linux-x86_64-3.8/scipy/fftpack/src/zrfft.o build/temp.linux-x86_64-3.8/scipy/fftpack/src/zfftnd.o build/temp.linux-x86_64-3.8/build/src.linux-x86_64-3.8/scipy/fftpack/src/dct.o build/temp.linux-x86_64-3.8/build/src.linux-x86_64-3.8/scipy/fftpack/src/dst.o build/temp.linux-x86_64-3.8/build/src.linux-x86_64-3.8/build/src.linux-x86_64-3.8/scipy/fftpack/fortranobject.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib -Lbuild/temp.linux-x86_64-3.8 -ldfftpack -lfftpack -lgfortran -o build/lib.linux-x86_64-3.8/scipy/fftpack/_fftpack.graalpy-38-native-x86_64-linux.so
ld.lld: error: undefined symbol: main
>>> referenced by /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../lib/Scrt1.o:(_start)

And I was under the impression that NUMPY and Pandas and SciPy would be supported already.

[msimacek]

There should be no difference between JDKs, certainly not in this area. Did you install the same GraalPy version with both? Numpy is supported and we're not aware of any build problems. It uses the build toolchain from your system, so problems like these are usually caused by version mismatches or configuration issues in the system toolchain. To determine if it's an issue with GraalPy or not, could you please first try if you can build the same version of numpy from source on CPython? Create a virtualenv with CPython 3.10 and run pip install --no-binary :all: numpy==1.23.5. If it succeeds on CPython, but fails on GraalPy, please post the error.

Since you opened a separate issue for scipy, let's focus just on numpy here.

[manticore-projects]

pip install --no-binary :all: numpy==1.23.5 Fails of JDK20/GraalPy23. Please see clean build log file attached.

Of course, native build works without problem.

numpy_build.txt

[manticore-projects]

Btw, Numpy 1.23.5 is not compatible to Python-3.10. Have you patched that version? Otherwise I would not understand how this was supposed to work.

[manticore-projects]

The particular error with latest Numpy 1.25.2:

are@ryzen /t/p/numpy_3e80e724154e4ca0aea7352415427f5c> gcc -DNDEBUG -D_GNU_SOURCE=1 -fPIC -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DHAVE_CBLAS -Ibuild/src.linux-x86_64-3.10/numpy/core/src/multiarray -Ibuild/src.linux-x86_64-3.10/numpy/core/src/common -Ibuild/src.linux-x86_64-3.10/numpy/core/src/umath -Inumpy/core/include -Ibuild/src.linux-x86_64-3.10/numpy/core/include/numpy -Ibuild/src.linux-x86_64-3.10/numpy/distutils/include -Ibuild/src.linux-x86_64-3.10/numpy/core/src/npysort -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/home/are/graalpy20/include -I/usr/lib/jvm/java-20-graalvm/languages/python/include/python3.10 -Ibuild/src.linux-x86_64-3.10/numpy/core/src/common -Ibuild/src.linux-x86_64-3.10/numpy/core/src/npymath -c numpy/core/src/multiarray/compiled_base.c -o build/temp.linux-x86_64-3.10/numpy/core/src/multiarray/compiled_base.o -MMD -MF build/temp.linux-x86_64-3.10/numpy/core/src/multiarray/compiled_base.o.d -msse -msse2 -msse3
In file included from numpy/core/src/common/npy_config.h:4,
                 from numpy/core/include/numpy/npy_common.h:10,
                 from numpy/core/include/numpy/ndarraytypes.h:4,
                 from numpy/core/include/numpy/ndarrayobject.h:12,
                 from numpy/core/include/numpy/arrayobject.h:5,
                 from numpy/core/src/multiarray/compiled_base.c:8:
build/src.linux-x86_64-3.10/numpy/core/include/numpy/config.h:3: warning: "SIZEOF_OFF_T" redefined
    3 | #define SIZEOF_OFF_T 8
      | 
In file included from /usr/lib/jvm/java-20-graalvm/languages/python/include/python3.10/Python.h:17,
                 from numpy/core/src/multiarray/compiled_base.c:5:
/usr/lib/jvm/java-20-graalvm/languages/python/include/python3.10/pyconfig.h:85: note: this is the location of the previous definition
   85 | #define SIZEOF_OFF_T __SIZEOF_SIZE_T__
      | 
numpy/core/src/multiarray/compiled_base.c: In function ‘arr_add_docstring’:
numpy/core/src/multiarray/compiled_base.c:1439:17: error: lvalue required as left operand of assignment
 1439 |             doc = docstr;                                               \
      |                 ^
numpy/core/src/multiarray/compiled_base.c:1449:9: note: in expansion of macro ‘_ADDDOC’
 1449 |         _ADDDOC(PyObject_GetDoc((PyObject*)(new)), new->m_ml->ml_name);
      |         ^~~~~~~

[manticore-projects]

The particular error with suggested Numpy 1.23.5:

are@ryzen /t/p/numpy_d2295e84552a4e4abad050b9964a5dc5> gcc -DNDEBUG -D_GNU_SOURCE=1 -fPIC -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Ibuild/src.linux-x86_64-3.10/numpy/core/src/multiarray -Ibuild/src.linux-x86_64-3.10/numpy/core/src/common -Ibuild/src.linux-x86_64-3.10/numpy/core/src/umath -Inumpy/core/include -Ibuild/src.linux-x86_64-3.10/numpy/core/include/numpy -Ibuild/src.linux-x86_64-3.10/numpy/distutils/include -Ibuild/src.linux-x86_64-3.10/numpy/core/src/npysort -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/home/are/graalpy20/include -I/usr/lib/jvm/java-20-graalvm/languages/python/include/python3.10 -Ibuild/src.linux-x86_64-3.10/numpy/core/src/common -Ibuild/src.linux-x86_64-3.10/numpy/core/src/npymath -c build/src.linux-x86_64-3.10/numpy/core/src/umath/loops.c -o build/temp.linux-x86_64-3.10/build/src.linux-x86_64-3.10/numpy/core/src/umath/loops.o -MMD -MF build/temp.linux-x86_64-3.10/build/src.linux-x86_64-3.10/numpy/core/src/umath/loops.o.d -msse -msse2 -msse3
In file included from numpy/core/src/common/npy_config.h:4,
                 from numpy/core/src/umath/loops.c.src:9:
build/src.linux-x86_64-3.10/numpy/core/include/numpy/config.h:3: warning: "SIZEOF_OFF_T" redefined
    3 | #define SIZEOF_OFF_T 8
      | 
In file included from /usr/lib/jvm/java-20-graalvm/languages/python/include/python3.10/Python.h:17,
                 from numpy/core/src/umath/loops.c.src:3:
/usr/lib/jvm/java-20-graalvm/languages/python/include/python3.10/pyconfig.h:85: note: this is the location of the previous definition
   85 | #define SIZEOF_OFF_T __SIZEOF_SIZE_T__
      | 
In file included from numpy/core/src/umath/loops.c.src:53:
numpy/core/src/umath/simd.inc.src: In function ‘AVX512F_conjugate_CFLOAT’:
numpy/core/src/umath/simd.inc.src:1119:13: warning: AVX512F vector return without AVX512F enabled changes the ABI [-Wpsabi]
 1119 |     @vindextype@ vindex = @vindexload@((@vindextype@*)index_ip1);
      |             ^~~~~~
numpy/core/src/umath/simd.inc.src: In function ‘AVX512F_conjugate_CDOUBLE’:
numpy/core/src/umath/simd.inc.src:1119:13: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
 1119 |     @vindextype@ vindex = @vindexload@((@vindextype@*)index_ip1);
      |             ^~~~~~
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/include/immintrin.h:53,
                 from numpy/core/src/umath/simd.inc.src:25:
/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/include/avx512fintrin.h: In function ‘AVX512F_square_CFLOAT’:
/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/include/avx512fintrin.h:314:1: error: inlining failed in call to ‘always_inline’ ‘_mm512_setzero_ps’: target specific option mismatch
  314 | _mm512_setzero_ps (void)
      | ^~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:1121:20: note: called from here
 1121 |     @vtype@ zeros = _mm512_setzero_@vsuffix@();
      |                    ^~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:740:1: error: inlining failed in call to ‘always_inline’ ‘avx512_get_full_load_mask_ps’: target specific option mismatch
  740 | avx512_get_full_load_mask_ps(void)
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:1120:27: note: called from here
 1120 |     @mask@ load_mask = avx512_get_full_load_mask_@vsuffix@();
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/include/avx512fintrin.h:6499:1: error: inlining failed in call to ‘always_inline’ ‘_mm512_loadu_si512’: target specific option mismatch
 6499 | _mm512_loadu_si512 (void const *__P)
      | ^~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:1119:22: note: called from here
 1119 |     @vindextype@ vindex = @vindexload@((@vindextype@*)index_ip1);
      |                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:752:1: error: inlining failed in call to ‘always_inline’ ‘avx512_get_partial_load_mask_ps’: target specific option mismatch
  752 | avx512_get_partial_load_mask_ps(const npy_int num_elem, const npy_int total_elem)
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:1125:25: note: called from here
 1125 |             load_mask = avx512_get_partial_load_mask_@vsuffix@(
      |                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1126 |                                     num_remaining_elements, @num_lanes@);
      |                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:782:1: error: inlining failed in call to ‘always_inline’ ‘avx512_masked_load_ps’: target specific option mismatch
  782 | avx512_masked_load_ps(__mmask16 mask, npy_float* addr)
      | ^~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:1130:18: note: called from here
 1130 |             x1 = avx512_masked_load_@vsuffix@(load_mask, ip);
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:764:1: error: inlining failed in call to ‘always_inline’ ‘avx512_masked_gather_ps’: target specific option mismatch
  764 | avx512_masked_gather_ps(__m512 src,
      | ^~~~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:1133:19: note: called from here
 1133 |             x1  = avx512_masked_gather_@vsuffix@(zeros, ip, vindex, load_mask);
      |                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/include/avx512fintrin.h:6373:1: error: inlining failed in call to ‘always_inline’ ‘_mm512_mask_storeu_ps’: target specific option mismatch
 6373 | _mm512_mask_storeu_ps (void *__P, __mmask16 __U, __m512 __A)
      | ^~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:1138:9: note: called from here
 1138 |         _mm512_mask_storeu_@vsuffix@(op, load_mask, out);
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:952:1: error: inlining failed in call to ‘always_inline’ ‘avx512_csquare_ps’: target specific option mismatch
  952 | avx512_csquare_@vsub@(@vtype@ x)
      | ^~~~~~~~~~~~~~~~~
numpy/core/src/umath/simd.inc.src:1136:22: note: called from here
 1136 |         @vtype@ out = @vectorf@_@vsuffix@(x1);
      |                      ^~~~~~~~~~~~~~~~~~~~~

[manticore-projects]

Hurray!

CC="gcc-9" CFLAGS='-march=native' CXXFLAGS='-march=native' pip install  --no-binary :all: numpy==1.23.5

Has finally worked. Maybe recommend this version on the Website. It's nowhere clear that your reference system is ancient (nothing wrong with that, just point it out somewhere please).

[msimacek]

Btw, Numpy 1.23.5 is not compatible to Python-3.10.

It is. The oldest currently supported numpy on 3.10 is 1.21.6 (source).

The particular error with latest Numpy 1.25.2

We patch numpy to fix small compatibility issues. If you install a newer version than what we have patches for, then it won't get patched and it fails like this. Not great, but expected.

Maybe recommend this version on the Website. It's nowhere clear that your reference system is ancient (nothing wrong with that, just point it out somewhere please).

Numpy builds fine on my laptop with gcc 12 and it had always built fine on all the past versions that I had throughout the years. I believe the same is true for everybody else in the team and we haven't heard of problems like this from anybody else in the community. I'm glad you figured it out, but the problem you had seems to be specific to your particular system, so it's not a reason for us to recommend people to use old compilers. The reference system is Oracle Linux 8 (equivalent to RHEL/Centos 8), there's nothing ancient about that.

Since you got it building, I'm going to close the issue, feel free to reopen if you have more problems.

[toroettg]

I recently struggled with the same issue (error: inlining failed) on ArchLinux with gcc-13 (13.2.1 20230801) but succeeded with gcc-12 (12.3.0). Unfortunately, I can't provide any further insights what might cause it. Thanks for the workaround @manticore-projects