Problem with make - ‘PyArray_GetArrayParamsFromObject’ was not declared in this scope

[ehlertdo]

Hi,

I have problems installing CRPropa on my system. The problem occurs both when attempting the installation on my laptop and on our local computing cluster. The python virtual environment is managed with miniforge/mamba. Earlier this year the issue did not occur but I have trouble understanding where it comes from.

Full make output CMakeCache.txt

Setting -DENABLE_SWIG_BUILTIN=ON does not solve the problem (curiously enough cmake still "finds" swig in the same path as when the flag is turned off).

System Information:

• Operating system: 6.10.0-3-MANJARO (64-bit)
• Laptop or computing cluster? Laptop (but same problem on HPC)
• Python version: 3.12.4
• Swig version: 4.2
• Cmake version: 3.30.2

Did you install CRPropa from git or download a release? I installed version 3.2.1-158-gbeee8ad4 from git/from the releases.

Cmake instructions and output NAME="crpropa-BH"

export CRPROPA_DIR=$HOME"/Software/"$NAME echo $CRPROPA_DIR

export VENV_DIR="/home/domenik/Software/miniforge3/envs/"$NAME

CMAKE_PREFIX_PATH=$CRPROPA_DIR cmake -DCMAKE_INSTALL_PREFIX=$CRPROPA_DIR -DFFTW3F_INCLUDE_DIR=$VENV_DIR"/include/" -DFFTW3F_LIBRARY=$VENV_DIR"/lib/libfftw3.so" -DZLIB_INCLUDE_DIR=$VENV_DIR"/include/" -DZLIB_LIBRARY=$VENV_DIR"/lib/libz.so" -DMUPARSER_INCLUDE_DIR=$VENV_DIR"/include/" -DMUPARSER_LIBRARY=$VENV_DIR"/lib/libmuparser.so" -DENABLE_HDF5=OFF ..

-- The Fortran compiler identification is GNU 14.1.1 -- The C compiler identification is GNU 14.1.1 -- The CXX compiler identification is GNU 14.1.1 -- Detecting Fortran compiler ABI info -- Detecting Fortran compiler ABI info - done -- Check for working Fortran compiler: /usr/bin/f95 - skipped -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Use --as-needed linker flags! With SIMD_EXTENSIONS "none" tricubic interpolation of vector grids is not possible. You should set SIMD_EXTENSION to a compatible value ("avx", "avx+fma", or -- depending on the build CPU -- "native"). Trilinear interpolation is still possible. -- Build Type: Release -- CRPropa version: 3.2.1-158-gbeee8ad4 beee8ad4171ff9ca6e2f8f8633e0f87126bfa201 refs/heads/master -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE Using provided EIGEN -- Found OpenMP_C: -fopenmp (found version "4.5") -- Found OpenMP_CXX: -fopenmp (found version "4.5") -- Found OpenMP_Fortran: -fopenmp (found version "4.5") -- Found OpenMP: TRUE (found version "4.5") -- Google perftools: Found! -- Include: /usr/include -- FFTW3 with single precision (FFTW3F): Found! -- Include: /home/domenik/Software/miniforge3/envs/crpropa-BH/include -- Library: /home/domenik/Software/miniforge3/envs/crpropa-BH/lib/libfftw3.so -- Found PkgConfig: /home/domenik/Software/miniforge3/envs/crpropa-BH/bin/pkg-config (found version "0.29.2") -- muParser: Found! -- Include: /home/domenik/Software/miniforge3/envs/crpropa-BH/include -- Library: /home/domenik/Software/miniforge3/envs/crpropa-BH/lib/libmuparser.so -- Found ZLIB: /home/domenik/Software/miniforge3/envs/crpropa-BH/lib/libz.so (found version "1.3.1") -- Use absolute RPATH /home/domenik/Software/crpropa-BH/lib -- Downloading data files from sciebo ~ 73 MB -- Extracting data file Error copying directory from "/home/domenik/Software/crpropa-BH/CRPropa3/build/data-2024-04-30/data" to "/home/domenik/Software/crpropa-BH/CRPropa3/build/data/". -- Build of documentation disabeled. Enable with BUILD_DOC=On -- Found Python: /home/domenik/Software/miniforge3/envs/crpropa-BH/bin/python3.12 (found suitable version "3.12.4", minimum required is "3.0") found components: Interpreter Development NumPy Development.Module Development.Embed -- Found SWIG: /home/domenik/Software/miniforge3/envs/crpropa-BH/bin/swig4.0 (found suitable version "4.2.1", minimum required is "3.0") -- Python: Found! -- version 3.12.4 -- executable: /home/domenik/Software/miniforge3/envs/crpropa-BH/bin/python3.12 -- libraries: /home/domenik/Software/miniforge3/envs/crpropa-BH/lib/libpython3.12.so -- headers: /home/domenik/Software/miniforge3/envs/crpropa-BH/include/python3.12 -- site packages: /home/domenik/Software/miniforge3/envs/crpropa-BH/lib/python3.12/site-packages -- development libraries: Found! -- package install directory: /home/domenik/Software/miniforge3/envs/crpropa-BH/lib/python3.12/site-packages -- NumPy: Found! -- headers: /home/domenik/Software/miniforge3/envs/crpropa-BH/lib/python3.12/site-packages/numpy/_core/include (version 2.0.1) -- Configuring done (4.1s) -- Generating done (0.0s) -- Build files have been written to: /home/domenik/Software/crpropa-BH/CRPropa3/build

[JulienDoerner]

hey @ehlertdo, this issue is related to the new numpy version (>= 2.0). If you downgrade to version 1.26 everything should still run.

[ehlertdo]

Ah indeed, after downgrading numpy to 1.26.4 it works! Thanks @JulienDoerner !

Seems like PyArray_GetArrayFromObject was deprecated in numpy release 1.19.0 and then completely removed with version 2.0.0.

[VasundharaShaw]

Hi @JulienDoerner

i am having the same issue even though I downgrade the numpy version

make[2]: Warning: File `CMakeFiles/crpropa-swig.dir/flags.make' has modification time 365 s in the future [ 71%] Building CXX object CMakeFiles/crpropa-swig.dir/crpropa_wrap.cxx.o /share/nas2/vshaw/CRPropa3/build/crpropa_wrap.cxx: In function ‘PyObject crpropa_MagneticLens_transformModelVector_numpyArray(crpropa::MagneticLens, PyObject*, double)’: /share/nas2/vshaw/CRPropa3/build/crpropa_wrap.cxx:8620:89: error: ‘PyArray_GetArrayParamsFromObject’ was not declared in this scope if (PyArray_GetArrayParamsFromObject(input, NULL, 1, &dtype, &ndim, dims, &arr, NULL) < 0) { ^ make[2]: [CMakeFiles/crpropa-swig.dir/crpropa_wrap.cxx.o] Error 1 make[1]: [CMakeFiles/crpropa-swig.dir/all] Error 2 make: *** [all] Error 2

My current numpy version is - NumPy version: 1.19.5

[ehlertdo]

@VasundharaShaw Is this also the numpy version that is discovered during the cmake step?

[VasundharaShaw]

@VasundharaShaw Is this also the numpy version that is discovered during the cmake step?

Yes, and I have not been able to fix it yet!

[JulienDoerner]

@VasundharaShaw, can you put the output of the cmake step here. Then we can check the dependencies and try to reproduce it.

[VasundharaShaw]

@VasundharaShaw, can you put the output of the cmake step here. Then we can check the dependencies and try to reproduce it.

Here you go - firstly I don't get why it complains about swig since I do have swig installed and I did give it the path

-- Use --as-needed linker flags!
With SIMD_EXTENSIONS "none" tricubic interpolation of vector grids is not possible. You should set SIMD_EXTENSION to a compatible value ("avx", "avx+fma", or -- depending on the build CPU -- "native"). Trilinear interpolation is still possible.
-- Build Type: Release
-- CRPropa version: 3.2.1-158-gbeee8ad beee8ad4171ff9ca6e2f8f8633e0f87126bfa201 refs/heads/master
Using provided EIGEN
-- Google perftools: NOT Found!
--   Include: GOOGLE_PERFTOOLS_INCLUDE_DIR-NOTFOUND
-- FFTW3 with single precision (FFTW3F): NOT Found!
--   Include:     FFTW3F_INCLUDE_DIR-NOTFOUND
--   Library:     FFTW3F_LIBRARY-NOTFOUND
-- muParser: NOT Found!
--   Include:     MUPARSER_INCLUDE_DIR-NOTFOUND
--   Library:     MUPARSER_LIBRARY-NOTFOUND
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Use absolute RPATH /home/vshaw/.local/lib
-- Downloading data files from sciebo ~ 73 MB
-- Extracting data file
Error copying directory from "/share/nas2/vshaw/CRPropa3/build/data-2024-04-30/data" to "/share/nas2/vshaw/CRPropa3/build/data/".
-- Build of documentation disabeled. Enable with BUILD_DOC=On
-- Python: Found!
--   version 3.10.14
--   executable: /home/vshaw/miniconda/envs/myenv/bin/python3.10
--   libraries: /home/vshaw/miniconda/envs/myenv/lib/libpython3.10.so
--   headers: /home/vshaw/miniconda/envs/myenv/include/python3.10
--   site packages: /home/vshaw/miniconda/envs/myenv/lib/python3.10/site-packages
--   development libraries: Found!
--   package install directory: /home/vshaw/miniconda/envs/myenv/lib/python3.10/site-packages
-- NumPy: Found!
--   headers: /home/vshaw/miniconda/envs/myenv/lib/python3.10/site-packages/numpy/_core/include (version 2.0.1)
-- Configuring done (4.0s)
CMake Warning at CMakeLists.txt:524 (add_library):
  Cannot generate a safe runtime search path for target crpropa-swig because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libz.so.1] in /usr/lib64 may be hidden by files in:
      /home/vshaw/miniconda/envs/myenv/lib

  Some of these libraries may not be found correctly.

-- Generating done (0.2s)
-- Build files have been written to: /share/nas2/vshaw/CRPropa3/build

~/miniconda/envs/crpropa-env/bin/swig - that's my swig

This is the main error

[ 70%] Built target crpropa make[2]: Warning: FileCMakeFiles/crpropa-swig-wrapper.dir/progress.make' has modification time 849 s in the future make[2]: warning: Clock skew detected. Your build may be incomplete. make[2]: Warning: File CMakeFiles/crpropa-swig-wrapper.dir/progress.make' has modification time 849 s in the future /share/nas2/vshaw/CRPropa3/include/crpropa/Random.h:95: Warning 560: Unknown Doxygen command: writeme. /share/nas2/vshaw/CRPropa3/include/crpropa/magneticField/turbulentField/TurbulentField.h:84: Warning 560: Unknown Doxygen command: f$l. /share/nas2/vshaw/CRPropa3/include/crpropa/magneticField/turbulentField/TurbulentField.h:84: Warning 560: Unknown Doxygen command: rm. /share/nas2/vshaw/CRPropa3/include/crpropa/magneticField/turbulentField/TurbulentField.h:84: Warning 560: Unknown Doxygen command: int. /share/nas2/vshaw/CRPropa3/include/crpropa/magneticField/turbulentField/TurbulentField.h:84: Warning 560: Unknown Doxygen command: infty. /share/nas2/vshaw/CRPropa3/include/crpropa/magneticField/turbulentField/TurbulentField.h:84: Warning 560: Unknown Doxygen command: langleB. /share/nas2/vshaw/CRPropa3/include/crpropa/magneticField/turbulentField/TurbulentField.h:84: Warning 560: Unknown Doxygen command: rangle. /share/nas2/vshaw/CRPropa3/include/crpropa/magneticField/turbulentField/TurbulentField.h:88: Warning 562: Expected Doxygen command: f$. /share/nas2/vshaw/CRPropa3/include/crpropa/module/Acceleration.h:143: Warning 560: Unknown Doxygen command: f$E. /share/nas2/vshaw/CRPropa3/include/crpropa/module/Acceleration.h:148: Warning 560: Unknown Doxygen command: f$R. /share/nas2/vshaw/CRPropa3/include/crpropa/module/Acceleration.h:148: Warning 560: Unknown Doxygen command: frac{E}{B. /share/nas2/vshaw/CRPropa3/include/crpropa/module/Acceleration.h:157: Warning 562: Expected Doxygen command: f$. make[2]: warning: Clock skew detected. Your build may be incomplete. [ 70%] Built target crpropa-swig-wrapper make[2]: Warning: FileCMakeFiles/crpropa-swig.dir/progress.make' has modification time 841 s in the future make[2]: warning: Clock skew detected. Your build may be incomplete. make[2]: Warning: File CMakeFiles/crpropa-swig.dir/progress.make' has modification time 841 s in the future [ 71%] Building CXX object CMakeFiles/crpropa-swig.dir/crpropa_wrap.cxx.o /share/nas2/vshaw/CRPropa3/build/crpropa_wrap.cxx: In function ‘PyObject* crpropa_MagneticLens_transformModelVector_numpyArray(crpropa::MagneticLens*, PyObject*, double)’: /share/nas2/vshaw/CRPropa3/build/crpropa_wrap.cxx:8620:89: error: ‘PyArray_GetArrayParamsFromObject’ was not declared in this scope if (PyArray_GetArrayParamsFromObject(input, NULL, 1, &dtype, &ndim, dims, &arr, NULL) < 0) { ^ make[2]: *** [CMakeFiles/crpropa-swig.dir/crpropa_wrap.cxx.o] Error 1 make[1]: *** [CMakeFiles/crpropa-swig.dir/all] Error 2 make: *** [all] Error 2

[JulienDoerner]

hey @VasundharaShaw,

It seems that you are still using the 2.0.1 numpy version

-- NumPy: Found!
--   headers: /home/vshaw/miniconda/envs/myenv/lib/python3.10/site-packages/numpy/_core/include (version 2.0.1)

You can try to clean up the build folder by running make clean or (if it doesn't help) remove the build folder and start installation again.

Also you have to check which envs from Conda you are using. From the path it seems that python is taken from the myenv while you want to use swig from crpropa-env

[VasundharaShaw]

hey @VasundharaShaw,

It seems that you are still using the 2.0.1 numpy version

-- NumPy: Found!
--   headers: /home/vshaw/miniconda/envs/myenv/lib/python3.10/site-packages/numpy/_core/include (version 2.0.1)

You can try to clean up the build folder by running make clean or (if it doesn't help) remove the build folder and start installation again.

Also you have to check which envs from Conda you are using. From the path it seems that python is taken from the myenv while you want to use swig from crpropa-env

I tried what you have suggested and did end up correcting for the environment but seems that I have new errors now.

(crpropa-env) [vshaw@galahad build]$ cmake --system-information | grep PYTHON
ENABLE_PYTHON:BOOL=ON
PYTHON_EXECUTABLE:UNINITIALIZED=/home/vshaw/miniconda/envs/crpropa-env/bin/python
PYTHON_VERSION:UNINITIALIZED=3.8
(crpropa-env) [vshaw@galahad build]$ cmake .. \
>   -DPYTHON_EXECUTABLE=$(which python) \
>   -DNUMPY_INCLUDE_DIRS=$(python -c "import numpy; print(numpy.get_include())") \
>   -DCMAKE_INSTALL_PREFIX=$HOME/.local
-- Use --as-needed linker flags!
With SIMD_EXTENSIONS "none" tricubic interpolation of vector grids is not possible. You should set SIMD_EXTENSION to a compatible value ("avx", "avx+fma", or -- depending on the build CPU -- "native"). Trilinear interpolation is still possible.
-- Build Type: Release
-- CRPropa version: 3.2.1-158-gbeee8ad beee8ad4171ff9ca6e2f8f8633e0f87126bfa201 refs/heads/master
Using provided EIGEN
-- Google perftools: NOT Found!
--   Include: GOOGLE_PERFTOOLS_INCLUDE_DIR-NOTFOUND
-- FFTW3 with single precision (FFTW3F): NOT Found!
--   Include:     FFTW3F_INCLUDE_DIR-NOTFOUND
--   Library:     FFTW3F_LIBRARY-NOTFOUND
-- muParser: NOT Found!
--   Include:     MUPARSER_INCLUDE_DIR-NOTFOUND
--   Library:     MUPARSER_LIBRARY-NOTFOUND
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Use absolute RPATH /home/vshaw/.local/lib
-- Downloading data files from sciebo ~ 73 MB
-- Extracting data file
Error copying directory from "/share/nas2/vshaw/CRPropa3/build/data-2024-04-30/data" to "/share/nas2/vshaw/CRPropa3/build/data/".
-- Build of documentation disabeled. Enable with BUILD_DOC=On
CMake Error at /home/vshaw/.local/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python (missing: Python_NumPy_INCLUDE_DIRS NumPy) (found
  suitable version "3.10.11", minimum required is "3.0")
Call Stack (most recent call first):
  /home/vshaw/.local/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /home/vshaw/.local/share/cmake-3.27/Modules/FindPython/Support.cmake:3824 (find_package_handle_standard_args)
  /home/vshaw/.local/share/cmake-3.27/Modules/FindPython.cmake:574 (include)
  CMakeLists.txt:449 (find_package)

-- Configuring incomplete, errors occurred!

[JulienDoerner]

Is NumPy installed in this environment? It seems that cmake can not find it. You can try to run pip show numpy in the env.

[VasundharaShaw]

Hi Julien - I downgraded numpy and python and added the DPYTHON_LIBRARY path to CMAKE command and it seems to work. @avvliet helped me out to figure this out. Thanks for your prompt replies :)