search

Nek5000 / nekRS

our next generation fast and scalable CFD code
https://nek5000.mcs.anl.gov/
Other
292 stars 77 forks source link

AMGx fails to configure with NV HPC_SDK #435

Closed thickAsABrick closed 2 years ago

thickAsABrick commented 2 years ago

I am pasting below the CMake error I see when I run nrsconfig:

-- Found CUDAToolkit: /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/cuda/11.6/include (found suitable version "11.6.112", minimum required is "10.0") 
This is a MPI build:TRUE
-- Performing Test COMPILER_C_SUPPORTS_MCMODEL
-- Performing Test COMPILER_C_SUPPORTS_MCMODEL - Success
-- Performing Test COMPILER_C_SUPPORTS_MCMODEL_FPIC
-- Performing Test COMPILER_C_SUPPORTS_MCMODEL_FPIC - Success
-- Performing Test COMPILER_Fortran_SUPPORTS_LARGE_DATA_THRES
-- Performing Test COMPILER_Fortran_SUPPORTS_LARGE_DATA_THRES - Success
-- Performing Test COMPILER_Fortran_SUPPORTS_CRAYPTR
-- Performing Test COMPILER_Fortran_SUPPORTS_CRAYPTR - Success
-- Found MPI_UNDERLYING_COMPILER: /opt/cray/pe/gcc/11.2.0/bin/g++

----------------- Summary -----------------
Installation directory: /home/bramesh/RECHERCHE/NekRS/v22.0/.local/nekrs
C compiler: /opt/cray/pe/craype/2.7.15/bin/cc
C++ compiler: /opt/cray/pe/craype/2.7.15/bin/CC
Fortran compiler: /opt/cray/pe/craype/2.7.15/bin/ftn
Default backend : CUDA
CPU backend compiler: g++ (flags: -O3 -g -march=native -mtune=native)
NVIDIA CUDA backend enabled (flags: -O3 --fmad=true -lineinfo)
GPU aware MPI support enabled
-------------------------------------------
-- Configuring done
CMake Error at CMakeLists.txt:244 (add_library):
  Target "nekrs-lib" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:244 (add_library):
  Target "nekrs-lib" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:244 (add_library):
  Target "nekrs-lib" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:244 (add_library):
  Target "nekrs-lib" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:244 (add_library):
  Target "nekrs-lib" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:244 (add_library):
  Target "nekrs-lib" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:294 (add_executable):
  Target "nekrs-bin" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:294 (add_executable):
  Target "nekrs-bin" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:294 (add_executable):
  Target "nekrs-bin" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:294 (add_executable):
  Target "nekrs-bin" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:294 (add_executable):
  Target "nekrs-bin" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:294 (add_executable):
  Target "nekrs-bin" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:299 (add_executable):
  Target "axhelm-bin" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:299 (add_executable):
  Target "axhelm-bin" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:299 (add_executable):
  Target "axhelm-bin" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:299 (add_executable):
  Target "axhelm-bin" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:299 (add_executable):
  Target "axhelm-bin" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:299 (add_executable):
  Target "axhelm-bin" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:303 (add_executable):
  Target "advsub-bin" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:303 (add_executable):
  Target "advsub-bin" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:303 (add_executable):
  Target "advsub-bin" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:303 (add_executable):
  Target "advsub-bin" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:303 (add_executable):
  Target "advsub-bin" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:303 (add_executable):
  Target "advsub-bin" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:307 (add_executable):
  Target "fdm-bin" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:307 (add_executable):
  Target "fdm-bin" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:307 (add_executable):
  Target "fdm-bin" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:307 (add_executable):
  Target "fdm-bin" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:307 (add_executable):
  Target "fdm-bin" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at CMakeLists.txt:307 (add_executable):
  Target "fdm-bin" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at 3rd_party/AMGX/CMakeLists.txt:352 (add_library):
  Target "amgxsh" links to target "CUDA::cublas" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at 3rd_party/AMGX/CMakeLists.txt:352 (add_library):
  Target "amgxsh" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at 3rd_party/AMGX/CMakeLists.txt:352 (add_library):
  Target "amgxsh" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at 3rd_party/AMGX/CMakeLists.txt:349 (add_library):
  Target "amgx" links to target "CUDA::cublas" but the target was not found.
  Perhaps a find_package() call is missing for an IMPORTED target, or an
  ALIAS target is missing?

CMake Error at 3rd_party/AMGX/CMakeLists.txt:349 (add_library):
  Target "amgx" links to target "CUDA::cusparse" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

CMake Error at 3rd_party/AMGX/CMakeLists.txt:349 (add_library):
  Target "amgx" links to target "CUDA::cusolver" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

The required libraries are in the LD_LIBRARY_PATH:

bramesh@x3107c0s13b0n0:~/RECHERCHE/NekRS/v22.0/nekRS> echo $LD_LIBRARY_PATH
/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/cuda/11.6/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/math_libs/11.6/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/cuda/11.6/extras/CUPTI/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/cuda/11.6/nvvm/lib64:/opt/cray/pe/gcc/11.2.0/snos/lib64:/opt/cray/pe/papi/6.0.0.14/lib64:/opt/cray/libfabric/1.11.0.4.87/lib64
bramesh@x3107c0s13b0n0:~/RECHERCHE/NekRS/v22.0/nekRS> ls -ltr /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/math_libs/11.6/lib64
lrwxrwxrwx 1 root root 24 Mar 17 20:30 /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/math_libs/11.6/lib64 -> targets/x86_64-linux/lib
bramesh@x3107c0s13b0n0:~/RECHERCHE/NekRS/v22.0/nekRS> cd /opt/nvidia/hpc_sdk/Linux_x86_64/22.3/math_libs/11.6/lib64
bramesh@x3107c0s13b0n0:/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/math_libs/11.6/lib64> ls
libcal.so                 libcufft.so.10                libcusolverMg.so.11          libcutensorMg.so.1.5.0
libcublasLt.so            libcufft.so.10.7.1.112        libcusolverMg.so.11.3.3.112  libcutensorMg_static.a
libcublasLt.so.11         libcufft_static.a             libcusolverMp.so             libcutensor.so
libcublasLt.so.11.8.1.74  libcufft_static_nocallback.a  libcusolver.so               libcutensor.so.1
libcublasLt_static.a      libcufftw.so                  libcusolver.so.11            libcutensor.so.1.5.0
libcublas.so              libcufftw.so.10               libcusolver.so.11.3.3.112    libcutensor_static.a
libcublas.so.11           libcufftw.so.10.7.1.112       libcusolver_static.a         liblapack_static.a
libcublas.so.11.8.1.74    libcufftw_static.a            libcusparse.so               libmetis_static.a
libcublas_static.a        libcurand.so                  libcusparse.so.11            libnvblas.so
libcufftMp.so             libcurand.so.10               libcusparse.so.11.7.2.112    libnvblas.so.11
libcufftMp.so.0           libcurand.so.10.2.9.55        libcusparse_static.a         libnvblas.so.11.8.1.74
libcufftMp.so.0.0.2       libcurand_static.a            libcutensorMg.so             stubs
libcufft.so               libcusolverMg.so              libcutensorMg.so.1

And, I am using CMake version > 3.18 

bramesh@x3107c0s13b0n0:/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/math_libs/11.6/lib64> cmake --version
cmake version 3.21.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
bramesh@x3107c0s13b0n0:/opt/nvidia/hpc_sdk/Linux_x86_64/22.3/math_libs/11.6/lib64>

I'd appreciate help figuring out why CMake is throwing an error.

kris-rowe commented 2 years ago

If you pass -DCMAKE_PREFIX_PATH="${NVHPC_PATH}/math_libs" to CMake, this seems to do the trick.

stgeke commented 2 years ago

Updating to newer CMake (e.g. 3.23) seems to resolve the issue.

thickAsABrick commented 2 years ago

I am afraid the problem of not being able to build with AMGX persists even with cmake-3.23.2. It may not be a CMake issue. Perhaps it has more to do with FindCUDAToolKit being unable to find all the CUDA relevant header files in version cudatoolkit-11.6. I do not have access to more than one platform that runs cudatoolkit-11.6. So, for now, this matter remains resolved.

thickAsABrick commented 2 years ago

This issue has been resolved.

stgeke commented 2 years ago

This is an AMGx not a nekRS issue. Feel free to open an issue here: https://github.com/NVIDIA/AMGX/issues