marsaev
commented
7 months ago can you check that thrust submodule is up to date?
git submodule update --init --recursive
Open s769 opened 7 months ago
marsaev
commented
7 months ago can you check that thrust submodule is up to date?
git submodule update --init --recursive
marsaev
commented
7 months ago I will also strongly recommend to use newer CUDA if possible, i don't think we still testing CUDA 11.x, and there should be quite a few improvements (compiler, cusparse, etc) since 11.4 which is 2 yeasrs old.
s769
commented
7 months ago The thrust submodule is up to date. I have switched to CUDA 12.0 now, and still seeing the error. Updated the original post.
marsaev
commented
7 months ago Thanks for confirming.
The configuration output, command lines and compiler command lines look fine for me. Considering that Intel MPI worked for you it seems that something might happen in the mvapich wrapper to host compiler. Could it be that mvapich wrapper addresses compiler other than Intel Compiler and they got mixed up?
Can you try configuring by providing regular host compilers (probably icc in your case), and hopefully FindMPI module will generate appropriate linker flags for the host compiler. So something like:
cmake -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icc -DCMAKE_CUDA_HOST_COMPILER=icc -DCUDA_ARCH=80 ..
s769
commented
7 months ago Here is the output of cmake -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icc -DCMAKE_CUDA_HOST_COMPILER=icc -DCUDA_ARCH=80 ..
-- The C compiler identification is Intel 19.1.0.20200306
-- The CXX compiler identification is Intel 19.1.0.20200306
-- The CUDA compiler identification is NVIDIA 12.0.140
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icc - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /opt/apps/cuda/12.0/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found MPI_C: /opt/apps/cuda/11.4/targets/x86_64-linux/lib/stubs/libcuda.so (found version "3.1")
-- Found MPI_CXX: /opt/apps/cuda/11.4/targets/x86_64-linux/lib/stubs/libcuda.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found OpenMP_C: -qopenmp (found version "5.0")
-- Found OpenMP_CXX: -qopenmp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
-- Found CUDAToolkit: /opt/apps/cuda/12.0/include (found suitable version "12.0.140", minimum required is "10.0")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found OpenMP_C: -qopenmp (found version "5.0")
-- Found OpenMP_CXX: -qopenmp (found version "5.0")
This is a MPI build:TRUE
-- Checking if arch 80 is supported...
-- Found libcudacxx: /work/08435/srvenkat/frontera/AMGX/thrust/dependencies/libcudacxx/lib/cmake/libcudacxx/libcudacxx-config.cmake (found suitable version "1.8.1.0", minimum required is "1.8.0")
-- Found Thrust: /work/08435/srvenkat/frontera/AMGX/thrust/thrust/cmake/thrust-config.cmake (found version "2.1.0.0")
-- Found CUB: /work/08435/srvenkat/frontera/AMGX/thrust/dependencies/cub/cub/cmake/cub-config.cmake (found suitable version "2.1.0.0", minimum required is "2.1.0.0")
-- Configuring done
CMake Warning at CMakeLists.txt:280 (add_library):
Cannot generate a safe runtime search path for target amgxsh because files
in some directories may conflict with libraries in implicit directories:
runtime library [libcusolver.so.11] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
/opt/apps/cuda/11.4/targets/x86_64-linux/lib
runtime library [libnvToolsExt.so.1] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
/opt/apps/cuda/11.4/targets/x86_64-linux/lib
Some of these libraries may not be found correctly.
CMake Warning at src/CMakeLists.txt:89 (add_executable):
Cannot generate a safe runtime search path for target amgx_tests_launcher
because files in some directories may conflict with libraries in implicit
directories:
runtime library [libcusolver.so.11] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
/opt/apps/cuda/11.4/targets/x86_64-linux/lib
runtime library [libnvToolsExt.so.1] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
/opt/apps/cuda/11.4/targets/x86_64-linux/lib
Some of these libraries may not be found correctly.
CMake Warning at examples/CMakeLists.txt:55 (add_executable):
Cannot generate a safe runtime search path for target generate_poisson
because files in some directories may conflict with libraries in implicit
directories:
runtime library [libcusolver.so.11] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
/opt/apps/cuda/11.4/targets/x86_64-linux/lib
runtime library [libnvToolsExt.so.1] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
/opt/apps/cuda/11.4/targets/x86_64-linux/lib
Some of these libraries may not be found correctly.
CMake Warning at examples/CMakeLists.txt:63 (add_executable):
Cannot generate a safe runtime search path for target
generate_poisson7_dist_renum because files in some directories may conflict
with libraries in implicit directories:
runtime library [libcusolver.so.11] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
/opt/apps/cuda/11.4/targets/x86_64-linux/lib
runtime library [libnvToolsExt.so.1] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
/opt/apps/cuda/11.4/targets/x86_64-linux/lib
Some of these libraries may not be found correctly.
-- Generating done
-- Build files have been written to: /work/08435/srvenkat/frontera/AMGX/build2If I run make VERBOSE=true all after this, I still get the same errors.
Describe the issue
A clear and concise description of what the issue is.
I am trying to build AMGX with MVAPICH2-GDR on the TACC Supercomputer Lonestar6 as the MPI. Building with Intel MPI works fine.
Environment information:
Intel 19.1.03.24.2CUDA 12.0MVAPICH2-GDR v2.3.7v 2.4.0 f280993e5df32701fb28c2636f49eea4bc5753a4Configuration information
Provide your
cmakecommand line that was used for configuration and it's full output.Output
Compilation information
Provide your make command
make all VERBOSE=1Output