search

alpaka-group / alpaka

Abstraction Library for Parallel Kernel Acceleration :llama:
https://alpaka.readthedocs.io
Mozilla Public License 2.0
355 stars 74 forks source link

`hostOnlyAPITest` fails #2328

Open fwyzard opened 3 months ago

fwyzard commented 3 months ago

I've building the alpaka tests with gcc 13 (tested also 12) and CUDA 12.5 (tested also 12.4) on Ubuntu 22.04, with

cmake /home/fwyzard/src/alpaka-group/alpaka \
  -G 'Unix Makefiles' \
  --log-level=VERBOSE \
  -Dalpaka_ACC_CPU_B_OMP2_T_SEQ_ENABLE='OFF' \
  -Dalpaka_ACC_CPU_B_SEQ_T_OMP2_ENABLE='OFF' \
  -Dalpaka_ACC_CPU_B_SEQ_T_SEQ_ENABLE='ON' \
  -Dalpaka_ACC_CPU_B_SEQ_T_THREADS_ENABLE='OFF' \
  -Dalpaka_ACC_CPU_B_TBB_T_SEQ_ENABLE='OFF' \
  -Dalpaka_ACC_CPU_DISABLE_ATOMIC_REF=OFF \
  -Dalpaka_ACC_GPU_CUDA_ENABLE='ON' \
  -Dalpaka_ACC_GPU_CUDA_ONLY_MODE='OFF' \
  -Dalpaka_ACC_GPU_HIP_ENABLE='OFF' \
  -Dalpaka_ACC_GPU_HIP_ONLY_MODE='OFF' \
  -Dalpaka_ACC_SYCL_ENABLE=OFF \
  -Dalpaka_ASSERT_ACC_ENABLE=ON \
  -Dalpaka_BLOCK_SHARED_DYN_MEMBER_ALLOC_KIB=47 \
  -Dalpaka_BUILD_BENCHMARKS=ON \
  -Dalpaka_BUILD_EXAMPLES=ON \
  -Dalpaka_CXX_STANDARD=17 \
  -Dalpaka_DEBUG=2 \
  -Dalpaka_DISABLE_VENDOR_RNG=ON \
  -Dalpaka_ENABLE_WERROR=OFF \
  -Dalpaka_FAST_MATH=DEFAULT \
  -Dalpaka_FTZ=DEFAULT \
  -Dalpaka_INSTALL_TEST_HEADER=OFF \
  -Dalpaka_RELOCATABLE_DEVICE_CODE=DEFAULT \
  -Dalpaka_USE_MDSPAN=OFF \
  -DBUILD_TESTING=ON \
  -DCMAKE_CXX_COMPILER='g++-13' \
  -DCMAKE_CXX_FLAGS='' \
  -DCMAKE_EXE_LINKER_FLAGS='' \
  -DCUDAToolkit_ROOT='/usr/local/cuda-12.5' \
  -DCMAKE_CUDA_COMPILER='/usr/local/cuda-12.5/bin/nvcc' \
  -DCMAKE_CUDA_ARCHITECTURES='86' \
  -DCMAKE_CUDA_FLAGS='' \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_VERBOSE_MAKEFILE=ON

hostOnlyAPITest fails to link with

[ 54%] Linking CXX executable hostOnlyAPITest
cd /home/fwyzard/src/alpaka-group/build/gcc13_cuda125/test/integ/hostOnlyAPI && /usr/bin/cmake -E cmake_link_script CMakeFiles/hostOnlyAPITest.dir/link.txt --verbose=1
/usr/bin/g++-13 -g -Og CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o -o hostOnlyAPITest  /usr/local/cuda-12.5/targets/x86_64-linux/lib/stubs/libcuda.so -ldl /usr/lib/x86_64-linux-gnu/librt.a /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.74.0 ../../../thirdParty/catch2/src/libCatch2Maind.a ../../../thirdParty/catch2/src/libCatch2d.a
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getErrorString(cudaError)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:208: undefined reference to `cudaGetErrorString'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getErrorName(cudaError)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:203: undefined reference to `cudaGetErrorName'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getLastError()':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:213: undefined reference to `cudaGetLastError'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getErrorString(cudaError)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:208: undefined reference to `cudaGetErrorString'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getErrorName(cudaError)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:203: undefined reference to `cudaGetErrorName'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getLastError()':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:213: undefined reference to `cudaGetLastError'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getDeviceCount(int*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:193: undefined reference to `cudaGetDeviceCount'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getLastError()':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:213: undefined reference to `cudaGetLastError'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::setDevice(int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:347: undefined reference to `cudaSetDevice'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::streamDestroy(CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:362: undefined reference to `cudaStreamDestroy'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::streamCreate(CUstream_st**)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:352: undefined reference to `cudaStreamCreate'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getLastError()':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:213: undefined reference to `cudaGetLastError'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getDeviceProperties(cudaDeviceProp*, int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:198: undefined reference to `cudaGetDeviceProperties_v2'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::memsetAsync(void*, int, unsigned long, CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:342: undefined reference to `cudaMemsetAsync_ptsz'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::streamSynchronize(CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:372: undefined reference to `cudaStreamSynchronize_ptsz'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getLastError()':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:213: undefined reference to `cudaGetLastError'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::hostFree(void*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:234: undefined reference to `cudaFreeHost'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::free(void*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:164: undefined reference to `cudaFree'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::freeAsync(void*, CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:170: undefined reference to `cudaFreeAsync_ptsz'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::setDevice(int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:347: undefined reference to `cudaSetDevice'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::memcpyAsync(void*, void const*, unsigned long, cudaMemcpyKind, CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:321: undefined reference to `cudaMemcpyAsync_ptsz'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::setDevice(int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:347: undefined reference to `cudaSetDevice'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::memcpyAsync(void*, void const*, unsigned long, cudaMemcpyKind, CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:321: undefined reference to `cudaMemcpyAsync_ptsz'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::setDevice(int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:347: undefined reference to `cudaSetDevice'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::memcpyAsync(void*, void const*, unsigned long, cudaMemcpyKind, CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:321: undefined reference to `cudaMemcpyAsync_ptsz'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::setDevice(int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:347: undefined reference to `cudaSetDevice'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::malloc(void**, unsigned long)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:265: undefined reference to `cudaMalloc'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::setDevice(int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:347: undefined reference to `cudaSetDevice'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::mallocAsync(void**, unsigned long, CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:279: undefined reference to `cudaMallocAsync_ptsz'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::hostMalloc(void**, unsigned long, unsigned int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:239: undefined reference to `cudaHostAlloc'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::setDevice(int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:347: undefined reference to `cudaSetDevice'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::streamCreateWithFlags(CUstream_st**, unsigned int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:357: undefined reference to `cudaStreamCreateWithFlags'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::getDeviceProperties(cudaDeviceProp*, int)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:198: undefined reference to `cudaGetDeviceProperties_v2'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::streamSynchronize(CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:372: undefined reference to `cudaStreamSynchronize_ptsz'
/usr/bin/ld: CMakeFiles/hostOnlyAPITest.dir/src/hostOnlyAPI.cpp.o: in function `alpaka::ApiCudaRt::streamDestroy(CUstream_st*)':
/home/fwyzard/src/alpaka-group/alpaka/include/alpaka/core/ApiCudaRt.hpp:362: undefined reference to `cudaStreamDestroy'
collect2: error: ld returned 1 exit status

CMake tries to link only /usr/local/cuda-12.5/targets/x86_64-linux/lib/stubs/libcuda.so, while it should like -L/usr/local/cuda-12.5/targets/x86_64-linux/lib -L/usr/local/cuda-12.5/targets/x86_64-linux/lib/stubs -lcudart -lcuda.

It does work if I change the linker script by hand.

Am I passing the wrong flags to CMake ? Or ?

psychocoderHPC commented 3 months ago

Maybe you need to set CUDA_CUDART. I have only a system where the cuda driver is installed and there all is fine and compiles.

I only need to call cmake ../alpaka -DBUILD_TESTING=ON -Dalpaka_ACC_GPU_CUDA_ENABLE='ON' all other flags for CUDA is derived automatically. On the system we set CMAKE_PREFIX_PATH=<CUDA_PATH>:$CMAKE_PREFIX_PATH

fwyzard commented 3 months ago

How does alpaka use CMake to find CUDA ? Does it use FindCUDAToolkit ? Or some other way ?

fwyzard commented 3 months ago

OK, after some testing, it looks like setting CMAKE_PREFIX_PATH or passing -DCUDAToolkit_ROOT='...' -DCMAKE_CUDA_COMPILER='.../bin/nvcc' are equivalent.

What seems to break the hostOnlyAPITest is passing -Dalpaka_DISABLE_VENDOR_RNG='ON'.

fwyzard commented 3 months ago

This seems to fix the issue:

diff --git a/cmake/alpakaCommon.cmake b/cmake/alpakaCommon.cmake
index ebaad1b9a80..d4155db366e 100644
--- a/cmake/alpakaCommon.cmake
+++ b/cmake/alpakaCommon.cmake
@@ -542,9 +542,11 @@ if(alpaka_ACC_GPU_CUDA_ENABLE)
             endif()
         endif()

+        target_link_libraries(alpaka INTERFACE CUDA::cudart)
+
         if(NOT alpaka_DISABLE_VENDOR_RNG)
             # Use cuRAND random number generators
-            target_link_libraries(alpaka INTERFACE CUDA::cudart CUDA::curand)
+            target_link_libraries(alpaka INTERFACE CUDA::curand)
         endif()
     else()
         message(FATAL_ERROR "Optional alpaka dependency CUDA could not be found!")
fwyzard commented 3 months ago

Fixed by #2329.

psychocoderHPC commented 3 months ago

How does alpaka use CMake to find CUDA ? Does it use FindCUDAToolkit ? Or some other way ?

We use native CMake CUDA language support.