search

ddemidov / amgcl

C++ library for solving large sparse linear systems with algebraic multigrid method
http://amgcl.readthedocs.org/
MIT License
723 stars 110 forks source link

using amgcl::amgcl target with a library containing *.cu files fails #202

Closed blattms closed 2 years ago

blattms commented 3 years ago

When used with target_link_library(lib PUBLIC amgcl::amgcl) compilation will fail if one of the source files is compiled with nvcc (e.g. a *.cu file):

nvcc fatal   : Unknown option 'fopenmp'
ddemidov commented 3 years ago

Hi Marcus,

I just tried to compile the solver_cuda example from amgcl examples, and it seems to work. Although the verbose build output shows that it does not actually use nvcc:

ninja -v solver_cuda
[1/2] cd /home/demidov/work/amgcl/build/examples/CMakeFiles/solver_cuda.dir && /usr/bin/cmake -E make_directory /home/demidov/work/amgcl/build/examples/CMakeFiles/solver_cuda.dir//. && /usr/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/home/demidov/work/amgcl/build/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o -D generated_cubin_file:STRING=/home/demidov/work/amgcl/build/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o.cubin.txt -P /home/demidov/work/amgcl/build/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.Release.cmake
[2/2] : && /usr/bin/c++ -DCL_TARGET_OPENCL_VERSION=120 -DCUB_IGNORE_DEPRECATED_CPP_DIALECT -DTHRUST_IGNORE_DEPRECATED_CPP_DIALECT -DBOOST_BIND_GLOBAL_PLACEHOLDERS -DCL_USE_DEPRECATED_OPENCL_1_2_APIS -DAMGCL_PROFILING -O3 -DNDEBUG -rdynamic examples/CMakeFiles/solver_cuda.dir/solver_cuda_generated_solver.cu.o -o examples/solver_cuda  -Wl,-rpath,/opt/cuda/lib64  /opt/cuda/lib64/libcudart_static.a  -ldl  -lrt  -lpthread  -fopenmp  /usr/lib/libboost_program_options.so.1.75.0  /opt/cuda/lib64/libcusparse.so && :

I am using cuda_add_executable: https://github.com/ddemidov/amgcl/blob/720f6b7bc325a8a87df9ae33da125a80069b1d5c/examples/CMakeLists.txt#L110-L120

g++ --version
g++ (GCC) 11.1.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0
ddemidov commented 3 years ago

Is it possible -fopenmp comes from outside amgcl::amgcl? It should be -Xcompiler -fopenmp when used with nvcc. Here is verbose build with makefiles instead of ninja, which is more clear:

make VERBOSE=1 solver_cuda
/usr/bin/cmake -S/home/demidov/work/amgcl -B/home/demidov/work/amgcl/build-make --check-build-system CMakeFiles/Makefile.cmake 0
make  -f CMakeFiles/Makefile2 solver_cuda
make[1]: Entering directory '/home/demidov/work/amgcl/build-make'
/usr/bin/cmake -S/home/demidov/work/amgcl -B/home/demidov/work/amgcl/build-make --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/demidov/work/amgcl/build-make/CMakeFiles 1
make  -f CMakeFiles/Makefile2 examples/CMakeFiles/solver_cuda.dir/all
make[2]: Entering directory '/home/demidov/work/amgcl/build-make'
make  -f examples/CMakeFiles/solver_cuda.dir/build.make examples/CMakeFiles/solver_cuda.dir/depend
make[3]: Entering directory '/home/demidov/work/amgcl/build-make'
[100%] Building NVCC (Device) object examples/CMakeFiles/solver_cuda.dir/solver_cuda_generated_solver.cu.o
cd /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir && /usr/bin/cmake -E make_directory /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//.
cd /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir && /usr/bin/cmake -D verbose:BOOL=1 -D build_configuration:STRING=RelWithDebInfo -D generated_file:STRING=/home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o -D generated_cubin_file:STRING=/home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o.cubin.txt -P /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.RelWithDebInfo.cmake
-- Removing /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o
/usr/bin/cmake -E rm -f /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o
-- Generating dependency file: /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.NVCC-depend
/opt/cuda/bin/nvcc -M -D__CUDACC__ /home/demidov/work/amgcl/build-make/examples/solver.cu -o /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.NVCC-depend -ccbin /usr/bin/cc -m64 -DAMGCL_BLOCK_SIZES=(3)(4) -DBOOST_ALL_NO_LIB -DBOOST_PROGRAM_OPTIONS_DYN_LINK -Xcompiler ,\"-DCUB_IGNORE_DEPRECATED_CPP_DIALECT\",\"-DBOOST_BIND_GLOBAL_PLACEHOLDERS\",\"-O2\",\"-g\",\"-DNDEBUG\" -Xcompiler -fopenmp -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -std=c++11 -Wno-deprecated-gpu-targets -Xcompiler -std=c++11 -Xcompiler -fPIC -Xcompiler -Wno-vla -DSOLVER_BACKEND_CUDA -DNVCC -I/opt/cuda/include -I/home/demidov/work/amgcl/examples/../tests -I/usr/include -I/home/demidov/work/amgcl
-- Generating temporary cmake readable file: /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.depend.tmp
/usr/bin/cmake -D input_file:FILEPATH=/home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.NVCC-depend -D output_file:FILEPATH=/home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.depend.tmp -D verbose=1 -P /usr/share/cmake-3.20/Modules/FindCUDA/make2cmake.cmake
-- Copy if different /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.depend.tmp to /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.depend
/usr/bin/cmake -E copy_if_different /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.depend.tmp /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.depend
-- Removing /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.depend.tmp and /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.NVCC-depend
/usr/bin/cmake -E rm -f /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.depend.tmp /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//solver_cuda_generated_solver.cu.o.NVCC-depend
-- Generating /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o
/opt/cuda/bin/nvcc /home/demidov/work/amgcl/build-make/examples/solver.cu -c -o /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o -ccbin /usr/bin/cc -m64 -DAMGCL_BLOCK_SIZES=(3)(4) -DBOOST_ALL_NO_LIB -DBOOST_PROGRAM_OPTIONS_DYN_LINK -Xcompiler ,\"-DCUB_IGNORE_DEPRECATED_CPP_DIALECT\",\"-DBOOST_BIND_GLOBAL_PLACEHOLDERS\",\"-O2\",\"-g\",\"-DNDEBUG\" -Xcompiler -fopenmp -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -std=c++11 -Wno-deprecated-gpu-targets -Xcompiler -std=c++11 -Xcompiler -fPIC -Xcompiler -Wno-vla -DSOLVER_BACKEND_CUDA -DNVCC -I/opt/cuda/include -I/home/demidov/work/amgcl/examples/../tests -I/usr/include -I/home/demidov/work/amgcl
Generated /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir//./solver_cuda_generated_solver.cu.o successfully.
cd /home/demidov/work/amgcl/build-make && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/demidov/work/amgcl /home/demidov/work/amgcl/examples /home/demidov/work/amgcl/build-make /home/demidov/work/amgcl/build-make/examples /home/demidov/work/amgcl/build-make/examples/CMakeFiles/solver_cuda.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/demidov/work/amgcl/build-make'
make  -f examples/CMakeFiles/solver_cuda.dir/build.make examples/CMakeFiles/solver_cuda.dir/build
make[3]: Entering directory '/home/demidov/work/amgcl/build-make'
[100%] Linking CXX executable solver_cuda
cd /home/demidov/work/amgcl/build-make/examples && /usr/bin/cmake -E cmake_link_script CMakeFiles/solver_cuda.dir/link.txt --verbose=1
/usr/bin/c++ -DCUB_IGNORE_DEPRECATED_CPP_DIALECT -DBOOST_BIND_GLOBAL_PLACEHOLDERS -O2 -g -DNDEBUG -rdynamic CMakeFiles/solver_cuda.dir/solver_cuda_generated_solver.cu.o -o solver_cuda  -Wl,-rpath,/opt/cuda/lib64 /opt/cuda/lib64/libcudart_static.a -ldl -lrt -lpthread -fopenmp /usr/lib/libboost_program_options.so.1.75.0 /opt/cuda/lib64/libcusparse.so 
make[3]: Leaving directory '/home/demidov/work/amgcl/build-make'
[100%] Built target solver_cuda
make[2]: Leaving directory '/home/demidov/work/amgcl/build-make'
/usr/bin/cmake -E cmake_progress_start /home/demidov/work/amgcl/build-make/CMakeFiles 0
make[1]: Leaving directory '/home/demidov/work/amgcl/build-make'
blattms commented 3 years ago

I am using nvcc 9.2 (maybe it is supported in newer versions?):

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148

and g++:

$ g++ --version
g++ (Debian 8.3.0-6) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

It comes from amgcl-targets.cmake, when installing amgcl. I am using find_package(amgcl) in my own project and wanted to use the target for linking to get the flags automatically.

# Create imported target amgcl::amgcl
add_library(amgcl::amgcl INTERFACE IMPORTED)

set_target_properties(amgcl::amgcl PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "\$<\$<CXX_COMPILER_ID:MSVC>:NOMINMAX>;\$<\$<CXX_COMPILER_ID:MSVC>:_USE_MATH_DEFINES>;\$<\$<CXX_COMPILER_ID:MSVC>:_VARIADIC_MAX=10>;\$<\$<CXX_COMPILER_ID:MSVC>:_SCL_SECURE_NO_WARNINGS>"
  INTERFACE_COMPILE_FEATURES "cxx_auto_type;cxx_range_for;cxx_rvalue_references;cxx_right_angle_brackets;cxx_static_assert"
  INTERFACE_COMPILE_OPTIONS "-fopenmp;\$<\$<CXX_COMPILER_ID:GNU>:>;\$<\$<CXX_COMPILER_ID:Clang>:>;\$<\$<CXX_COMPILER_ID:MSVC>:/bigobj>;\$<\$<CXX_COMPILER_ID:MSVC>:/wd4715>"
  INTERFACE_INCLUDE_DIRECTORIES "/usr/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "\$<\$<CXX_COMPILER_ID:GNU>:-fopenmp>;\$<\$<CXX_COMPILER_ID:Clang>:-fopenmp>;\$<\$<CXX_COMPILER_ID:Intel>:-fopenmp>"
  INTERFACE_SYSTEM_INCLUDE_DIRECTORIES "/usr/include"
)

I also could not find a reference in that file to vexcl. But I can just search for that, too.

ddemidov commented 3 years ago

Could you provide a minimal CMakeLists.txt and a .cu file that would let me reproduce this?

Also, could you try to compile solver_cuda target from amgcl examples on your system, using CMakeLists.txt from amgcl (it is compiled from examples/solver.cpp, cmake copies it to solver.cu first)? Just to see if that works, or it is indeed an nvcc version problem.

VexCL targets are built like this: https://github.com/ddemidov/amgcl/blob/720f6b7bc325a8a87df9ae33da125a80069b1d5c/examples/CMakeLists.txt#L71-L92

vexcl_add_executables is a helper function (from vexcl cmake) that creates multiple targets linked with VexCL::OpenCL, VexCL::CUDA, and the other backends supported by the current system: https://github.com/ddemidov/vexcl/blob/5097dc6b6614235dcf2ce59f402e63618dfbaab1/cmake/VexCLTools.cmake#L4-L27

blattms commented 3 years ago

Will do next week. Have a nice weekend Dennis