search

parthenon-hpc-lab / athenapk

AthenaPK: a performance portable version of Athena++ built on Parthenon and Kokkos
BSD 3-Clause "New" or "Revised" License
53 stars 13 forks source link

KOKKOS_ENABLE_CUDA defined but the compiler is not defining the __CUDACC__ macro as expected #42

Closed tbhaxor closed 1 year ago

tbhaxor commented 1 year ago

Getting this error in <mesh/mesh.hpp> of kh.cpp file.

File external/Kokkos/core/src/setup/Kokkos_Setup_Cuda.hpp

#if defined(KOKKOS_ENABLE_CUDA) && !defined(__CUDACC__)
#error \
    "KOKKOS_ENABLE_CUDA defined but the compiler is not defining the __CUDACC__ macro as expected"
// Some tooling environments will still function better if we do this here.
#define __CUDACC__
#endif /* defined(KOKKOS_ENABLE_CUDA) && !defined(__CUDACC__) */

I have also tried -D__CUDACC__=external/Kokkos/bin/nvcc_wrapper, but still same error. Though code works fine and compiles without any error but it complains (false positive messages) in the code editor while working with the codebase.

BenWibking commented 1 year ago

__CUDACC__ is defined by the CUDA compiler. This most likely means there is an issue with finding the CUDA compiler. Can you post the output of the CMake configuration?

tbhaxor commented 1 year ago

The command I used to configure build

$ cmake -DKokkos_ARCH_TURING75=ON -DKokkos_ENABLE_CUDA=ON -DCUDA_ROOT=/opt/cuda \
  -DCMAKE_CXX_COMPILER=external/Kokkos/bin/nvcc_wrapper \
  -DCMAKE_CXX_FLAGS:STRING="-I/opt/cuda/targets/x86_64-linux/include/ -L/opt/cuda/targets/x86_64-linux/lib/" \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE -DCMAKE_BUILD_TYPE=Release -Bbuild -G Ninja

This is the output

CMake Warning at external/parthenon/cmake/MachineCfg.cmake:61 (message):
  Not using any machine configuration.  Consider creating a configuration
  file following the examples in /cmake/machine_cfgs/ and then point the
  MACHINE_CFG cmake or environment variable to your custom file.Note, that
  the machine file can be placed in any directory (also outside the repo).
Call Stack (most recent call first):
  CMakeLists.txt:9 (include)

-- The CXX compiler identification is GNU 11.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /mnt/Projects/athenapk/external/Kokkos/bin/nvcc_wrapper - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- clang-format --version: 15.0.7
-- black --version: black, 23.1.0 (compiled: yes)
Python (CPython) 3.10.9

-- Found MPI_CXX: /usr/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1") found components: CXX 
CMake Warning at external/parthenon/cmake/MachineCfg.cmake:61 (message):
  Not using any machine configuration.  Consider creating a configuration
  file following the examples in /mnt/Projects/athenapk/cmake/machine_cfgs/
  and then point the MACHINE_CFG cmake or environment variable to your custom
  file.Note, that the machine file can be placed in any directory (also
  outside the repo).
Call Stack (most recent call first):
  external/parthenon/CMakeLists.txt:21 (include)

-- The C compiler identification is GNU 12.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- black --version: black, 23.1.0 (compiled: yes)
Python (CPython) 3.10.9

-- Found HDF5: /usr/lib/libhdf5.so;/usr/lib/libsz.so;/usr/lib/libz.so;/usr/lib/libdl.a;/usr/lib/libm.so (found version "1.12.2") found components: C 
-- Setting default Kokkos CXX standard to 17
-- The project name is: Kokkos
-- Compiler Version: 11.8.89
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=c++17 for C++17 standard as feature
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::Cuda
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
-- 
-- Architectures:
--  TURING75
-- Found CUDAToolkit: /opt/cuda/include (found version "11.8.89") 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found TPLCUDA: TRUE  
-- Found TPLLIBDL: /usr/include  
-- Using internal desul_atomics copy
-- Kokkos Devices: CUDA;SERIAL, Kokkos Backends: CUDA;SERIAL
-- Using Kokkos source from Kokkos_ROOT=/mnt/Projects/athenapk/external/Kokkos
-- PAR_LOOP_LAYOUT='MANUAL1D_LOOP' (default par_for wrapper layout)
-- PAR_LOOP_INNER_LAYOUT='TVR_INNER_LOOP' (default par_for_inner wrapper layout)
-- Found Python3: /usr/bin/python3.10 (found version "3.10.9") found components: Interpreter 
-- Found Git: /usr/bin/git (found version "2.39.2") 
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
-- Checking for Python modules (numpy;unyt;matplotlib;h5py;scipy) required for regression tests.
   Check can be disabled by setting PARTHENON_ENABLE_PYTHON_MODULE_CHECK=OFF but then
   tests are not guaranteed to work anymore.
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/build

EDIT with /usr/bin/g++-11 as compiler

CMake Warning at external/parthenon/cmake/MachineCfg.cmake:61 (message):
  Not using any machine configuration.  Consider creating a configuration
  file following the examples in /cmake/machine_cfgs/ and then point the
  MACHINE_CFG cmake or environment variable to your custom file.Note, that
  the machine file can be placed in any directory (also outside the repo).
Call Stack (most recent call first):
  CMakeLists.txt:9 (include)

-- The CXX compiler identification is GNU 11.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++-11 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- clang-format --version: 15.0.7
-- black --version: black, 23.1.0 (compiled: yes)
Python (CPython) 3.10.9

-- Found MPI_CXX: /usr/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1") found components: CXX 
CMake Warning at external/parthenon/cmake/MachineCfg.cmake:61 (message):
  Not using any machine configuration.  Consider creating a configuration
  file following the examples in /mnt/Projects/athenapk/cmake/machine_cfgs/
  and then point the MACHINE_CFG cmake or environment variable to your custom
  file.Note, that the machine file can be placed in any directory (also
  outside the repo).
Call Stack (most recent call first):
  external/parthenon/CMakeLists.txt:21 (include)

-- The C compiler identification is GNU 12.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- black --version: black, 23.1.0 (compiled: yes)
Python (CPython) 3.10.9

-- Found HDF5: /usr/lib/libhdf5.so;/usr/lib/libsz.so;/usr/lib/libz.so;/usr/lib/libdl.a;/usr/lib/libm.so (found version "1.12.2") found components: C 
-- Setting default Kokkos CXX standard to 17
-- The project name is: Kokkos
-- Compiler Version: 11.8.89
-- kokkos_launch_compiler (/mnt/Projects/athenapk/external/Kokkos/bin/kokkos_launch_compiler) is enabled...
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=c++17 for C++17 standard as feature
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::Cuda
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
-- 
-- Architectures:
--  TURING75
-- Found CUDAToolkit: /opt/cuda/include (found version "11.8.89") 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found TPLCUDA: TRUE  
-- Found TPLLIBDL: /usr/include  
-- Using internal desul_atomics copy
-- Kokkos Devices: CUDA;SERIAL, Kokkos Backends: CUDA;SERIAL
-- Using Kokkos source from Kokkos_ROOT=/mnt/Projects/athenapk/external/Kokkos
-- PAR_LOOP_LAYOUT='MANUAL1D_LOOP' (default par_for wrapper layout)
-- PAR_LOOP_INNER_LAYOUT='TVR_INNER_LOOP' (default par_for_inner wrapper layout)
-- Found Python3: /usr/bin/python3.10 (found version "3.10.9") found components: Interpreter 
-- Found Git: /usr/bin/git (found version "2.39.2") 
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
-- Checking for Python modules (numpy;unyt;matplotlib;h5py;scipy) required for regression tests.
   Check can be disabled by setting PARTHENON_ENABLE_PYTHON_MODULE_CHECK=OFF but then
   tests are not guaranteed to work anymore.
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/build
BenWibking commented 1 year ago

-DCMAKE_CXX_COMPILER should point to the host compiler (g++), not the CUDA compiler or wrapper.

tbhaxor commented 1 year ago

-DCMAKE_CXX_COMPILER should point to the host compiler (g++), not the CUDA compiler or wrapper.

Yes, with g++ It is not working so I had to use g++-11. Still same error. nvcc_wrapper I did it after in search of fix

These are cache variables from the initial build (i did it weeks ago)

$ grep -n CMAKE_CXX_COMPILER CMakeCache.txt
48:CMAKE_CXX_COMPILER:FILEPATH=/usr/bin/g++-11
52:CMAKE_CXX_COMPILER_AR:FILEPATH=/usr/bin/gcc-ar-11
56:CMAKE_CXX_COMPILER_RANLIB:FILEPATH=/usr/bin/gcc-ranlib-11
1115://ADVANCED property for variable: CMAKE_CXX_COMPILER
1116:CMAKE_CXX_COMPILER-ADVANCED:INTERNAL=1
1117://ADVANCED property for variable: CMAKE_CXX_COMPILER_AR
1118:CMAKE_CXX_COMPILER_AR-ADVANCED:INTERNAL=1
1119://ADVANCED property for variable: CMAKE_CXX_COMPILER_RANLIB
1120:CMAKE_CXX_COMPILER_RANLIB-ADVANCED:INTERNAL=1
BenWibking commented 1 year ago

What does your CMake configure output look like now?

tbhaxor commented 1 year ago

CMake configure output look like now?

You mean console output? that is already shared here https://github.com/parthenon-hpc-lab/athenapk/issues/42#issuecomment-1482902073

BenWibking commented 1 year ago

That has the wrong setting for CMAKE_CXX_COMPILER.

tbhaxor commented 1 year ago

That has the wrong setting for CMAKE_CXX_COMPILER.

@BenWibking Ok here is with g++ compiler

$ cmake -DKokkos_ARCH_TURING75=ON -DKokkos_ENABLE_CUDA=ON -DCUDA_ROOT=/opt/cuda \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-11 \
  -DCMAKE_CXX_FLAGS:STRING="-I/opt/cuda/targets/x86_64-linux/include/ -L/opt/cuda/targets/x86_64-linux/lib/" \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE -DCMAKE_BUILD_TYPE=Release -Bbuild -G Ninja
CMake Warning at external/parthenon/cmake/MachineCfg.cmake:61 (message):
  Not using any machine configuration.  Consider creating a configuration
  file following the examples in /cmake/machine_cfgs/ and then point the
  MACHINE_CFG cmake or environment variable to your custom file.Note, that
  the machine file can be placed in any directory (also outside the repo).
Call Stack (most recent call first):
  CMakeLists.txt:9 (include)

-- The CXX compiler identification is GNU 11.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++-11 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- clang-format --version: 15.0.7
-- black --version: black, 23.1.0 (compiled: yes)
Python (CPython) 3.10.9

-- Found MPI_CXX: /usr/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1") found components: CXX 
CMake Warning at external/parthenon/cmake/MachineCfg.cmake:61 (message):
  Not using any machine configuration.  Consider creating a configuration
  file following the examples in /mnt/Projects/athenapk/cmake/machine_cfgs/
  and then point the MACHINE_CFG cmake or environment variable to your custom
  file.Note, that the machine file can be placed in any directory (also
  outside the repo).
Call Stack (most recent call first):
  external/parthenon/CMakeLists.txt:21 (include)

-- The C compiler identification is GNU 12.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- black --version: black, 23.1.0 (compiled: yes)
Python (CPython) 3.10.9

-- Found HDF5: /usr/lib/libhdf5.so;/usr/lib/libsz.so;/usr/lib/libz.so;/usr/lib/libdl.a;/usr/lib/libm.so (found version "1.12.2") found components: C 
-- Setting default Kokkos CXX standard to 17
-- The project name is: Kokkos
-- Compiler Version: 11.8.89
-- kokkos_launch_compiler (/mnt/Projects/athenapk/external/Kokkos/bin/kokkos_launch_compiler) is enabled...
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=c++17 for C++17 standard as feature
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::Cuda
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
-- 
-- Architectures:
--  TURING75
-- Found CUDAToolkit: /opt/cuda/include (found version "11.8.89") 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found TPLCUDA: TRUE  
-- Found TPLLIBDL: /usr/include  
-- Using internal desul_atomics copy
-- Kokkos Devices: CUDA;SERIAL, Kokkos Backends: CUDA;SERIAL
-- Using Kokkos source from Kokkos_ROOT=/mnt/Projects/athenapk/external/Kokkos
-- PAR_LOOP_LAYOUT='MANUAL1D_LOOP' (default par_for wrapper layout)
-- PAR_LOOP_INNER_LAYOUT='TVR_INNER_LOOP' (default par_for_inner wrapper layout)
-- Found Python3: /usr/bin/python3.10 (found version "3.10.9") found components: Interpreter 
-- Found Git: /usr/bin/git (found version "2.39.2") 
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
-- Checking for Python modules (numpy;unyt;matplotlib;h5py;scipy) required for regression tests.
   Check can be disabled by setting PARTHENON_ENABLE_PYTHON_MODULE_CHECK=OFF but then
   tests are not guaranteed to work anymore.
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/build
BenWibking commented 1 year ago

Can you try deleting these flags: -DCMAKE_CXX_FLAGS:STRING="-I/opt/cuda/targets/x86_64-linux/include/ -L/opt/cuda/targets/x86_64-linux/lib/"

tbhaxor commented 1 year ago

Still same error

tbhaxor commented 1 year ago

Did you get error? I have tried it with microsoft c++ extension, clangd in vscode and also in qtcreator, just thinking maybe it is because of editor or language server specific but same error in all of them

BenWibking commented 1 year ago

No, I haven't seen this error before.

tbhaxor commented 1 year ago

Which code editor you use?

BenWibking commented 1 year ago

I use emacs, but I'm not sure how that's relevant here. Are you building from the command line? I assumed so, since you listed:

$ cmake -DKokkos_ARCH_TURING75=ON -DKokkos_ENABLE_CUDA=ON -DCUDA_ROOT=/opt/cuda \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-11 \
  -DCMAKE_CXX_FLAGS:STRING="-I/opt/cuda/targets/x86_64-linux/include/ -L/opt/cuda/targets/x86_64-linux/lib/" \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE -DCMAKE_BUILD_TYPE=Release -Bbuild -G Ninja

for your CMake configure step.

tbhaxor commented 1 year ago

Yes

tbhaxor commented 1 year ago

@BenWibking does this relate to the current problem https://github.com/kokkos/kokkos/issues/5139

BenWibking commented 1 year ago

I am able to build successfully with 

$ cmake .. -DKokkos_ARCH_VOLTA70=ON -DKokkos_ENABLE_CUDA=ON -GNinja

I get:

-- Setting default Kokkos CXX standard to 17
-- The project name is: Kokkos
-- Compiler Version: 11.8.89
-- kokkos_launch_compiler (/mnt/home/wibkingb/test/athenapk/external/Kokkos/bin/kokkos_launch_compiler) is enabled...
-- Using -std=c++17 for C++17 standard as feature
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::Cuda
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
--
-- Architectures:
--  VOLTA70
-- Using internal desul_atomics copy
-- Kokkos Devices: CUDA;SERIAL, Kokkos Backends: CUDA;SERIAL
-- Using Kokkos source from Kokkos_ROOT=/mnt/home/wibkingb/test/athenapk/external/Kokkos
-- PAR_LOOP_LAYOUT='MANUAL1D_LOOP' (default par_for wrapper layout)
-- PAR_LOOP_INNER_LAYOUT='TVR_INNER_LOOP' (default par_for_inner wrapper layout)
-- Checking for Python modules (numpy;unyt;matplotlib;h5py;scipy) required for regression tests.
   Check can be disabled by setting PARTHENON_ENABLE_PYTHON_MODULE_CHECK=OFF but then
   tests are not guaranteed to work anymore.
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/home/wibkingb/test/athenapk/build

For some reason I do not understand, your CMake configuration does not have the line

-- kokkos_launch_compiler (/mnt/home/wibkingb/test/athenapk/external/Kokkos/bin/kokkos_launch_compiler) is enabled...

I think this is the issue. I can't reproduce it, though, so I don't know how to fix it.

tbhaxor commented 1 year ago

@BenWibking It is there

Screenshot_20230326_093320

tbhaxor commented 1 year ago

@BenWibking Ohk problem is solved, I don't know why it took me so long to solve it. But the indexing engine was using clangd. I switch to g++ and now there are no errors.

Please close this once you acknowledge this comment.

tbhaxor commented 1 year ago

Also code works fine, this is some IDE problem.