search

LLNL / RAJA

RAJA Performance Portability Layer (C++)
BSD 3-Clause "New" or "Revised" License
484 stars 102 forks source link

why won't CAMP set ENABLE_TARGET_OPENMP? #1145

Open jeffhammond opened 3 years ago

jeffhammond commented 3 years ago

I must be stupid, because I set -DENABLE_TARGET_OPENMP=On or -DENABLE_TARGET_OPENMP=1 and CAMP does not set it. I'd like some sort of diagnostic here, rather than just having the build fail and having to print-debug the preprocessor.

jehammond@dgx-a100-math:/local/jehammond/RAJA/raja/build$ git clean -dfx ; cmake .. -DCMAKE_C_COMPILER=nvc -DCMAKE_CXX_COMPILER=nvc++ -DCMAKE_CXX_FLAGS="-mp -target=gpu"  -DENABLE_TARGET_OPENMP=On && make
warning: failed to remove ./: Invalid argument
Removing ./cmake_install.cmake
Removing ./RAJA.pc
Removing ./tests
Removing ./test
Removing ./CMakeFiles
Removing ./CTestTestfile.cmake
Removing ./tpl
Removing ./lib
Removing ./share
Removing ./Testing
Removing ./exercises
Removing ./bin
Removing ./include
Removing ./DartConfiguration.tcl
Removing ./CMakeCache.txt
Removing ./Makefile
Removing ./examples
Removing ./compile_commands.json
Removing ./blt
-- The CXX compiler identification is NVHPC 21.9.0
-- The C compiler identification is NVHPC 21.9.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvc++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test COMPILER_SUPPORTS_c++17
-- Performing Test COMPILER_SUPPORTS_c++17 - Success
Using C++ standard: c++17
-- BLT Version: 0.4.1
-- CMake Version: 3.21.1
-- CMake Executable: /usr/local/bin/cmake
CMake Deprecation Warning at blt/SetupBLT.cmake:73 (cmake_policy):
  The OLD behavior for policy CMP0076 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.
Call Stack (most recent call first):
  CMakeLists.txt:167 (include)

-- Found Git: /usr/bin/git (found version "2.25.1") 
-- Git Support is ON
-- Git Executable: /usr/bin/git
-- Git Version: 2.25.1
-- MPI Support is OFF
-- OpenMP Support is On
-- Found OpenMP_C: -mp  
-- Found OpenMP_CXX: -mp  
-- Found OpenMP: TRUE   
-- OpenMP Compile Flags: -mp
-- OpenMP Link Flags:    -mp
-- CUDA Support is Off
-- HIP Support is OFF
-- HCC Support is OFF
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Sphinx support is ON
-- Failed to locate Sphinx executable (missing: SPHINX_EXECUTABLE) 
-- Valgrind support is ON
-- Failed to locate Valgrind executable (missing: VALGRIND_EXECUTABLE) 
-- AStyle support is ON
-- Failed to locate AStyle executable (missing: ASTYLE_EXECUTABLE) 
-- ClangFormat support is ON
-- Failed to locate ClangFormat executable (missing: CLANGFORMAT_EXECUTABLE) 
-- Uncrustify support is ON
-- Failed to locate Uncrustify executable (missing: UNCRUSTIFY_EXECUTABLE) 
-- Yapf support is ON
-- Failed to locate Yapf executable (missing: YAPF_EXECUTABLE) 
-- CMakeFormat support is ON
-- Failed to locate CMakeFormat executable (missing: CMAKEFORMAT_EXECUTABLE) 
-- Cppcheck support is ON
-- Failed to locate Cppcheck executable (missing: CPPCHECK_EXECUTABLE) 
-- ClangQuery support is ON
-- Failed to locate ClangQuery executable (missing: CLANGQUERY_EXECUTABLE) 
-- ClangTidy support is ON
-- Failed to locate ClangTidy executable (missing: CLANGTIDY_EXECUTABLE) 
-- C Compiler family not set!!!
-- Adding optional BLT definitions and compiler flags
-- Standard C++17 selected
-- Enabling all compiler warnings on all targets.
-- Fortran support disabled.
-- CMAKE_C_FLAGS flags are:    -Wall -Wextra 
-- CMAKE_CXX_FLAGS flags are:  -mp -target=gpu     -Wall -Wextra 
-- CMAKE_EXE_LINKER_FLAGS flags are:  
-- Google Test Support is ON
-- Google Mock Support is OFF
CMake Deprecation Warning at blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/CMakeLists.txt:56 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- Found PythonInterp: /usr/bin/python (found version "3.8.5") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- OpenMP Enabled
-- Desul Atomics support is OFF
-- Using RAJA CAMP submodule.
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for std::aligned_alloc
-- Looking for std::aligned_alloc - not found
-- Looking for _mm_malloc
-- Looking for _mm_malloc - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /local/jehammond/RAJA/raja/build
Scanning dependencies of target RAJA
[  0%] Building CXX object CMakeFiles/RAJA.dir/src/AlignedRangeIndexSetBuilders.cpp.o
"/local/jehammond/RAJA/raja/tpl/camp/include/camp/defines.hpp", line 117: warning: #warning directive: NO ENABLE_TARGET_OPENMP
  #warning NO ENABLE_TARGET_OPENMP
   ^

"/local/jehammond/RAJA/raja/tpl/camp/include/camp/resource.hpp", line 41: warning: #warning directive: WTF
  #warning WTF
   ^

"/local/jehammond/RAJA/raja/include/RAJA/util/macros.hpp", line 133: warning: variable "errtemp" was set but never used
    const char * errtemp = nullptr;
                 ^

[  0%] Building CXX object CMakeFiles/RAJA.dir/src/DepGraphNode.cpp.o
"/local/jehammond/RAJA/raja/tpl/camp/include/camp/defines.hpp", line 117: warning: #warning directive: NO ENABLE_TARGET_OPENMP
  #warning NO ENABLE_TARGET_OPENMP
   ^

[  0%] Building CXX object CMakeFiles/RAJA.dir/src/LockFreeIndexSetBuilders.cpp.o
"/local/jehammond/RAJA/raja/tpl/camp/include/camp/defines.hpp", line 117: warning: #warning directive: NO ENABLE_TARGET_OPENMP
  #warning NO ENABLE_TARGET_OPENMP
   ^

"/local/jehammond/RAJA/raja/tpl/camp/include/camp/resource.hpp", line 41: warning: #warning directive: WTF
  #warning WTF
   ^

"/local/jehammond/RAJA/raja/include/RAJA/util/macros.hpp", line 133: warning: variable "errtemp" was set but never used
    const char * errtemp = nullptr;
                 ^

[  0%] Building CXX object CMakeFiles/RAJA.dir/src/MemUtils_CUDA.cpp.o
[  0%] Building CXX object CMakeFiles/RAJA.dir/src/MemUtils_HIP.cpp.o
[  0%] Building CXX object CMakeFiles/RAJA.dir/src/MemUtils_SYCL.cpp.o
[  1%] Building CXX object CMakeFiles/RAJA.dir/src/PluginStrategy.cpp.o
"/local/jehammond/RAJA/raja/include/RAJA/util/macros.hpp", line 133: warning: variable "errtemp" was set but never used
    const char * errtemp = nullptr;
                 ^

"/local/jehammond/RAJA/raja/tpl/camp/include/camp/defines.hpp", line 117: warning: #warning directive: NO ENABLE_TARGET_OPENMP
  #warning NO ENABLE_TARGET_OPENMP
   ^

"/local/jehammond/RAJA/raja/tpl/camp/include/camp/resource.hpp", line 41: warning: #warning directive: WTF
  #warning WTF
   ^

[  1%] Linking CXX static library lib/libRAJA.a
[  1%] Built target RAJA
Scanning dependencies of target gtest
[  1%] Building CXX object blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[  1%] Linking CXX static library ../../../../lib/libgtest.a
[  1%] Built target gtest
Scanning dependencies of target gtest_main
[  1%] Building CXX object blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[  1%] Linking CXX static library ../../../../lib/libgtest_main.a
[  1%] Built target gtest_main
Scanning dependencies of target blt_gtest_smoke
[  2%] Building CXX object blt/tests/smoke/CMakeFiles/blt_gtest_smoke.dir/blt_gtest_smoke.cpp.o
[  2%] Linking CXX executable ../../../tests/blt_gtest_smoke
[  2%] Built target blt_gtest_smoke
Scanning dependencies of target blt_openmp_smoke
[  2%] Building CXX object blt/tests/smoke/CMakeFiles/blt_openmp_smoke.dir/blt_openmp_smoke.cpp.o
[  2%] Linking CXX executable ../../../tests/blt_openmp_smoke
[  2%] Built target blt_openmp_smoke
Scanning dependencies of target test-plugin-forall-OpenMP.exe
[  2%] Building CXX object test/integration/plugin/CMakeFiles/test-plugin-forall-OpenMP.exe.dir/test-plugin-forall-OpenMP.cpp.o
"/local/jehammond/RAJA/raja/tpl/camp/include/camp/defines.hpp", line 117: warning: #warning directive: NO ENABLE_TARGET_OPENMP
  #warning NO ENABLE_TARGET_OPENMP
   ^

"/local/jehammond/RAJA/raja/include/RAJA/util/macros.hpp", line 133: warning: variable "errtemp" was set but never used
    const char * errtemp = nullptr;
                 ^

"/local/jehammond/RAJA/raja/tpl/camp/include/camp/resource.hpp", line 41: warning: #warning directive: WTF
  #warning WTF
   ^

"/local/jehammond/RAJA/raja/include/RAJA/util/resource.hpp", line 114: error: namespace "camp::resources" has no member "Omp"
      using type = camp::resources::Omp;
                                    ^

"/local/jehammond/RAJA/raja/include/RAJA/util/resource.hpp", line 119: error: namespace "camp::resources" has no member "Omp"
      using type = camp::resources::Omp;
                                    ^

"/local/jehammond/RAJA/raja/include/RAJA/util/resource.hpp", line 124: error: namespace "camp::resources" has no member "Omp"
      using type = camp::resources::Omp;
                                    ^

"/local/jehammond/RAJA/raja/include/RAJA/util/resource.hpp", line 129: error: namespace "camp::resources" has no member "Omp"
      using type = camp::resources::Omp;
                                    ^

"/local/jehammond/RAJA/raja/include/RAJA/util/resource.hpp", line 134: error: namespace "camp::resources" has no member "Omp"
      using type = camp::resources::Omp;
                                    ^

"/local/jehammond/RAJA/raja/include/RAJA/util/resource.hpp", line 154: error: namespace "RAJA::resources" has no member "Omp"
      template <> struct is_resource<resources::Omp> : std::true_type {};
                                                ^

"/local/jehammond/RAJA/raja/include/RAJA/policy/openmp_target/forall.hpp", line 35: error: namespace "RAJA::resources" has no member "Omp"
  RAJA_INLINE resources::EventProxy<resources::Omp> forall_impl(resources::Omp omp_res,
                                               ^

"/local/jehammond/RAJA/raja/include/RAJA/policy/openmp_target/forall.hpp", line 35: error: namespace "RAJA::resources" has no member "Omp"
  RAJA_INLINE resources::EventProxy<resources::Omp> forall_impl(resources::Omp omp_res,
                                                                           ^

"/local/jehammond/RAJA/raja/include/RAJA/policy/openmp_target/forall.hpp", line 75: error: namespace "RAJA::resources" has no member "Omp"
  RAJA_INLINE resources::EventProxy<resources::Omp> forall_impl(resources::Omp omp_res,
                                               ^

"/local/jehammond/RAJA/raja/include/RAJA/policy/openmp_target/forall.hpp", line 75: error: namespace "RAJA::resources" has no member "Omp"
  RAJA_INLINE resources::EventProxy<resources::Omp> forall_impl(resources::Omp omp_res,
                                                                           ^

"/local/jehammond/RAJA/raja/test/include/RAJA_test-camp.hpp", line 38: error: namespace "camp::resources" has no member "Omp"
  using OpenMPTargetResourceList = camp::list<camp::resources::Omp>;
                                                               ^

11 errors detected in the compilation of "/local/jehammond/RAJA/raja/build/test/integration/plugin/test-plugin-forall-OpenMP.cpp".
make[2]: *** [test/integration/plugin/CMakeFiles/test-plugin-forall-OpenMP.exe.dir/build.make:75: test/integration/plugin/CMakeFiles/test-plugin-forall-OpenMP.exe.dir/test-plugin-forall-OpenMP.cpp.o] Error 2
make[1]: *** [CMakeFiles/Makefile2:3193: test/integration/plugin/CMakeFiles/test-plugin-forall-OpenMP.exe.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

my pathetic attempts to debug this

jehammond@dgx-a100-math:/local/jehammond/RAJA/raja/build$ git diff --submodule=diff
Submodule tpl/camp contains modified content
diff --git a/tpl/camp/include/camp/defines.hpp b/tpl/camp/include/camp/defines.hpp
index 502107b..4ccf3e3 100644
--- a/tpl/camp/include/camp/defines.hpp
+++ b/tpl/camp/include/camp/defines.hpp
@@ -107,11 +107,14 @@ namespace camp

 #if defined(ENABLE_TARGET_OPENMP)
 #if _OPENMP >= 201511
+#warning HELLO
 #define CAMP_HAVE_OMP_OFFLOAD 1
 #else
-#define CAMP_HAVE_OMP_OFFLOAD 0
-#warning Compiler does NOT support OpenMP Target Offload even though user has enabled it!
+#error Compiler does NOT support OpenMP Target Offload even though user has enabled it!
+//#define CAMP_HAVE_OMP_OFFLOAD 0
 #endif
+#else
+#warning NO ENABLE_TARGET_OPENMP
 #endif

 // This works for:
diff --git a/tpl/camp/include/camp/resource.hpp b/tpl/camp/include/camp/resource.hpp
index e9dfdc1..8ad9977 100644
--- a/tpl/camp/include/camp/resource.hpp
+++ b/tpl/camp/include/camp/resource.hpp
@@ -31,7 +31,14 @@ http://github.com/llnl/camp
 #endif

 #if defined(CAMP_HAVE_OMP_OFFLOAD)
+#if CAMP_HAVE_OMP_OFFLOAD
+#warning TRUE
+#else
+#warning FALSE
+#endif
 #include "camp/resource/omp_target.hpp"
+#else
+#warning WTF
 #endif
davidbeckingsale commented 3 years ago

Hey @jeffhammond - setting -D on your Cmake line will only set that as a CMake option. Camp is looking for a compiler macro define. Can you try adding -DENABLE_TARGET_OPENMP to CMAKE_CXX_FLAGS?

jeffhammond commented 3 years ago

I just hard-coded it in the header, which is equivalent to the flags suggestion, and that works, but I wonder how RAJA OpenMP Target ever worked, unless everyone who uses it knows that CMake doesn't propagate this token.

rhornung67 commented 3 years ago

@jeffhammond actually, the RAJA OpenMP Target back-end is mostly for cussing at compilers and amusement purposes. Please no wagering.... :smile:

rchen20 commented 3 years ago

Just seeing this thread now, sorry about this . . . I added that check to prevent OpenMP Target from being built on machines without GPUs. We did not have the #if defined(ENABLE_TARGET_OPENMP) check before, and CAMP would attempt to build omp-target if the compiler was capable.

rchen20 commented 2 years ago

Hi @jeffhammond, does the latest camp/main (https://github.com/LLNL/camp) solve this problem for you? We put some recent changes into the Camp repo which hopefully have taken care of this, and you're welcome to try it out.