search

LLNL / RAJA

RAJA Performance Portability Layer (C++)
BSD 3-Clause "New" or "Revised" License
484 stars 102 forks source link

Fix "exhaustive" atomic tests #1017

Open rhornung67 opened 3 years ago

rhornung67 commented 3 years ago

The tests: test-forall-atomic-basic-OpenMP.exe test-forall-AtomicRefMinMax-OpenMP.exe

fail when we build and run exhaustive tests. They work with the default test builds.

@rchen20 assigning this to you to look at when you have time. You won't see these test failures on the develop branch so they are likely due to me adding in new OpenMP execution policies to test. A PR for these changes is coming soon.

rhornung67 commented 2 years ago

@rchen20 is this still an issue? I tried a couple of compilers (gnu, clang) and everything passed.

rchen20 commented 2 years ago

I'll try this with XL because that's usually the most problematic compiler.

rchen20 commented 2 years ago

Strange, I'm running in to 2 different compilation issues with test-forall-atomic-basic-OpenMP.exe. One is the excessive recursion on our templating (saw this in https://github.com/LLNL/RAJA/pull/1165), the other is the camp::resources::Omp not found issue. I'll need to dig further in to why these are happening.

nvcc/10.1.243 + xl/2021.12.22 -

cd /usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-nvcc10.1.243-sm_70-xl2021.12.22/test/functional/forall/atomic-basic && /usr/tce/packages/cuda/cuda-10.1.243/bin/nvcc -ccbin=/usr/tce/packages/xl/xl-2021.12.22/bin/xlc++_r -DGTEST_HAS_DEATH_TEST=1 -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/include -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/functional/forall/atomic-basic/tests -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-nvcc10.1.243-sm_70-xl2021.12.22/include -I/usr/tce/packages/cuda/cuda-10.1.243/include -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/tpl/camp/include -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-nvcc10.1.243-sm_70-xl2021.12.22/tpl/camp/include -isystem=/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/include -isystem=/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/tpl/cub -restrict -arch sm_70 --expt-extended-lambda --expt-relaxed-constexpr -Xcudafe "--display_error_number" -O3 -Xcompiler -O3 -Xcompiler -qxlcompatmacros -Xcompiler -qalias=noansi -Xcompiler -qsmp=omp -Xcompiler -qhot -Xcompiler -qnoeh -Xcompiler -qsuppress=1500-029 -Xcompiler -qsuppress=1500-036 -Xcompiler=-fPIE -Xcompiler=-qsmp=omp -std=c++14 -x cu -c /usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-nvcc10.1.243-sm_70-xl2021.12.22/test/functional/forall/atomic-basic/test-forall-atomic-basic-OpenMP.cpp -o CMakeFiles/test-forall-atomic-basic-OpenMP.exe.dir/test-forall-atomic-basic-OpenMP.cpp.o

/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/include/gtest/internal/gtest-internal.h(731): error #456: excessive recursion at instantiation of function "testing::internal::TypeParameterizedTest<Fixture, TestSel, Types>::Register [with Fixture=ForallAtomicBasicTest, TestSel=testing::internal::TemplateSel, Types=testing::internal::Types<camp::list<RAJA::policy::omp::omp_parallel_for_guided_exec<-1>, RAJA::policy::cuda::cuda_atomic_explicit, camp::resources::v1::Host, RAJA::Index_type, unsigned long long>, camp::list<RAJA::policy::omp::omp_parallel_for_guided_exec<-1>, RAJA::policy::cuda::cuda_atomic_explicit, camp::resources::v1::Host, RAJA::Index_type, float>, camp::list<RAJA::policy::omp::omp_parallel_for_guided_exec<-1>

xl/2021.12.22_omptarget -

[ 66%] Building CXX object test/functional/forall/atomic-basic/CMakeFiles/test-forall-atomic-basic-OpenMP.exe.dir/test-forall-atomic-basic-OpenMP.cpp.o cd /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/test/functional/forall/atomic-basic && /usr/tce/packages/xl/xl-2021.12.22/bin/xlc++_r -+ -DGTEST_HAS_DEATH_TEST=1 -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/functional/forall/atomic-basic/tests -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/tpl/camp/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/tpl/camp/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/include -qthreaded -std=c++14 -O3 -qxlcompatmacros -qlanglvl=extended0x -qalias=noansi -qsmp=omp -qhot -qpic -qsuppress=1500-029 -qsuppress=1500-036 -qpic -qoffload -qsmp=omp -qalias=noansi -std=c++1y -o CMakeFiles/test-forall-atomic-basic-OpenMP.exe.dir/test-forall-atomic-basic-OpenMP.cpp.o -c /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/test/functional/forall/atomic-basic/test-forall-atomic-basic-OpenMP.cpp In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/test/functional/forall/atomic-basic/test-forall-atomic-basic-OpenMP.cpp:11: In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/include/RAJA_test-base.hpp:15: In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include/RAJA/RAJA.hpp:44: In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include/RAJA/pattern/forall.hpp:64: In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include/RAJA/policy/MultiPolicy.hpp:31: /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include/RAJA/util/resource.hpp:119:35: error: no type named 'Omp' in namespace 'camp::resources' using type = camp::resources::Omp;

rhornung67 commented 2 years ago

FWIW, I tried compiling earlier with multiple different versions of clang-ibm and OpenMP target enabled and saw a bunch of similar errors about no 'Omp' in 'camp::resources' namespace. I recall that there was an issue where ENABLE_OPENMP was not getting propagated to camp and I thought it was fixed a while back.

The compilation that I reported as successful above was on toss3. I just built with clang-ibm-12.0.1 on rzansel and everything compiled, but the OpenMP sort tests failed with segfaults. All tests pass with a couple of vanilla clang compiler versions I tried on rzansel. Build errors in our examples with xl/2021.12.22 coming from camp's tuple.hpp file -- unable to access private class members.

Is there something messed up in the CMake logic that prevents camp from having OpenMP enabled in some cases, are we not passing the right compiler flags for XL, or are XL compilers broken?

rchen20 commented 2 years ago

Adding -DCAMP_ENABLE_TARGET_OPENMP looks like it's helping; I'm just waiting for XL to build it . . .

Once that's confirmed to work, I'll put a PR up. Still not sure what we can do about the excessive recursion in the compiler though.

rhornung67 commented 2 years ago

should we add something to our CMake to set that flag when OpenMP target is enabled in RAJA?

rchen20 commented 2 years ago

Indeed, I'm looking in to that option!

davidbeckingsale commented 2 years ago

You can probably copy and tweak this snippet from Umpire:

https://github.com/LLNL/Umpire/blob/develop/src/umpire/tpl/CMakeLists.txt#L92-L97

rchen20 commented 2 years ago

Thanks @davidbeckingsale, that worked for the clang-omptarget tests. Still waiting on XL to see if the test works, but made a PR for this anyway because we should have it https://github.com/LLNL/RAJA/pull/1207.