Update desul and assess atomics

[rhornung67]

• [x] Build up atomic capability in RAJA using desul atomics to coincide with existing atomic capabilities.
• [x] Build up tests of desul atomics to coincide with existing atomic tests. Make sure we have all atomic operations covered and both sets of atomics pass all the tests.
• [ ] Assess performance, interface differences for RAJA users, etc.
• [ ] If performance is acceptable, make desul atomics the default (need to update desul submodule!) and deprecate the existing ones at the next release and remove at the release following that.

This should be done after https://github.com/LLNL/RAJA/issues/1515

[rhornung67]

RAJA API does not expose scope or memory order features.

Notion of "sequential" atomic now exists.

Discuss how to stage deployment/API changes to users.

[rchen20]

@rhornung67 @trws The latest Desul+BLT+RAJA+Camp combination works on RZANSEL when OpenMP=Off (and still works everywhere else when OpenMP=On, i.e. EAS3, TOSS4 Intel).

Notes for future Desul debugging:

• Works on all platforms when Desul=Off.
• RZANSEL fails with kernel launch failures and seg faults when built with OpenMP=On + nvcc/10.1.243 + gcc/8.3.1.
• RZANSEL fails with garbage answers when built with OpenMP=On + nvcc/11.8.0 + gcc/11.2.1.
• Failing tests are:

      174 - test-forall-atomic-basic-Cuda.exe (Failed)
      175 - test-forall-atomic-basic-unsigned-Cuda.exe (Failed)
      180 - test-forall-AtomicView-Cuda.exe (Failed)
      181 - test-forall-AtomicMultiView-Cuda.exe (Failed)
      195 - test-forall-AtomicRefAdd-Cuda.exe (Failed)
      196 - test-forall-AtomicRefSub-Cuda.exe (Failed)
      198 - test-forall-AtomicRefCAS-Cuda.exe (Failed)
      199 - test-forall-AtomicRefMinMax-Cuda.exe (Failed)
      200 - test-forall-AtomicRefLogical-Cuda.exe (Failed)
      221 - test-kernel-basic-fission-fusion-loop-Cuda.exe (Failed)
      224 - test-kernel-conditional-fission-fusion-loop-Cuda.exe (Failed)
      682 - test-atomic-incdec.exe (Failed)
      685 - test-atomic-ref-exchanges.exe (Failed)
      686 - test-atomic-ref-addsub.exe (Failed)
      687 - test-atomic-ref-minmax.exe (Failed)
      688 - test-atomic-ref-bitwise.exe (Failed)

[trws]

Is this with or without the patch to raja usage of desul that properly propagates its OpenMP flag? #1563

[rchen20]

Is this with or without the patch to raja usage of desul that properly propagates its OpenMP flag? #1563

@trws This is without #1563, do we want to merge that into raja/develop?

[rhornung67]

@rchen20 we do not want to merge that PR into develop. It contains hacks in desul, BLT, and RAJA I believe that we do not want as well as changes to GitLab CI configuration that we do not want. I think we only want the OpenMP flag thing that @trws asked about.

[rchen20]

Ah I see, we just want the change to RAJA's CMakeLists.txt. Let me give that a try.

[trws]

Yeah, it should just be that. All the other ones are ensuring it got the newer blt version.

[rchen20]

Yeah, it should just be that. All the other ones are ensuring it got the newer blt version.

@trws It worked after I also changed some of the Desul CMake. Here's a rough sketch of what I changed:

• RAJA CMakeLists.txt from #1563.
• Desul CMakeLists.txt from https://github.com/desul/desul/pull/117/
• Desul atomics/CMakeLists.txt from https://github.com/desul/desul/pull/117/

The latter 2 Desul modifications are mostly to ensure that the OpenMP flag is recognized and properly set, and I excluded the changes to desul/cmake/blt_boilerplate.cmake. Do you want me to create PRs in RAJA and Desul for this fix? I feel like I may have missed something, or created some extra CMake flags.

[trws]

Thanks for that reminder @rchen20, I left that one sitting when we were waiting on the BLT update. I'll ask Damien about getting a review so we can move that through. If there's something that should be fixed in 117, make a PR to that branch or post the modifications there and I'll add them in no problem. Thanks for chasing this down.