[Kokkos] Build system does not support Fujitsu Compiler in clang mode

[andreipoe]

Summary

The Fujitsu compiler for the A64FX has two modes of operations: trad (the default), in which it uses a proprietary frontend and CLI flags, and clang, based on the clang frontend. To use clang mode, all commands needs to include -Nclang and clang (not trad) flags. The current BUDE Kokkos build system doesn't support a way to invoke the Fujitsu compiler in clang mode.

What should happen

The user should be able to specify the compiler to be either FCC (for trad mode) or FCC -Nclang (for clang mode). The selected mode needs to be used for both compiling objects and linking.

What actually happens

Including -Nclang in CXX_EXTRA_FLAGS results in CMake detecting (and using) FCC in the default trad mode:

$ cmake -DCMAKE_CXX_COMPILER='FCC'  -DCXX_EXTRA_FLAGS=-Nclang ...
-- The CXX compiler identification is Fujitsu

$ make VERBOSE=1
[  4%] Building CXX object kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_CPUDiscovery.cpp.o
/opt/FJSVstclanga/cp-1.0.20.04/bin/FCC -I... -fopenmp -march=armv8.2-a+sve -std=c++14 -o ...

Notice that the -Nclang flag is not included above. This will lead to objects being compiled in trad mode. The linker, however, will be passed the CXX_EXTRA_FLAGS, and so will try to link in clang mode, which will fail with unresolved symbols in the standard library.

On the other hand, manually setting CXXFLAGS detects the compiler as clang and applies the right flags:

$ CXX=FCC CXXFLAGS=-Nclang cmake ...
-- The CXX compiler identification is Clang 7.1.0

$ make VERBOSE=1
[  4%] Building CXX object kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_CPUDiscovery.cpp.o
/opt/FJSVstclanga/cp-1.0.20.04/bin/FCC -I...  -Nclang --mcpu=a64fx -O3 -fopenmp=libomp ...

This leads to a complete build, but setting the environment variable this way completely overrides the flags passed to CXX_EXTRA_FLAGS, which is used in our portability scripts.

Setting CMAKE_CXX_COMPILER='FCC -Nclang' is not accepted by CMake.

Proposed fix

Ideally, the CXX_EXTRA_FLAGS should be passed to CMake early, so that it detects FCC in clang mode successfully. If this is not possible, then we will need to have a documented workaround for this specific compiler...

[tom91136]

We now have a FORWARD_CXX_EXTRA_FLAGS_TO_KOKKOS flag for forwarding CXX_EXTRA_FLAGS to Kokkos, the option defaults to OFF.

This is done for compatibility reasons as Kokkos sets flags based on -DKokkos_*, if we unconditionally forward CXX_EXTRA_FLAGS, it might break the build with duplicate or incompatible options.

We can revisit this if we ever need to pass completely different flags for Kokkos and Bude, although I don't see why this would be required.