switch to using openmp target in cmake

[mkstoyanov]

Co-authored-by: ahojukka5@gmail.com

Switch to using OpenMP target inside CMake

closes #46

[mkstoyanov]

@ahojukka5 can you verify that this fixes the issue? I'll merge it as soon as you do.

[ahojukka5]

Unfortunately, looks like some more work is still needed.

So yes, the compile error did vanish, so this modification works. But I just noticed that Cmake is finding the wrong compiler (I was a bit wondering already why CMAKE_CXX_COMPILER_ID is GNU ...):

17:39 $ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -D Heffte_ENABLE_AVX=ON -D Heffte_ENABLE_FFTW=ON -D Heffte_ENABLE_PYTHON=ON
-- The CXX compiler identification is GNU 7.5.0
...

I guess the right compiler should be

17:44 $ which CC
/opt/cray/pe/craype/2.7.23/bin/CC
✔ ~/dev/heffte/heffte [pr-47 L|✚ 1…1] 
18:33 $ CC --version
Cray clang version 16.0.1  (6d4824324d375100ba18ca639dfc956fe6546d06)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/16.0.1/cce-clang/x86_64/share/../bin

So if I compile again, this time with right compiler:

18:34 $ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -D Heffte_ENABLE_AVX=ON -D Heffte_ENABLE_FFTW=ON -D Heffte_ENABLE_PYTHON=ON -D CMAKE_CXX_COMPILER=CC -D MPI_CXX_COMPILER=CC
-- The CXX compiler identification is Clang 16.0.6
-- Cray Programming Environment 2.7.23 CXX
...

We are back in the beginning, or in the "else" part of the if-else branch:

[ 19%] Linking CXX executable convolution
ld.lld: error: undefined reference due to --no-allow-shlib-undefined: omp_get_thread_num
>>> referenced by /opt/cray/pe/fftw/3.3.10.5/x86_milan/lib/libfftw3_omp.so

ld.lld: error: undefined reference due to --no-allow-shlib-undefined: omp_get_num_threads
>>> referenced by /opt/cray/pe/fftw/3.3.10.5/x86_milan/lib/libfftw3_omp.so

However, if I add, as you proposed:

if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" OR "${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang") ...

It will actually compile. Can you clarify, what you mean by writing

... enabling OpenMP for clang will use the Intel libiomp symbols while FFTW will require the libgomp symbols

? This is just a suggestion, but could this issue be solved by having a separate option to choose whether to use OpenMP or not. And if this option is not explicitly set, then string comparison is used to determine the correct usage (in automatic manner)? So we could have something like

# respect user provided FFTW_USE_OPENMP
if (NOT DEFINED FFTW_USE_OPENMP)
    message(STATUS "FFTW_USE_OPENMP is not defined, will search for OpenMP if CMAKE_CXX_COMPILER_ID is GNU")
    if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
        message(STATUS "CMAKE_CXX_COMPILER_ID is GNU, will search for OpenMP")
        set(FFTW_USE_OPENMP TRUE)
    else()
        message(STATUS "CMAKE_CXX_COMPILER_ID is not GNU, will not search for OpenMP")
        set(FFTW_USE_OPENMP FALSE)
    endif()
else()
    message(STATUS "FFTW_USE_OPENMP is defined, will search for OpenMP if FFTW_USE_OPENMP is TRUE")
    message(STATUS "FFTW_USE_OPENMP = ${FFTW_USE_OPENMP}")
endif()

and

...
    if (FFTW_USE_OPENMP)
        find_package(OpenMP REQUIRED)
        list(APPEND FFTW_LIBRARIES OpenMP::OpenMP_CXX)
    else()
...

[mkstoyanov]

I think we are converging to the solution here.

GNU compilers use libgomp.so for OpenMP while Clang uses libiomp.so

The question is which compiler was used for libfftw_omp. On the Cray machine that you have, I suppose they used their own Cray compiler, hence using the Clang OpenMP version works just fine.

The latest commit that I pushed will always assume that if fftw_omp is present then it is using the OpenMP from the same CXX compiler. I added an option that will override that and allow to manually select the library and mix-and-match between the compilers.

[mkstoyanov]

I confirm that this is now compiling in the Cray environment. But is this now also changing the default behavior of Cmake file? The consequence would be that e.g. before mentioned ROCm clang compiler users need to explicitly define -lgomp from now on to compile software.

Doing a mix-and-match between the compilers should be treated as the exception rather than the rule. I tested on a docker environment and Ubuntu seems to pickup the correct dependencies even without the -lgomp. I don't even remember who reported this long time ago but I'm sure it was on some exotic system.