OpenMP detection with Clang is broken in 2.6.0

[barracuda156]

--->  Configuring dbcsr
Executing:  cd "/opt/local/var/macports/build/_opt_svacchanda_SonomaPorts_math_dbcsr/dbcsr/work/build" && /opt/local/bin/cmake -G "Ninja" -DCMAKE_BUILD_TYPE=MacPorts -DCMAKE_INSTALL_PREFIX="/opt/local" -DCMAKE_INSTALL_NAME_DIR="/opt/local/lib" -DCMAKE_SYSTEM_PREFIX_PATH="/opt/local;/usr" -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" -DCMAKE_OBJC_COMPILER="$CC" -DCMAKE_OBJCXX_COMPILER="$CXX" -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_POLICY_DEFAULT_CMP0060=NEW -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_COLOR_MAKEFILE=ON -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_MAKE_PROGRAM=ninja -DCMAKE_MODULE_PATH="/opt/local/share/cmake/Modules" -DCMAKE_PREFIX_PATH="/opt/local/share/cmake/Modules" -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DCMAKE_INSTALL_RPATH="/opt/local/lib" -Wno-dev -DPython_EXECUTABLE=/opt/local/bin/python3.11 -DFYPP_EXECUTABLE=/opt/local/bin/fypp-3.11 -DUSE_MPI=ON -DUSE_OPENMP=ON -DUSE_SMM=blas -DWITH_C_API=ON -DWITH_CUDA_PROFILING=OFF -DBUILD_TESTING=ON -DTEST_OMP_THREADS=2 -DTEST_MPI_RANKS=4 -DWITH_EXAMPLES=OFF -DCMAKE_OSX_ARCHITECTURES="arm64" -DCMAKE_OSX_DEPLOYMENT_TARGET="14.0" -DCMAKE_OSX_SYSROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk" -DBLA_VENDOR=OpenBLAS -DBLAS_LIBRARIES=/opt/local/lib/libopenblas.dylib -DLAPACK_LIBRARIES=/opt/local/lib/libopenblas.dylib /opt/local/var/macports/build/_opt_svacchanda_SonomaPorts_math_dbcsr/dbcsr/work/dbcsr-2.6.0 
-- The C compiler identification is Clang 17.0.6
-- The CXX compiler identification is Clang 17.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /opt/local/bin/mpicc-mpich-clang17
-- Check for working C compiler: /opt/local/bin/mpicc-mpich-clang17 - works
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - failed
-- Check for working CXX compiler: /opt/local/bin/mpicxx-mpich-clang17
-- Check for working CXX compiler: /opt/local/bin/mpicxx-mpich-clang17 - works
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The Fortran compiler identification is GNU 13.2.0
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - no
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - failed
-- Check for working Fortran compiler: /opt/local/bin/mpif90-mpich-clang17
-- Check for working Fortran compiler: /opt/local/bin/mpif90-mpich-clang17 - works
-- Checking whether /opt/local/bin/mpif90-mpich-clang17 supports Fortran 90
-- Checking whether /opt/local/bin/mpif90-mpich-clang17 supports Fortran 90 - no
CMake Error at /opt/local/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
Call Stack (most recent call first):
  /opt/local/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /opt/local/share/cmake-3.28/Modules/FindOpenMP.cmake:580 (find_package_handle_standard_args)
  CMakeLists.txt:129 (find_package)

-- Configuring incomplete, errors occurred!
Command failed:  cd "/opt/local/var/macports/build/_opt_svacchanda_SonomaPorts_math_dbcsr/dbcsr/work/build" && /opt/local/bin/cmake -G "Ninja" -DCMAKE_BUILD_TYPE=MacPorts -DCMAKE_INSTALL_PREFIX="/opt/local" -DCMAKE_INSTALL_NAME_DIR="/opt/local/lib" -DCMAKE_SYSTEM_PREFIX_PATH="/opt/local;/usr" -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" -DCMAKE_OBJC_COMPILER="$CC" -DCMAKE_OBJCXX_COMPILER="$CXX" -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_POLICY_DEFAULT_CMP0060=NEW -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_COLOR_MAKEFILE=ON -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_MAKE_PROGRAM=ninja -DCMAKE_MODULE_PATH="/opt/local/share/cmake/Modules" -DCMAKE_PREFIX_PATH="/opt/local/share/cmake/Modules" -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DCMAKE_INSTALL_RPATH="/opt/local/lib" -Wno-dev -DPython_EXECUTABLE=/opt/local/bin/python3.11 -DFYPP_EXECUTABLE=/opt/local/bin/fypp-3.11 -DUSE_MPI=ON -DUSE_OPENMP=ON -DUSE_SMM=blas -DWITH_C_API=ON -DWITH_CUDA_PROFILING=OFF -DBUILD_TESTING=ON -DTEST_OMP_THREADS=2 -DTEST_MPI_RANKS=4 -DWITH_EXAMPLES=OFF -DCMAKE_OSX_ARCHITECTURES="arm64" -DCMAKE_OSX_DEPLOYMENT_TARGET="14.0" -DCMAKE_OSX_SYSROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk" -DBLA_VENDOR=OpenBLAS -DBLAS_LIBRARIES=/opt/local/lib/libopenblas.dylib -DLAPACK_LIBRARIES=/opt/local/lib/libopenblas.dylib /opt/local/var/macports/build/_opt_svacchanda_SonomaPorts_math_dbcsr/dbcsr/work/dbcsr-2.6.0 
Exit code: 1

Adding this sort of fix works: https://github.com/ValerioMa/moveit/commit/46d318bbe4208cf43299ac475fe7ec266b188e05 (perhaps only libomp is relevant for Clang, others can be dropped).

[hfp]

Is your Clang installed by package manager or self-built?

[alazzaro]

So I understand you want to mix LLVM (clang) for C/C++ and GNU (gfortran) for Fortran compilers. As you pointed out, in this way you mix the two OpenMP runtimes (libomp and libgomp, respectively), which is not good (see for example https://cpufun.substack.com/p/is-mixing-openmp-runtimes-safe ). For the same reason, the proposed solution would not work (here we have gfortran in the game). Although the safest solution would be to disalbe OpenMP in DBCSR, I wonder if we can escape the check on the OpenMP on C if we are not using C at all (only Fortran)... I will investigate more.

[barracuda156]

@alazzaro Thank you for responding!

It is not really I want, it is Macports default setting for Intel and arm64. I would be happy to avoid Clangs altogether and use GCC, like I do on PowerPC systems. But it is not something I can decide globally. (And there is substance to the argument to use Clang on newer macOS, since GCC is largely untested with libc++, and this is the OS runtime library. For Clang it is the default library, on the other hand. At the same time, LLVM fortran (flang / lfortran) compilers are, at least reportedly, not production-quality yet.)

To be honest I am not sure which OpenMP library gfortran links to (if to any at all) on systems using Clangs. I can check that. In practical terms using Clang with gfortran seems to work fine (configure problems aside).

[alazzaro]

Well, @dev-zero suggested a way to avoid the OpenMP C and C++ if we don't use ACCEL and C_API... I will try to implement it. However, there are two orthogonal problems here: CMAKE cannot find OpenMP (which is your issue) and DBCSR to mix LLVM and GNU OpenMP runtime (which is a DBCSR issue). For your case, I see people reporting that this is a Xcode issue (see https://discourse.cmake.org/t/how-to-find-openmp-with-clang-on-macos/8860/3 and https://mac.r-project.org/openmp/). So you can give a try?

For the DBCSR problem, I would like to enforce a better way to link OpenMPI libraries in DBCSR, but again this is not your primary issue... Of course, it would be great if we can use flang for compiling DBCSR, but it doesn't work yet...

[mtaillefumier]

gfortran links with the runtime included in the gcc distribution. I do not know if it is possible to change it. Clang might offer the possibility to switch between clang openmp runtime and the gcc implementation. Mixing the two implementations is surely not going to work in the long run if it even run at all (we tried).

according to the CMakeLists.txt, -DWITH_EXAMPLES=OFF -DUSE_OPENMP=OFF should be enough provided that GPU acceleration is not needed.

[alazzaro]

gfortran links with the runtime included in the gcc distribution. I do not know if it is possible to change it. Clang might offer the possibility to switch between clang openmp runtime and the gcc implementation. Mixing the two implementations is surely not going to work in the long run if it even run at all (we tried).

according to the CMakeLists.txt, -DWITH_EXAMPLES=OFF -DUSE_OPENMP=OFF should be enough provided that GPU acceleration is not needed.

Unfortunately no, even if C/C++ is not used, the findOpenMP still check for C and C++ OpenMP libraries... Need to protect for that case. Still, this is not the main issue of this ticket. The problem is that OpenMP is not found clang on macos, as reported at https://mac.r-project.org/openmp/.

Concerning OpenMP runtime in clang, as far as I know, libomp is the only supported (see https://openmp.llvm.org/design/Runtimes.html#openmp-runtimes ). Then, as pointed out at https://cpufun.substack.com/p/is-mixing-openmp-runtimes-safe , you can use libomp with gcc linking...

[barracuda156]

@mascguy @eborisch @cjones051073 Could someone please clarify how is our OpenMP supposed to work when Clang is used (with/without MPICH) and also Gfortran is? Which will be the case with almost everything on x86 and aarch64.

I.e., Clang is linking to libomp, while Gfortran apparently to libgomp, how does it work?

[mtaillefumier]

it is very risky to mix both runtimes. I would really advice against. Hint here https://cpufun.substack.com/p/is-mixing-openmp-runtimes-safe

the findopenmp issue is solved with this

find_package(OpenMP REQUIRED COMPONENTS Fortran)

by default find_package(OpenMP REQUIRED) will search for C and C++

[dev-zero]

@mtaillefumier yes, that's what Alfio and I have already planned: switch to requiring only the Fortran component, and the others only iff accelerator or C API is requested. The default macOS clang does not have OpenMP. Mixing OpenMP runtime libraries is not something we will support. Only enable building on macOS with a system clang (and without C API and accel support) and a brew/macports gcc with OpenMP.

[barracuda156]

@dev-zero Got it, thank you. I have switched dbcsr now to GCC, and it builds on most of systems: https://ports.macports.org/port/dbcsr/details (did not yet look through logs why some still fail).