Building openmp on Apple Silicon fails with missing symbol `___divdc3`

[keith]

This is a new issue for the same error described here https://github.com/llvm/llvm-project/issues/54596 but this is a more common use case.

The core issue is that openmp relies on ___divdc3 and ___divsc3 which when building normally on macOS implicitly come from the compiler_rt that is shipped with Xcode. You can see this by linking libomp.dylib and producing a map file:

[ 36] /Applications/Xcode-15.4.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.osx.a[arm64][47](divdc3.c.o)
[ 37] /Applications/Xcode-15.4.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.osx.a[arm64][53](divsc3.c.o)

When you build in the release process, the multi-stage nature of those builds is such that we do not produce a compiler rt alongside the clang that is used in the final stage of the build, and therefore the omp link cannot implicitly pick up this library. This results in the failure.

There are a few ways to reproduce this, using the test-release script is one, but a probably easier method is to build clang as usual with cmake, making sure not to build compiler rt, then point a second cmake build directory to your first clang like this:

CC=/Users/ksmiley/dev/llvm/llvm-project/final/Phase2/Release/llvmCore-18.1.8-final.install/usr/local/bin/clang CXX=/Users/ksmiley/dev/llvm/llvm-project/final/Phase2/Release/llvmCore-18.1.8-final.install/usr/local/bin/clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_ENABLE_RUNTIMES= ../llvm --fresh

Then run ninja omp and you will see this error:

Undefined symbols for architecture arm64:
  "___divdc3", referenced from:
      ___kmpc_atomic_cmplx8_div in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx8_div in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div_cmplx8 in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div_cmplx8 in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx8_div_cpt in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx8_div_cpt in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx10_div_cpt in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx10_div_cpt in kmp_atomic.cpp.o
      ...
  "___divsc3", referenced from:
      ___kmpc_atomic_cmplx4_div in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div_cpt in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div_cpt in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div_cpt in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div_cpt in kmp_atomic.cpp.o
      ...

I verified that the fix referenced in the other issue https://github.com/llvm/llvm-project/pull/83767 does fix this issue @ HEAD. This still repros @ HEAD today.

This blocks producing macOS arm64 release binaries that include openmp.

[keith]

cc @shiltian

[llvmbot]

@llvm/issue-subscribers-openmp

Author: Keith Smiley (keith)

This is a new issue for the same error described here https://github.com/llvm/llvm-project/issues/54596 but this is a more common use case. The core issue is that openmp relies on `___divdc3` and `___divsc3` which when building normally on macOS implicitly come from the compiler_rt that is shipped with Xcode, you can see this by linking `libomp.dylib` and producing a map file: ``` [ 36] /Applications/Xcode-15.4.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.osx.a[arm64][47](divdc3.c.o) [ 37] /Applications/Xcode-15.4.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.osx.a[arm64][53](divsc3.c.o) ``` When you build in the release process, the multi-stage nature of those builds is such that we do not produce a compiler rt alongside the clang that is used in thee final stage of the build, and therefore the omp link cannot implicitly pick up this library. This results in the failure. There are a few ways to reproduce this, using the test-release script is one, but a probably easier method is to build clang as usual with cmake, making sure _not_ to build compiler rt, then point a second cmake build directory to your first clang like this: ``` CC=/Users/ksmiley/dev/llvm/llvm-project/final/Phase2/Release/llvmCore-18.1.8-final.install/usr/local/bin/clang CXX=/Users/ksmiley/dev/llvm/llvm-project/final/Phase2/Release/llvmCore-18.1.8-final.install/usr/local/bin/clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_ENABLE_RUNTIMES= ../llvm --fresh ``` Then run `ninja omp` and you will see this error: ``` Undefined symbols for architecture arm64: "___divdc3", referenced from: ___kmpc_atomic_cmplx8_div in kmp_atomic.cpp.o ___kmpc_atomic_cmplx8_div in kmp_atomic.cpp.o ___kmpc_atomic_cmplx4_div_cmplx8 in kmp_atomic.cpp.o ___kmpc_atomic_cmplx4_div_cmplx8 in kmp_atomic.cpp.o ___kmpc_atomic_cmplx8_div_cpt in kmp_atomic.cpp.o ___kmpc_atomic_cmplx8_div_cpt in kmp_atomic.cpp.o ___kmpc_atomic_cmplx10_div_cpt in kmp_atomic.cpp.o ___kmpc_atomic_cmplx10_div_cpt in kmp_atomic.cpp.o ... "___divsc3", referenced from: ___kmpc_atomic_cmplx4_div in kmp_atomic.cpp.o ___kmpc_atomic_cmplx4_div in kmp_atomic.cpp.o ___kmpc_atomic_cmplx4_div in kmp_atomic.cpp.o ___kmpc_atomic_cmplx4_div_cpt in kmp_atomic.cpp.o ___kmpc_atomic_cmplx4_div_cpt in kmp_atomic.cpp.o ___kmpc_atomic_cmplx4_div_cpt in kmp_atomic.cpp.o ___kmpc_atomic_cmplx4_div_cpt in kmp_atomic.cpp.o ... ``` I verified that the fix referenced in the other issue https://github.com/llvm/llvm-project/pull/83767 does fix this issue @ HEAD. This still repros @ [HEAD](https://github.com/llvm/llvm-project/commit/33f3ebc86e7d3afcb65c551feba5bbc2421b42ed) today. This blocks producing macOS arm64 release binaries that include openmp.

[jhuber6]

Presumably you'd need -DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp and then make sure that the compile jobs are using -rtlib=compiler-rt. That will build compiler-rt before openmp and then use it when compiling openmp.

[keith]

Yea that makes sense. I guess my question is if we should be doing that in the release process because of this, or if there's a different solution we should prefer (potentially like the PR linked above)

[shiltian]

I'd suggest to build libomp using system/Xcode LLVM instead of the in-tree LLVM (or phase one LLVM). Don't use LLVM_ENABLE_RUNTIMES on macOS because there is no point of using in-tree LLVM here. If we do need to use phase one LLVM, build compiler-rt in phase one as well.

[keith]

I assumed there was a strong reason to do the split phases during the release binary production? Can you elaborate on why you think that shouldn't be necessary? Or are you suggesting splitting out omp specifically and continuing to build everything else with the phaseN compiler and only exclude omp from that?

[keith]

Looks like flang actually has this issue too, at least on the 18.x branch:

: && /Users/ksmiley/dev/llvm-project/final/Phase2/Release/llvmCore-18.1.7-final.install/usr/local/bin/clang++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -Wno-deprecated-copy -Wno-string-conversion -Wno-ctad-maybe-unsupported -Wno-unused-command-line-argument -Wstring-conversion           -Wcovered-switch-default -Wno-nested-anon-types -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode-15.4.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk -mmacosx-version-min=11.0 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -Wl,-dead_strip tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Allocatable.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/ArrayConstructor.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/BufferTest.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/CharacterTest.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/CommandTest.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Complex.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/CrashHandlerFixture.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Derived.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/ExternalIOTest.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Format.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Inquiry.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/ListInputTest.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/LogicalFormatTest.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Matmul.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/MatmulTranspose.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/MiscIntrinsic.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Namelist.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Numeric.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/NumericalFormatTest.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Pointer.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Ragged.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Random.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Reduction.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/RuntimeCrashTest.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Stop.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Time.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/TemporaryStack.cpp.o tools/flang/unittests/Runtime/CMakeFiles/FlangRuntimeTests.dir/Transformational.cpp.o -o tools/flang/unittests/Runtime/FlangRuntimeTests  lib/libLLVMSupport.a  lib/libllvm_gtest_main.a  lib/libllvm_gtest.a  lib/libFortranRuntime.a  lib/libLLVMSupport.a  -lm  /Applications/Xcode-15.4.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk/usr/lib/libz.tbd  /opt/homebrew/lib/libzstd.a  /Applications/Xcode-15.4.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk/usr/lib/libcurses.tbd  lib/libLLVMDemangle.a  -lpthread  lib/libFortranDecimal.a && :
ld: warning: ignoring duplicate libraries: 'lib/libLLVMSupport.a'
Undefined symbols for architecture arm64:
  "___divdc3", referenced from:
      double complex tgpowi<double complex, int>(double complex, int) in libFortranRuntime.a[9](complex-powi.cpp.o)
      double complex tgpowi<double complex, long long>(double complex, long long) in libFortranRuntime.a[9](complex-powi.cpp.o)
  "___divsc3", referenced from:
      float complex tgpowi<float complex, int>(float complex, int) in libFortranRuntime.a[9](complex-powi.cpp.o)
      float complex tgpowi<float complex, long long>(float complex, long long) in libFortranRuntime.a[9](complex-powi.cpp.o)

[shiltian]

I understand the reason to do split phases. The complication here is, the LLVM used to build libomp doesn't have compiler-rt. libomp on no matter macOS or Linux requires compiler-rt or something similar (which is libgcc.so on Linux). That's why on Linux it doesn't matter whether compiler-rt is available in the second phase build, because if it is not (which is probably always the case), it has libgcc.so. On macOS, it seems like compiler-rt is not available in the previous phase LLVM. This LLVM is also not able to (or should not) use compiler-rt from the system LLVM. That's why I suggest, either use the system LLVM to use libomp, or enable compiler-rt in previous phase LLVM build.