Cross-compiling libomp fails with missing symbol "___divdc3"

[ishmael1851]

Cross-compiling libomp for MacOSX on a Linux machine gives the following error. Any ideas? Source code was downloaded from here: https://github.com/llvm/llvm-project/releases/download/llvmorg-14.0.0/openmp-14.0.0.src.tar.xz

[35/36] Linking C shared library runtime/src/libomp.dylib
FAILED: runtime/src/libomp.dylib
: && /usr/local/osxcross/bin/../bin/aarch64-apple-darwin21.1-clang -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-extra -Wno-ped
antic -O3 -DNDEBUG -mmacosx-version-min=11.0 -dynamiclib -Wl,-headerpad_max_install_names -current_version 5.0 -compatibility_version 5.0 -o runtime/src/libomp.
dylib -install_name @rpath/libomp.dylib runtime/src/CMakeFiles/omp.dir/kmp_alloc.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_atomic.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_csupport.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_debug.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_itt.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_environment.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_error.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_global.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_i18n.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_io.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_runtime.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_settings.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_str.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_tasking.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_threadprivate.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_utility.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_barrier.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_wait_release.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_affinity.c
pp.o runtime/src/CMakeFiles/omp.dir/kmp_dispatch.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_lock.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_sched.cpp.o runtime/src/
CMakeFiles/omp.dir/z_Linux_util.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_gsupport.cpp.o runtime/src/CMakeFiles/omp.dir/thirdparty/ittnotify/ittnotify_static.cpp
.o runtime/src/CMakeFiles/omp.dir/kmp_taskdeps.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_cancel.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_ftn_cdecl.cpp.o runtime/
src/CMakeFiles/omp.dir/kmp_ftn_extra.cpp.o runtime/src/CMakeFiles/omp.dir/kmp_version.cpp.o runtime/src/CMakeFiles/omp.dir/ompt-general.cpp.o runtime/src/CMakeF
iles/omp.dir/z_Linux_asm.S.o   && cd /tmp/Build/runtime/src && /usr/local/bin/cmake -E create_symlink libomp.dylib libgomp.dylib && /usr/local/bin/cmake -E crea
te_symlink libomp.dylib libiomp5.dylib
Undefined symbols for architecture arm64:
  "___divdc3", referenced from:
      ___kmpc_atomic_cmplx8_div in kmp_atomic.cpp.o
      ___kmpc_atomic_cmplx4_div_cmplx8 in kmp_atomic.cpp.o
  "___divsc3", referenced from:
      ___kmpc_atomic_cmplx4_div in kmp_atomic.cpp.o
ld: symbol(s) not found for architecture arm64
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

[llvmbot]

@llvm/issue-subscribers-openmp

[AndreyChurbanov]

No idea what the problem might be. Googling shows people had similar problem about a year ago (https://github.com/tpoechtrager/osxcross/issues/267), and the suggestion was: either build natively or use pre-built binaries.

For native builds people have same linking problem when outdated xcode used, but I am not sure if it is related to cross build somehow.

[rorth]

I'm seeing the same issue natively, though: trying to build LLVM 16.0.0 rc1/rc2 on arm64-apple-darwin21.6.0 with Command Line Tools for Xcode 14.2 (the latest version) shows the same link error. When I continue with ninja -k0 check-all, the issue hits again:

FAILED: tools/flang/unittests/Runtime/FlangRuntimeTests
[...]
Undefined symbols for architecture arm64:
  "___divdc3", referenced from:
      double complex tgpowi<double complex, int>(double complex, int) in libFortranRuntime.a(complex-powi.cpp.o)
      double complex tgpowi<double complex, long long>(double complex, long long) in libFortranRuntime.a(complex-powi.cpp.o)
  "___divsc3", referenced from:
      float complex tgpowi<float complex, int>(float complex, int) in libFortranRuntime.a(complex-powi.cpp.o)
      float complex tgpowi<float complex, long long>(float complex, long long) in libFortranRuntime.a(complex-powi.cpp.o)
ld: symbol(s) not found for architecture arm64
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)

I've not yet investigated more closely, though.

[rorth]

What I've now found is that those missing symbols live in libclang_rt.osx.a: usually clang always links with that archive library implicitly, irrespective of creating an executable or dynamic library. For some weird reason, this doesn't happen when linking libomp.dylib.

I tried with a Debug build of llvm 16.0.0 rc3 to check what happens in clang/lib/Driver/ToolChains/Darwin.cpp:DarwinClang::AddLinkRuntimeLibArgs and AddLinkRuntimeLib(Args, CmdArgs, "builtins");. Unfortunately, I cannot debug clang with the bundled lldb in an ssh session:

error: process exited with status -1 (this is a non-interactive debug session, cannot get permission to debug processes.)

[h-vetinari]

I've encountered this with 18.1.0-rc1 in conda-forge, with a build recipe that has successfully cross-compiled previous versions (17, 16, 15, ...) for osx-arm64 from osx-64. It looks like somewhere a linkage to compiler-rt isn't being inserted correctly anymore...?

[shiltian]

We generally don't need any other project to build OpenMP but Apple Silicon platform is an exception because certain functions are not provided by the system. Compiler-rt happens to have them. I think we need to "copy" the implementation of those functions to OpenMP.

[Torrekie]

this happens when not linking libclang_rt, a manual way to solve this is add target clang_rt library to linker flags to where having this problem

[dbabokin]

Any plans to resolve this in the upcoming 18.1 release?

[shiltian]

It's not in the plan yet but I'll at least fix it in trunk soon. If that is merged before 18.1 release, we can cherry pick it.

[shiltian]

83767 can fix the issue. I think we can make it into 18.1 release.

[shiltian]

I did a detailed check. Now I think compiler-rt is required to compile libomp on macOS. Basically on a Linux system, libomp will link against libgcc.a, which is the compiler rt for GCC. For LLVM/OpenMP on macOS, we will need a clang that has compiler-rt. The system clang definitely has so there is no problem building OpenMP with it.

[Torrekie]

it is possible when self bootstrapping scene, at least I met this problem when bootstrapping on iOS, where host compiler have clang_rt but stage1 clang may not automatically passing it to linker, even there is a stage1 compiler-rt. But I only triggers this error when linking flang

[shiltian]

Does putting openmp into LLVM_ENABLE_RUNTIMES help?

Update: probably not since compiler-rt might not be installed into clang's directory.

[Torrekie]

Haven't tried that yet, but libunwind configure failed due to no target clang_rt path passed to its CMake args, while this sub project flags was not handled by user but the top Makefile/build.ninja

[shiltian]

You don't have to explicitly pass argument to use compiler-rt IIUC. As long as it is installed properly into ${llvm_install_dir}/lib/${version}/lib/${os}/, clang can pick it up automatically.

[Torrekie]

Then this might be the problem that clang_rt not installed to stage1 clang search path after build, thus affecting all stage2 subprojects that using it

[jprotze]

Does putting openmp into LLVM_ENABLE_RUNTIMES help?

Update: probably not since compiler-rt might not be installed into clang's directory.

It will have the same issue as building libomptarget (with c++ default rtl set to libc++) on a system where you don't have libc++ installed yet. compiler-rt and libc++ are also built as LLVM_ENABLE_RUNTIMES and not available at the time you build libomptarget.

[shiltian]

Yeah, the installation will only happen after the runtime build.

[keith]

I filed a new issue for this since it is still a problem https://github.com/llvm/llvm-project/issues/105997