mstorsjo / llvm-mingw

An LLVM/Clang/LLD based mingw-w64 toolchain
Other
1.96k stars 187 forks source link

OpenMP fails with missing dlfcn library #298

Closed falhumai96 closed 1 year ago

falhumai96 commented 2 years ago

I am using Fedora 36.

Not sure what am I missing. Here is the sample out I get (I stripped the output from build-all.sh to just the output for configuring and building OpenMP):

-- The C compiler identification is Clang 14.0.0
-- The CXX compiler identification is Clang 14.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/falhumai96/Documents/Projects/Install/llvm-mingw/x86_64-unknown-linux-gnu/bin/i686-w64-mingw32-clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/falhumai96/Documents/Projects/Install/llvm-mingw/x86_64-unknown-linux-gnu/bin/i686-w64-mingw32-clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test OPENMP_HAVE_WALL_FLAG
-- Performing Test OPENMP_HAVE_WALL_FLAG - Success
-- Performing Test OPENMP_HAVE_WERROR_FLAG
-- Performing Test OPENMP_HAVE_WERROR_FLAG - Success
-- Performing Test OPENMP_HAVE_WCAST_QUAL_FLAG
-- Performing Test OPENMP_HAVE_WCAST_QUAL_FLAG - Success
-- Performing Test OPENMP_HAVE_WFORMAT_PEDANTIC_FLAG
-- Performing Test OPENMP_HAVE_WFORMAT_PEDANTIC_FLAG - Success
-- Performing Test OPENMP_HAVE_WIMPLICIT_FALLTHROUGH_FLAG
-- Performing Test OPENMP_HAVE_WIMPLICIT_FALLTHROUGH_FLAG - Success
-- Performing Test OPENMP_HAVE_WSIGN_COMPARE_FLAG
-- Performing Test OPENMP_HAVE_WSIGN_COMPARE_FLAG - Success
-- Performing Test OPENMP_HAVE_WNO_EXTRA_FLAG
-- Performing Test OPENMP_HAVE_WNO_EXTRA_FLAG - Success
-- Performing Test OPENMP_HAVE_WNO_PEDANTIC_FLAG
-- Performing Test OPENMP_HAVE_WNO_PEDANTIC_FLAG - Success
-- Performing Test OPENMP_HAVE_WNO_MAYBE_UNINITIALIZED_FLAG
-- Performing Test OPENMP_HAVE_WNO_MAYBE_UNINITIALIZED_FLAG - Failed
-- Performing Test OPENMP_HAVE_STD_CPP14_FLAG
-- Performing Test OPENMP_HAVE_STD_CPP14_FLAG - Success
-- Found PythonInterp: /usr/bin/python (found version "3.10.5") 
-- Cannot find llvm-lit.
-- Please put llvm-lit in your PATH, set OPENMP_LLVM_LIT_EXECUTABLE to its full path, or point OPENMP_LLVM_TOOLS_DIR to its directory.
CMake Warning at cmake/OpenMPTesting.cmake:22 (message):
  The check targets will not be available!
Call Stack (most recent call first):
  cmake/OpenMPTesting.cmake:51 (find_standalone_test_dependencies)
  CMakeLists.txt:54 (include)

-- The ASM_MASM compiler identification is unknown
-- Found assembler: /home/falhumai96/Documents/Projects/Install/llvm-mingw/x86_64-unknown-linux-gnu/bin/llvm-ml
-- Performing Test LIBOMP_HAVE_FNO_EXCEPTIONS_FLAG
-- Performing Test LIBOMP_HAVE_FNO_EXCEPTIONS_FLAG - Success
-- Performing Test LIBOMP_HAVE_FNO_RTTI_FLAG
-- Performing Test LIBOMP_HAVE_FNO_RTTI_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_CLASS_MEMACCESS_FLAG
-- Performing Test LIBOMP_HAVE_WNO_CLASS_MEMACCESS_FLAG - Failed
-- Performing Test LIBOMP_HAVE_WNO_COVERED_SWITCH_DEFAULT_FLAG
-- Performing Test LIBOMP_HAVE_WNO_COVERED_SWITCH_DEFAULT_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_FRAME_ADDRESS_FLAG
-- Performing Test LIBOMP_HAVE_WNO_FRAME_ADDRESS_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_STRICT_ALIASING_FLAG
-- Performing Test LIBOMP_HAVE_WNO_STRICT_ALIASING_FLAG - Success
-- Performing Test LIBOMP_HAVE_WSTRINGOP_OVERFLOW_FLAG
-- Performing Test LIBOMP_HAVE_WSTRINGOP_OVERFLOW_FLAG - Failed
-- Performing Test LIBOMP_HAVE_WNO_STRINGOP_TRUNCATION_FLAG
-- Performing Test LIBOMP_HAVE_WNO_STRINGOP_TRUNCATION_FLAG - Failed
-- Performing Test LIBOMP_HAVE_WNO_SWITCH_FLAG
-- Performing Test LIBOMP_HAVE_WNO_SWITCH_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_UNINITIALIZED_FLAG
-- Performing Test LIBOMP_HAVE_WNO_UNINITIALIZED_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_UNUSED_BUT_SET_VARIABLE_FLAG
-- Performing Test LIBOMP_HAVE_WNO_UNUSED_BUT_SET_VARIABLE_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_RETURN_TYPE_C_LINKAGE_FLAG
-- Performing Test LIBOMP_HAVE_WNO_RETURN_TYPE_C_LINKAGE_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_CAST_QUAL_FLAG
-- Performing Test LIBOMP_HAVE_WNO_CAST_QUAL_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_INT_TO_VOID_POINTER_CAST_FLAG
-- Performing Test LIBOMP_HAVE_WNO_INT_TO_VOID_POINTER_CAST_FLAG - Success
-- Performing Test LIBOMP_HAVE_MSSE2_FLAG
-- Performing Test LIBOMP_HAVE_MSSE2_FLAG - Success
-- Performing Test LIBOMP_HAVE_FTLS_MODEL_FLAG
-- Performing Test LIBOMP_HAVE_FTLS_MODEL_FLAG - Success
-- Performing Test LIBOMP_HAVE_MMIC_FLAG
-- Performing Test LIBOMP_HAVE_MMIC_FLAG - Failed
-- Performing Test LIBOMP_HAVE_M32_FLAG
-- Performing Test LIBOMP_HAVE_M32_FLAG - Success
-- Performing Test LIBOMP_HAVE_MRTM_FLAG
-- Performing Test LIBOMP_HAVE_MRTM_FLAG - Success
-- Looking for shm_open
-- Looking for shm_open - not found
-- Looking for shm_open
-- Looking for shm_open - not found
-- Looking for xmmintrin.h
-- Looking for xmmintrin.h - found
-- Performing Test LIBOMP_HAVE__MM_MALLOC
-- Performing Test LIBOMP_HAVE__MM_MALLOC - Success
-- Looking for aligned_alloc
-- Looking for aligned_alloc - not found
-- Looking for posix_memalign
-- Looking for posix_memalign - not found
-- Looking for _aligned_malloc
-- Looking for _aligned_malloc - found
-- Performing Test LIBOMP_HAVE_SAFESEH_FLAG
-- Performing Test LIBOMP_HAVE_SAFESEH_FLAG - Failed
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Looking for immintrin.h
-- Looking for immintrin.h - found
-- Performing Test LIBOMP_HAVE_ATTRIBUTE_RTM
-- Performing Test LIBOMP_HAVE_ATTRIBUTE_RTM - Success
-- Performing Test LIBOMP_HAVE_ATTRIBUTE_WAITPKG
-- Performing Test LIBOMP_HAVE_ATTRIBUTE_WAITPKG - Success
-- Performing Test LIBOMP_HAVE_WAITPKG_INTRINSICS
-- Performing Test LIBOMP_HAVE_WAITPKG_INTRINSICS - Success
-- Performing Test LIBOMP_HAVE_RTM_INTRINSICS
-- Performing Test LIBOMP_HAVE_RTM_INTRINSICS - Success
-- Found Perl: /usr/bin/perl (found version "5.34.1") 
-- Performing Test LIBOMP_HAVE_VERSION_SYMBOLS
-- Performing Test LIBOMP_HAVE_VERSION_SYMBOLS - Failed
-- Performing Test LIBOMP_HAVE___BUILTIN_FRAME_ADDRESS
-- Performing Test LIBOMP_HAVE___BUILTIN_FRAME_ADDRESS - Success
-- Performing Test LIBOMP_HAVE_WEAK_ATTRIBUTE
-- Performing Test LIBOMP_HAVE_WEAK_ATTRIBUTE - Success
-- Performing Test LIBOMP_HAVE_PSAPI
-- Performing Test LIBOMP_HAVE_PSAPI - Success
-- LIBOMP: Operating System     -- Windows
-- LIBOMP: Target Architecture  -- i386
-- LIBOMP: Build Type           -- Release
-- LIBOMP: Library Kind         -- SHARED
-- LIBOMP: Library Type         -- normal
-- LIBOMP: Fortran Modules      -- FALSE
-- LIBOMP: Build                -- 20140926
-- LIBOMP: Use Stats-gathering  -- FALSE
-- LIBOMP: Use Debugger-support -- FALSE
-- LIBOMP: Use ITT notify       -- TRUE
-- LIBOMP: Use OMPT-support     -- FALSE
-- LIBOMP: Use OMPD-support     -- FALSE
-- LIBOMP: Use Adaptive locks   -- TRUE
-- LIBOMP: Use quad precision   -- FALSE
-- LIBOMP: Use Hwloc library    -- FALSE
-- Looking for sqrt in m
-- Looking for sqrt in m - found
-- Looking for __atomic_load_1
-- Looking for __atomic_load_1 - not found
-- Looking for __atomic_load_1 in atomic
-- Looking for __atomic_load_1 in atomic - not found
-- check-libomp does nothing.
-- check-ompt does nothing.
-- check-openmp does nothing.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/falhumai96/Documents/Projects/Src/llvm-mingw/llvm-project/openmp/build-i686
[29/36] Building ASM_MASM object runtime/src/CMakeFiles/omp.dir/z_Windows_NT-586_asm.asm.obj
llvm-ml: warning: ignoring unsupported 'coff' option
[36/36] Linking C shared library runtime/src/libomp.dll
FAILED: runtime/src/libomp.dll runtime/src/libomp.dll.a 
: && /home/falhumai96/Documents/Projects/Install/llvm-mingw/x86_64-unknown-linux-gnu/bin/i686-w64-mingw32-clang -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-extra -Wno-pedantic -O3 -DNDEBUG  /home/falhumai96/Documents/Projects/Src/llvm-mingw/llvm-project/openmp/build-i686/runtime/src/libomp.def -msse2 -shared -o runtime/src/libomp.dll -Wl,--out-implib,runtime/src/libomp.dll.a -Wl,--major-image-version,5,--minor-image-version,0 runtime/src/CMakeFiles/omp.dir/kmp_alloc.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_atomic.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_csupport.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_debug.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_itt.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_environment.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_error.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_global.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_i18n.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_io.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_runtime.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_settings.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_str.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_tasking.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_threadprivate.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_utility.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_barrier.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_wait_release.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_affinity.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_dispatch.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_lock.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_sched.cpp.obj runtime/src/CMakeFiles/omp.dir/z_Windows_NT_util.cpp.obj runtime/src/CMakeFiles/omp.dir/z_Windows_NT-586_util.cpp.obj runtime/src/CMakeFiles/omp.dir/thirdparty/ittnotify/ittnotify_static.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_taskdeps.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_cancel.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_ftn_cdecl.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_ftn_extra.cpp.obj runtime/src/CMakeFiles/omp.dir/kmp_version.cpp.obj runtime/src/CMakeFiles/omp.dir/z_Windows_NT-586_asm.asm.obj runtime/src/CMakeFiles/omp.dir/libomp.rc.res  -lpthread -lpsapi  -ldl  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && :
lld: error: unable to find library -ldl
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
longnguyen2004 commented 2 years ago

What's your CMake version?

mstorsjo commented 2 years ago

I reproduced this issue - it's caused by Fedora applying a custom, nonstandard patch on CMake.

In https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/openmp/runtime/src/CMakeLists.txt#L150-L153, the OpenMP runtime adds linking against ${CMAKE_DL_LIBS}, with the intent that if the platform requires linking against a non-default library to do dynamic loading, link against that library.

Fedora's CMake packaging includes a patch which sets CMAKE_DL_LIBS to dl on mingw platforms, despite the fact that there's no default shipped dl library on mingw platforms: https://src.fedoraproject.org/rpms/cmake/blob/rawhide/f/cmake-mingw-dl.patch

This issue has been discussed and argued upstream in CMake in https://gitlab.kitware.com/cmake/cmake/-/issues/17600 and https://gitlab.kitware.com/cmake/cmake/-/merge_requests/1642. The argument is that CMAKE_DL_LIBS is documented as Name of library containing dlopen and dlclose. - and if you want dlopen/dlclose on mingw, you need a third party library such as https://github.com/dlfcn-win32/dlfcn-win32.

The issue is that lots and lots of cmake code includes CMAKE_DL_LIBS even on platforms where they don't specifically are going to be calling dlopen and dlclose, as noted in the merge request above as it broke lots of code when the patch was tested out - the llvm OpenMP runtime seems to be yet another case of this.

The discussion hinted that third party code shouldn't be including CMAKE_DL_LIBS if they really aren't going to be calling dlopen anyway, and patches were written (but not merged) to this effect as part of that merge request, but the upstream merge request was closed.

Despite this, fedora still carries this disruptive patch downstream.

falhumai96 commented 2 years ago

What's your CMake version?

Apologies for the late reply. My CMake version is 3.22.2.

falhumai96 commented 2 years ago

I reproduced this issue - it's caused by Fedora applying a custom, nonstandard patch on CMake.

In https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/openmp/runtime/src/CMakeLists.txt#L150-L153, the OpenMP runtime adds linking against ${CMAKE_DL_LIBS}, with the intent that if the platform requires linking against a non-default library to do dynamic loading, link against that library.

Fedora's CMake packaging includes a patch which sets CMAKE_DL_LIBS to dl on mingw platforms, despite the fact that there's no default shipped dl library on mingw platforms: https://src.fedoraproject.org/rpms/cmake/blob/rawhide/f/cmake-mingw-dl.patch

This issue has been discussed and argued upstream in CMake in https://gitlab.kitware.com/cmake/cmake/-/issues/17600 and https://gitlab.kitware.com/cmake/cmake/-/merge_requests/1642. The argument is that CMAKE_DL_LIBS is documented as Name of library containing dlopen and dlclose. - and if you want dlopen/dlclose on mingw, you need a third party library such as https://github.com/dlfcn-win32/dlfcn-win32.

The issue is that lots and lots of cmake code includes CMAKE_DL_LIBS even on platforms where they don't specifically are going to be calling dlopen and dlclose, as noted in the merge request above as it broke lots of code when the patch was tested out - the llvm OpenMP runtime seems to be yet another case of this.

The discussion hinted that third party code shouldn't be including CMAKE_DL_LIBS if they really aren't going to be calling dlopen anyway, and patches were written (but not merged) to this effect as part of that merge request, but the upstream merge request was closed.

Despite this, fedora still carries this disruptive patch downstream.

I see. So, the best approach now I think is to download a custom CMake, build it, and use it (e.g., by modifying PATH in the build scripts to use the custom one) for building LLVM-MinGW, or there could be instructions on README to use a specific modified CMake for building setup by the developer.

falhumai96 commented 2 years ago

I suspected it's a dl wrapper required for MinGW.

mstorsjo commented 2 years ago

I see. So, the best approach now I think is to download a custom CMake, build it, and use it (e.g., by modifying PATH in the build scripts to use the custom one) for building LLVM-MinGW, or there could be instructions on README to use a specific modified CMake for building setup by the developer.

I guess that's the simplest solution for now. Alternatively, you can patch the openmp code to stop setting CMAKE_DL_LIBS - I posted https://reviews.llvm.org/D130892 which does that, but let's see how it fares.

mstorsjo commented 2 years ago

FWIW, the OpenMP patch was accepted and merged - let's see if I can get it backported into the upcoming 15.0.0 release too.

falhumai96 commented 2 years ago

FWIW, the OpenMP patch was accepted and merged - let's see if I can get it backported into the upcoming 15.0.0 release too.

Awesome. I guess we can close this issue once we can build on other distros such as Fedora.

mstorsjo commented 1 year ago

FWIW, the OpenMP patch was accepted and merged - let's see if I can get it backported into the upcoming 15.0.0 release too.

The fix was backported to 15.0.0 in the end: https://github.com/llvm/llvm-project/commit/bab8af8ea062f6332b5c5d13ae688bb8900f244a

Awesome. I guess we can close this issue once we can build on other distros such as Fedora.

Ok, closing it now.