search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
29.11k stars 12.01k forks source link

Build issue with openmp when using Polly flags (with stack trace) on LLVM-17 git #63448

Open ms178 opened 1 year ago

ms178 commented 1 year ago

While running a build for instrumenting for BOLT, I came across this build failure in openmp on LLVM/Clang-17 (7c2604ca196c3ba0247509c0fde350e23f0cccb0) when using Polly.

This is the script used for that specific stage that reproduces the issue. build_stage3-bolt-without-sampling.bash.txt

One additional note, normally I don't use Polly in the script and could not reproduce the issue without the Polly flags.

[2039/6224] Linking C shared library lib/libomp.so
FAILED: lib/libomp.so 
: && /home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/clang-17 -fPIC -O3 -march=native -mtune=native -maes -mllvm -inline-threshold=500 -mllvm -polly -mllvm -polly-position=early -mllvm -polly-dependences-computeout=50000000 -mllvm -polly-detect-profitability-min-per-loop-insts=40 -mllvm -polly-tiling=true -mllvm -polly-prevect-width=256 -mllvm -polly-vectorizer=stripmine -mllvm -polly-invariant-load-hoisting -mllvm -polly-loopfusion-greedy -mllvm -polly-run-inliner -mllvm -polly-run-dce -mllvm -polly-enable-delicm=true -mllvm -extra-vectorizer-passes -mllvm -enable-interleaved-mem-accesses -mllvm -enable-masked-interleaved-mem-accesses -mllvm -enable-cond-stores-vec -mllvm -slp-vectorize-hor-store -mllvm -enable-loopinterchange -mllvm -enable-loop-distribute -mllvm -enable-unroll-and-jam -mllvm -enable-loop-flatten -mllvm -interleave-small-loop-scalar-reduction -mllvm -unroll-runtime-multi-exit -mllvm -aggressive-ext-opt -fno-math-errno -fno-trapping-math -falign-functions=32 -fno-semantic-interposition -fomit-frame-pointer -mllvm -vp-counters-per-site=6 -fcf-protection=none -mharden-sls=none -flto=thin -fwhole-program-vtables -freroll-loops -mllvm -adce-remove-loops -mllvm -enable-ext-tsp-block-placement=1 -mllvm -enable-gvn-hoist -mllvm -enable-dfa-jump-thread -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-enum-constexpr-conversion -Wno-extra -Wno-pedantic -O3 -DNDEBUG  -Wl,--lto-O3,-O3,-Bsymbolic-functions,--as-needed -fuse-ld=lld -fcf-protection=none -mharden-sls=none -flto=thin -fwhole-program-vtables -Wl,--thinlto-jobs=all -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld -Wl,--color-diagnostics  -Wl,--as-needed -Wl,--version-script=/home/marcus/toolchain/llvm/llvm-project/openmp/runtime/src/exports_so.txt -static-libgcc -Wl,-z,noexecstack  -Xlinker --dependency-file -Xlinker projects/openmp/runtime/src/CMakeFiles/omp.dir/link.d -shared -Wl,-soname,libomp.so -o lib/libomp.so projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_alloc.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_atomic.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_csupport.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_debug.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_itt.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_environment.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_error.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_global.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_i18n.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_io.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_runtime.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_settings.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_str.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_tasking.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_threadprivate.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_utility.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_barrier.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_wait_release.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_affinity.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_dispatch.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_lock.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_sched.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_collapse.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/z_Linux_util.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_gsupport.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/thirdparty/ittnotify/ittnotify_static.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_taskdeps.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_cancel.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_ftn_cdecl.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_ftn_extra.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_version.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/ompt-general.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/ompd-specific.cpp.o projects/openmp/runtime/src/CMakeFiles/omp.dir/z_Linux_asm.S.o  -lm  -ldl && cd /home/marcus/toolchain/llvm/stage3-without-sampling/lib && /usr/bin/cmake -E create_symlink libomp.so libgomp.so && /usr/bin/cmake -E create_symlink libomp.so libiomp5.so
Instruction does not dominate all uses!
  %521 = add nsw i32 %374, 1
  %523 = phi i32 [ %521, %520 ], [ %374, %373 ], [ %521, %589 ]
LLVM ERROR: Broken module found, compilation aborted!
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x0000558d0e805387 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x3205387)
 #1 0x0000558d0e80581d SignalHandler(int) Signals.cpp:0:0
 #2 0x00007f8b4e021f10 (/usr/lib/libc.so.6+0x221f10)
 #3 0x00007f8b4e08fbcd pthread_kill (/usr/lib/libc.so.6+0x28fbcd)
 #4 0x00007f8b4e021e52 raise (/usr/lib/libc.so.6+0x221e52)
 #5 0x00007f8b4e0004a0 abort (/usr/lib/libc.so.6+0x2004a0)
 #6 0x0000558d0e154edd llvm::report_fatal_error(llvm::Twine const&, bool) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x2b54edd)
 #7 0x0000558d0d5e3845 (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x1fe3845)
 #8 0x0000558d0e3ef6c5 (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x2def6c5)
 #9 0x0000558d0e3ef60d llvm::detail::PassModel<llvm::Module, llvm::VerifierPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) LTOBackend.cpp:0:0
#10 0x0000558d0de94d09 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x2894d09)
#11 0x0000558d0e19816d llvm::lto::opt(llvm::lto::Config const&, llvm::TargetMachine*, unsigned int, llvm::Module&, bool, llvm::ModuleSummaryIndex*, llvm::ModuleSummaryIndex const*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x2b9816d)
#12 0x0000558d0e192666 llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, std::vector<std::pair<llvm::StringRef, llvm::BitcodeModule>, std::allocator<std::pair<llvm::StringRef, llvm::BitcodeModule>>>>*, std::vector<unsigned char, std::allocator<unsigned char>> const&)::$_1::operator()(llvm::Module&, llvm::TargetMachine*, std::unique_ptr<llvm::ToolOutputFile, std::default_delete<llvm::ToolOutputFile>>) const LTOBackend.cpp:0:0
#13 0x0000558d0e191c2e llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, std::vector<std::pair<llvm::StringRef, llvm::BitcodeModule>, std::allocator<std::pair<llvm::StringRef, llvm::BitcodeModule>>>>*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x2b91c2e)
#14 0x0000558d0e191486 (anonymous namespace)::InProcessThinBackend::runThinLTOBackendThread(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, std::function<llvm::Expected<std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>> (unsigned int, llvm::StringRef, llvm::Twine const&)>, unsigned int, llvm::BitcodeModule, llvm::ModuleSummaryIndex&, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const&, llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo, void>> const&, std::map<unsigned long, llvm::GlobalValue::LinkageTypes, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, llvm::GlobalValue::LinkageTypes>>> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, std::vector<std::pair<llvm::StringRef, llvm::BitcodeModule>, std::allocator<std::pair<llvm::StringRef, llvm::BitcodeModule>>>>&)::'lambda'(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>)::operator()(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>) const LTO.cpp:0:0
#15 0x0000558d0e190e19 std::_Function_handler<void (), std::_Bind<(anonymous namespace)::InProcessThinBackend::start(unsigned int, llvm::BitcodeModule, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const&, llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo, void>> const&, std::map<unsigned long, llvm::GlobalValue::LinkageTypes, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, llvm::GlobalValue::LinkageTypes>>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, std::vector<std::pair<llvm::StringRef, llvm::BitcodeModule>, std::allocator<std::pair<llvm::StringRef, llvm::BitcodeModule>>>>&)::'lambda'(llvm::BitcodeModule, llvm::ModuleSummaryIndex&, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const&, llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo, void>> const&, std::map<unsigned long, llvm::GlobalValue::LinkageTypes, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, llvm::GlobalValue::LinkageTypes>>> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, std::vector<std::pair<llvm::StringRef, llvm::BitcodeModule>, std::allocator<std::pair<llvm::StringRef, llvm::BitcodeModule>>>>&) (llvm::BitcodeModule, std::reference_wrapper<llvm::ModuleSummaryIndex>, std::reference_wrapper<llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const>, std::reference_wrapper<llvm::DenseSet<llvm::ValueInfo, llvm::DenseMapInfo<llvm::ValueInfo, void>> const>, std::reference_wrapper<std::map<unsigned long, llvm::GlobalValue::LinkageTypes, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, llvm::GlobalValue::LinkageTypes>>> const>, std::reference_wrapper<llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const>, std::reference_wrapper<llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, std::vector<std::pair<llvm::StringRef, llvm::BitcodeModule>, std::allocator<std::pair<llvm::StringRef, llvm::BitcodeModule>>>>>)>>::_M_invoke(std::_Any_data const&) LTO.cpp:0:0
#16 0x0000558d0e4f8fd2 std::_Function_handler<void (), llvm::ThreadPool::createTaskAndFuture(std::function<void ()>)::'lambda'()>::_M_invoke(std::_Any_data const&) Writer.cpp:0:0
#17 0x0000558d0e5019ff llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/marcus/toolchain/llvm/stage2-prof-use-lto/install/bin/ld.lld+0x2f019ff)
#18 0x0000558d0e501821 void* llvm::thread::ThreadProxy<std::tuple<llvm::ThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#19 0x00007f8b4e08dc92 (/usr/lib/libc.so.6+0x28dc92)
#20 0x00007f8b4e1389fc (/usr/lib/libc.so.6+0x3389fc)
clang-17: error: unable to execute command: Aborted
clang-17: error: linker command failed due to signal (use -v to see invocation)
aaupov commented 1 year ago

Is it failing with BOLT-instrumented Clang?

ms178 commented 1 year ago

@aaupov I am a bit unsure what you mean, the error is seen while instrumenting Clang for consumption in BOLT while in the stage-3 phase but the script uses the LTO+PGOed stage-2 compiler for that instrumentation step and that compiler wasn't BOLTEd yet. The error happens within the training phase of that script however when compiling some LLVM-projects as training data, so BOLT already ran once.

For more details on my llvm-bolt-scripts, take a look at them in my repo.

To save time, I usually build just the stage-1-compiler and the stage-2-compiler with an older PGO-file, as seen in the build_stage2-prof-use-lto.sh file. If I understand the stack trace correctly (I am not a programmer), it points to a problem with lld or lto. However, the issue only gets exposed when I use the polly flags as included in the top post file and also train on openmp. With a reduced set of the polly flags (without all of the vectorizer parameters) and without openmp, all went fine in that step (see the attached file here). build_stage3-bolt-without-sampling.sh.FINE.txt

hiraditya commented 1 year ago

can you try without profile data and see if it still crashes?

llvmbot commented 1 year ago

@llvm/issue-subscribers-polly

hiraditya commented 1 year ago

One additional note, normally I don't use Polly in the script and could not reproduce the issue without the Polly flags.

It could be polly miscompiling the program? cc @jdoerfert @tobiasgrosser @Meinersbur

ms178 commented 1 year ago

can you try without profile data and see if it still crashes?

You seem to confuse this issue with the other one discussed today (both might share the same root cause, though). For clarification, please note that I have two completly different LLVM-Toolchains in my toolchain-experimental repo that are used and optimized for different purposes. The (lib32-)llvm-git package(s) serve as a system compiler replacement, wheras the llvm-bolt-scripts generate a super-optimized Clang that I use for compiling specific packages only and gets build in a completely different way.

The issue reported here was seen with my llvm-bolt-scripts while the other issue discussed today was seen with the llvm-git package. In the script discussed here, I don't use any profile data in that particular stage, however it is used to generate a profile for later consumption by BOLT.