search

InBetweenNames / gentooLTO

A Gentoo Portage configuration for building with -O3, Graphite, and LTO optimizations
GNU General Public License v2.0
571 stars 97 forks source link

llvm-11 and friends fails with lto #619

Open hedmo opened 3 years ago

hedmo commented 3 years ago

llvm-11 has been released and it fails with. affected packages : ` / # cat /etc/portage/package.cflags/ltolocalworkarounds.conf

BEGIN: LTO workarounds

sys-devel/llvm FLAGS-=-flto sys-devel/clang FLAGS-=-flto sys-libs/libomp FLAGS-=-flto

END: LTO not recommended

/ # `

perfect7gentleman commented 3 years ago

Build LLVM/Clang with LLVM/Clang.

hedmo commented 3 years ago

Build LLVM/Clang with LLVM/Clang.

what do youn mean?

elsandosgrande commented 3 years ago

Build Clang 11 with Clang 10, same with LLVM.

perfect7gentleman commented 3 years ago

++ all LLVM tools and libs

AnonymousRetard commented 3 years ago

I have successfully built both clang:11 and libomp-11.0.0 with -flto. It's only llvm:11 that fails. It seems to build a broken llvm-tablegen binary which then crashes during the build when it tries to use it to generate files needed to continue building.

I tried switching compiler to clang like this: /etc/portage/env/clang: CC="clang" CXX="clang++" AR="llvm-ar" NM="llvm-nm" RANLIB="llvm-ranlib"

/etc/portage/package.env: sys-devel/llvm clang

Then it builds but clang doesn't recognize any of these flags: -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fno-semantic-interposition -flto=4 -fuse-linker-plugin

So then it builds without lto. As a final test I tried using this instead: /etc/portage/env/clang-lto: CC="clang" CXX="clang++" CFLAGS="${CFLAGS} -flto" CXXFLAGS="${CXXFLAGS} -flto" LDFLAGS="-Wl,-O3 -Wl,--as-needed" AR="llvm-ar" NM="llvm-nm" RANLIB="llvm-ranlib"

Which then tries to build llvm:11 with clang and -flto but then the build fails in the same way (it builds a broken llvm-tablegen which crashes when launched).

khuei commented 3 years ago

I got them to successfully build with LTO by disabling --as-needed for llvm and clang

Ninpo commented 3 years ago

I got them to successfully build with LTO by disabling --as-needed for llvm and clang

Confirm this worked for me and building libomp/llvm/clang-11 with clang-10 allowed libomp to build with LTO.

To whom should a bug report be sent for the no-as-needed switch with llvm/clang?

AnonymousRetard commented 3 years ago

I can also confirm that building llvm:11 with all flags from this overlay: CFLAGS="-march=znver1 -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -flto=4 -fuse-linker-plugin -pipe -fomit-frame-pointer -fno-finite-math-only -Wl,-O1" works as long as I remove --as-needed from LDFLAGS. But not if I try to build llvm:11 with clang:11, it only works with GCC 10.2.0. I could also get it to build with clang:11 if I disable LTO but removing --as-needed only works for GCC. With clang:11 it gets me a lot further but it still fails in the final linking step. I have not tried clang:10.

Additionally I never had any problems with building clang, only llvm. As for libomp I have the "cuda" USE-flag enabled for that package which forces me to use <= GCC 9.

InBetweenNames commented 3 years ago

I went ahead and disabled LTO on these packages as I'm strapped for time at the moment to deep dive and figure out where they are breaking. Normally LLVM builds fine with LTO, so this is surprising.

leandrolnh commented 3 years ago

Maybe you can try -flto=thin instead, my setup works fine with this:

/etc/portage/env/compiler-clang.conf:
CC="clang"
CXX="clang++"
AR="llvm-ar"
NM="llvm-nm"
RANLIB="llvm-ranlib"
CFLAGS="-march=native -O3 -pipe -flto=thin"
CXXFLAGS="${CFLAGS} -stdlib=libc++"
LDFLAGS="-Wl,-O2 -Wl,--as-needed -fuse-ld=lld -stdlib=libc++ -rtlib=compiler-rt -unwindlib=libunwind -flto=thin -Wl,--thinlto-jobs=8"
/etc/portage/package.env:
sys-devel/llvm-common compiler-clang.conf
sys-devel/llvm compiler-clang.conf
sys-devel/llvmgold compiler-clang.conf
sys-devel/lld compiler-clang.conf
sys-devel/clang compiler-clang.conf
sys-devel/clang-common compiler-clang.conf
sys-libs/compiler-rt-sanitizers compiler-clang.conf
sys-libs/compiler-rt compiler-clang.conf
sys-devel/clang-runtime compiler-clang.conf
sys-libs/libcxx compiler-clang.conf
sys-libs/libcxxabi compiler-clang.conf
sys-libs/libomp compiler-clang.conf
sys-libs/llvm-libunwind compiler-clang.conf
dev-lang/rust compiler-clang.conf
media-libs/mesa compiler-clang.conf
hedmo commented 3 years ago

in my opinion.i dont think it is a good idea to mixing compilers.well on local basics it is okay but not global like here in lto-overlay.I went ahead and disabled LTO on these packages as i have more package in my system that has a mix off clang and gcc when it is compiling.BTW i am keeping : sys-devel/llvm *FLAGS-="${IPAPTA}" to just in case.

barolo commented 3 years ago

they compile fine with clang --flto=full and -fwhole-program-vtables

perfect7gentleman commented 3 years ago

@barolo , how long will it take to build llvm and clang with -flto=full and -fwhole-program-vtables against -flto=thin ?

barolo commented 3 years ago

@perfect7gentleman genlop says that it took 53 min for llvm and 54 for clang [ multilib ], on my 4 core APU

killbox998 commented 3 years ago

I'm still getting build failures even with lto disabled

FAILED: include/llvm/IR/IntrinsicsPowerPC.h cd /var/tmp/portage/sys-devel/llvm-11.0.0/work/llvm-11.0.0_build-abi_x86_32.x86 && /var/tmp/portage/sys-devel/llvm-11.0.0/work/llvm-11.0.0_build-abi_x86_32.x86/bin/llvm-tblgen -gen-intrinsic-enums -intrinsic-prefix=ppc -I /var/tmp/portage/sys-devel/llvm-11.0.0/work/llvm/include/llvm/IR -I /usr/include/libxml2 -I /var/tmp/portage/sys-devel/llvm-11.0.0/work/llvm-11.0.0_build-abi_x86_32.x86/include -I /var/tmp/portage/sys-devel/llvm-11.0.0/work/llvm/include /var/tmp/portage/sys-devel/llvm-11.0.0/work/llvm/include/llvm/IR/Intrinsics.td --write-if-changed -o include/llvm/IR/IntrinsicsPowerPC.h -d include/llvm/IR/IntrinsicsPowerPC.h.d terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1

https://dpaste.com/FFGJR9WXS

perfect7gentleman commented 3 years ago

so, you wanna say that -flto=1 means it's disabled?

khuei commented 3 years ago

/etc/portage/package.cflags/ltoworkarounds.conf -> sys-devel/llvm: FLAGS-=-flto* # Issue #619 temporarily disabled for now due to build errors

I think it's because it's missing the prefix *, so it should be *FLAGS-=-flto* rather than FLAGS-=-flto*

killbox998 commented 3 years ago

Yeah, That was the issue

Hello71 commented 3 years ago

llvm requires -Wl,--no-as-needed due to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55394#c7, fixed in gcc 11. it causes terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1.

alanswanson commented 3 years ago

Testing -Wl,--no-as-needed it's only required for tablegen with sys-devel/llvm and sys-devel/clang and not sys-libs/libomp (when using LTO). Only build difference was libLLVM-11.so being linked to librt.so.

hedmo commented 3 years ago

llvm requires -Wl,--no-as-needed due to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55394#c7, fixed in gcc 11. it causes terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1.

okej . I Will remove my overides and check .

hedmo commented 3 years ago

i can confirm that llvm-11 and friends compiles with full LTO on gcc-11

laomaiweng commented 3 years ago

FWIW, this issue is tracked in https://bugs.gentoo.org/749162, and there are patches there available for both LLVM and Clang that have been reported to fix LTO builds with GCC<11.

Tomte-Fan commented 1 year ago

With llvm 16.0.4 I have to turn off IPAPTA or it will fail to build with a stack trace.