search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
29.08k stars 11.99k forks source link

cmake fails with LIBCXX_INCLUDE_TESTS=ON #49658

Closed 4084477f-e1df-4024-a410-168c80096e18 closed 2 years ago

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago
Bugzilla Link 50314
Resolution INVALID
Resolved on Oct 23, 2021 09:31
Version 12.0
OS Windows NT
CC @ldionne,@mclow,@tstellar

Extended Description

Looks like somening is not finished

CMake Error at benchmarks/CMakeLists.txt:204 (configure_lit_site_cfg): Unknown CMake command "configure_lit_site_cfg".

ldionne commented 2 years ago

mentioned in issue llvm/llvm-bugzilla-archive#50315

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

1) I'm using as native compiler gcc

2) Is it any problem with building clang using gxx that yoy've mentioon about boostrap?

ldionne commented 3 years ago

Use one of the two documented ways to build libc++:

  1. The "default build", where you build using a system compiler: https://libcxx.llvm.org/BuildingLibcxx.html#id3

  2. The "Bootstrapping build", where you build the runtimes using the just-built Clang compiler: https://libcxx.llvm.org/BuildingLibcxx.html#bootstrapping-build

Both these builds assume a full check-out of the LLVM monorepo.

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

So may I ask how this project should be supported if using LIBCXX_INCLUDE_TESTS=ON iss unsuppoerted?

ldionne commented 3 years ago

Closing as INVALID since the reporter is using an unsupported configuration. Please open new bug reports if you want us to make related improvements and I will be happy to take a look.

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

I think that I found solution. I've added in my libcxx.spec to %cmake parames "-D CMAKE_MODULE_PATH=%{_libdir}/cmake/llvm" and it caused that cmake was able to find AddLLVM cmake module.

I've done that so far in case libcxxabi. After that whole projesty has been build however ctest was not able to find test suite :/

[tkloczko@barrel x86_64-redhat-linux-gnu]$ ctest -vv Test project /home/tkloczko/rpmbuild/BUILD/libcxxabi-13.0.0rc4.src/x86_64-redhat-linux-gnu No tests were found!!!

So look slike in case libcxxabi even if it is passed "-D LIBCXXABI_INCLUDE_TESTS=ON" there is no test suite.

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

Going back to the the issue that cmake is not able to locate AddLLVM cmake module ..

Just checked what I have in my llvm-devel package

[tkloczko@barrel SPECS]$ rpm -ql llvm-devel | grep AddLLVM /usr/lib64/cmake/llvm/AddLLVM.cmake /usr/lib64/cmake/llvm/AddLLVMDefinitions.cmake

So looks like all what is needed is to correct cmake include path. That should be one line fix in (I think)

ldionne commented 3 years ago

This is starting to be counterproductive.

Tomasz: I was not the one to decide that LLVM was going to move to a monorepo. The community decided to do that a couple years ago. If you think that's a bad decision, please bring it up on a mailing list (preferably llvm-dev) and have a discussion there.

Tom: My definition of a Standalone build for libc++ is a build where the root of the CMake invocation is the libcxx/ directory - nothing more. Same goes for libc++abi -- the libcxxabi/ directory is the root of the CMake invocation. It follows trivially that one needs at least two invocations of CMake to build both libc++ and libc++abi Standalone, one in each directory. You can see precisely what I mean here in libc++'s CI build script: https://git.io/Jz41h. That sort of Standalone build is supported only for legacy, and we're going to remove it. Please move off of that in favour of one of the builds documented in https://libcxx.llvm.org/BuildingLibcxx.html.

It's just that we load the targets from /usr/lib64/llvm/cmake instead of the build directory.

Understood. For the runtimes, we don't currently install CMake targets files that can be found with find_package(libcxx) (I assume that's what you are referring to). Even if we did, that wouldn't be sufficient because we need implementation detail files that are not installed with libc++ in order to test libc++abi. Yes, that could be fixed, but there's quite a few circular dependencies in there and it's a larger project that it may seem.

I'm willing to make a step in the direction of not requiring the full monorepo checkout, but if there is no appetite for it, I'll close this as invalid and file a new bug report about removing the libcxx source tarballs from the releases, since they are only misleading.

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

If may I add some 5c .. Guys please have look on KDE. They have set of shared between projects cmake modules. Each main CMakeLists.txt imprts that module. Whatever else is used is kept in cmake/ of the project as private modules. This is really not a rocket science.

With separated versioning that set of macros/modules (extra-cmake-modules) is possible to share the same set of -D FOO_SWITCH={ON|OFF} to enable build and install documentation, define and execute test suite and many more. Each project on area of provide test suite, build and install documentation has almost declarative description of those bits .. all thanks to move whole functional definition to extra-cmake-modules.

Each of you can control only limited number of variables. Abstraction and define interface is the best method of keep under max possible threshold of variables on analysing any issues.

LLVM will be only growing (entropy .. physics is bitch). If anything can be named as "swimming against the current" it is definitely combining everything into one big tree. Sooner or later such tree will be so big and complicated with many not obvious internal dependencies between subparts that only way to restore proper control would be slice everything into smaller pieces. That what you can learn from QT, KDE, Gnome, X11, Perl, Python or few even older like LiveTeX/LaTeX.

If you want to avoid doom of gcc where only relatively small group of highly specialising developers eating gcc code on the breakfast would be able to add even simple fix please do not follow they path. Putting to much code into one basket caused that now there are binutils, gcc, gdb mono repos .. and number of problems is only growing. All because there is no well defined boundaries between subparts. Whoever will be able control that gigantic tree will left the project (even by random case like road accident) will leave other in the middle of the forest of the dependencies.

There are few projects outside of LLVM. With well defined set of cmake modules like in case of KDE it would be possible to simplify streamline many tasks outside current strict area of the LLVM. That could be REAL example of reusability.

And yet another small bit .. LLVM has yet another small brain split. it is related to lit python module. Usually that module is few weeks behind of what lands as sdist lit pypi archive. After asking few times ago to push updated lit module to pypi that forced me package into rpm/Solaris IPS packages lit build from llvm tree.

Because lit is quite core part and comes effectively with llvm I would suggest start pushing lit sdist tar ball to pypi and really stop maintaining it as part of the LLVM mono repo and keep versioning of that part more frequent than rest of the LLVM. The best way to keep that code is use setuptool/pep517 but outside of the cmake controlled tree. All because cmake is not well suited to manage/maintain python code (I mean build, install, generated documentation and test module before actual installation). Python testing procedures can be even extended now without touching tested code of testing description by using pluggable pytest extensions. In other words here it is IMO yet another example of "swimming against the current" ..

tstellar commented 3 years ago

Louis, I agree with you that we probably shouldn't be shipping tarballs that don't work without patching. Although distros like Fedora are still using them. Would it make sense to merge libcxx, libcxxabi, and libunwind into one tarball?

Unfortunately no, because we need more than just that. For instance, we need llvm/utils/lit if you want to enable the tests, and we're about to also require <root>/cmake, where some shared CMake utilities are going to live. There's also <root>/runtimes, which we'd like to move towards being the canonical way to build libc++ and the other runtimes.

In other words, trying to support a subsetted monorepo is just swimming against the current, and it creates a maintenance burden by adding constraints to how we can reuse code across projects, for the sole benefit (AFAICT) of having a smaller source tarball.

Now, in practice, we're never actually going to require some directories like <root>/clang or <root>/mlir for libc++ to be buildable and testable. That would just be silly. I'd be OK with maintaining a script that subsets the repository and keeps only those things that we need for the runtimes to work - but I'd have to be in control of that script and the tarballs would be generated using it, so that it doesn't tie our hands if we want to change that in the future. Would that be a reasonable deal?

Or maybe we could agree to support standalone builds in experimental mode like we do for targets where it is OK to break it?

No. Standalone builds are a mess and they complicate everything because you can't rely on any CMake targets from other projects, you have to make every path customizable, and you need to triplicate several CMake settings (LIBCXX_ENABLE_EXCEPTIONS, LIBCXXABI_ENABLE_EXCEPTIONS, LIBUNWIND_ENABLE_EXCEPTIONS). That's just poor engineering - instead, what we want is a single way to build the runtimes that does not depend on the rest of the CMake machinery in <root>/llvm (which, I agree, is terribly heavyweight and overkill for the needs of the runtimes). We're getting there and Standalone builds are just getting in the way.

We may be working on different definitions of standalone builds, because when we do a standalone build of clang for example, we rely on the cmake targets for other projects. It's just that we load the targets from /usr/lib64/llvm/cmake instead of the build directory.

Moving shared code into /cmake would actually make it much easier for the kind of standalone builds we do, because we can put those files into their own package, install it on the system and then just flip the cmake include paths from ../cmake to /usr/lib64/llvm/cmake.

ldionne commented 3 years ago

Louis, I agree with you that we probably shouldn't be shipping tarballs that don't work without patching. Although distros like Fedora are still using them. Would it make sense to merge libcxx, libcxxabi, and libunwind into one tarball?

Unfortunately no, because we need more than just that. For instance, we need llvm/utils/lit if you want to enable the tests, and we're about to also require <root>/cmake, where some shared CMake utilities are going to live. There's also <root>/runtimes, which we'd like to move towards being the canonical way to build libc++ and the other runtimes.

In other words, trying to support a subsetted monorepo is just swimming against the current, and it creates a maintenance burden by adding constraints to how we can reuse code across projects, for the sole benefit (AFAICT) of having a smaller source tarball.

Now, in practice, we're never actually going to require some directories like <root>/clang or <root>/mlir for libc++ to be buildable and testable. That would just be silly. I'd be OK with maintaining a script that subsets the repository and keeps only those things that we need for the runtimes to work - but I'd have to be in control of that script and the tarballs would be generated using it, so that it doesn't tie our hands if we want to change that in the future. Would that be a reasonable deal?

Or maybe we could agree to support standalone builds in experimental mode like we do for targets where it is OK to break it?

No. Standalone builds are a mess and they complicate everything because you can't rely on any CMake targets from other projects, you have to make every path customizable, and you need to triplicate several CMake settings (LIBCXX_ENABLE_EXCEPTIONS, LIBCXXABI_ENABLE_EXCEPTIONS, LIBUNWIND_ENABLE_EXCEPTIONS). That's just poor engineering - instead, what we want is a single way to build the runtimes that does not depend on the rest of the CMake machinery in <root>/llvm (which, I agree, is terribly heavyweight and overkill for the needs of the runtimes). We're getting there and Standalone builds are just getting in the way.

And that sharing could be done always by use well defined interface between different parts!

This is a common misunderstanding which I personally shared before LLVM moved to a monorepo. I came to change my mind. It is possible to have well-defined interfaces between different parts without enforcing a physical separation of the source code. Yes, physical proximity of the source code can make API layers easier to violate in some cases, however proper engineering still dictates that those boundaries aren't crossed.

On the other hand, physical proximity of source code means that when you do need to share something between projects, the barrier to doing so is smaller. In the past, we've done terrible things like duplicate header files between libc++ and libc++abi because we could not rely on libc++ being available while building libc++ (or vice versa). We've had bugs because of that sort of duplication. That's just poor engineering - sometimes it's better to acknowledge that you're sharing some implementation details and do it properly than to come up with inferior solutions or duplicate code just to avoid sharing.

Summary: Tom, I can write a script that will subset the monorepo for the needs of libc++, libc++abi and libunwind. I'll maintain that and ensure that it works. Then, the release tarballs can be generated using that script. Does that sound like a plan? If I need to include new directories or move stuff around, I'll maintain that and everything will just work. How does that sound? I've actually got a use case for doing that, since we already subset the monorepo internally :-)

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

Tomasz, is there any reason why you can't just use the libcxx package shipped in Fedora? If not, you can copy how we build it for Fedora.

Because I'm working on my own distribution which has tons of issues fixed which no one on even is aware in Fedora (or sometimes wants to fix those issue). That project is already advanced in about 90% and probably within 6-9 months should be ready for public presentation.

Biggest problem of the Fedora is lack of detailed tracing ABI changes and making upgrades of the packages without assessing actual impact of those changes across other packages depending on upgraded packages. That is why on each Mass Rebuild suddenly many people are able of see that some new packages are no longer even building. Those fails are at that point completely detached from actual cause in form of other packages upgrades. With my distribution build automation I'm performing more that 10-50 times control rebuilds only to confirm that just introduced upgrade is not causing cascades of fails in builds of other packages. That area in Fedora is maintained almost entirely manually which with ~20k source packages is not possible to trace using only actual human resources.

FYI: I'm already maintainer of the few packages in Fedora and my eye are sometimes literally are bleeding when I see not fixed many issues.

Just to let you know .. I'm not new in that are. If you will do "rpm -qa --changelog|grep kloczek" in any RH distribution or derivates (like Oracle Linux) you can find traces of my work staring from 1997.

tstellar commented 3 years ago

Building (some of the) projects in standalone mode e.g. from the individual is still supposed to be a supported configuration (this is how we build packages for Fedora).

It's true that the libcxx build from the tarball does not work out of the box, we patch it in Fedora so that it builds correctly.

Tomasz, is there any reason why you can't just use the libcxx package shipped in Fedora? If not, you can copy how we build it for Fedora.

Louis, I agree with you that we probably shouldn't be shipping tarballs that don't work without patching. Although distros like Fedora are still using them. Would it make sense to merge libcxx, libcxxabi, and libunwind into one tarball? Or maybe we could agree to support standalone builds in experimental mode like we do for targets where it is OK to break it?

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

And that sharing could be done always by use well defined interface between different parts!

llvm, clang, lldb and few more already is possible to build and test without using mono repo.

ldionne commented 3 years ago

With all due respect, there was a community decision (which I wasn't a part of) of moving LLVM to a monorepo.

One of the benefits of doing that is that we can share more code across different projects and streamline some processes such as doing the CMake configuration. We want to standardize around having 1-2 ways of building libc++, not 4-5 like we do today. The Standalone build you're using is not supported anymore, please move off of it. Having so many different ways to build things is what's creating a maintenance nightmare.

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

Tom, how could we stop shipping the sources for libc++, libc++abi and libunwind as part of https://github.com/llvm/llvm-project/releases?

Why not just fix that by install by llvm install target that missing AddLLVM cmake module or find any other proper solution for not reversing separation between different bits of the LLVM?

Really building smaller parts like libcxxabi, libcxx only from mono repo is a bit odd.

I would say that testing everything out of mono repo makes everything less isolated, harder to control and correlate where cause of some issues may be located.

LLVM as project is rally huge and looks like it already reached the same point as in the past X11 which have been build for quite long time from single source tree. Today whole X11 stack is build out of many smaller subprojects. Which makes everything easier to maintain, build and test.

The same illness is rotting SELnux :/

Different parts of LLVM project are developed in different speed. Versioning everything the same way sooner or later will be creating big overhead of some maintenance man/hour resources :/

For example I would like to build mlir independently from whole LLVM but there is no separated source tar ball for that part .. and IMO miir is far more important than libcxxabi or libcxx :(

One of the biggest advantages of the LLVM stack over gcc is that it is much more modular than gcc one.

Instead reversing that modularity probably not only I would love to see better and deeper modularisation.

ldionne commented 3 years ago

Bug llvm/llvm-bugzilla-archive#50315 has been marked as a duplicate of this bug.

ldionne commented 3 years ago

Okay, so it seems like you're shooting yourself in the foot. There's a reason why we have that check in place.

I 100% agree that it's stupid to provide sources for libc++, libc++abi and libunwind since those can't be built on their own. I'll see if we can fix that.

Tom, how could we stop shipping the sources for libc++, libc++abi and libunwind as part of https://github.com/llvm/llvm-project/releases?

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

I just checked one more time my build procedure.

I'm using parch from fedora:

From fc46c464795f965c4cdbc321058f24191af8e40c Mon Sep 17 00:00:00 2001 From: Tom Stellard tstellar@redhat.com Date: Tue, 11 Aug 2020 07:55:18 -0700 Subject: [PATCH] [PATCH][libcxx] Remove monorepo requirement

libcxx/CMakeLists.txt | 9 --------- 1 file changed, 9 deletions(-)

diff --git a/libcxx/CMakeLists.txt b/libcxx/CMakeLists.txt index 9bf1a02..5348a15 100644 --- a/libcxx/CMakeLists.txt +++ b/libcxx/CMakeLists.txt @@ -1,10 +1,3 @@ -# See https://libcxx.llvm.org/docs/BuildingLibcxx.html for instructions on how -# to build libcxx with CMake.

-if (NOT IS_DIRECTORY "${CMAKE_CURRENT_LIST_DIR}/../libcxxabi")

Issue is that if libcxx must be build from mono repo providing separated source tar ball like https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0-rc3/libcxx-13.0.0rc3.src.tar.xz is completely waste of disk space and time because that tar ball is dysfunctional :/

In other words that dist tar ball is broken. Probably other as well.

I have similar ticket for libcxxabi llvm/llvm-bugzilla-archive#50315

ldionne commented 3 years ago

This is what I get: (cd libcxx-13.0.0rc3.src && rm -rf build-repro && cmake -B build-repro -DBUILD_SHARED_LIBS=ON -S . -DCXX_ENABLE_RTTI=ON -DLIBCXXABI_USE_LLVM_UNWINDER=OFF -DLIBCXX_CXX_ABI_INCLUDE_PATHS=/usr/include -DLIBCXX_CXX_ABI=libcxxabi -DLIBCXX_ENABLE_SHARED=ON -DLIBCXX_ENABLE_STATIC=OFF -DLIBCXX_INCLUDE_TESTS=ON -DLIBCXX_STANDALONE_BUILD=ON -DLIBCXX_USE_COMPILER_RT=OFF -DLIBCXX_INCLUDE_BENCHMARKS=ON) CMake Error at CMakeLists.txt:5 (message): libc++ now requires being built in a monorepo layout with libcxxabi available

You must have some other folders lying around, like at least libcxxabi alongside the libcxx directory -- please describe that.

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

I'm not using mono repo tar ball. I'm using using https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0-rc3/libcxx-13.0.0rc3.src.tar.xz

ldionne commented 3 years ago

Can you please describe the directory layout of the monorepo you're using?

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

Just tested and looks like it still fails the same way

CMake Error at benchmarks/CMakeLists.txt:207 (configure_lit_site_cfg): Unknown CMake command "configure_lit_site_cfg".

ldionne commented 3 years ago

Awesome, please re-open if you still see the issue using one of the supported ways of building!

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

I just started moving from 12.0.1 to 13.0.0-rc3 and I have only build llvm, clang and lldb. Will back shortly with results on top of 13.0.0-rc3 :)

4084477f-e1df-4024-a410-168c80096e18 commented 3 years ago

I just started moving from 12.0.1 to 13.0.0-rc3 and I have only build llvm, clang and lldb. Will back shortly with results on top of 13.0.0-rc3 :)

ldionne commented 3 years ago

Sorry for the delay, but are you still seeing this?

If so, please move to one of the documented ways of building libc++ (as documented in https://libcxx.llvm.org/BuildingLibcxx.html) and let us know if that doesn't solve your issue.