search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
28.34k stars 11.7k forks source link

16.0.3 `libcxx`, `libcxxabi`: circular build dependencies #55632

Closed kloczek closed 10 months ago

kloczek commented 2 years ago

Trying to build unpatched libcxx it fails because it requires being built in a monorepo layout with libcxxabi: 

+ /usr/bin/cmake -B x86_64-redhat-linux-gnu -D BUILD_SHARED_LIBS=ON -D CMAKE_AR=/usr/bin/gcc-ar -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_C_FLAGS_RELEASE=-DNDEBUG -D CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -D CMAKE_Fortran_FLAGS_RELEASE=-DNDEBUG -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_NM=/usr/bin/gcc-nm -D CMAKE_RANLIB=/usr/bin/gcc-ranlib -D CMAKE_VERBOSE_MAKEFILE=ON -D INCLUDE_INSTALL_DIR=/usr/include -D LIB_INSTALL_DIR=/usr/lib64 -D LIB_SUFFIX=64 -D SHARE_INSTALL_PREFIX=/usr/share -D SYSCONF_INSTALL_DIR=/etc -S . -D CXX_ENABLE_RTTI=ON -D LIBCXXABI_USE_LLVM_UNWINDER=OFF -D LIBCXX_CXX_ABI_INCLUDE_PATHS=/usr/include -D LIBCXX_CXX_ABI=libcxxabi -D LIBCXX_ENABLE_ABI_LINKER_SCRIPT=ON -D LIBCXX_ENABLE_ASSERTIONS=OFF -D LIBCXX_ENABLE_DEBUG_MODE_SUPPORT=OFF -D LIBCXX_ENABLE_EXCEPTIONS=ON -D LIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF -D LIBCXX_ENABLE_SHARED=ON -D LIBCXX_ENABLE_STATIC_ABI_LIBRARY=OFF -D LIBCXX_ENABLE_STATIC=OFF -D LIBCXX_ENABLE_STDIN=ON -D LIBCXX_ENABLE_STDOUT=ON -D LIBCXX_ENABLE_THREADS=ON -D LIBCXX_ENABLE_WERROR=OFF -D LIBCXX_INCLUDE_TESTS=OFF -D LIBCXX_INSTALL_HEADERS=ON -D LIBCXX_INSTALL_LIBRARY=ON -D LIBCXX_INSTALL_SHARED_LIBRARY=ON -D LIBCXX_STANDALONE_BUILD=ON -D LIBCXX_USE_COMPILER_RT=OFF -D LLVM_EXTERNAL_LIT=/usr/bin/lit -D PYTHON_EXECUTABLE=/usr/bin/python3 -D PYTHONINTERP_FOUND=ON -D LIBCXX_LIBDIR_SUFFIX=64
CMake Error at CMakeLists.txt:5 (message):
  libc++ now requires being built in a monorepo layout with libcxxabi
  available

-- Configuring incomplete, errors occurred!

I found fedora patch

From fc46c464795f965c4cdbc321058f24191af8e40c Mon Sep 17 00:00:00 2001
From: Tom Stellard <tstellar@redhat.com>
Date: Tue, 11 Aug 2020 07:55:18 -0700
Subject: [PATCH] [PATCH][libcxx] Remove monorepo requirement

---
 libcxx/CMakeLists.txt | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/libcxx/CMakeLists.txt b/libcxx/CMakeLists.txt
index 9bf1a02..5348a15 100644
--- a/libcxx/CMakeLists.txt
+++ b/libcxx/CMakeLists.txt
@@ -1,10 +1,3 @@
-# See https://libcxx.llvm.org/docs/BuildingLibcxx.html for instructions on how
-# to build libcxx with CMake.
-
-if (NOT IS_DIRECTORY "${CMAKE_CURRENT_LIST_DIR}/../libcxxabi")
-  message(FATAL_ERROR "libc++ now requires being built in a monorepo layout with libcxxabi available")
-endif()
-
 #===============================================================================
 # Setup Project
 #===============================================================================
@@ -39,8 +32,6 @@
 include(GNUInstallDirs)

 if (LIBCXX_STANDALONE_BUILD)
-  # Find the LLVM sources and simulate LLVM CMake options.
-  include(HandleOutOfTreeLLVM)

   find_package(Python3 COMPONENTS Interpreter)
   if(NOT Python3_Interpreter_FOUND)
--
1.8.3.1

However it did not helped and looks like on building libcxx and libcxxabi usign non-nomo repo tar balls libcxx cmake passes without installed libcxxabi devel resources

```console + /usr/bin/cmake -B x86_64-redhat-linux-gnu -D BUILD_SHARED_LIBS=ON -D CMAKE_AR=/usr/bin/gcc-ar -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_C_FLAGS_RELEASE=-DNDEBUG -D CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -D CMAKE_Fortran_FLAGS_RELEASE=-DNDEBUG -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_NM=/usr/bin/gcc-nm -D CMAKE_RANLIB=/usr/bin/gcc-ranlib -D CMAKE_VERBOSE_MAKEFILE=ON -D INCLUDE_INSTALL_DIR=/usr/include -D LIB_INSTALL_DIR=/usr/lib64 -D LIB_SUFFIX=64 -D SHARE_INSTALL_PREFIX=/usr/share -D SYSCONF_INSTALL_DIR=/etc -S . -D CXX_ENABLE_RTTI=ON -D LIBCXXABI_USE_LLVM_UNWINDER=OFF -D LIBCXX_CXX_ABI_INCLUDE_PATHS=/usr/include -D LIBCXX_CXX_ABI=libcxxabi -D LIBCXX_ENABLE_ABI_LINKER_SCRIPT=ON -D LIBCXX_ENABLE_ASSERTIONS=OFF -D LIBCXX_ENABLE_DEBUG_MODE_SUPPORT=OFF -D LIBCXX_ENABLE_EXCEPTIONS=ON -D LIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF -D LIBCXX_ENABLE_SHARED=ON -D LIBCXX_ENABLE_STATIC_ABI_LIBRARY=OFF -D LIBCXX_ENABLE_STATIC=OFF -D LIBCXX_ENABLE_STDIN=ON -D LIBCXX_ENABLE_STDOUT=ON -D LIBCXX_ENABLE_THREADS=ON -D LIBCXX_ENABLE_WERROR=OFF -D LIBCXX_INCLUDE_TESTS=OFF -D LIBCXX_INSTALL_HEADERS=ON -D LIBCXX_INSTALL_LIBRARY=ON -D LIBCXX_INSTALL_SHARED_LIBRARY=ON -D LIBCXX_STANDALONE_BUILD=ON -D LIBCXX_USE_COMPILER_RT=OFF -D LLVM_EXTERNAL_LIT=/usr/bin/lit -D PYTHON_EXECUTABLE=/usr/bin/python3 -D PYTHONINTERP_FOUND=ON -D LIBCXX_LIBDIR_SUFFIX=64 CMake Warning at CMakeLists.txt:24 (message): The Standalone build is deprecated in this release. Please use one of the ways described at https://libcxx.llvm.org/BuildingLibcxx.html for building libc++. -- The CXX compiler identification is GNU 12.1.1 -- The C compiler identification is GNU 12.1.1 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/g++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/gcc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Could NOT find FFI (missing: FFI_LIBRARIES HAVE_FFI_CALL) -- Could NOT find Terminfo (missing: Terminfo_LIBRARIES Terminfo_LINKABLE) -- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR) -- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR) -- Found Python3: /usr/bin/python3.8 (found version "3.8.13") found components: Interpreter -- Performing Test LIBCXX_SUPPORTS_UNWINDLIB_NONE_FLAG -- Performing Test LIBCXX_SUPPORTS_UNWINDLIB_NONE_FLAG - Failed -- Looking for fopen in c -- Looking for fopen in c - found -- Looking for __gcc_personality_v0 in gcc_s -- Looking for __gcc_personality_v0 in gcc_s - found -- Performing Test LIBCXX_SUPPORTS_NOSTDLIBXX_FLAG -- Performing Test LIBCXX_SUPPORTS_NOSTDLIBXX_FLAG - Failed -- Performing Test LIBCXX_SUPPORTS_NODEFAULTLIBS_FLAG -- Performing Test LIBCXX_SUPPORTS_NODEFAULTLIBS_FLAG - Success -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Looking for ccos in m -- Looking for ccos in m - found -- Looking for clock_gettime in rt -- Looking for clock_gettime in rt - found -- Looking for __atomic_fetch_add_8 in atomic -- Looking for __atomic_fetch_add_8 in atomic - found -- Looking for cxxabi.h in /usr/include -- Looking for cxxabi.h in /usr/include - not found CMake Warning at cmake/Modules/HandleLibCXXABI.cmake:77 (message): Failed to find cxxabi.h in /usr/include Call Stack (most recent call first): cmake/Modules/HandleLibCXXABI.cmake:118 (setup_abi_lib) CMakeLists.txt:511 (include) -- Looking for __cxxabi_config.h in /usr/include -- Looking for __cxxabi_config.h in /usr/include - not found CMake Warning at cmake/Modules/HandleLibCXXABI.cmake:77 (message): Failed to find __cxxabi_config.h in /usr/include Call Stack (most recent call first): cmake/Modules/HandleLibCXXABI.cmake:118 (setup_abi_lib) CMakeLists.txt:511 (include) -- Performing Test LIBCXX_SUPPORTS_FALIGNED_ALLOCATION_FLAG -- Performing Test LIBCXX_SUPPORTS_FALIGNED_ALLOCATION_FLAG - Failed -- Performing Test LIBCXX_SUPPORTS_NOSTDINCXX_FLAG -- Performing Test LIBCXX_SUPPORTS_NOSTDINCXX_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_FVISIBILITY_INLINES_HIDDEN_FLAG -- Performing Test LIBCXX_SUPPORTS_FVISIBILITY_INLINES_HIDDEN_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WALL_FLAG -- Performing Test LIBCXX_SUPPORTS_WALL_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WEXTRA_FLAG -- Performing Test LIBCXX_SUPPORTS_WEXTRA_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_W_FLAG -- Performing Test LIBCXX_SUPPORTS_W_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WWRITE_STRINGS_FLAG -- Performing Test LIBCXX_SUPPORTS_WWRITE_STRINGS_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WNO_UNUSED_PARAMETER_FLAG -- Performing Test LIBCXX_SUPPORTS_WNO_UNUSED_PARAMETER_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WNO_LONG_LONG_FLAG -- Performing Test LIBCXX_SUPPORTS_WNO_LONG_LONG_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WERROR_EQ_RETURN_TYPE_FLAG -- Performing Test LIBCXX_SUPPORTS_WERROR_EQ_RETURN_TYPE_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WEXTRA_SEMI_FLAG -- Performing Test LIBCXX_SUPPORTS_WEXTRA_SEMI_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WUNDEF_FLAG -- Performing Test LIBCXX_SUPPORTS_WUNDEF_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WFORMAT_NONLITERAL_FLAG -- Performing Test LIBCXX_SUPPORTS_WFORMAT_NONLITERAL_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WNO_LITERAL_SUFFIX_FLAG -- Performing Test LIBCXX_SUPPORTS_WNO_LITERAL_SUFFIX_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WNO_CXX14_COMPAT_FLAG -- Performing Test LIBCXX_SUPPORTS_WNO_CXX14_COMPAT_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WNO_NOEXCEPT_TYPE_FLAG -- Performing Test LIBCXX_SUPPORTS_WNO_NOEXCEPT_TYPE_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WNO_SUGGEST_OVERRIDE_FLAG -- Performing Test LIBCXX_SUPPORTS_WNO_SUGGEST_OVERRIDE_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_WNO_ERROR_FLAG -- Performing Test LIBCXX_SUPPORTS_WNO_ERROR_FLAG - Success -- Performing Test LIBCXX_SUPPORTS_EHSC_FLAG -- Performing Test LIBCXX_SUPPORTS_EHSC_FLAG - Failed -- Performing Test LIBCXX_SUPPORTS_ZL_FLAG -- Performing Test LIBCXX_SUPPORTS_ZL_FLAG - Failed -- Performing Test LIBCXX_SUPPORTS_NODEFAULTLIB_FLAG -- Performing Test LIBCXX_SUPPORTS_NODEFAULTLIB_FLAG - Failed CMake Warning at benchmarks/CMakeLists.txt:7 (message): The libc++ benchmarks are not available in a standalone build. Please migrate to an official build as documented in https://libcxx.llvm.org/BuildingLibcxx.html. -- Configuring done -- Generating done ```

However after that build fails on

```console [ 92%] Building CXX object src/CMakeFiles/cxx_shared.dir/exception.cpp.o cd /home/tkloczko/rpmbuild/BUILD/libcxx-14.0.3.src/x86_64-redhat-linux-gnu/src && /usr/bin/g++ -DNDEBUG -D_LIBCPP_BUILDING_LIBRARY -D_LIBCPP_DISABLE_NEW_DELETE_DEFINITIONS -D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER -I/home/tkloczko/rpmbuild/BUILD/libcxx-14.0.3.src/x86_64-redhat-linux-gnu/include/c++build -I/home/tkloczko/rpmbuild/BUILD/libcxx-14.0.3.src/src -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -DNDEBUG -fPIC -DLIBCXX_BUILDING_LIBCXXABI -nostdinc++ -fvisibility-inlines-hidden -Wall -Wextra -W -Wwrite-strings -Wno-unused-parameter -Wno-long-long -Werror=return-type -Wextra-semi -Wundef -Wformat-nonliteral -Wno-literal-suffix -Wno-c++14-compat -Wno-noexcept-type -Wno-suggest-override -Wno-error -I/home/tkloczko/rpmbuild/BUILD/libcxx-14.0.3.src/x86_64-redhat-linux-gnu/include/c++/v1 -std=c++20 -MD -MT src/CMakeFiles/cxx_shared.dir/exception.cpp.o -MF CMakeFiles/cxx_shared.dir/exception.cpp.o.d -o CMakeFiles/cxx_shared.dir/exception.cpp.o -c /home/tkloczko/rpmbuild/BUILD/libcxx-14.0.3.src/src/exception.cpp /home/tkloczko/rpmbuild/BUILD/libcxx-14.0.3.src/src/exception.cpp:14:12: fatal error: cxxabi.h: No such file or directory 14 | #include | ^~~~~~~~~~ compilation terminated. ```

On building libcxxabi cmake requires -D LIBCXXABI_LIBCXX_INCLUDES so in that case that requires to have installed libcxx devel resources. In other words there is circular dependency between those two when

philnik777 commented 2 years ago

Are you trying to build libc++ outside the monorepo? If yes: it's not supported anymore. If it doesn't work inside the monorepo layout could you clarify what the problem is?

kloczek commented 2 years ago

Issue is that for both LLVM components are created official dist source tar balls so it suggest that it should be possible to do that. It wold be really good to have proper separation between LLVM components. This project already is huge and IMO sooner or later it will be necessary to divide it into separated subprojects. Because everything is in single repo it is really hard to across all activities like issues, PRs commits, discussions. Currently everything lands on single pile and only that I'm guessing maintenance must be harder and harder ..

philnik777 commented 2 years ago

That is indeed weird. @ldionne should the separate sources still exist? Or should there be a runtimes download? It looks like there is currently no way to build libc++ without downloading all of the LLVM project, since there is no way to get the runtimes/ subdirectory otherwise.

kloczek commented 2 years ago

gentle ping .. 😋

kloczek commented 1 year ago

Gentle ping .. issue still is around 😞

ldionne commented 1 year ago

We shouldn't be distributing individual sources anymore, no. In fact I don't think it makes sense to distribute any sources that is a subset of the whole monorepo since most parts of LLVM use shared infrastructure.

So for example, there's no hope of being able to build libc++ without libcxxabi/, runtimes/, cmake/, llvm/utils/lit and probably others I am forgetting. I am not sure what generates these source tarballs during releases.

@tru @tstellar Do you know what controls the generation of source tarballs during releases? https://github.com/llvm/llvm-project/releases/tag/llvmorg-16.0.0 has a bunch of small tarballs for libcxx, libcxxabi and others, and they don't really make any sense IMO.

ldionne commented 1 year ago

Hmm, I just read the comment in https://discourse.llvm.org/t/building-llvm-out-of-per-subproject-tar-balls-still-is-not-working/72829/8:

The per sub-project tar balls are really meant to be used the same way one would use sparse checkouts from the git repository. There are still dependencies between sub-projects that make it impossible to build some of the sub-projects with a single tarball. For llvm, clang, and lld, the required sub-directories are documented here.

If the goal of providing those tarballs in the release is sparse checkouts, then I think this can be closed as NTBF (but also I think that the intent isn't super clear).

kloczek commented 1 year ago

We shouldn't be distributing individual sources anymore, no. In fact I don't think it makes sense to distribute any sources that is a subset of the whole monorepo since most parts of LLVM use shared infrastructure.

I cannot agree with above. LLVM as the project is HUGE. In case if it will be something to fix by apply single small patch on clang code it wille be possible to rebuild only clang without wasting time on rebuilding everything. Issue is only with libcxx and libcxxabi which requires each other on build stage so it makes sense to merge them. Such project even combined will be still one of the smallest chunks of the LLVM.

philnik777 commented 1 year ago

You don't need to build everything just because you have all the source, so I don't see your point. (Most) libc++ developers only build libc++/libc++abi/libunwind but have all the sources available. There is no need to build anything else.

kloczek commented 1 year ago

You don't need to build everything just because you have all the source, so I don't see your point. (Most) libc++ developers only build libc++/libc++abi/libunwind but have all the sources available. There is no need to build anything else.

Yes it is possible to do that however each such sub package will have included like in case of Fedora llvm-bolt FULL LLVM source tree tar ball. In other words all what you rote it is only trying to find kind of excuse to not solve that issue in civilised way. Problem will not disappear if it will be not treated as the issue. Dependency between those two components is on build stage and separating them into two separated source tar balls CREATES problem.

The same situation is with llvm, third-party and llvm-cmake source dist tar balls. llvm is effectively first LLVM component which needs to be build and forming additional llvm-cmake and third-party source tar balls does not solve any problems but it ONLY CREATES new ones complication packaging procedures.

libunwind is well isolated and so far nothing needs to be changed here.

Endilll commented 1 year ago

In other words all what you rote it is only trying to find kind of excuse to not solve that issue in civilised way. Problem will not disappear if it will be not treated as the issue.

As it was pointed out to you in the Discourse thread, standalone builds are not working because of some accident. Here are past discussions of this: Streamlining support for standalone builds, Standalone build for clang & lld, fix or remove?.

Maybe @mgorny can chime in and share his experience, since he expressed interest in maintaining standalone builds.

Also be mindful of our Code of Conduct.

kloczek commented 1 year ago

As it was pointed out to you in the Discourse thread, standalone builds are not working because of some accident. Here are past discussions of this: Streamlining support for standalone builds, Standalone build for clang & lld, fix or remove?.

If that was accident why since May no one have been trying to solve that issue? 🤔 Is it any problem with fixing that? 🤔 (I'm really only honestly asking because I don't see any changes in 16.x branch and 17.x as well .. but maybe I've lost some commits)

Endilll commented 1 year ago

I'm sorry for not being clear enough. I meant that it's not an accident that there are troubles with standalone builds.

kloczek commented 1 year ago

Just FTR: samples of use those per sub project dist tar balls it is possible to find across many Linux distributions so even LLVM is not (probably) testing those tar balls it is IMO enough sources details about what still needs to be fixed.

In https://github.com/llvm/llvm-project/issues/61502#issuecomment-1605297704 I've even proposed some set of minimal changes to fix some most obvious issues and so far (sorry so maybe a bit rude wording) no one even farted about that. That is really discouraging .. 😞

tru commented 1 year ago

In #61502 (comment) I've even proposed some set of minimal changes to fix some most obvious issues and so far

Try to submit it as diff in Phabricator and you might get a review. Adding a diff to a issue is probably easily going to be missed by the people that would review something like that.

kloczek commented 1 year ago

May I ask for some instruction how to do that? 🤔 (I'm not familiar with LLVN PR workflow) Will try to do that ASP 😋

tru commented 1 year ago

Maybe you can wait until later this week, we are currently migrating to GitHub Pull Requests and it might be more familiar to you.

kloczek commented 1 year ago

OK will wait 👍 😄

mgorny commented 1 year ago

We have no problems with standalone builds in Gentoo. You just have to unpack a bunch of directories from the big tarball and run cmake from runtimes directory these days. A little hackery may be required, especially if you want to run tests and avoid cyclic dependencies but nothing very hard.

Here's our logic: https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-libs/libcxx/libcxx-17.0.0_rc3.ebuild?id=d29628d23bc1b4f988fbbde0c6ab16b87d1292f3 https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-libs/libcxxabi/libcxxabi-17.0.0_rc3.ebuild?id=0c2ab945dd43a0b441f9237fb96a4a2ee73eecba

barracuda156 commented 10 months ago

So for example, there's no hope of being able to build libc++ without libcxxabi/, runtimes/, cmake/, llvm/utils/lit

Since it is possible to use libc++ with GCC, it certainly makes sense to allow building it without LLVM and Clang.

kloczek commented 10 months ago

We have no problems with standalone builds in Gentoo. You just have to unpack a bunch of directories from the big tarball and run cmake from runtimes directory these days. A little hackery may be required, especially if you want to run tests and avoid cyclic dependencies but nothing very hard.

What you've described is nothing more than just classic JFDI solution. Again: as long as between those two LLVM components is circular dependency does not make to much sense keep that code separately.

ldionne commented 10 months ago

FWIW I think one solution here might be to merge libcxxabi into libcxx (just their CMake projects -- this doesn't have any bearing on whether we should be able to swap a different C++ ABI library under libc++). We run into many problems because of that, from this here to code duplication problems to bugs in a few cases.

However, at the end of the day we currently do not support building libc++ and libc++abi as standalone projects. We used to and we removed that a while back because there was way too much complexity around the 16 different ways you could build the runtimes. Nowadays we support few of them but we (try to) support them well.

I am going to close this because this "behaves correctly" w.r.t. our intent not to support standalone builds. Like I said, in the future this will hopefully become easier if we end up merging libcxx and libcxxabi (which is not an easy task, just to set the expectations).

barracuda156 commented 10 months ago

@ldionne Could you refer to some sensible procedure to build current/recent versions of libc++ and libc++abi without the rest of LLVM software? (I understand it is not supported presently, so I am not gonna open extra tickets if something fails, but I reckon it cannot be impossible, given that it worked before.)

tstellar commented 10 months ago

@barracuda156 https://src.fedoraproject.org/rpms/libcxx/blob/rawhide/f/libcxx.spec

kloczek commented 10 months ago

To be honest I still do not understand propose of creating LLVM per subproject dist tar balls and keeping them in unusable state for quite long time😞

Currently useability of those archives is limited because there are lots of inter submodules dependencies like:

On top of that since at least 13.0.x actually nothing has been done to address all those issues and looks like there is no to much intention to even review what already has been submitted (which is odd). From one side slicing into subprojects should allow and actually allows now (after apply some patches) to not waste hours to build whole LLVM stack again in case of any build/testing issues in the middle. I can submit batch of my fixes .. but many of these fixes would be worthless if installed by llvm/ cmake files would be correctly integrated into cmake/ submodule or better if cmake/ would be integrated under llvm/.

In other words .. currently except cmake/ NONE of the per subproject dist tar balls are useable without patching or using with other dist tar balls.

I'm not expecting to have all those (and few other minor) issues solved for yesterday however it would be really good to have from LLVM core maintainers some sketch of plan of how all those issues LLVM are going to be addressed.

Than with that plan would be possible to apply proper set of consistent solutions in which I'm 100% sure different distros maintainers would be able actively participate reusing at lease partially already used patches.

I fully understand as well that all possible fixes needs to preserve mono repo build as highest priority.

ldionne commented 10 months ago

To be honest I still do not understand propose of creating LLVM per subproject dist tar balls

Neither do I, to be honest. That's why I suggested that shipping the separate tarballs was misleading and we should stop doing that.

In other words .. currently except cmake/ NONE of the per subproject dist tar balls are useable without patching or using with other dist tar balls.

That's cause you're swimming against the current. We are in a monorepo now, and that's the way we build stuff. If you do things differently, you're on your own and as you noticed it's not a well supported path.

@ldionne Could you refer to some sensible procedure to build current/recent versions of libc++ and libc++abi without the rest of LLVM software? (I understand it is not supported presently, so I am not gonna open extra tickets if something fails, but I reckon it cannot be impossible, given that it worked before.)

I think things should mostly work if you ensure you have the following paths in their respective monorepo-like locations:

libcxx
libcxxabi
llvm/cmake
llvm/utils/llvm-lit
llvm/utils/lit
runtimes
cmake
third-party/benchmark

I think our builds don't really touch other parts of the monorepo so I think it should work. But none of this is officially supported, everything is subject to change at any time, etc.

kloczek commented 10 months ago

That's cause you're swimming against the current. We are in a monorepo now, and that's the way we build stuff. If you do things differently, you're on your own and as you noticed it's not a well supported path.

Sooner or later someone will find this as much more problematic as same as it was with XFree86/X11.

kloczek commented 10 months ago

Number of issue tickets in this repo makes LLVM ALREADY driven all to the state in which controlling all those tickets in single repo is unsustainable.

ldionne commented 10 months ago

If you want to change things, I would recommend directing your energy towards producing a well-written RFC to post on discourse.llvm.org. I'm certain you're not the only one who has issues with the monorepo. I'm not 100% sold myself, it definitely causes difficulties in a few cases, but it's what we have right now.

Endilll commented 10 months ago

Number of issue tickets in this repo makes LLVM ALREADY driven all to the state in which controlling all those tickets in single repo is unsustainable.

I'd say lack of manpower at least some projects experience is what makes this unsustainable, same goes for long review queues. I think standalone builds are in a sorry state for the same reason. I don't see how splitting the monorepo is going to address any of that.

barracuda156 commented 10 months ago

@Endilll As often, what matters is “not seen”. The current mono setup creates troubles which developers have to solve for themselves and projects they maintain. This results in duplicate efforts, less time to contribute and ultimately less motivation either.

tru commented 10 months ago

I don't think this issue is the right place to discuss the monorepos "be or not to be", it would have to go to the discourse and be exposed to the wider community in order to get any movement, so anyone that want to argue for a change in the LLVM infrastructure should post there instead.

kloczek commented 10 months ago

If you want to change things

As I wrote I really want however if none of the core LLVM core maintainers would want such change my effort would be worthless.

I would recommend directing your energy towards producing a well-written RFC

Issue is that exact change can vary depends on some goals which ae in case of LLV unclear. This is why I've asked to clarify few things about general goals. If those goals will be clear THAN it would be possible to propose exact change.

So far I don't see any comment from anyone who could plot direction of exact change.

In other words I'm not going to write all possible permutations of RFCs.

I'd say lack of manpower at least some projects experience is what makes this unsustainable, same goes for long review queues

Situations is more and more like in old communist time Polish joke. "A guy walking over the street found a very strange situation on roadside construction. Construction workers have been running in the loop with empty barrels between constructed buildings and piles of bricks. He asked one of the workers resting for a few minutes "What you are doing here?". The worker answered that that they don't know but they were so busy that they had no time to load and unload the bricks."

IIRC a year ago number of LLVM opened issue tickets was ~16k. Now that number is approaching to 20k. Number of issue tickets labels grew as well.

IMO dividing whole llvm/llvm repo into smaller pieces, delegate exact maintainers to each repo and create at least second layer chain of command, solving problems etc. would help:

Other things which IMO would be good to do to save preciouses time:

[1] currently testing per subproject does not work as well.

kloczek commented 10 months ago

Just in case if someone would be asking .. create subprojects with its own repos does not mean instant abandoning mono repo build and testing. IMO it is possible to reorganise whole code to use parent for example llvm/monorepo in which other subprojects will by imported as git submodules to be able on transition time reuse current set of build and testing methodologies.

BTW: divide whole LLVM into set of subprojects will DRAMATICALLY improve visibility of the git changes because it will be no longer one gigantic git log.

ldionne commented 10 months ago

https://discourse.llvm.org/t/rfc-do-something-with-the-subproject-tarballs-in-the-release-page