Manu343726 commented 4 years ago

Package Details

Package Name/Version: libclang/9.0.0
Website: https://llvm.org/
Source code: https://github.com/llvm/llvm-project

Description Of The Library / Tool

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.

I'm registering the issue as requesting libclang, since this is the most requested user library part of the llvm project. This is also the case for me and for other people that has reached me regarding llvm packaging (@lefticus, @flexferrum, etc). That said, libclang is only one library of the framework and the goal is to package and publish the full framework with one package per library. That's what I've been working on (See https://github.com/conan-io/wishlist/issues/197) and what will be discussed and documented in this issue.

I will redirect any conversation regarding llvm packaging to this thread, effectively deprecating https://github.com/conan-io/wishlist/issues/197. I encourage people to do the same to keep all the effort in one place only to ease with development and feedback. Also, if you find any conversation in conan, bincrafters, or cpplang chat in general please redirect people here. Thank you.

Manu343726 commented 4 years ago

I'm in the process of migrating the project with my recipes (https://gitlab.com/Manu343726/clang-conan-packages) from gitlab to GitHub so that we can open issues to track dev status and bugs in the repo and have these cross referenced in this request issue. The original gitlab project will be kept for continuous integration only until the recipes are merged into conan-center-index.

EDIT: Repo available here

Manu343726 commented 4 years ago

I've added a project to keep track of progress: https://github.com/Manu343726/clang-conan-packages/projects/1

Manu343726 commented 4 years ago

Windows compiled packages appear to be HUUUUUUUUUUUUUUUGE. Please conan masters be kind with me and be ready to increase my bintray account storage til we merge in the index :)

Here's LLVM 9.0.1 compiled with Visual Studio 2019: https://bintray.com/beta/#/manu343726/conan-packages/llvm:Manu343726/9.0.1:testing?tab=files

Manu343726 commented 4 years ago

@Croydon I'm trying to get the recipes merged here in the index. I appreciate your work (https://github.com/bincrafters/conan-llvm/commit/69ce9561fd304d9a27392cb87f4877c05ca9fac8) but I would prefer to focus on merging here and using the official Conan build matrix than integrating my recipes into the bincrafters infrastructure. Because, eventually, bincrafters packages will have to be merged into the index, right? So I see no point in doing the work with an extra level of indirection :)

Manu343726 commented 4 years ago

@Croydon that said, if you want to continue the bincrafters way I will be glad to hear from your feedback. Anyway, I'm afraid the recipes will diverge too much to be of much help :( (for example, today I patched all recipes to use the new python requires syntax, which required a lot of breaking changes in the recipes)

Croydon commented 4 years ago

@Manu343726 As I said here: https://github.com/conan-io/wishlist/issues/197#issuecomment-573464944

CCI isn't ready yet for a bulk recipes collection like LLVM or modual Boost or X11. You would need one sub-directory for each recipe, with a full set of files. For initial integration and every single update you would need to create a pull request for each and every one of them

today I patched all recipes to use the new python requires syntax

CCI doesn't support python requires, so we shouldn't use them for the LLVM recipes

Anyway, I'm afraid the recipes will diverge too much to be of much help

I have invited you to the repository as I believe you have shown interest in the past to get this into Bincrafters. So you can directly push into the testing/<version> branches (and create experimental/<version> if you want to).

Please wait with pushes into the testing/6.0.1 branch, until I have made some CI adoption changes. But feel free to create one experimental/<version> branches right now as you like 😄

To summarize: Yes, the (long-term) strategy is to get everything into CCI, but CCI is still in early stage and not ready for all recipes. And not all recipes are ready for CCI. It is my offering to you that we get LLVM mature in Bincrafters and when LLVM and CCI are ready we get it merged here

Thanks for all your work!

jgsogo commented 4 years ago

Hi! We've been thinking a lot about how to make LLVM into CCI after this PR https://github.com/conan-io/conan-center-index/pull/1075. Our main concern is to not introduce something that we cannot afford in the future. I hope this is understandable to all of you.

1075 is a concrete request for one component of LLVM, maybe not a core library but we need to think about as it was and the solution should be valid for all of them. We need to take the risk and make decisions and this is what we have come up with:

individual packages are useful because you are downloading only 5Mb instead of the 2Gb from the full framework
individual packages for components of a framework are ok, if those components make sense isolated (not for the purpose of having just more recipes here)
we do not want to model the dependencies between the components, we've learned from the feedback of modular boost that it is real pain not just to create the graph, but to maintain it and update all the modules in order.

Our proposal then is to re-package the components from a master package: LLVM will we compiled once in llvm/10.0.0 and then this package llvm-openmp will take all the binaries it needs from that master package and create a new package from it:

(not real dependencies)

This doesn't mean that we need llvm/10.0.0 first in order to package llvm-openmp, it means that once we have llvm we should rewrite the recipe for llvm-openmp from scratch. This way we can merge #1075 now and modify it in the future.

Nevertheless there are problems with this approach and current Conan version. I will enumerate here some of them:

if we are using several components of llvm in a graph, all of them should belong to the same version, otherwise ABI will be compromised. To ensure it, the recipes repackaging need to close a diamond, something like this:

class LLVMOpenMP(ConanFile):
   name = "llvm-openmp"

   def requirements(self):
       self.requires("llvm/{}".format(self.version))

   def configure(self):
       if self.version != self.deps_cpp_info["llvm"].version:
           raise ConanInvalidConfiguration("LLVM version mismatch")

repeated internal libraries: if every component repackages all the binaries it needs, there would be libraries (artifacts) duplicated. Given (1) all of them belong to the same version, so they should be identical the same artifact:
- when linking with them, will the linker identify them as the same artifact and use only one of them?
- regarding dynamic libraries, runtime will load just one of them without error?
Updating llvm requires us to update (to run CI again) for all the components. Rebuilding dependencies is something in our roadmap for ConanCenter, so it is not something new given this scenario.
With the current graph model, we can use different types of requirements to model the dependency between the components and the master package: if we use build_requires or requires(private) Conan won't close the diamond and we'll fail with (1). Using regular requires Conan will propagate the information from the master recipe to the consumers, the problem is that it will contain both, the component and the master recipe (and it will download binaries for both of them). We would need a requirement we don't have: something that closes the diamond, but doesn't propagate information.

As you see, there is an open problem, so IMHO the best we can do is to move forward #1075 and, at this moment, delegate on the user the responsibility of checking that the version is the same (eventually we will have several llvm-xxx components before having the master recipe). Then, when the master recipe is available, implement it in terms of Conan components and recommend the single one over the others, reimplement llvm-xxxx with repackaging (maybe with build_requires and make the user responsible of not mixing version) and when Conan allows it, use a proper requirement.

Any feedback is welcome. Thanks!

Manu343726 commented 4 years ago

I just want to comment that while I like the approach, I'm worried about disk space (It is a problem if you have multiple versions and toolchains, like it's the case in my company for example). The other approach, re-packaging all the components and writing requirements between them (i.e. modelling the full components dependency graph) solves this issue but makes CI setup harder (For reference check my gitlab CI, which does that. I'm sure it's linked in one of the comments above).

Manu343726 commented 4 years ago

For reference, this is a du output of my laptop conan local cache:

    5,4 GiB [##########] /qt                                                                                                                                                                
    3,6 GiB [######    ] /clang
    3,1 GiB [#####     ] /boost
    2,9 GiB [#####     ] /llvm
    2,1 GiB [###       ] /kfc
    1,0 GiB [#         ] /libclang
    1,0 GiB [#         ] /clang_executables
  532,6 MiB [          ] /compiler-rt
  449,4 MiB [          ] /tinyrefl-tool
  128,2 MiB [          ] /libcxx
  108,6 MiB [          ] /icu
   78,5 MiB [          ] /tracy_client
   65,7 MiB [          ] /tinyrefl
   48,6 MiB [          ] /idkit
   44,6 MiB [          ] /libmysqlclient
   34,5 MiB [          ] /glib
   33,4 MiB [          ] /clang_sema
   32,3 MiB [          ] /libx11
   30,4 MiB [          ] /tracy
   29,5 MiB [          ] /fontconfig
   28,1 MiB [          ] /fmt
   26,0 MiB [          ] /libxcb
   23,2 MiB [          ] /openssl
   22,7 MiB [          ] /gtest
   20,8 MiB [          ] /llvm_codegen
   19,7 MiB [          ] /pistache
   18,9 MiB [          ] /clang_static_analyzer_checkers
   16,7 MiB [          ] /clang_arc_migrate
   16,3 MiB [          ] /clang_ast
   15,9 MiB [          ] /llvm_headers
   14,4 MiB [          ] /llvm_analysis
   14,1 MiB [          ] /llvm_x86_codegen
   13,6 MiB [          ] /cppast
   13,0 MiB [          ] /llvm_scalar_opts
   12,4 MiB [          ] /odbc
   11,4 MiB [          ] /clang_headers
   11,2 MiB [          ] /llvm_selection_dag
   11,0 MiB [          ] /llvm_core
   10,8 MiB [          ] /clang_basic
   10,7 MiB [          ] /backward
    8,6 MiB [          ] /clang_static_analyzer_core
    8,6 MiB [          ] /zlib
    8,6 MiB [          ] /house-of-cards
    8,5 MiB [          ] /concurrentqueue
    8,1 MiB [          ] /clang_driver
    7,5 MiB [          ] /llvm_x86_desc

The libclang package takes ~1GB because it also includes the clang executables (Like the clang_executables next to it). The libclang library itself takes just 1MB

As you can see, the clang root package is ~3GB (Around 2GB for the build folder, 1GB for the package). But then the different modules are just a few MB each, because these packages include each module library only. If we follow the @jgsogo's approach we will end up with a ~300MB libclang package, a ~300MB libtooling package, and so on only for one version and toolchain.

I'm listing the most usual llvm/clang libraries requested by users as an example, but I expect people asking for more packages as the thing starts to work and known in the community

I'm not saying the approach is wrong, neither that mine is better. Both have its pros and cons. I'm just trying to share my concerns with you and see what solution we could find.

Manu343726 commented 4 years ago

More reference for the debate :) Full clang 6.0.1 dependencies graph: https://github.com/Manu343726/clang-conan-packages/blob/master/clang_6.0.1_lddtree.txt

Manu343726 commented 4 years ago

I'm tempted to write a tool to automate the process. Build llvm/clang/qt/whatever as dynamic libs, extract the dep graph of each compiled library, and automatically generate a repackaging recipe for each.

jgsogo commented 4 years ago

This is a message from the future: from the POV of a user, if I'm going to use several modules of LLVM I won't use the repackaged recipes (n*300Mb) but the master one with all the LLVM (~1Gb).

What about recipes in CCI that require a module of LLVM? Should they be modified to use the master LLVM recipe? Probably yes, but not sure 😓, recipes for modules should be a leaf in the CCI tree to avoid the scenario you are worried about.

Manu343726 commented 4 years ago

if I'm going to use several modules of LLVM I won't use the repackaged recipes (n*300Mb) but the master one with all the LLVM (~1Gb).

Trust me, it's exactly the opposite. Because using llvm with current tech is a mess that involves custom CMake macros to find the dependencies of the modules. The main advantage that conan provides is that you can use llvm modules like any other library, just require the library you need, leaving the dependency mess to conan to solve.

With current approaches you end up doing something like this: https://github.com/jfultz/libtooling_step_by_step/blob/master/CMakeLists.txt

With conan you just do:

[requires]
clang_libtooling/6.0.1

[generators]
cmake_find_package

find_package(clang_libtooling REQUIRED)
target_link_libraries(mytool PRIVATE clang_libtooling::clang_libtooling) # Magic

TL;DR: I don't expect anybody to use the root LLVM package directly (Which requires to deal with the deps mess) once the repackaging is available. Note that what I'm talking about applies to both repackaging approaches.

Oh, also note that I'm not theorizing here, that is what I have been doing for two years with my conan packages :P

Manu343726 commented 4 years ago

BTW we need libtinfo (Part of ncurses iirc) for the linux build, it would be awesome if someone from bincrafters could port the package to conan-center-index.

Croydon commented 4 years ago

ncurses is a conan-community recipe, not a Bincrafters' one

Porting is work-in-progress here: https://github.com/conan-io/conan-center-index/pull/994

jgsogo commented 4 years ago

@Manu343726 , regarding your comment here https://github.com/conan-io/conan-center-index/issues/683#issuecomment-619213342

With the single package and the components® feature in Conan, consuming clang will look like:

[requires]
clang/6.0.1

[generators]
cmake_find_package

find_package(clang REQUIRED)
target_link_libraries(mytool PRIVATE clang::libtooling) # More magic

We think this is a better approach and this is what we are trying to achieve (hopefully Conan v1.26), wdyt?

newlawrence commented 4 years ago

Maybe a little bit late to the conversation. In my company, we are following the same approach as @jgsogo said above, just we are only using the core libraries though.

I've found that, in this particular case, you don't need to compile any shared libraries to manually resolve the dependency graph for the package_info section as @Manu343726 said in #683 (comment). You can just gather that exact information by looking at the LLVMExports.cmake file if you install the library by cmake means in the package section instead of manually copying the binaries and the headers. That file contains the list of the targets and their dependencies being pretty easy to parse.

That is how we are doing to implement the magic of #683 (comment).

Manu343726 commented 4 years ago

@newlawrence you owe me a link to a github repo... :P

newlawrence commented 4 years ago

There you are @Manu343726. Sorry it took so long... 😅

As you can see, I use networkx to resolve the linking order in just one line. I'm such a lazy person... 😁. So that's it, for the whole clang+llvm project the same could be done using the new components to model each one of the tools and libraries.

One thing I've been discussing with @jgsogo is that maybe it is not necessary to install the libraries to resolve the dependencies after all. You can ask cmake for the relations between the targets it has, but I'm not sure sure it is going to be as easy to parse though.

Manu343726 commented 4 years ago

I'm sorry I have been away from this, it's been a mess these last six months. So @newlawrence I finally took a look at your lexicographical sorting black magic, I like it. As you say, we can model components out of this. What do you think @jgsogo, what if we push @newlawrence's version (It's battle-tested), we give our dieses to the Linux version, and we write a little roadmap here for:

Tools (llvm tools and clang compilers)
Windows support
Mac support

that way we push the hard part to the index and more people could help with these 3 points.

newlawrence commented 4 years ago

I've being fixing some issues with my recipe these holidays. In the process I also paved the way for modeling each independent library as a conan component (they will come in handy if someone wants to link against the big all-in-one dynamic library instead of the individual static ones). The only thing left is to give support for multiple versions in the recipe.

One thing to keep in mind is that the recipe does not follow conan-center's guidelines, as I'm installing and shipping all of cmake's stuff along the binaries (which I use to gather the information about the interdependencies in the first place); so we need to figure it out another way of getting those relationships beforehand. We can also just remove that stuff once it has served its purpose but, come on, there has to be a more clever way to do that.

Also remember that this is only a recipe for the core libraries at the moment.

newlawrence commented 4 years ago

I've been playing a little bit more with the recipe along the weekend. Using conan components to handle the dependency graph there is no need to rely on an external library like networkx. The recipe is cleaner now.

A little comparison between using components instead of the whole library can be found here. While shared builds use the big single library approach recommended (see LLVM_BUILD_LLVM_DYLIB), for static builds you can choose to link against those components you need.

newlawrence commented 4 years ago

I've finally submitted a pull request for the core libraries to conan-center-index.

As discussed in this other pull request, I think we could just go for monolithic builds of the full llvm repo and then just remove all the stuff that does not belong to each package. I know that will mean to compile the same binaries several times across different packages, just to be removed later on; but, it will do the trick. What do you think?

We can continue the discussion in the aforementioned pull requests.

xahon commented 2 years ago

What's the state of libclang? llvm-core doesn't contain libclang's libraries in it

qqiangwu commented 11 months ago

Are there any updates? I'm trying to write a static analyzer for C++, which relies on the recipe.

jusito commented 11 months ago

@qqiangwu Do you need llvm 13 to 17 or version 9 in particular? If you can work with a version of that range, please try https://github.com/conan-io/conan-center-index/pull/17509. It's unfinished, but clang libs should work because I need them.

qqiangwu commented 11 months ago

@jusito Awesome! Thank you for your hard work. I'll try it after I figure out how to consume recipes in your fork.

conan-io / conan-center-index

[request] libclang/9.0.0 #683

Package Details

Description Of The Library / Tool

1075 is a concrete request for one component of LLVM, maybe not a core library but we need to think about as it was and the solution should be valid for all of them. We need to take the risk and make decisions and this is what we have come up with: