Support using Abseil via CMake find_package

[nolange]

Hello,

I have a bit of trouble using a local Abseil installation, as I dont want to depend on external packages with some fixes URL nor some fixed filesystem paths. Cmake already HAS the necessary infrastructure to allow building dependend modules.

For using Abseil in your Project, just this would be necessary:

# cctz pulls in some gtest and benchmark suites otherwise
set (BUILD_TESTING OFF)
find_package(Abseil REQUIRED
)

And, given the worst-case that abseil and cctz are in no directory that can be found automatically, configuration would add the respective paths: cmake -DAbseil_DIR=<local path> -DCctz_DIR=<local path> ...

Basic support would be added by this patch (note that cctz would need to be fixed in a similar fashion):

diff --git a/AbseilConfig.cmake b/AbseilConfig.cmake
new file mode 100644
index 0000000..4476b98
--- /dev/null
+++ b/AbseilConfig.cmake
@@ -0,0 +1,2 @@
+# Macros
+include(${CMAKE_CURRENT_LIST_DIR}/CMake/AbseilMacros.cmake)
diff --git a/CMake/AbseilMacros.cmake b/CMake/AbseilMacros.cmake
new file mode 100644
index 0000000..94c6886
--- /dev/null
+++ b/CMake/AbseilMacros.cmake
@@ -0,0 +1,8 @@
+# TODO: this should be another find_package(Cctz)
+add_subdirectory(${Cctz_DIR} _libcctz
+   EXCLUDE_FROM_ALL
+)
+
+add_subdirectory("${CMAKE_CURRENT_LIST_DIR}/.." _libabseil
+   EXCLUDE_FROM_ALL
+)

[JonathanDCohen]

Hey there! This looks reasonable at first glance -- rght now our CMake support is still very much experimental, but it should stabilize hopefulyy in the next couple of quarters as the team gains more cmake knowledge and experience.

My understanding is that we should also provide a FindAbseil.cmake file to go with this, yes?

[nolange]

FindAbseil.cmake would be necessary if Abseil does not provide CMake support itself. This is then typically used in the packages using Abseil or provided with CMake itself.

There might be exceptions to the rule, but generally you provide the AbseilConfig.cmake file to be able to build it as submodule (illustrated above). Then you add support to installing some CMake helpers via the install(EXPORT ... mechanism, so find_package will automatically find the system-installed package.

The most annoying thing right now is the cctz dependency which will try to use gmock, gtest and some benchmarking software...

[JonathanDCohen]

Okay, getting back around to this. Just to make sure I'm understanding your proposal correctly, since I'm still learning CMake -- this would, via install mechanism, compile abseil from source into a system libabseil library, which then could be found automatically via find_package?

Would this library get rebuilt on each compile? A main concern of ours whenever install() rules come up is that we'd like to avoid if at all possible users depending on a precompiled version of abseil, since we don't promise ABI compatibility. It looks like EXCLUDE_FROM_ALL forces abseil to be rebuilt at compile time, if I'm reading the docs correctly.

Am I understanding this correctly?

[nolange]

My proposal is making CMake support better.

• The patch from my first post adds support for compiling with your project (some compiler-flags and everything)
• The install portion would be an additional step, common to CMake project are helpers who do what you apparently don't want: install a static or dynamic library for building dependent projects. I haven't used this much, but there can be separate version for debug and release.
• What you want, is some other install target which more or less copies everything to a directory, and possibly adds some CMake-glue so a find_package(Abseil) will locate the copied package.

To me the first two use-cases are important, the first is fitting for using a prepared or automatically downloaded version. The second would be interesting for distribution support, ie. debian typically recommends using dynamic libs, those are locked in an compatible version for each release (ie. just bugfixes). Compromising on that likely makes it difficult for projects using abseil to be built in a conforming way (isolated builders). The third is just a variant of the first, shouldn't be to hard to get this going.

As for my personal opinion, you try to use a project more or less as a header-only library, but have some more "heavy stuff" like the time conversion functions using an additional library. If you could try separating those into a clean small base library without further dependencies and some special purpose modules with maybe ABI compatible helper functions then the solution(s) would be easier.

[Mizux]

please look at #38 TLDR you should depends on abseil as add_subdirectory() than trying to install it first. That's why abseil team don't want to provide install rules AFAIK.

you can look at this example if you want: https://github.com/Mizux/cmake-abseil Please note the windows/alpine issues spotted (^v^);

[JonathanDCohen]

add_subdirectory("${CMAKE_CURRENT_LIST_DIR}/.."

-> -> -> _libabseil <- <- <-

EXCLUDE_FROM_ALL

The part marked with arrows is what we don't want to support. Because as soon as one of your dependencies has a different libabseil, you're swimming in an ocean of ODR violations. We don't provide ABI stability -- if you want to use a compiled version of ABseil, that's your business, but we don't want to support it.

Being able to specify some other abseil directory, on the other hand, is great. If you can then point all your dependencies at the same absl/ directory, then everyone is happy. Would that be reasonable for you?

[nolange]

The part marked with arrows is what we don't want to support...

You got that completely wrong, that's the directory libabseil.a will be built in, the directory where the Abseil sources are is given at commandline with -DAbseil_DIR=<local path>. I would want to use Abseil with a cross-compiled project, so you don' t need to preach about the downsides of pre-compiled libraries in my direction =)

This already works with the patch in the very first post, maybe just spend a minute testing it?

@Mizux already mentioned some issues with add_subdirectory however. Not really fit for suggesting another solution, but I would not want to depend on an internet connection for building. Likely will just place Abseil in a subdirectory and be done with it

[Mizux]

Hi, So first we are all agreed to have a single abseil (not prebuilt !) in your super-super-cmake build project. So my current workflow currently for incorporating a dependencies in my project is: 1) Check if the cmake target (with cmake namespace if possible) is present 2) if not, use ExternalProject to retrieve the source and incorporate it in my CMake build.

note: 1 is needed if several deps (B, C) of my project A could depends on D and one of them already load it. note: 2 if you don't like depending on url for external project you could integrate abseil directly in your project and add_subdirectory() it first so other deps will find yours and will not try to download it (if they use 1) cons: you could have a "old" version of abseil if you don't update regularly but at least you'll have one abseil version. cons: if you integrate everything as cmake subproject you could have a super mega giga CMake project which can take time to compile everything so you reach the 4MB log limit in Travis CI or the 50min timeout.

Examples: In cpu_features we check targets gtest / gmock when building tests if not found let's incorporate googletest cf https://github.com/google/cpu_features/blob/master/CMakeLists.txt#L89 In or-tools which depends on gflags, glog, zlib, protobuf (and soon abseil) so I do the same for all our dependencies cf. https://github.com/google/or-tools/blob/master/cmake/external/CMakeLists.txt#L47 please also note that glog depends on gflags, and protobuf on zlib and... they use find_package() -> so I overload find_package() to no op if target present and add the necessary alias in zlib/gflags (via ExternalProject patches or PR) thus find_package become a kind of find_target() (here you start to reach the limit of Modern CMake IMHO) cf https://gist.github.com/mbinna/c61dbb39bca0e4fb7d1f73b0d66a4fd1#using-a-library-defined-in-the-same-cmake-tree-should-look-the-same-as-using-an-external-library

Bonus question: if you build a standalone "sdk/software suite" which depends on abseil you'll have to provide at least abseil headers (if they are part of your api), since abseil don't provide install rules (to avoid build abseil as standalone and have several versions in your system I know) what to do ? Also it become tedious to provide semantic version for your software suite while using a moving dependencies target (e.g. abseil master in an external project...).

[nolange]

I am still trying to get my head around this, but your approach seems rather sensible, thanks @Mizux . There aren't much indepth guides discussion the option of managing depend projects with CMake.

ie. those 3 options should be transparent to the rest of your project (configured via CMake defintions on the cmdline):

1. use dependency from source from a path (potentially subdirectory / git submodule)
2. use precompiled dependency from a path
3. use precompiled dependency from your system installation. (in absence of any arguments, this should be tried first)

So first we are all agreed to have a single abseil (not prebuilt !) in your super-super-cmake build project. The only thing I would object here is that it should be mostly transparent whether you would want to use abseil by compiling it from source, compiling it once for a CMake super-project, or using a prebuilt one. Think of creating an root filesystem for embedded devices - here I wont run into any compatibility troubles and might want to use shared libraries for saving memory (not sure whether is is an argument that holds with an mostly-header library like abseil). Whatever independent projects I compile for this rootfs will/should use the same libraries - hence ideally I would install the libraries once in the rootfs (this could be shared, static or source-libraries). there are projects like buildroot which automate that, means add abseil there and any other projects should use this version by default.

That would fit into your "sdk/software suite" aswell. a "install target" for this need would really help, but they would not necessary end up installing shared libraries but static ones or a CMakeLists.txt describing a source library (object library?)

[Mizux]

Yes, I guess Yocto community will be happy to have an abseil-cpp recipe (they already have CMake plugin provided the CMakeLists.txt contains install rules)

@nolang I guess 1) "dependency from source" could be split in two topics: 1a. use dependencies from source given an "env_DIR" variable 1b. automatically retrieve dependencies sources and incorporate it using ExternalModule (pro:
user just run cmake and every deps are retrieved,
you could reasonably use CI to guaranty compilation and build being reproducible for this setup) note: gflags for example checks if currently configured as subproject or as standalone to choose static/shared enable/disable unit test by default etc...

For instance currently:

• I don't know which version of cctz abseil-cpp has been tested against using public CI nor kokoro.
• cmake . just crash so people can't built it as standalone project (for just running unit test/collaborate for example) they have to provide a Meta-CMakeLists.txt on top...
• Software Embedded user building "reproducible distro" (using buildroot or Yocto) for embedded device (e.g. SoftBank Robotics Pepper Robot) usually are [source] package based system (and don't have Google resources to rebuild every deps for every pkgs...)

[nolange]

I have troubles with 1b, namely that this will either need to download during configuring, or you will have to hardcode correct paths/dependencies as you cant use a CMakeLists.txt that ain't there yet.

Usually I would want to always write CMakefiles that use the installed targets (eg. Hugo::Hugo instead of hugo). Which means to install all dependencies (that aren't already available).

The idea I am currently playing with, is having a separate CMakeLists.txt file taking care of all dependencies. that would be a manual 2-step process but it seems the most sane to me (outside of throwing everything in subdirectories - but that doesn't scale for libs used multiple times and git submodule has its own ugly warts).

(CMake's ExternalProject is rather useless for cross-compiling aswell)

Might have to checkout hunter, sounds like this covers similar ground

[Mizux]

Sorry if I miss something but with target alias (if provided -_-) you can use Hugo::Hugo even when add_subdirectory() deps.

separate CMakeLists.txt did you means something like https://github.com/google/or-tools/blob/master/cmake/external/CMakeLists.txt (cf directory for deps) ? TLDR: you use ExternalProject/cmake command to download only deps then you integrate CMakeLists.txt deps on the fly at configure time...

I use ExternalProject just for cloning deps in a "portable way" (pros: don't have the issue with submodule overwrite by a git merge)

libs use multiple times accross your "project" or the "distro" ?

[nolange]

yeah, I already used the ALIAS trick, still that's extra code that someone has to maintain. In a perfect world a helper like ExternalProject could just use the install statements to create such shims. Its actually what I expected of CMake before I started doing some more complicated setups.

libs would be used across multiple projects ideally, not distro (might be a different cpu architecture or OS). Lets just call it a staging area. Yes, the file you linked is basically a template for that.

before building your project, you would build/update the staging area to contain the modules you need - either in source form or as library. this could be used for multiple projects (as long as the target matches and compiler/-settings agree).

[JonathanDCohen]

Hey @Mizux

So your idea is basically what we want to go for. Looking at other projects (folly, really) it looks like what we need to do from our end is an install EXPORT rule so you can reference abseil targets, and then an install(DIRECTORY) to copy abseil's files over into your source directory.

One unfortunate thing is that our headers and .cc files are in teh same folders because that's just how we structure projects at Google -- Bazel lets you declare private headers which aren't visible throguh your target, so there's less need for us to have includes/ directories.

My question then is what else do we need to do to support this use case.

@nolange -- I'm still a little bit wary of your find_+package approach. I'm imagining a project directly depending on abseil throguh their libabseil.a, but then depending on another project which uses a different libabseil.a and everything explodes. Am I missing something?

Thanks everyone for your patience with this. I took lead on this project precisely because I knew ~nothing about this space and wanted to learn through doing.

[nolange]

@nolange -- I'm still a little bit wary of your find_+package approach. I'm imagining a project directly depending on abseil throguh their libabseil.a, but then depending on another project which uses a different libabseil.a and everything explodes. Am I missing something?

yes everything will explode if you use different incompatible versions. you are missing out that it is completely unrelated whether find_package was involved, you will get the same mess with any other mechanism if you don't specify the same package version.

find_package to me, is the canonical way to reuse software modules in CMake, so this would be the first attempt to get it working. You can then add a whole rube-goldberg-apparatus to download the correct versions, but to me building and preparing/installing dependencies is a separate prior step. The build using abseil should not be affected and just gets passed a variable so find_package will search in the correct directory.

In case you aren't familiar, I would recommend you to create a rootfs with buildroot, and then understand how a package using CMake will automatically pick the libraries and headers not from your OS but from the created rootfs which might not even be the same CPU architecure. CMake is popular exactly because find_package will "do the right thing" aslong you pass along a few parameters (mostly a toolchain file and path to such a rootfs).

consider building a set of apps using abseil would use a similar prepared "rootfs" to solve API/ABI issues, so the problem is the same.

Thanks everyone for your patience with this. I took lead on this project precisely because I knew ~nothing about this space and wanted to learn through doing. Thats great, can I hire at google and lead their AI efforts - I know nothing about AI and would like to learn ?(sorry, I had to bite - this is what it sounds like, I am sure it was meant differently ;) )

[Mizux]

Sorry fall sick last week... @JonathanDCohen I guess this topic could be close as now more or less a duplicate of #38. (i.e. you need install rules with Config.cmake to be able to use find_package (note: yes you can export(EXPORT) build tree to use find_package without install rules but I want to keep it simple...))

also vcpkg (Microsoft backed library package manager more/less based on CMake) have a good starting point for install rules -> https://github.com/Microsoft/vcpkg/blob/master/ports/abseil/CMakeLists.txt (and this is why you should provide your install rules directly) (note: as statute you still need to ship the dll deps with your apps)

Concerning private header you'll have to REMOVE_ITEM (e.g. here) them from GLOB_RECURSE list if their are mixed with internal, I fear here you have to deal with bazel/cmake implicit directory layout convention...

Personally for google/or-tools which is a library and a standalone sdk I guess I'll have to support few ways of integration: 1) build ortools as standalone sdk so I'll need to package also all its dependencies to create a big fat standalone archive... (currently we have Makefile rules which provide only this) -> need all deps (i.e. abseil-cpp) to have install rules and incorporate them all as sub_directory EDIT: I thinking I could do it in an external git project than trying to have my ortools CMakeLists support this workflow and the following one... Also I need this because we want to provide pypi/nuget/maven packages so it easier to have all deps as static library and integrate in our ortools.lib with SWIG wrapper for creating a standalone binary package for this languages.

2) Support customer who want to just use ortools in their own local CMake project/apps using the distro prebuilt package (i.e. using find_package ?) -> in this case I think, I MUST just check my deps cmake target are availabe (i.e. abseil-cpp target(s)) and don't try to ExternalProject/add_subdirectory them (could be done in the Config.cmake) 2bis) Also, it means that I need to support the build of ortools with dependencies provided by distro/system packages (aka find_package) which can be in differents version according to distro release politics/rolling-release/etc than the one I test when I build everything from source in 1). requirement: all my deps are available in the distro otherwise I'll need to support them (i.e. create a package for my deps A) if I want to provide my package ortools...

3) Support people who want to incorporate a specific version of ortools (or abseil?) in their build (or can't/want rely on distro package version): 3.1) by being able to incorporate ortools as sub_directory (e.g. using Gtest ExternalProject tricks) 3.2) by building ortools specific version first then consume its install output in their build (i.e. using CMAKE_PREFIX_PATH to retrieve this artifact what #38 guys want basically) I think this workflow is just the wrapping of 2) in a customer project.

I guess abseil-cpp need to support 3.1 and 2/3.2. 3.1 is cool for "live at head" (and it's the only way to use a lib which doesn't provide install rules). 2/3.2 who need install rules but could also be use to "live at head" or for maintainer to provide a distro package

Actually 1 and 3.1 is only good when install rules / config.cmake is bad so it's better to incorporate deps source code directly than trying build/install/consume them (I think about cross compilation and passing all stuff to deps to prebuilt them with the same conf than "me", my current lib)

[JonathanDCohen]

I guess abseil-cpp need to support 3.1 and 2/3.2.

Yeah, pretty much this is where I stand too

so it's better to incorporate deps source code directly than trying build/install/consume them

This is also where I stand. Especially because absiel is about to drop our cctz dependency and so become self-contained in non-test builds, I don't really see an advantage of build/install/consume libabseil.a or whatever when you can just build straight from source without issues.

[JonathanDCohen]

This is now tracked in #111. I'll keep this open until that issue is resolved, since there is a lot of good discussion here.

[Faisalsalipada]

Ok

[milahu]

FindAbseil.cmake would be necessary if Abseil does not provide CMake support itself.

cmake/FindAbseil.cmake

note: this will not use the system-wide install of abseil, but will download abseil into the workdir