Project-Based Package Management for Dependencies

[elsiehupp]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Expected behaviour

Rather than pestering a contributor to install dependencies on a system-wide basis before completing a build, the build system should manage them within the scope of the project. Additionally, the user should not need to worry about version compatibility, which can be a persistent and annoying issue. For example, OpenSSL conflicts with the system's LibreSSL on macOS, and Linux distributions such as Ubuntu have been know to ship comedically outdated versions of packages.

Actual behaviour

CMake pesters the user about dependencies, and setting up the build environment is unnecessarily complicated. For example, the user needs to manually specify the install paths of OpenSSL and certain Qt libraries on macOS, and this specification requires an ad hoc install script, cluttering the user's shell environment, or an onerously long series of qualifiers in the CMake invocation

Steps to reproduce

1. Clone the repository
2. Create the build folder and run CMake
3. Oh look there are other things you might need to install besides CMake (and maybe Xcode)

Client configuration

Operating system: any/all (but especially macOS) OS language: any/all

Possible Approaches

There are a number of project-based package managers designed around (among other things) C++. These run the gamut from hooking to CMake to entirely replacing it. While there are advantages to using an integrated package manager and build system, a more incremental approach might be more feasible in the immediate future. Among the options I looked into, Hunter seemed the most lightweight, due to the fact that it is written entirely in CMake and does not create any additional dependencies for the user to install.

[FlexW]

That is in general a good idea. But I don't think that this is CMake's fault. CMake's scope is just to create build files (and it does this quite well in my opinion:)). It's not a dependency management tool. I would keep CMake for the creation of the build files like it is now. For dependency management I would go for Conan as it seems to become a standard. But it will be not a small undertaking to move from our current system to conan.

[elsiehupp]

Conan does seem to be quite popular. The reason I suggested Hunter, though, is that it slots neatly into CMake (since it literally just consists of CMake scripts), so it would be more feasible to implement in the somewhat nearer future. I mean, I could try doing it myself once your update to the main build scripts gets merged (if it hasn't been already). Or I could throw caution to the wind and try and implement it even if your merge hasn't happened, yet, and then deal with the rebasing and/or awkward merge when the time comes to it.

FWIW, the page I linked earlier explains Hunter like this:

The Hunter client is a collection of CMake-only modules (i.e. it’s not a binary like apt-get or script like brew) so it supports out-of-the-box all platforms/generators/IDEs which CMake can handle, like Visual Studio, Xcode, Android Studio, QtCreator, NMake, Ninja, Cygwin or MinGW. It works fine with CMake-GUI too.

The prime directive used for adding package to the current root is hunter_add_package which companioning CMake’s find_package. For example:

hunter_add_package(Boost COMPONENTS system filesystem iostreams)
find_package(Boost CONFIG REQUIRED system filesystem iostreams)

It even generally works in most existing CI environments, too!

Again, it does seem more ideal to switch to an integrated build environment of one sort or another (and the blog post I linked to does go into this), but something lightweight like Hunter in the nearer future need not be mutually exclusive with a larger revamp down the line.

[FlexW]

Conan does seem to be quite popular. The reason I suggested Hunter, though, is that it slots neatly into CMake (since it literally just consists of CMake scripts), so it would be more feasible to implement in the somewhat nearer future. I mean, I could try doing it myself once your update to the main build scripts gets merged (if it hasn't been already). Or I could throw caution to the wind and try and implement it even if your merge hasn't happened, yet, and then deal with the rebasing and/or awkward merge when the time comes to it.

Integrating conan dirty with our CMake scripts would be roughly one line. It's pretty simple. That's one of the reason why I would go for conan.

What I think takes more time is to make sure all the installers are created correctly and that all our build environments must be adjusted.

Again, it does seem more ideal to switch to an integrated build environment of one sort or another (and the blog post I linked to does go into this), but something lightweight like Hunter in the nearer future need not be mutually exclusive with a larger revamp down the line.

I'm unsure I understand what the benefit of buckaroo in comparison to CMake + Conan would be.

[nicolasfella]

In KDE we have a very similar status quo where there is virtually no dependency management in the (CMake) build system itself. This has the advantage of being very flexible and makes it easy to accommodate for various platforms and packaging formats that often have their own ideas about how dependencies should be handled (classic Linux distros, Flatpak, Snap, Appimage, Windows, macOS, homebrew, Yocto etc just to name a few). This works fairly well for the case of Linux distros assuming all dependencies are packaged in a recent enough version. For Windows and macOS the developer experience gets worse. We address that by offloading the dependency management to a tool (Craft). That works well enough and could also be an option for Nextcloud. I belive ownCloud also uses Craft (which makes sense given the ownCloud desktop maintainer also maintains Craft).

With the current setup adding a new third-party like for example a KDE Framework (we currently have copies of individual files from them in the tree) would add some significant friction to the build process. If we can reduce that friction by better automatic dependency management that would be a very welcome change in my book.

I don't have any particular experience with any of the proposed solutions so I can't compare them. The only somewhat hard requirements I see are that the solution should allow Linux distributions to use their system packages instead and should avoid building Qt from source if avoidable, since that takes hours even on a very decent machine and that is a burden I do not want to impose on prospective contributors.

[elsiehupp]

An alternative to Hunter that I came across is CPM.cmake, which is explicitly inspired by npm. My understanding with CPM vs Hunter is that CPM allows for specifying particular package versions, much like npm allows. Of course, both Hunter and CPM allow for downloading arbitrary web assets, as well.

This Reddit thread goes into some more details about the pros and cons of Hunter in particular (and is how I found out about CPM). One thing in particular that seems appealing is how one of the commenters describes using Hunter as an optional backend for the existing find_package() function in CMake, so that the CMake scripts generally don't have to be aware if Hunter's existence. That is, the CMake scripts could use Hunter for caching package downloads.

Again, one of the biggest advantages of Hunter and CPM.cmake is that they are both literally just CMake scripts, whereas Conan, for example, requires Python. Nextcloud's documentation build already requires Python, so it isn't strictly an additional dependency, but it kind of is if you're just describing the main build process.

It's also worth mentioning that Homebrew is already a recommended part of the process of setting up a build environment on Mac (though Homebrew could be used on Linux, as well). Having something like Hunter or CMake could obviate the need for installing Homebrew, though on macOS Homebrew is still the easiest way of installing CMake itself due to the lack of apt or something similar. (I mean, if it comes down to it, we could just have the instructions tell Mac users to download from the cmake.org, though they don't offer a simple "latest" download link like Homebrew does.)

Also FWIW I'm pretty sure macOS comes with Ruby preinstalled, so using a macOS-specific Ruby interface in a cross-platform bootstrapping shell script would probably be safe. The only Ruby package manager off the top of my head, though, is Homebrew, and off the top of my head I don't know if Homebrew supports project-based package management. Again, the safest thing would probably be a bootstrapping shell script that would take care of everything (at least starting with CMake) since Linux and macOS have compatible shells.

[elsiehupp]

Oh, here are two introductions to CPM.cmake, if you're curious.

[elsiehupp]

It appears that you can install Homebrew anywhere by using the following command:

$ curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C <folder_name>

rather than:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

though running Homebrew in an alternative location does apparently have the potential to cause unexpected behaviors. But my understanding is that Homebrew installed in a nonstandard location will install any packages it uses in the same folder that Homebrew itself is installed in.

The main reason I bring this up is that Homebrew includes a ridiculously large number of installation formulae, so it can provide essentially static URLs for downloading various packages, such as Qt. This way any script invoking Homebrew wouldn't need the exact Qt download URL with the version number and everything and wouldn't need to build it from scratch. Though on the other hand it might be preferable to use the very specific download URLs in order to stick to very specific versions of certain packages.

FYI, the download URL Homebrew provides for Qt@5 (yes, they now default to Qt 6) is the following:

https://download.qt.io/official_releases/qt/5.15/5.15.2/single/qt-everywhere-src-5.15.2.tar.xz

Which presumably could be used by CPM or Hunter without involving Qt. That is, we could just use Homebrew's formulae to glean direct binary download URLs where necessary rather than actually running Homebrew itself during the bootstrapping process.

Oh, and it would probably be a good thing to make the CMake scripts work on, for example, ARM, since x64 isn't as much of a monopoly on desktop as it has been for the past decade or two. Like, uh, eventually it would good for the CMake scripts to support Windows and Linux on ARM and macOS on Apple Silicon (which is also ARM).

[FlexW]

We address that by offloading the dependency management to a tool (Craft). That works well enough and could also be an option for Nextcloud. I belive ownCloud also uses Craft (which makes sense given the ownCloud desktop maintainer also maintains Craft).

That sounds good too.

I don't have any particular experience with any of the proposed solutions so I can't compare them. The only somewhat hard requirements I see are that the solution should allow Linux distributions to use their system packages instead and should avoid building Qt from source if avoidable, since that takes hours even on a very decent machine and that is a burden I do not want to impose on prospective contributors.

Yes, that's an important point and I think it should be no problem with Conans CMake find_package generator to allow Linux distributions (or anyone else) to use their own packages. If we use Conan-Center we should have binary packages for Qt available too.

for the existing find_package() function in CMake, so that the CMake scripts generally don't have to be aware of Hunter's existence.

It's the same for Conan :)

An alternative to Hunter that I came across is CPM.cmake

I think a big con for CPM is that it requires that the packages are built locally and I bet you don't want to build Qt locally if you just want to develop a Qt application. Since it takes a lot of time and is nearly impossible if you don't have strong hardware:)

[FlexW]

We decided for craft.

[elsiehupp]

@FlexW could you link to a related issue or PR for craft? thx

[FlexW]

For example https://github.com/nextcloud/desktop-client-blueprints/tree/master/nextcloud-client and https://github.com/nextcloud/desktop/commits/master/appveyor.yml

[elsiehupp]

Thanks. Is there a related comment thread for those, though? Hence why I asked for an issue or pull request, since those tend to include more commentary and explanation. (I could probably dig up the pull requests from the Git history, but I don’t remember how to do so at the moment.)

[FlexW]

No, there is no comment thread. What explanation do you need?

[elsiehupp]

I mean, are there updated build instructions because of this, or anything like that?

[FlexW]

No, it's early. You should build like before.

[elsiehupp]

Oh, okay. Thanks. Just, when you do release it, please do update the documentation. I might be able to help with build instructions again, if you’d like.

[mgallien]

Oh, okay. Thanks. Just, when you do release it, please do update the documentation. I might be able to help with build instructions again, if you’d like.

I have not yet had much time to look at the current state of building the desktop client using craft on MacOS but generally that would be mostly generic craft documentation I have added a specific recipes repository for the recipes such that they are not in KDE's default repositories. The fact is that indeed there is no documentation. Sorry I will have to take care of that. Once this is done, do you want to help us test it on MacOS ?

[elsiehupp]

Once this is done, do you want to help us test it on MacOS ?

Sure! Since I helped with the documentation earlier, I can help with it again!