Building woes

[OvermindDL1]

Currently when checking out / downloading the archive of one of the branches then we try to cmake <path-to-extracted-directory> -D... it fails pretty quick because it cannot find corrade:

Make Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find Corrade (missing: CORRADE_INCLUDE_DIR
  _CORRADE_CONFIGURE_FILE)

Looking in the CMakeLists.txt file it seems that it is assuming that CORRADE, MAGNUM, and occasional other things like SDL2 are installed either globally or have their base paths passed explicitly to the cmake command, all of which is painful (especially as there are no version restrictions, so what if, say, someone's SDL2 was too old or too new?).

Using any kind of cmake based downloading and build manager (I tend to prefer hunter.sh nowadays) would make this a simple process as it could acquire everything it needs at the correct versions of what is needed without filling up the system package scope and/or passing in huge path lines to various directories after manually(!) building a host of dependencies.

Compare this to, say if using hunter.sh, then first you'd just need to include the gate file (scaffolding, rarely updates, it grabs everything else it needs automatically), put it in a cmake or so directory of the project or where-ever you want to store such cmake utility files. Then you call in to it while declaring what version of the main package repository you want (you can define your own repositories as well, hosted anywhere, even the 'main' one is on github and is optional unless you want something from it), such as:

# Load the gate file:
include("cmake/HunterGate.cmake")
# Bring in the primary repository (this is cached globally by default, user-configurable)
HunterGate(
    URL "https://github.com/ruslo/hunter/archive/v0.19.218.tar.gz"
    SHA1 "edaeb585f5ba709311c828facf9ea9e83754fd4b"
)

Then you bring in whatever dependencies you want (like SDL2, you can specify a specific version to use as well, by default it just uses the latest that the package snapshot knows about):

hunter_add_package(SDL2)
# If corrade was in the repo, or a custom one added above:
hunter_add_package(Corrade)
# Ditto with magnum
hunter_add_package(Magnum)

At this point you just use them as normal, nothing special here:

find_package(SDL2 CONFIG REQUIRED)
find_package(Magnum CONFIG REQUIRED
    SceneGraph
    Sdl2Application)

set_directory_properties(PROPERTIES CORRADE_USE_PEDANTIC_FLAGS ON)

add_executable(MyApplication MyApplication.cpp)
target_link_libraries(MyApplication
    Magnum::Application
    Magnum::Magnum
    Magnum::SceneGraph)

Everything gets built and cached globally by default (again, it's configurable, defaults to ~/.hunter) based on the build parameters (so multiple projects that bring in, say, magnum will use the shared built version if they all want it with the same build parameters, else each set of distinct builds get their own hashed location so no worry of conflicts).

This works wonderfully with CI especially as you just have to cache the ~/.hunter directory and you can clone and build all you want very quickly (sans the first time when all the dependencies are built or when they are updated or build options are changed).

Even if not using hunter.sh, just using anything that does not require anything more than CMake and the C++ compiler/linker/etc... would significantly ease a lot of setup on using this.

A few extra nice bits, hunter.sh also has a repo full of toolchains for (cross-)compiling to about everything in existence in any form wanted, if those would ever be useful to add more platforms on.

[mosra]

Hi!

Sorry, I'm terrible at answering early enough -- wanted to give this the full attention but didn't have time until now.

Regarding the first few paragraphs: there's a Getting Started Guide in the documentation that instructs you how to build the bootstrap project without having the dependencies installed externally. That still doesn't handle SDL2, though, but SDL2 is in many package managers already, so I don't see that as a big problem.

Regarding Hunter: I'd like to give people a freedom of choice, not forcing them to anything. For example I'm personally using ArchLinux packages of Magnum (or Homebrew packages on macOS), different users may want to use different things. Which means that Hunter support should be completely optional and documented just as one of the alternatives in the documentation.

Thank you for this in-depth introduction, unfortunately at the moment I don't have time or capacity to try integrating Hunter support myself -- just by looking at the docs it seems like quite a lot of effort. So if you would be so kind and submit a PR with patches for Hunter support to Corrade and Magnum and then submitting the packages to Hunter package repo, I would be very happy (and I'm sure other people too) :smiley:

[DougRogers]

I am trying to build this according to the link you provided, but the bootstrap project does not have a CMakeLists.txt file, so it fails to build.

[DougRogers]

I was able get it to build by creating the CMakeLists.txt . with this

cmake_minimum_required(VERSION 2.8.12) project(MyApplication)

set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${PROJECT_SOURCE_DIR}/modules/")

add_subdirectory(corrade) add_subdirectory(magnum)

[mosra]

Well, the master branch doesn't have a CMakeLists.txt (as stated in the docs) -- you have to use one of the other branches, for example the base branch ;)

[DougRogers]

From your directions "Then open the CMakeLists.txt file in the root of bootstrap project and add these two new subdirectories using add_subdirectory() so the file looks like this:"

[mosra]

Just a tiny teeny bit above: ;)

[DougRogers]

I see. It is not clear that a bootstrap repository does not contain the bootstrap project. I would suggest that a CMakeLists.txt file be added to the bootstrap repository

[DougRogers]

Also the top level project should not contain the src dir

add_subdirectory(corrade) add_subdirectory(magnum) add_subdirectory(src)

[mosra]

Well, the thing is that every branch contains a completely different setup, there just can't be a single CMakeLists.txt file that would cover all the cases and still could be readable by normal humans. That's why it's separated to different branches.

I could put a CMakeLists.txt file into master that would contain just this:

message(FATAL_ERROR "Please download one of the base, base-glfw, base-qt, scenegraph2D,
    ... etc. branches of the magnum-bootstrap project instead of master to get an actual 
    bootstrap project. The master branch doesn't contain any CMake project. See the 
    documentation for more information.")

Would that make this clearer?

[DougRogers]

A more useful cmake would include all the options that you support and allow people to enable and disable what they want. I'm just trying give helpful feedback. It is not intuitive to have to handcraft a cmake file for a project you are not familiar with.

[mosra]

Yes, thank you for the feedback. I'm trying to be helpful as well and trying very hard to make the initial steps obstacle-free :) But, see, I think we are misunderstanding each other in some core aspect of all this. Let me try to explain again from the ground up:

• you can either download, build and install magnum itself via a package
• or you can do it manually and then you have all the freedom to enable or disable what you want
• or, if you don't want to install a package, there's the option to use it as a cmake subproject via add_subdirectory(), as you just posted above, from your already existing cmake project. Then you also can enable or disable what you want.

Or, if you don't have some "already existing cmake project", to make things even easier, in this repository are the bootstrap projects (but in the other branches, not in master), where you don't have to handcraft anything, you just just download a snapshot from ---> this link <----, for example, extract it and use it. Then you have a full setup, all the CMakeLists.txt and other stuff is there. Then, you can either use the packages/installation you already installed above ^, or add the dependencies as cmake subprojects by cloning them via git and adding those two add_subproject() lines, as stated in the docs.

Yes, there's always a bit of manual work involved, but it's done this way because everybody has a different opinion and wants a different setup and I don't want to force anybody to alien workflows. It's done this way so people still have a complete control over everything and everything is transparent.

I hope it's a bit clearer for you now.... :)

[OvermindDL1]

where you don't have to handcraft anything, you just just download a snapshot from ---> this link <----,

Entirely fine for many uses, but even then it still requires substantial setup of the other libraries when it all should just be something like git clone <url> && cd <path> && mkdir build && cd build && cmake <opts> .. && make && ./<executable> (even that seems like too much really, but it's the common C++ way anymore, can leave out the build paths if you want to do an in-source build, though I prefer not to).

EDIT: Or in the case of a dependency management thing then that would involve adding one line to my own cmake script (with whatever options to configure how the library is built on that line), which makes it even far more easy for those that prefer to setup their own projects.

[mosra]

Hey, thanks for getting back to this :)

I understand the point behind Hunter, but as I said above, I won't be able to tackle that myself until a bit later. We were working on polishing vcpkg support recently, for example, it's now just a matter of a single command to install all the things and it makes a real difference for Windows users. Hunter will follow later, but if you (or someone else) could provide initial patches and docs for Hunter support before I get to it myself, it would be very appreciated.

Thank you.

[OvermindDL1]

Hunter will follow later, but if you (or someone else) could provide initial patches and docs for Hunter support before I get to it myself, it would be very appreciated.

I don't really care if it is hunter or not, as long as it is a dependency system where I do not have to install anything else (Hunter is install-less, it is just a cmake script). :-)

Really though, supporting projects with hunter does not require much (though if they have a cmake build system that installs properly then it is significantly simplified to literally a couple lines of cmake code), and having a system use hunter is just a couple of lines as well. In addition using or supporting hunter does not mess with anything else an end user may move I'd might be able to PR it when I get time sometime, ping me (often). ^.^

vcpkg

Not heard of that one, googling... And I'm back a half-hour later, that was a rabbit hole of pain... I don't see the point of it as cross-platform dependency systems (like hunter if you don't want to install anything else, or conan if you do) already handle the things it does but even more effectively... o.O

[mosra]

Really though, supporting projects with hunter does not require much

I get that the part with integrating hunter into the project is not much, I am more scared of the other part (integrating a project into the hunter package repos) -- starting here and then going on for 68 more pages or so :) Or maybe I'm missing something.

I'd might be able to PR it when I get time sometime, ping me (often). ^.^

Thank you, really. I will ;)

vcpkg

Yes, it's Windows-only. The nice thing about it is that Windows-centric developers that have Visual Studio as the primary IDE can just use the packages by adding a #include in their code. Everything else like finding libraries, linking and deploying proper DLLs and whatnot, is done automatically behind the scenes. (Yes, really.) I find that magical.

Lots of people coming from Windows (imagine, students just starting with C++ at universities) complain even about having to use CMake (haven't really heard such complaints from people coming from Linux or macOS), and vcpkg solves that in the most elegant way.

[OvermindDL1]

Yes, it's Windows-only. The nice thing about it is that Windows-centric developers that have Visual Studio as the primary IDE can just use the packages by adding a #include in their code. Everything else like finding libraries, linking and deploying proper DLLs and whatnot, is done automatically behind the scenes. (Yes, really.) I find that magical.

From what I've read you had to do things like run vcpkg integrate install to 'install' it (the heck?), then you had to make a new toolchain cmake file (wth?), then you could include it (with no versioning information?! In hunter everything is versioned, you will not get sudden surprises because some library you depend on upgraded in a bad way...). That is a lot of setup...

Though making a package to distribute with it, though verbose, doesn't seem much harder than hunter's (though hunter's is all done through a single cmake file, which you can host yourself, or add to the main repo, andI don't have experience with conan).

Lots of people coming from Windows (imagine, students just starting with C++ at universities) complain even about having to use CMake (haven't really heard such complaints from people coming from Linux or macOS)

Actually I do work at such a college (my day job ^.^), and cmake is taught with C++ here, though we use Clion (on windows sadly though) and not visual studio (though we used to) as visual studio's intellisense and speed kind of blows in comparison while not supporting the latest C++ standards as well (plus a whole lot more costly, better git support (though I hear VS has gotten a 'little' bit better lately), large plugin ecosystem, etc...). Hunter is not something taught in class though it is mentioned. :-)

And yeah, CMake is a far cry better from the old Makefiles. Using CMake's Ninja generator and using ninja instead of make to build a project is crazy-fast as well, yet to see any build system resolve files and just 'work' faster than ninja, it is amazing (and it's supported by clion nicely enough)! :-)

But yeah, not heard of vcpkg here and this is a windows school (though that is 'slowly' changing over time to include more linux).

[mosra]

That is a lot of setup...

I think of it as a system package manager (homebrew, pacman, ... also need to be installed system-wide). Once it gets bundled with VS installation (i expect this to happen at some point), things get simpler. But I think I don't need to try convincing you any further ;)

Actually I do work at such a college (my day job ^.^), and cmake is taught with C++ here

Consider yourself very lucky :) At my university, back when I was studying there, C++ was taught as a language that one enters into the Visual Studio editor and lets the IDE do the magic, with only a very abstract introduction into compilation process (that of course nobody remembered), and completely skipping buildsystems, lib dependency handling or general source code management.

My latest effort, including vcpkg, is making Magnum easy to get into for beginners, so this university could use it for their GPU lectures.

[OvermindDL1]

I think of it as a system package manager (homebrew, pacman, ... also need to be installed system-wide). Once it gets bundled with VS installation (i expect this to happen at some point), things get simpler. But I think I don't need to try convincing you any further ;)

Heh, I'm very much not for installing libraries system wide that are used to build a singular project, especially as other projects may use the same library but a different/incompatible version, system installed variants either break poorly then or you have to start doing things like adding the version to the name of the libraries and folders and such. ^.^;

Consider yourself very lucky :) At my university, back when I was studying there, C++ was taught as a language that one enters into the Visual Studio editor and lets the IDE do the magic, with only a very abstract introduction into compilation process (that of course nobody remembered), and completely skipping buildsystems, lib dependency handling or general source code management.

Ohmyg... uh, how do you learn that way?! o.O

My latest effort, including vcpkg, is making Magnum easy to get into for beginners, so this university could use it for their GPU lectures.

Well it wouldn't work at others, where something like the hunter style (not necessarily hunter, but its style) does not require installing any additional software, does not worry about conflicts, you'd never need to worry about an incorrect version of a library being used (since the default version is defined on the version of the repo you reference, so no surprising changes), and involves them adding a single call to an existing cmake project to bring in (and one call to grab the dependency repo if not already grabbed), compare that to installing a second program that will not work on other systems (and even involves putting in hardcoded absolute paths in a toolchain, that is still a bit of a wtf there...) and is not configurable in how it builds the libraries and etc... I'm just confused on how that is more user-friendly?

[mosra]

To make myself clear: I'm not implying one is better than the other, I'm not forcing you away from hunter and into vcpkg or anything like that, I'm just saying that each tool solves a problem in a slightly different way for a slightly different subset of people, and the tools don't conflict with each other, so why not support both :)

I see your point, I am well aware of the problems with (and the advantages of) system packages and now I think I know what problems hunter-style package management solve (thanks). I'm not extreme in pushing for one particular approach, though. I took vcpkg only as an example of something different that was added recently to support the "open visual studio, create a fresh project and start typing c++ right in" use case.

Ohmyg...

Yes, not everything is perfect. I'm trying to change that, but one little step at a time :)

[OvermindDL1]

To make myself clear: I'm not implying one is better than the other, I'm not forcing you away from hunter and into vcpkg or anything like that, I'm just saying that each tool solves a problem in a slightly different way for a slightly different subset of people, and the tools don't conflict with each other, so why not support both :)

Hear hear. :-)

now I think I know what problems hunter-style package management solve (thanks)

My main thing about it is the rebuildability. As far as I can see vcpkg does not constrain a version, so if someone makes a project that uses a dependency it may build 'now', but when they clone it in 6 months and the dependency has updated since then, then the project suddenly cannot build and needs fixing when it did work before. The hunter style (not just hunter, 'most' work this way, which is why vcproj surprises me so much, the 'system' install way has long since known to be really broken) makes repeatable builds, whether now or a year from now. :-)

Yes, not everything is perfect. I'm trying to change that, but one little step at a time :)

Lol, I can attest that the college I actually graduated from (years ago, not the one I work at now) was a bit more similar to yours (though still not visual studio, that's not something I see taught at any place I've ever been, but rather the whole 'style'). I'm constantly amazed at just how awesome the place I work at is. :-)

[ras0219-msft]

Not to interrupt too badly, but we actually care a lot about reproducibility in Vcpkg!

We completely support having as many copies of Vcpkg on a machine as needed, and it's 100% supported for a project to provide their own set of dependencies that way. This gives projects complete control over how things are built, including applying custom patches to the dependencies if needed. I'd be happy to go into more detail about this if either of you'd like to hear :)

However, as you've both pointed out above, there are many different user scenarios -- both those who want tight control per project, as well as users who are new to C++ and are delighted to just get a triangle on screen! We care deeply about both cases in Vcpkg 😄.

[OvermindDL1]

We completely support having as many copies of Vcpkg on a machine as needed, and it's 100% supported for a project to provide their own set of dependencies that way. This gives projects complete control over how things are built, including applying custom patches to the dependencies if needed. I'd be happy to go into more detail about this if either of you'd like to hear :)

Cool, I'm curious then, the toolchain file in the tutorial I saw referenced an absolute path to a vcpkg, could that just be pointed to the local project directory (a relative path to, say, ./.deps or so)? And could that be left 'out' of the git repository so as to no pollute it, and as such would it download the deps on the next compile of the exact and precise versions wanted? This kind of stuff is very important to keeping build times low on CI's while enforcing perfectly repeatable builds without polluting the git repo with non-direct-project files as one big example.

Plus it needs linux support. ^.^;

However, as you've both pointed out above, there are many different user scenarios -- both those who want tight control per project, as well as users who are new to C++ and are delighted to just get a triangle on screen! We care deeply about both cases in Vcpkg smile.

I'm curious how that compares to the hunter style then, where you just copy the 'Gate' file into your cmake folder (it's a simple hoister that grabs the rest of hunter automatically), include it in the cmake file, then just call it's function with the snapshot set of packages to work with (if you do not version a library you use from it then it uses the 'latest' versions at the time of that snapshot, thus ensuring repeatability in all cases, otherwise you can pick any older version as well). You can include different snapshot listings from other repo's as well, not just the primary hunter repo (so you can make your own repo with all the dependencies you want at whatever versions you want).

I.E, hunter involves downloading one file once, adding 2 calls in a cmake file to set it up, then using a dependency from then on involves one extra call per library you want to bring in, that call can contain various arguments to define what/how to bring in from it, the version and how to compile/use it, but usually the defaults suffice, then it 'just works' like it were a system installed library, on any system, even windows. Where from what I see from vcpkg it seems it requires acquiring vcpkg itself, choosing a directory (potentially one per project), acquiring the packages somehow (whether via an automatic header inclusion, of which I don't see how that can supply any options on as to 'how' to build the dependency or what version or anything, or you have to vcpkg install ... or so, which still does not seem to support options on the dependency itself), and you seem to have to do this on each place you want to compile (so on a CI it seems you have to pre-setup vcpkg and maybe even the dependencies where hunter requires doing nothing beyond just having cmake as you would already), 'and' you have to have a unique toolchain file (does this mean you have to edit every toolchain file you want to use for every potential build you want to do to various platforms, like what if you are compiling for android, or emscripten to javascript, or even 32 or 64 bit builds, plus it seems to be very windows-only currently?).

So hunter takes less work compared to vcpkg as well as it is by default repeatable and requires no software installation other than the build system you already have, unlike vcpkg on all those points? Mostly I'm curious what vcpkg gains over something like hunter as I'm not seeing much, in all cases it seems more simple, less installation, etc...?

/me is obviously going off-topic in here, can quit if so desired, but I am honestly quite curious as I do believe that C++ needs a fantastic dependency system that most other languages already have and so far think hunter is the 'closest' thing so far to 'just work everywhere'

A bit o' disclaimer, I'm not a big windows user, I dabble at work, but otherwise primarily linux, and I compile to windows from linux (I actually make a surprising bit of windows programs, made on linux...), to android from linux, to various microchips (propeller, rpy, etc...) from linux, to occasional on-windows compilation (using powershell and the windows MSVC compiling tools, no visual studio ide itself), and hunter works with everything as long as the libraries I use do too (which I've not had many issues with actually), and it's all always just adding the gate file, adding 2 lines to the top cmake file to load the gate and load the default hunter dep repo, and bring in the deps with 1 line per dep. If I could find a dependency system that simplifies that even further with no loss of functionality I would be so ecstatic! :-)

[mosra]

Closing as resolved. For further discussion and PRs regarding Conan / Hunter / other packaging stuff please use the main Magnum repository.

A designed-to-fail CMakeLists.txt was added to the master branch in eb66bc9aee250f8ce5cb2c4c3cc39577274736c0.