Initial build experience: frustrating

For a tiny bit of background, I'm a medium Linux person. I'm not a distro-developer, but I've been using Linux for 25 years, I've written person-years of code under Linux in C++, and also less in NodeJS, Python, etc. and recently I've spent a year and a half doing FPGA / Verilog, development.

My job today has been to evaluate a set of 3D engines (C++ OpenGL) for ease of use, functionality and so on. I've installed built and run bs:framework, Diligent Engine, BGFX. Also for fun, I included NVIDIA Cuda Examples and a few others. All easy experiences. Building and installing code is something I do everyday.

But... my Magnum experience stood out. For someone like me, coming in cold to the project, the level of complexity of the "Getting Started" experience was prohibitive. Most projects have a pattern as follows:

sudo apt get [prerequisites] git clone [awesome project] cd [awesome project] mkdir build cd build cmake .. make sudo make install cd examples ./example

Later there's a "if you need some special stuff do this other thing"

Magnum, by making the initial startup so complex has excluded what looks like a great project from my evaluation. I tried, but I don't have 3x the time to work through Magnum procedures. Maybe take a look at the directions you ask people to follow. Put yourself in the position of a person who is comfortable in the world of cmake, apt, make, gcc, but who is busy. And someone who has a lot of great other options!

Maybe I'm not alone in experiencing this. Maybe your project would be more approachable to the average working stiff like me if the "Getting Started" experience was more conventional and easily comprehensible.

Anyway! I offer this feedback in good faith and in an attempt to be helpful. I really want to see what this system is like. Best wishes!

Hi @davidthings !

It's a shame that you had this experience and would love to know which guide you followed to get started. From that we can figure out what leads to frustration during the setup.

In Getting Started you are provided with the subproject option, which would not require the install step as you mentioned for the pattern of projects. But apart from that it's pretty much exactly that for both Option A & B mentioned in that guide.

Additionally, if you don't want to build magnum yourself, you can use one of the 7 package types (apt/debian is part of that. It's just not in the official repository)

Maybe the issue is the separation into Corrade, Magnum, Corrade, Examples, Plugins and Extras which makes the setup more complicated?

Looking forward to your reply,

Best, Jonathan

Thanks for your attention, Jonathan.

Also, I want to recognize how much work somebody has done to make all the doc that is there. The site looks beautiful and there's a lot of information there.

My observations concern the path I took through them to get to a place where I had compiled and running example code on my machine to evaluate.

I was just going to leave my comments above as a hint for someone to take a look. But since you seem surprised at my reaction, and you wrote me a nice reply, I thought I'd give Magnum another go, when I wasn't tired, grumpy and hadn't been drinking.

TLDR - I think this reads more as a "writing your first project" than "getting started". It assumes my very first act is to write a mini app. It isn't! In order to learn I want to see examples and run code. The web-based examples are nice but I want them in the environment that I'm going to use them. And I want to get there as fast as possible. Remember I know almost nothing about your project, and my evaluation time is precious - at least to me. I did eventually succeed-ish, but not without more frustration (clear headed, sober frustration).

I thought it might be helpful for me to run through the steps I followed, along with my reactions commentary. This is long. It took me a long time. I hope it's useful. There are more summary comments at the end if the format below is too exhausting.

MAIN SITE

"Beautiful! Now... where can I see code, build, and run it? Ah! Getting Started, that looks right!"

GETTING STARTED

"Good"

Download the bootstrap project

"Huh? That's weird - something that helps me load the project in? [reads more] Oh... Huh? Master doesn't even contain the actual projects? That's odd. But why do I need that to look at examples?" - maybe this could be relegated to a later place when you talk about creating your own projects?

[Goes to the repo and git clones it]

1a. Option A: add Magnum as a CMake subproject

"OK. Now what? More options? Something about including Magnum in my project vs using libraries. Again... why is this important to me trying to get a feel for the project. So many words! It will be a while (if ever) before I make my own code. And what? I have to edit the CMakeLists.txt too? Nah!"

[Skips this. I like libraries]

1b. Option B install Magnum separately and let CMake find it

"OK this looks better. Wow. Many words. Oh boy. I have to go and do separate Corrade and Magnum builds now? "

[Goes to Corrade, without reading further]

DOWNLOADING AND BUILDING CORRADE

[skipping over all the package stuff...]

"I already said I wanted to build and make libraries! Could that link have dropped me somewhere better? More scrolling. Manual Build! Yay!"

Manual Build

[clones Corrade repo]

[skips CMake primer...]

"why do that cmake explanation here - put it somewhere else? Wow - there are hundreds of options, why not pick one and let people change that later?"

mkdir build && cd build
cmake ..
cmake --build .

"Money!"

[Sees more cmake tutorial]

"Why??!"

sudo make install

Whew!

[Back button on browser! Click on Magnum]

"Wait where am I, is this the first page again? No... no... we're somewhere new"

[Skipping hundreds of other platform options]

"Wow."

Manual Build

"Whew!"

[mkdir build... etc... etc.. make install]

"Money! Again..."

[Presses back button...]

"OK now what.. "

[Scrolls a bit...]

"How the heck do I get something to run?"

3. Build the Project

"What project? That empty bootstrap project? Why?"

4. Run the application

"What application?"

[Scrolls]

"Blank screen?"

[Scrolls]

"Blue screen?"

[Scrolls]

"Using other toolkits? I don't care!!"

[Scrolls]

Additional Information

"Finally Examples!!!"

[Clicks link Downloading and building examples]

DOWNLOADING AND BUILDING EXAMPLES

Prepared packages

"Nope! - build build build!"

[Scroll scroll scroll]

Manual Build

[Clones the examples directory]

"Only one example is going to be built? Argh! I need to get into Cmake and click links and laboriously gather up dependencies.... Oh no..."

[mkdir build... blah blah... make. BONK - no SDL2Application]

"What? What the hell is SDL2Application? Is that part of SDL2?"

[Goes back, searches for SDL2Application online, nothing.]

"Huh?"

[Goes back to - Getting started. This is hard. The navigation drop downs don't seem to have everything listed! Finally reads "don't forget to enable WITH_SDL2APPLICATION when building Magnum so the bootstrap project can correctly find and use it."]

"Wow. I would not have expected that. Make the damn option on by default if SDL2 is installed, or put the damn thing in the examples repo. Argh!"

[Back to Magnum, re-cmake, check WITH_SDL2APPLICATION. sudo make install. Runs magnum-triangle]

"Woot! A triangle.[sigh] Now. Let me look into trying to get something else to run"

[Looks at the example list - trying to find some for which their dependencies are met on this machine]

"Oh boy! Nearly every example has different dependencies"

[Works through list of apps in cmake-gui. Attempts a configure. Nope]

"Argh! Something is requiring ImGUI"

[Searches source code for references to ImGUI. Many]

"Argh!! So many"

[Unchecks Fluid Simulation projects. Attempts a configure. Better. Nope]

"Argh! Still missing dependencies"

[Unchecks Sokol example. Attempts a configure. Better. Nope]

"Argh! Looks like TRIANGLE PLAIN GLFW needs to go.

[Unchecks. Attempts a configure. Better. Completes]

"Better try to generate and build"

[Generates; Success!]

OK. That's enough blow-by-blow.

After all that, I have the following

magnum-arcball
magnum-motionblur
magnum-mouseinteraction
magnum-picking
magnum-primitives
magnum-shadows
magnum-triangle

Hard won examples. They all work and all look beautiful.

I can now see how smooth everything looks, look at the coding style of each app and start to get a feel for how the engine works. For reference, this took a LONG time, the other engines I looked at took seconds to get to this point. No long cmake lectures, not having to go to several different places, not having to change CMake files, etc.

Because UI is important to me, I was eager to see how fonts and the UI examples looked so I went back in search of those.

But where to get that working? Much searching. Magnum Extras? Where's that? How do I build that? Abandon doc. Go to github. Find extras. Clone, Configure. Do it several times until WITH_UI_GALLERY etc. appears. make. make install. Back to examples. Now failing with no ImGUI lib. Go find that. Git Clone. Read almost nothing. cd examples. cd example_glfw_opengl3. make. run! (Time from download to running code? 30s). Anyway, I guess there's no install with imGUI, so back to Magnum with an idea of manually banging the dir in somewhere. Added it to examples. Doesn't work. Extras need to know about it too? Searching for ImGUI integration. Not finding anything helpful. "Requires integration". Back to GitHub. Oh wow! There's a Magnum Integration repo too! Clone. mkdir build. cd build. cmake-gui.. Add ImGUI source. In about 10 places! Configures OK. Get brave and add Eigen and Bullet. Have no Dart, GLM or OVR. Generate. make. make install. Back to examples. cmake looks good. Add more examples (BULLET, Fluid2D and 3D) now we have more hopes of completion. Generate. ARGH! Need to add imgui_DIR. Where is that? Add pointer to the imgui directories that satisfied the integration code. Also need to add imgui include directory. Add that.

OK. No love from bullet

BulletExample.cpp:(.text+0x1374): undefined reference to `btDiscreteDynamicsWorld::stepSimulation(float, int, float)'

No love from ImGUI. Something weird with the imgui_DIR value in cmake?

FluidSimulation2DExample.cpp:(.text+0x487): undefined reference to `ImGui::CreateContext(ImFontAtlas*)'
FluidSimulation2DExample.cpp:(.text+0x491): undefined reference to `ImGui::StyleColorsDark(ImGuiStyle*)'

So that's a few hours done. Pretty frustrating in the end. I hope this was all useful. Out of it at least I get to see some example code working. I'm really sad I couldn't get any UI up after trying for quite a while. That's draining stuff to struggle through all the cmake code, etc. looking for clues.

I'm left with the feeling that all the extra repos could be combined. Even the examples. One nice cmake for the library - all of it, and one more for examples.

And the doc could be refactored to quickly get a coder from repo to running example, rather than blending in all the other information.

Maybe try a doc for Getting Started - Manual Build, and others for the other flavors of package. That way we're not reading other people's business all the time.

The site itself is really nice looking but the navigation is a little confusing. When inside some getting started pages I couldn't see how to get to others.

I'm really looking forward to reading some code now then seeing how the project develops.

Now I need a drink!

Cheers.

@davidthings That is incredible feedback, I really appreciate how much effort you put into this! 🙏

Half way through, when you discovered the examples repository, I started laughing, it is somewhat comical indeed. I guess it's something I got used to over the time, but still remember from my own beginnings with Magnum, 5+ years ago.

There are good reasons for a lot of the build and repository setup side, that someone who uses Magnum for a larger application values a lot, while I fully understand how they get into the way of a "Getting Started" experience:

Repositories are split to maintain different levels of code quality. Magnum + Corrade are outstanding to insane in this regard (to me at least), while examples and extras are a little less strict, especially in terms of testing.
Repositories are split for git clone speed. This is especially useful when building a project on CI, but you don't need integration or extras, or especially the examples with your own project. There are libraries out there that download crazy amounts of megabytes of example files with the main repository.
Build only what you really need: all those CMake options allow getting really small WebAssembly binaries for example, by just cutting out what you don't need. Needless to say this help compilation speeds of the library and again CI times, if you have to recompile magnum for every run
- ...

Nevertheless, this of course doesn't help anybody, if noone can be convinced to use Magnum in the first place, because they have a hard time compiling the examples.

I am very interested in seeing what @mosra has to say about this, he's usually the first to answer any issue, but he's swamped by work currently, so his reply might pop up sometime in the upcoming week.

I see two possible options to improve on the issues you mention:

Have magnum repos as subproject/submodule of magnum-examples and let the CMake project set the required CMake flags on the submodules when an example gets enabled (+ build more examples by default?). My usual thought-pattern when compiling a library is "what can I disable", not "what can I enable".
Or at least build more libs by default on most projects. IMO, it should be advanced users adding more flags (i.e. to disable libs for a more tailored build) than new users adding more libs (for a more comprehensive build that allows building all examples). This will make a "building all the examples" guide more straight forward.

I also like the modularization and customization aspects of the current setup! (but I have also seen how complex it is to people new to Magnum).

Have magnum repos as subproject/submodule of magnum-examples and let the CMake project set the required CMake flags on the submodules when an example gets enabled .

That sounds quite nice to me and would remove much of the pain @davidthings went through ;) Magnum-as-subproject is also my go-to solution when I help someone bootstrap a Magnum application exactly for this reason.

The magnum-examples CMake code could also try to detect additional libraries such as SDL/ImGui/etc and enable the corresponding samples & Magnum options.

Or at least build more libs by default on most projects. IMO, it should be advanced users adding more flags (i.e. to disable libs for a more tailored build) than new users adding more libs (for a more comprehensive build that allows building all examples). This will make a "building all the examples" guide more straight forward.

I like that additional functionality has to be enabled explicitly for the Magnum core / plugins - that way there are no surprises e.g. on a Magnum update. But putting auto-detection code in magnum-examples would not hurt.

I'd like to echo that frustration, from the Windows side.

Magnum looks great, and it's always reassuring to find a VCPKG port because that suggests installation & usage should be a breeze. So VCPKG seems to have built Magnum etc. just fine - hooray!

But now I can't build the examples.

Using Git and CMake the usual way just returns errors about being unable to find CORRADE_INCLUDE_DIR. Just like the world was before VCPKG came along! I have never known what to do about CMake errors.

The documentation just says to read the documentation, and that building this thing is like building the other thing. I do not know what to do with that documentation, either.

I'm sure that all this is a very low barrier to entry when viewed from the other side, but I'm pretty rusty, so any barrier at all is enough to make me think I'm probably not meant to be here.

I could share with you a script that I made for automating the process of compiling the dependencies of all magnum ecosystem.

It's a shell script that I wrote. Be aware that the script only builds for static libraries, not dynamic ones.

Steps to follow:

Save the script from below to the folder containing the repos with your preferred name and make it executable by invoking chmod +x <name of the script>
Create a folder containing the required repos. You'll have to clone there the following repos: glfw, corrade, magnum, magnum-plugins, magnum-extras, magnum-integration, imgui and basis_universal
cd <name of the folder containing the repos>

invoke manually the script to compile each repo like below:

./<name of the script> glfw
./<name of the script> corrade
...

You can also list the repos the script can compile with ./<name of the script> list

If you want to update the repos and then compile the updated repos, the script can do it for you, just run the script without arguments. Be aware that if corrade is updated and compiled, but not the other magnum repos, then you have to compile again the rest of the magnum repos to make it work properly (I refer to the previous step).
You can find the libraries and includes in the <name of the folder containing the repos>/final
In your own cmake project, you have to invoke cmake with the extra argument -DCMAKE_PREFIX_PATH="<name of the folder containing the repos>/final"
Be aware that I wrote the script on macOS, so the sed utility differs a bit from the Linux one, so the arguments fed to it within the script must be adjusted accordingly (in the bash function git_check()).
In the script you have to edit the repocommand value for every repo which represents the cmake arguments for every repo, so pick your own preferences.

That's it.


#!/usr/bin/bash
current_folder=$(PWD)
#uncomment the below line to see the compiler/linker invocations in case you encountered an error in the previous compile session
#makefile_verbose='-DCMAKE_VERBOSE_MAKEFILE=ON'

repo[0]='glfw'
repocommand[0]='-DBUILD_SHARED_LIBS=OFF -DGLFW_BUILD_EXAMPLES=OFF -DGLFW_BUILD_TESTS=OFF -DGLFW_BUILD_DOCS=OFF'
repo[1]='corrade'
repocommand[1]='-DBUILD_STATIC=ON -DBUILD_STATIC_PIC=ON'
repo[2]='magnum'
repocommand[2]='-DBUILD_STATIC=ON -DBUILD_STATIC_PIC=ON -DBUILD_PLUGINS_STATIC=ON -DWITH_GL=ON -DWITH_ANYIMAGEIMPORTER=ON -DWITH_ANYIMAGECONVERTER=ON -DWITH_ANYSCENEIMPORTER=ON -DWITH_GL_INFO=ON -DWITH_FONTCONVERTER=ON -DWITH_DISTANCEFIELDCONVERTER=ON -DWITH_MAGNUMFONT=ON -DWITH_MAGNUMFONTCONVERTER=ON -DWITH_OBJIMPORTER=OFF -DWITH_MESHTOOLS=ON -DWITH_PRIMITIVES=ON -DWITH_SCENEGRAPH=ON -DWITH_SHADERS=ON -DWITH_TEXT=ON -DWITH_TEXTURETOOLS=ON -DWITH_SDL2APPLICATION=OFF -DWITH_GLFWAPPLICATION=ON -DWITH_TGAIMPORTER=OFF -DWITH_TGAIMAGECONVERTER=OFF'
repo[3]='magnum-plugins'
repocommand[3]='-DBUILD_STATIC=ON -DBUILD_PLUGINS_STATIC=ON -DBUILD_STATIC_PIC=ON -DBASIS_UNIVERSAL_DIR=${current_folder}/basis_universal -DWITH_BASISIMPORTER=ON -DWITH_DDSIMPORTER=OFF -DWITH_FREETYPEFONT=ON -DWITH_HARFBUZZFONT=ON -DWITH_JPEGIMPORTER=OFF -DWITH_PNGIMPORTER=OFF -DWITH_STANFORDIMPORTER=OFF -DWITH_STBIMAGEIMPORTER=OFF -DWITH_STBTRUETYPEFONT=OFF'
repo[4]='magnum-extras'
repocommand[4]='-DBUILD_STATIC=ON -DBUILD_STATIC_PIC=ON -DWITH_UI_GALLERY=OFF -DWITH_UI=ON'
repo[5]='magnum-integration'
repocommand[5]='-DWITH_IMGUI=ON -DBUILD_STATIC=ON -DIMGUI_DIR=${currentfolder}/imgui'

function compile_with_cmake() {
    cd $1
    if [ -e CMakeLists.txt ]; then
    echo "==========================================================================="
    echo "Compiling $1 with CMake"
    echo "==========================================================================="
    echo "==========================================================================="
    echo "Uninstalling old version from final folder"
    if [ -d build ]; then
        cd build
        if [ -f install_manifest.txt ]; then
            xargs rm < install_manifest.txt
        fi
        cd ..
    fi
    echo "==========================================================================="
    if [ -d build ]; then
        echo "Emptying the build folder ..."
        echo "==========================================================================="
        rm -rf build
        mkdir build
    else 
        mkdir build
    fi
    cd build
    local output_folder=$current_folder'/final'
    local library_path=$output_folder'/lib'
    local pkgconfigpath=$library_path'/pkgconfig'
    local args_to_be_passed_to_cmake="$2"
    export LT_SYS_LIBRARY_PATH="$library_path"
    export LDFLAGS="$library_path"
    export PKG_CONFIG_PATH="${pkgconfigpath}"
    LDFLAGS="-L${output_folder}/lib"  cmake .. $makefile_verbose -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=TRUE -DCMAKE_BUILD_WITH_INSTALL_RPATH=FALSE -DCMAKE_INSTALL_RPATH=$library_path -DCMAKE_BUILD_TYPE="Release" -DCMAKE_PREFIX_PATH=$output_folder -DCMAKE_INSTALL_PREFIX=$output_folder $args_to_be_passed_to_cmake
    cmake --build .
    cmake --build . --target install
    echo "==========================================================================="
    cd ..
    fi
    cd ..
}

function git_check() {
    [ $(git rev-parse HEAD) = $(git ls-remote $(git rev-parse --abbrev-ref @{u} | sed 's/\// /g') | cut -f1) ] && return 0 || return 1
}

function update_and_compile() {
    cd $1
    echo "Checking $1"
    git_check
    local res=$?
    if [ $res -eq 1 ]; then
    git pull
    cd ..
    compile_with_cmake "$1" "$2"
    cd $1
    fi
    cd ..
}

package="$1"
empty=""
if [ "$package" = "$empty" ]
then
    package="all"
fi

length=${#repo[*]}

if [ "$package" = "all" ]
then
    for (( i=0; i<=$(( $length -1 )); i++ ))
    do
    update_and_compile ${repo[$i]} "${repocommand[$i]}"
    done
else
    if [ "$package" = "list" ]
    then
    for (( i=0; i<=$(( $length -1 )); i++ ))
    do
        echo "${repo[$i]}"
    done
    else
    for (( i=0; i<=$(( $length -1 )); i++ ))
    do
        if [ "$package" = "${repo[$i]}" ]
        then 
        compile_with_cmake ${repo[$i]} "${repocommand[$i]}"
        fi
    done
    fi
fi

mosra / magnum

Initial build experience: frustrating #428