[POLL] Build system change

[andlabs]

I had four reasons for choosing flat makefiles:

1. I wanted libui to not have any extra dependencies (GNU make on Windows was already pushing it, but nmake is just way too primitive).
2. I wanted to ensure that every generated file goes somewhere clean that I can predict away from the actual source code, in this case to just .obj and out directories (obviating the need for #13 in my opinion, at least)
3. I'm not really a fan of other build systems to begin with, either because of their complexity or their mess (see point 2 above)
4. I wanted to know exactly what was going on in the build process

However it seems that this decision is causing problems for lots of people, mostly on Windows where the command-line Visual Studio tools either behave unpredictably or require special handling depending on the configuration. @kainjow in #15 makes a really good point: most Windows users are likely more familiar with an IDE than with the command-line build system, to the point that command-line systems like cmake are more accessible.

While it's true I originally resisted changing, thinking make would be simpler, the number of build issues keeps growing, especially on other platforms. So now I'm willing ot hear arguments for specific build systems.

I genuinely do not have a preference for any one build system over the other, so please make your case for one you prefer. As part of your case, you should demonstrate what a setup for libui, the test/ folder, and each example would look like, as well as (if possible) a rule to build all examples. Feel free to point out any other advantages or disadvantages.

Here is a page detailing what an ideal build system would be like: https://gist.github.com/andlabs/243d5d817d291fa62722c33a9c58c6ab

Right now the following are being considered:

• cmake
• waf
• scons

I'll make my decision and switch build systems if one appeals to me more than the others.

Thanks for your feedback! I do appreciate it all, and I want to make sure libui works for everyone :)

[kainjow]

I'm maintaining my own CMakeLists.txt file here for ease of development for myself. I plan on adding Windows and GTK support to it when I get a chance.

Currently libui has nearly 20 files that are part of the Makefile. Using more modern versions of CMake (3.1 and later), you can manage multiple targets in a single file very easily.

Some examples of big projects using CMake are Qt and LLVM.

The great thing about CMake is it generates the cross platform projects for you, so one can work in Xcode, Visual Studio, Makefiles, Eclipse, and more.

CMake is fully open source. It can be built from source, or the binaries can be installed via Homebrew on OS X, Chocolatey on Windows, apt-get on Debian/Ubuntu, or downloaded directly for OS X and Windows.

With my example CMakeLists.txt file, you would put that in the root directly, then run these commands (minimum needed), which works on every platform (CMake will choose a default generator based on what it finds):

cmake .
cmake --build .

[andlabs]

For the OS X build, what files does cmake generate, and how?

[kainjow]

On *nix platforms, if you don't specify a generator it makes a Makefile (and other junk/cache files for detecting compiler features), so then you can make as normal. On OS X for example if you pass -G Xcode it makes an Xcode project. cmake --build is an easy built-in way to build whatever CMake generates without having to know the native command (make, msbuild, xcodebuild, etc.).

[andlabs]

And everything is created in .? (So I would do mkdir build; cd build; cmake .. to drop build files elsewhere?)

[kainjow]

Yep, exactly. With CMake it's usually preferred to build out-of-source to keep things clean.

[HalosGhost]

Honestly, I deeply prefer flat makefiles over most build-systems as well. But I have taken a special liking to tup. It builds incredibly fast, has brilliant dependency resolution, has incredibly terse build rule syntax, makes reproducible builds much simpler, is cross-platform and supports Windows, OSX and Linux.

The only things to know are that it is not makefile-generating, so it is a shift away from makefiles (where things like CMake are a layer on top of makefiles) and that it is truly a build-system and nothing more; e.g., it does not have install/uninstall rules.

In my personal projects, I tend to use Tup as my build-system and a flat makefile on top of it to deal with things like install/uninstall rules cleanly.

[moosingin3space]

Agree with @HalosGhost -- tup is nice and simple, and with a Makefile wrapper, you can easily add other targets.

The current system of doing cross-OS builds could be emulated by sticking a Tuprules.tup file in each platform directory, which would assign values to !cc and !ld targets (or something like that), then make $OS could be implemented by copying the correct Tuprules.tup file into the project root and then running tup upd. I'm attempting to do something like this in one of my projects that uses cross-platform C.

[hajimehoshi]

I prefer qo (https://github.com/andlabs/qo). You will like it if you like Go-way to build something.

[SevenBits]

I definitely prefer flat Makefiles. On Windows it might make things more difficult, but in the long term it is probably a good idea to avoid external dependencies; once you add one, then it isn't too long before you concede to adding another, and soon the whole project is overflowing and bloated.

[HalosGhost]

@SevenBits, I agree, though I think it is fair to classify external dependencies in two ways: makedeps and runtime deps; adding makedeps is less of a concern in my opinion as far as bloat.

[andlabs]

The thing about tup that I'm not thrilled about is its use of a database that you have to constantly update yourself (and which I'm not even sure is safe to check into source control); copying build scripts around especially sounds unattractive.

[HalosGhost]

@andlabs, database? When do you need to manually update it?

In addition, you definitely would not need to copy around build-scripts to support multiple targets.

[moosingin3space]

Yeah, I didn't read the Variants section of the tup manual when suggesting that.

[sekrause]

+1 for CMake from me.

I especially like the fact that it simply generates files for the platform's "native" build system, e.g. Makefile on Unix and a Visual Studio project on Windows.

[waddlesplash]

CMake. I hate it, but there's nothing better (yet -- there are a few projects in the works, but they're way behind what libui wants as of now.)

[NattyNarwhal]

I would recommend against CMake - it's way too complex. Makefiles are simple and GNU Make isn't that much of a requirement.

[RotsiserMho]

Makefiles are simple until you want to do even just a little bit more than build the source. For example: running tests. CMake works well for something like that because you don't have to worry about platform-specific path formats, etc.

[aoloe]

on top of what has already being said, there are two reasons for me to prefer cmake:

• it would make very easy to embed libui inside of projects that already use cmake. probably, without hurting the rest of the world.
• most people know about it and you can compile cmake based projects without having to deal with the way a programmer found cool at the time he wrote the library.

i can live with libui's flat makefiles; i would prefer if libui would avoid "hesoteric" (for c++ projects) pre-processors, even if they are superior. and i could have saved much time if libui were already using cmake...

[kainjow]

@NattyNarwhal what makes CMake complex to you?

[SevenBits]

My thought is that whatever built system that is chosen needs to be relatively simple to setup and use on all of the supported platforms. I've seen build systems (OpenSSL comes to mind) where you can't just simply run make or something similar; you have to engage in this ridiculous and complex dance involving multiple sets of scripts, etc before you can build, or at least you did have to when I last did it.

I love that libui is using flat Makefiles currently; I downloaded the source archive, opened it in my Mac's terminal, and ran make. All done. No hassle. It then produced a single file that I can copy into my project or tie into my existing Makefiles. That should be how this project continues. I'd hate to have to do more than that; I've used enough GUI libraries requiring lots of build time dependencies. So far, libui has been able to avoid that, and I think that that's fantastic.

[jamierocks]

I've only used it once but it seemed good, but gradle's cpp plugin.

[waddlesplash]

I've only used it once but it seemed good, but gradle's cpp plugin.

Gradle is a hulking resource hog of a build system, essentially the complete antithesis of libui. It wouldn't be a good match.

[kainjow]

Seems like building libui as a static library is a popular request. In CMake, when you use add_library to create a library target, it by default builds static. However if BUILD_SHARED_LIBS is defined and true (which the user can easily set on the command line) all libraries instead build as shared (unless explicitly set).

[bkaradzic]

GENie (not build system, rather project generator) https://github.com/bkaradzic/GENie#genie---project-generator-tool

[Vdragon]

I really against any build system that is not space/non-ASCII containing path compatible, GNU Make & it's friends are on my blacklist for this reason.

[jasom]

While I really like tup (mentioned by others) it is really a make replacement, so at least some of the issues you are currently having now with windows will still be around.

However, it is smart about exporting the variables visual studio needs, and it provides a varaible for which host you are on, so "do X if on windows and Y otherwise" is trivial to add.

[andlabs]

So far I've only seen cmake and tup mentioned here. Any advantages to any of the others? Someone else mentioned gyp somewhere else, for instance.

[bkaradzic]

@andlabs I mentioned GENie. Advantage is that it's not build system, just project generator so your build is done by actual build system. CMake does something similar, but GENie uses .lua which is nicer to work with than CMake script.

[andlabs]

Ah, I missed that mention. How would it work for libui? Can you provide a quick demonstration?

[sadolit]

What about waf? CMake is somewhat complex, while waf introduces virtually no external dependencies, except Python.

[bkaradzic]

For example this file: https://github.com/andlabs/libui/blob/master/windows/GNUfiles.mk

Becomes something like (if you want libui_windows.a):

projects "libui_windows"
    includes {
        "windows/**.hpp",
    }

    files {
        "windows/**.cpp",
    }

Common stuff like:

ifneq (,$(STATIC))
CFLAGS += -D_UI_STATIC
CXXFLAGS += -D_UI_STATIC
RCFLAGS += -D _UI_STATIC
endif

Would be put in main script that would apply things to all projects.

Also with GENie you can put all these scripts into common directory (for example /scipts) instead being all over the place.

[andlabs]

@sadolit same question to you. I'll look into seeing what libui would look like in cmake and genie and maybe tup in the meantime... after the next packaged release

[joa-quim]

I vote for Cmake.

[andlabs]

The limitations of MinGW that prevented me from allowing it for a shared library don't apply to a static library, and since mixing and matching isn't really possible (I tried), I'm going to have to provide separate static libraries for MinGW and MSVC (as an Alpha 3.1).

So you decide, within the next 12-24 hours, do I

1. make one last megahack to the makefiles to allow MinGW Windows builds, since the makefiles were never designed for this scenario
2. give up and devote my immediate effort toward switching to a buildsystem — I'll have to decide which myself at this point, given libui's tesseract-matrix of configurations; thanks in the meantime for your input!

There's still enough pure-Makefile people here to warrant this emergency poll.

[billyquith]

I also vote for cmake. It is the ubiquitous cross-platform project generator. The syntax is a hairy at first but once you see a few projects with a simple examples it isn't that difficult to pick up. I'm maintaining a couple of projects if you would like examples:

• https://github.com/billyquith/ponder - C++ library
• https://github.com/billyquith/GWork - a cross platform GUI (not using native widgets)

There are also various templates on Github. Some help notes for cmake (work in progress!): Cheetsheet

The advantage really is that it gives the user options over how they want to use the project with cmake support. I.e. they can choose if they want makefiles, or Ninja, or they want to use an IDE so have Xcode, Visual Studio, etc. Personally I use a different build system depending on what I am doing. Plus it has a GUI to edit the options if needs be.

• Generators supported

[andlabs]

tup seems to be the most promising so far. Is it possible to have different variable declaration rules for different compilers? For instance, a Tuprules.tup for gcc and a Tuprules.tup for msvc, and possibly the same tup upd OS=windows TOOLCHAIN=gcc STATIC=1 syntax I have right now? (And yeah, I could just add TOOLCHAIN to my current makefiles, but there's a lot of Unix assumptions in the gcc rules right now.)

I'll keep evaluating in the meantime (including cmake).

[billyquith]

Yes, you can set compile options per target and you know the platform from the config detection. Cmake runs a configure-style setup before generating the build files so you know platform, compiler, compiler features, etc.

if(UNIX)
  add_definitions(-DFOO)
elseif(WIN32)
  target_compile_options(...)
elseif(APPLE)
endif()

cmake also has a quite powerful generator expressions syntax so that you can specialise variables per target.

However, let's not get carried away. I'm currently building cmake with this CMakeLists.txt. As you can see it is quite simple, like a makefile.

Other areas where cmake simplifies things:

• finding dependent packages
• static/dynamic libraries
• installation - where should installed build products go?
• packaging (zipping build products as distribution)

[jasom]

and possibly the same tup upd OS=windows TOOLCHAIN=gcc STATIC=1 syntax I have right now?

This part isn't possible with tup; tup does not take arguments on the command line.

The good news is that you wouldn't need to pass OS=windows as tup already has a PLATFORM variable internally. For the other options you would use a configuration file. tup supports multiple configuration files, each output to a separate directory, and has a command to automatically generate an output directory from a config file.

So, the flow would be more like:

This step done once only

<posibly edit mingw.config file>
tup variant mingw

Then afterwards:

tup upd

Similarly:

tup variant msvc
tup upd

[andlabs]

@billyquith does cmake have built-in flags for static vs. shared and toolchains? and is it possible to a) build an executable as a flat binary instead of a bundle b) with a custom filename?

@jasom cool, thanks.

waf is also on the table for the record gyp is not; it seems to be hardcoded to use msvc only on windows and seems to only generate IDE files :/

[billyquith]

Static vs shared libraries are fairly simple.

add_library(<name> [STATIC | SHARED | MODULE]
            [EXCLUDE_FROM_ALL]
            source1 [source2 ...])

Toolchains can be implemented by redefining all of the cmake defaults (CMAKE_* variables, which are the defaults for the configuration). E.g.

• Allegro supports many platforms and toolchains.
• OpenCV also uses.

Bundles are just specific to OSX. This cmake config is one I copied from @kainjow (who also seems to use a Mac) so this would need expanding for the other platforms, if they had specific options. The default is a flat file.The name can be changed by changing the target options (OUTPUT_NAME). It can also be changed per-config (e.g. debug libs/exes can have a different name).

[HalosGhost]

@andlabs, it is also worth mentioning that, if you really need to support multiple systems like that, it is trivial to write a configure script to modify Tuprules.tup with sed.

Eventually, tup is meant to support pulling environment variables in with a directive (cf. this issue), but as that has not landed yet, I tend to use the configure mentioned above in my projects for things as simple as the compiler declaration.

[jasom]

Also, having used both tup and cmake, I would make a few observations:

• Tup is a significantly improved make; pretty much everything make does it does better. The only two regressions are:
  • There is no way for it to handle paths with whitespace (make is already bad at this though)
  • It cannot write outside the toplevel directory of the project, so no "make install"
  • I've never found this to be an issue; either wrap tup with make, or just add an install shell script
  • It is somewhat new and less widely used than other build tools
  • While I find it extremely easy to use, I had to solve a couple of problems on my own as the community is smaller
• If tup is a significantly improved make, Cmake is more like a significantly improved Autotools
  • It is much more complicated than tup because it tries to solve a lot of problems tup doesn't
  • It has a configuration file syntax that only a mother could love
  • this is mitigated by the fact that once it's working, modifying the config files should be relatively rare
  • It is fairly robust and there will be examples for pretty much any combination of build tools and platforms you want

My summary is that tup will be easy to use to configure your build exactly as you want it, but you'll have to already know how you want to build on each platform. It is possible you will run into an issue that nobody using tup has encountered before. The upside is that there won't be any black boxes in the build system

If you use cmake, you almost certainly won't encounter issues that other users of cmake have had, but unless you spend some time investment into learning cmake's internals, parts of the build system will be a black box. The upside is that they will be a black box because they have solved problems for you that you otherwise would have had to solve manually. It can even generate nmake files for you, so you could remove the windows dependency on gnu make without having to ever edit nmake files yourself.

I haven't used waf.

[andlabs]

Right now I'm going all in and writing build system scripts for each of what I want to try, starting with cmake. This is what I have so far:

# 30 may 2016
cmake_minimum_required(VERSION 2.8.12)
project(libui)

if(APPLE)
    set(_OSDIR darwin)
    set(_OSSRCEXT m)
    set(_SETVERSION TRUE)
    set(_VERSION "A")
    set(_LIBUI_LDFLAGS "-framework Foundation -framework AppKit")

    # always use our rpath
    set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
    # the / is required by some older versions of OS X
    set(CMAKE_INSTALL_RPATH "@executable_path/")
    set(CMAKE_MACOSX_RPATH TRUE)
elseif(WINDOWS)
    set(_OSDIR windows)
    set(_OSSRCEXT cpp)
    set(_SETVERSION FALSE)
else(APPLE)
    set(_OSDIR unix)
    set(_OSSRCEXT c)
    set(_SETVERSION TRUE)
    set(_VERSION "0")

    # always use our rpath
    set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
endif(APPLE)

include_directories(. common ${_OSDIR} test)
file(GLOB SOURCES
    common/*.c
    ${_OSDIR}/*.${_OSSRCEXT})

macro(libui _name _mode _setver _exclude)
    add_library(${_name} ${_mode} ${SOURCES})
    set_target_properties(${_name} PROPERTIES
        OUTPUT_NAME ui)
    if(${_setver})
        set_target_properties(${_name} PROPERTIES
            SOVERSION ${_VERSION})
    endif(${_setver})
    # omit libui-static from default builds
    if(${_exclude})
        set_target_properties(${_name} PROPERTIES
            EXCLUDE_FROM_ALL 1)
    endif(${_exclude})
    set_target_properties(${_name} PROPERTIES
            LINK_FLAGS "${_LIBUI_LDFLAGS}")
endmacro(libui)

libui(libui SHARED ${_SETVERSION} FALSE)
libui(libui-static STATIC FALSE TRUE)

include_directories(test)
file(GLOB TESTSOURCES test/*.c)

macro(test _name _libui _static)
    add_executable(${_name} ${TESTSOURCES})
    set_target_properties(${_name} PROPERTIES
        OUTPUT_NAME test
        EXCLUDE_FROM_ALL 1)
    target_link_libraries(${_name} ${_libui})
    # be sure to include libui libraries in the output
    if(${_static})
        set_target_properties(${_name} PROPERTIES
            LINK_FLAGS "${_LIBUI_LDFLAGS}")
    endif(${_static})
endmacro(test)

test(tester libui FALSE)
test(tester-static libui-static TRUE)

and once you do cmake .. you build with make [libui] and make tester for shared and make libui-static and libui tester-static for static. Not yet complete, and it seems as if toolchains have to be chosen at cmake time rather than at make time, but I guess that's not much of an issue. And only one CMakeLists.txt file surprises me, given all the cmake projects I've seen have multiple — only one file is optimal for me.

If all else fails this will work, but I still want to give others a fair chance. The real question is can this be described in other build systems...

[billyquith]

it seems as if toolchains have to be chosen at cmake time rather than at make time

Well you either edit config and regenerate the project using cmake again, or have multiple build folders. So:

mkdir other && cd other  # (build used) so all project files in another folder 
cmake .. -G Ninja
ninja all
# edit CMakeCache.txt to change options
cmake ..
ninja all

or

mkdir other && cd other
cmake .. -G Ninja
ninja all
cd ..
mkdir other2 && cd other2
cmake .. -G Xcode
cmake --build

[billyquith]

And only one CMakeLists.txt file surprises me, given all the cmake projects I've seen have multiple — only one file is optimal for me.

You can put it all in one file, and maybe that is the best idea for a small project.

I didn't know you were doing that and have been adding my own cmake files! You can see them split up here: https://github.com/billyquith/libui/tree/cmake

EDIT: obviously that is for Mac at the moment. All the bundle and static options can be set/not-set using variables.

It tends to be more tidy like this and more scalable. Also, a nice feature of cmake is that each subdirectory acts like a scope, so the top level one defines "global" features and the leaves can't interfere with each other.

[billyquith]

@jasom

There is no way for it to handle paths with whitespace (make is already bad at this though)

Really? That's pretty poor. Could be an issue with a lot of Windows users.

It is much more complicated than tup because it tries to solve a lot of problems tup doesn't

But, as you say, it does that robustly.

It has a configuration file syntax that only a mother could love

Beauty is in the eye of the bee holder, as they say. I think most build systems have pretty terrible syntax or nasty gotchas. I admit to start with it is pretty horrendous, but you get to understand the quirks and then it's as quick as anything else to knock up a project.

I was a proponent of premake (another project file generator) not so long ago but I find the development pace of that too glacial. It still doesn't support many platforms. cmake is out there used on thousands of projects and has a very active community.

The other advantage of a project generator, rather than a build system is that you can be straight into a debugger from an IDE, or not, depending on your requirements.

[billyquith]

It's generally frowned on to do this in cmake (although it's perfectly possible to do):

file(GLOB SOURCES
    common/*.c
    ${_OSDIR}/*.${_OSSRCEXT})

It is preferred to list all the files. The reason is that cmake throws an error if a file is missing. You also don't accidentally get files in your project that weren't supposed to be there.

I've listed the platform specific files in the subprojects, so you could just add_subdirectory(PLATFORM) and then all the platforms specific stuff goes in that config file.

[andlabs]

That could work. It'd avoid fiddling with variables, anyway...

[jasom]

@billyquith

There is no way for it to handle paths with whitespace (make is already bad at this though)

Really? That's pretty poor. Could be an issue with a lot of Windows users.

It's actually a non-issue as tup never has outputs outside of your build tree (see my comment on no make install equivalent), and always uses relative pathnames. You can build project foo in c:\my Projects\foo just fine, as long as foo contains no files with spaces (which is completely controllable by the owners of foo)

[kainjow]

You should be able to simplify the CMake syntax a little so that these lines:

else(APPLE)
endif(APPLE)
endmacro(test)

become:

else()
endif()
endmacro()