Consider moving the buildsystem to Meson or another build system

[RiverMesa]

Meson's website

I know, this is a bit of a hot topic, or at least a "nobody really wants to discuss it because nobody wants to actually do it" topic, so I'm putting massive emphasis here on the "consider" part. If the end result of this discussion is that Godot's build system shall stay the way it is, so be it.

So currently, Godot uses SCons, and I get why that is - it's a very powerful build system (since it's basically using Python to write build scripts), and the way it's setup in Godot makes it extremely easy to use. However, it's not without its downsides - primarily, that it's quite slow (being basically that SCons build files are Python scripts, and most Python interpreters do not particularly excel at execution speed), and because of its relative lack of popularity, difficult to find many resources or documentation about it. Plus, I don't personally know how maintainable or easy-to-work-with the buildsystem code, but in my opinion it shouldn't be the sort of situation of "don't touch this code ever, or you risk breaking it and having to do boring and painful work just to get the project working again".

Now, I realize that Godot's project structure is not simple and that's a huge reason why SCons is what was chosen as the build system (and why there has been massive reluctance to move to another system), but I don't really believe that it's the only build system that can possibly satisify Godot's requirements. Feel free to prove me wrong, though.

So what could a migration to Meson offer?

• Better baseline build speeds (and potentially that much better if unity or threaded builds are taken advantage of) [Some sample benchmarks]

• The use of Meson's dependency system, wrap, would help reduce the repository size (third party dependencies would no longer have to be included with the project, and would certainly also help when said dependencies need to be updated to new versions. Use of wrap would even allow for modules (both official and custom) to be simply included as 'subprojects' (and offloaded to their own repositories), further reducing the main repo's size. [On wrap]

• I don't know where SCons stands regarding this, but it seems like Meson is designed to play nicely with CI tools such as Travis and AppVeyor, which might be of note. [CI]

• Meson also appears to some degree of support for localization, but it appears to be gettext based which I believe differs from the way Godot handles this, so I'm not sure how compatible this is here. [Localisation]

• Right now Godot does not seem to make much use of unit tests (the godot-tests repo seems to be rather low-activity), but Meson has full support for performing such tests, which may be valuable. [Unit tests]

• In addition to providing a way to generate the relevant files for both Visual Studio and XCode, Meson comes with an API that allows for integration with IDEs and other build tools, which again seems like something of value for those who don't use VS or XCode. [IDE integration]

This is all in addition to supporting (almost?) everything that Godot already uses and requires - cross-compilation support, source generation, running external commands, et cetera. It's evidently a buildsystem that is capable of handling both small and big and complex projects - complex projects like systemd, GNOME, elementaryOS, Xorg, Mesa.

No promises, but I may be able to try and work on a proof-of-concept of this, to see how it fares. Also if there are any Meson experts/enthusiasts reading this, please feel free to chime in, as I probably got a few things wrong (both wrt Meson and SCons).

Cheers!

[bruxisma]

@isaachier that benchmark is between make and ninja, not ninja and scons, so you'll have to forgive me if I don't want to pay attention to it. ;)

[rminderhoud]

I guess the best way to prove what is better is to write a new on and see :)

I agree, there is no point in bike shedding. If you have an opinion on a better build system, implement it and prove it's better. Otherwise this is a discussion of opinion and will lead nowhere. @fire has already done a partial test with bazel: https://github.com/godotengine/godot/issues/18518.

[isaachier]

@slurps-mad-rips ya totally different topic, not SCons. I just meant make vs. ninja isn't crazy difference.

[bruxisma]

Slight update: I'm giving up on trying to move to meson. It's been piecemeal and while the build files are slightly easier to read, staying up to date with the current system is where most of my (now rare and in short supply) free time is spent, instead of moving scons to meson.

Right now, a lot of code and operations are placed directly in the scons scripts and this makes moving to another build system more difficult. If anything I'd argue time should be spent decoupling the existing python scripts that generate code from scons, as this would make a move to another build system like meson, cmake, or even bazel a hell of a lot easier.

[fire]

Good thing a patch was just merged to move all the code generation out. https://github.com/godotengine/godot/pull/17595 Don't give up!

[bdbaddog]

SCons Project Co-Manager here. A couple notes: 1 - SCons is under active development and since our migration to Github we're seeing an increased rate of pull requests. 2 - We've identified a few specific parts of SCons which are responsible for about 50% of the null incremental build time and are actively working on addressing that. (The Subst functionality) 3 - We've started supporting Python 3.5+ (as well as 2.7.x) with SCons 3.0 4 - Let me suggest that instead of attempting to port to N different build systems, contributing some time to help improve SCons might be a more effective use of developer time 5 - We identified and addressed via profiling some of the low hanging fruit in improving performance which was released in 3.0.0 and 3.0.1. If you're using an earlier version the null incremental build can be 5-15% slower. (assuming Python 2.7.x)

As always the SCons project is ready and willing to help projects who use SCons with issues they run into.

[Griefchief]

Some "stats" for SCons & godot.

I used (and still do) a Q6600 which is a 4 core cpu from 2008 or so with 2.4GHZ (and a lower instructions per cycle count) than the processors we are "all" using today (i7s).

On Q6600, a built took between 3 to 5 minutes (or more, but exact num not that important), out of which ~35-40 seconds was spent by SCons parsing the tree of dependencies and stuff (the prep work)... so, 80% of the time spent running cl.exe (MSVC compiler) process over and over again.

So, performance & optimisation wise, unless SCons (python really, not SCons) has a terrible process calling system that is taking more time than it should or some other tools is not compiling one file per process (like SCons currently is) and the 1 process per one file overhead is significant (I'm not an expert on cl.exe compliation internals & startup overheads), the only performance optimisation a faster build system would produce is probably in the building of dependency tree phase, and other prepwork for complication.

So, we are optimizing those 35-40 seconds out of a 5min build (on i7's now we are probably optimising a 20 second single core setup step out of a 3 min multi core build).

So, to recap, if i remember correctly, 3 to 5 min is actually a 4 core multicore build, while a single core build takes around 10? I need to rerun my tests...

So, "optimizing" SCons away means more or less optimizing those first 35 seconds of the builds start...

That's as far as "optimizations" are concerned when it comes to SCons vs other faster build system... It's possible that the actual compilation could be optimised a bit by avoiding calling a single cl.exe for every obj file, but that's only if the process overhead is significant (I fear HDD will always be the actual IO bottleneck there (and not the CPU), which is not avoidable/mitigatable without an SSD)

All of these paragraphs above consider a "full build". That said, SCons "cheats" if everything is already built and passes it's heavy handed consistency checks so if you change a single file it will actually compile a "single file" and then link everything back in, making a "full build" into somewhat of an incremental build). This however requires those 35-40 seconds of full dependency tree precomputation...

So, an "incremental build" is around 45 seconds on my old computer, and I thought when I was a noob 2 years ago that it would be CPU bound, but actually, it might be IO bound (checking several thousand files on the hdd) for a full dependency sweep/tree rebuild... So, if it IS IO bound, no "faster buidlsystem" is going to solve that one...

My hackish idea from 2 years ago was to try to attach a filesystem "watcher" to SCons, have SCons run & recompile a single changed file (and it's dependencies, that's automatic), and then just have it link everything back again... although I realise today that linking would still probably run a full dependency sweep... that said, the dependency tree sweep/rebuilding can be killed with an option and then SCons will use it's cached dep tree... which saves around 15-20 seconds of the 35 sec precomputation step (the first 15 seconds seem unavoidable) although that's might not guarantee a "perfect" build always like SCons does (though that might not be important for incremental builds in dev... if you decide that speed is a worthy tradeoff).

I now have enough knowledge that I could probably hack it through npm and npm watch... and or other system that I'm not currently familiar with... But that's for later, and when I have time (not currently).

This is getting a bit long so I'll end it here, but the point of these paragraphs was to give you some "statistical" information rather then to discourage anyone (go ahead if you like working on build systems), like I myself liked fixing some stuff in Godot's SCons. I hope some information was useful for you guys here.

If you wish to replicate/redo my statistics, just browse through SCons docs till you that turn on statistical info (timers stuff) and/or find a way to turn off dependency redolution (those first 20 to 30-35 seconds of "precomputation")...

Or for a slightly less manual solution, timer debugging info should already be present (edit: it's prob not, somebody did what looks like a sweet update to the vs project generation, this paragraph might be stale info) in the automated VS project generated by SCons (at the end of SConstruct file)... you can find info on how to generate a VS Project in the "Windows compilation tutorial" on the godot website. You can then run SCons from VS, I believe the generated project should still probably work, though I have not tested it in Godot 3... It's an option, but tailing scons to a > log.txt is also another option (etc)...

Here's hoping somebody found this info helpful.

[bojidar-bg]

@Griefchief Just to clear things up a bit, I think we aren't trying to optimize full-build times, since those are dominated by compiler time anyway. Instead, we want to optimize incremental builds where only a single source file is changed.

[bdbaddog]

Indeed the "null incremental build" (nothing actually needs to be rebuilt, but scons will still need to process the whole dependency tree to ensure that) is the case we're working on.

Some profiling has pointed out some inefficiencies in processing the command lines (which have to be done for every target to ensure that they have not changed) have been identified. The bulk of that is the Subst() logic. This is some tricky code and thus far has been implemented in a simpler and safe, but not terrible efficiently way.

Another hot spot of time is the reading and writing of the sconsign files, currently implemented with cpickle and always read and written as a whole. We have some ideas on how to either speed up and/or make more incremental. Anyway watch future releases for improvements.

On a side note, you might try MSVC_BATCH to see if that speeds up your windows builds:

MSVC_BATCH When set to any true value, specifies that SCons should batch compilation of object files when calling the Microsoft Visual C/C++ compiler. All compilations of source files from the same source directory that generate target files in a same output directory and were configured in SCons using the same construction environment will be built in a single call to the compiler. Only source files that have changed since their object files were built will be passed to each compiler invocation (via the $CHANGED_SOURCES construction variable). Any compilations where the object (target) file base name (minus the .obj) does not match the source file base name will be compiled separately.

[Griefchief]

@bojidar-bg yes, yes, I was actually talking about incremental builds as well, thank you for the tip :D

@bdbaddog Hi baddog, just for your info, I actually wrote this big comment of mine before I saw your comment, so, it doesn't actually refer to anything in you comment specifically... but you did manage mention some of the "concerns" I had (information for other people really) while we were both writing our comments at the same time...

And I didn't want to ping you in it cause I didn't wanna "waste" your time :D, but I guess I will if spend some more time on godot build/scons in the future (I do plan to, i'm just very busy))

Thank you for your input here!

[Griefchief]

@bdbaddog BD, since you're providing support, if I may ask here:

If I attach npm watch to the filesystem (godot's source) and know which exact file has been changed, would that bring me any benefits with SCons?

Now I can run scons through npm watch and push the file to it...

scons [godot options] the/file/that/was/changed.

this will obviously recompile that file, probably avoiding Godot's whole dep tree? correct? So now dep tree traversal won't take 20 sec, but less then 1 prob, but I get 1 updated .obj file (or more, depends on what you change). It won't actually go through the whole godot dep tree for that one file? I do have an advantage in less dep tree traversal in this example (and possibly other advantages in the precompute phase that can not be gained through reusing the dep tree cache, like less files overall to go through?)?

Now, if I do have an advantage, and if I wanted to just link everything again, would I need a full tree traversal for a "simple" linking phase?

I understand this usecase might not be currently supported by Scons (just link everything again), but I'm asking both from a practical and theoretical standpoint? I realize that cached dep tree could be reused with that "don't do dep tree" option (I forgot it, it's been two years), and that would work for some usecases, and that usecase is no new files added in, correct? (npm watch could alert me to new file addition and I could then do a full dep tree refresh if I detect that... in other words, kind of push the "ensure dep tree consistency" back to a filewatcher like npm watch, automate typing in that "use dependency cache" option done manually by the user to a program like npm watch... let the program worry about consistency, not the user, and do that in "real time", the moment the user has saved a file, giving lots more time for scons to do it's thing)

Is there something in my idea that would not work as an optimisation(practically and theoretically), and do you have other suggestions (that do not have to guarantee a perfect build if something like npm watch can take care of stuff, it's fine as well)? Can you advise please? Thank you!

[Griefchief]

To recap, the usecase is similar to this:

1) run a filewatcher like npm watch on godot's source directory 2) user saves/modifies an old file 3) scons is immediately run for that file by npm. 4) if compilation is successful, npm instructs scons to run the linker and link the executable.

4.1) if dep tree is cashed (a full build was already run), npm can instruct the linker to just link at this point with a cached version of dep tree 4.1) if no cached dep tree is detected, npm watch runs a full godot build 4.2) if cache invalidation was detected, like adding a new file, perform a full dep tree traversal (npm won't use the "use cached dep tree" option with scons).

I hope this makes my idea a bit easier to understand, please let me know what's wrong with it at this point ;D

P.S. I believe this is the command that shaves of 15 seconds off of the 35 null incremental build, with it's "obvious" consequences... I basically want to automate what is described here (more or less, + let scons know which exact file was modified if that helps it in any way (or it could help it)):

https://www.scons.org/doc/latest/HTML/scons-user/ch06s04.html

[bruxisma]

So a few things here (in no particular order, but I'm using numbers because yolo

1) Incremental null builds are basically nothing under any tool that generates ninja files. Chrome has, AFAIK, a less than 2 second null incremental build. This is of important note, as a 35 second drop to 15 under an npm watch is still very large compared to a meson or cmake based Ninja build. Ninja also does the same thing that Scons does in that it does perform a rebuild in the event that the command line changes (make, of course, does not do this)

2) CMake recently implemented the ability to use CONFIGURE_DEPENDS as an argument to the glob system. This has always been recommended against because tools like xcodebuild and msbuild have historically (and presently) not supported directory change detection. This is, afaik, one thing that Scons could improve upon, but I have neither the time nor patience to deep dive into the project to implement this. Basically, every OS updates a directory when its contents change, be it an added file, modified file, or removed file. When making incremental changes, one can simply check which directories have changed and glob only those on a reconfigure check. This can reduce work done as you're not globbing the entire tree, but smaller bits. For small projects this is fine. Whether this would benefit godot or not would require some testing and that's a lot of work to check if it would even help.

3) While it's nice to see that Scons is receiving more attention, I personally think it is better for the community at large for godot to move to a more used build system. I cannot think outside of godot of any large project that uses Scons. GNOME/GTK recently moved to meson (skipping CMake), KDE has been on CMake for quite some time. While it's anecdotal evidence (and thus a basic confirmation bias), I know of several C++ developers who would like to try godot but would rather not have to touch scons ever again. Removing the nasty experiences and thus the sour taste left in developers mouths is difficult, and I do honestly wish the Scons project the best. But I do know that the C++ community at large already hates long compile time. Long incremental null builds (or even long incremental builds of 1 file) are a nightmare in the modern world.

I do intend to try again, but I might target CMake for this. Moving from CMake to meson is a hell of a lot easier thanks to meson having a conversion script that can do a lot of work. CMake might make handling dependencies a bit easier with the new FetchContent module. We'll see how it works out. I will say, I'm glad these codegen scripts were moved to separate files. Using CMake will be nice if only because moving to a different C++ standard is as simple as target_compile_features(<target> (PUBLIC|PRIVATE|INTERFACE) cxx_std_<number>)

Obviously any build system related stuff is going to be contentious, but trying can't hurt.

To be quite honest, (and this might come off as callous) I have no desire to deep dive into the Scons project to improve it. Maybe after 2020 when Python 2.7 is officially finally really dead and the project can move forward to Python 3.5+ only it'll be worth investigating or improving, possibly with async operations. Until then, however, I'd rather not have to touch it.

[Faless]

Maybe after 2020 when Python 2.7 is officially finally really dead and the project can move forward to Python 3.5+ only it'll be worth investigating or improving

Scons 3 does support Python 3.5+ : https://scons.org/tag/releases.html .

I don't really understand how one can dislike Scons so much and like the configuration mess that is CMake, where it's even hard to define your own build settings if you are a user, hard to understand what are the options and their types and what are internal defines, it only brought pain to me, and I was actually relieved that Godot didn't use it.

In any case, if you feel like it's going to be a better solution go on and try, I might end up changing my view on CMake.

[OvermindDL1]

where it's even hard to define your own build settings if you are a user,

If that is asked then CMake is being used wrong. All toolchain settings should be in a toolchain file and any project specific configurations are passed via flags (or gui or edit the cache file or whatever). Everything is documented and typed, even the project authors can well document and type their exposed configuration. You should read CGold and the modern CMake standards.

[isaachier]

@Faless no one denies the CMake language is terrible. But the implementation is best in class. I've been considering the possibility of writing a transpiler from a better language to alleviate the issue.

[isaachier]

@OvermindDL1 love the suggestion to read CGold.

[Faless]

If that is asked then CMake is being used wrong

Well I guess that all the programs I encountered where doing it wrong then.

To be honest, I hate having to change flags via -DI_LOVE_CMAKE_AND_MAKE_DEFINES_LONG=on (really? on?). Not to mention the idea of editing the cache file O_o . Or that, if I want to change a flag, I basically end up having to delete the build folder and start from scratch (because, you know, cache differs!)... so much for incremental builds...

EDIT: Also, one thing I really liked about Scons (in Godot at least, and this might possible with other build system of course), is that I didn't actually read any standard or doc, I just run scons -h in the Godot folder

[isaachier]

95% of build files I have seen on GitHub are wrong. People get lazy about build systems and follow cargo cult practices to cobble something together.

[Faless]

@isaachier , just as a metric, could you provide 2 examples of a bad and good CMake file? How would you consider this: https://github.com/ARMmbed/mbedtls/blob/development/CMakeLists.txt ?

[isaachier]

Better than most I've seen. Still, if you are not using CMake >= 3.0, then your experience with is not with modern CMake.

[OvermindDL1]

Well I guess that all the programs I encountered where doing it wrong then.

I tend to see it done well nowadays.

To be honest, I hate having to change flags via -DI_LOVE_CMAKE_AND_MAKE_DEFINES_LONG=on (really? on?). Not to mention the idea of editing the cache file O_o . Or that, if I want to change a flag, I basically end up having to delete the build folder and start from scratch (because, you know, cache differs!)... so much for incremental builds...

You don't have to use on, you can use true, 1, and a variety of other things. It is to make it so you can choose an appropriate boolean name that flows well with flag name.

And you should not ever need to rebuild the build folder if you change options, cmake is exceptionally good at rebuilding when necessary.

Still though, everyone touching C/C++ build systems should read CGold as well as the official manual of CMake. It is really easy to use CMake right, it is often a great deal shorter as well. The ancient CMake2 methodologies should not be being used. Really though, anyone that depends on a CMake 2.x version is absolutely doing things wrong (not using targets and toolchains properly, not setting options properly, etc... etc...).

[Faless]

Thanks for the explanations, while I remain sceptical, I'm looking forward to see how it would work for Godot :).

[OvermindDL1]

@slurps-mad-rips Your post with the supposed CMake example code for enabling C++11 mode is entirely non-standard CMake for a variety of reasons, including but not limited to:

• Setting unexposed variables.
• Toolchain arguments set in a non-toolchain file.
• Absolutely not how you do something like enable C++11 mode.
• Inefficient.

Given the shown self-described-as-simplified meson code in that post of:

cxx = meson.get_compiler('cpp')
# This is an extremely simplified approach. One can do a different option when dealing with MSVC and gcc support.
args = compiler.get_supported_arguments('-std=c++11', '-stdlib=libc++')
add_project_arguments(args)

The equivalant simplified (in that you can add more properties as well) CMake would be:

set_target_properties(godot PROPERTIES CXX_STANDARD 11)

And that will make sure that the toolchain used knows that it needs to set that mode (and you can specify how to handle failures if using a compiler that doesn't support it as well). Assuming special compiler-specific command line flags in the build file is extremely poor form as those should only exist in toolchain files and the fact the meson example shown in the prior post actually specifies specific arguments essentially means you are hardcoding the supported compilers.

Considering there is a pre-built CMake Toolchain file for just about any toolchain you can think of (including all the standards of MSVC of various varieties, GCC/Clang, android, iOS, emscripten, etc... etc...) you should not be re-inventing the wheel. If the user that is compiling something like godot wants to use their own custom compiler with unique commandline arguments (like take Intel's compiler or maybe to some dedicated chip or just a simple RPi) then the build script shouldn't need to care about it or need any changes at all (the code might of course, but optimally it should not ever either as CMake can tell it what all is supported via definitions) and the user can supply their own custom toolchain file for the build to use.

EDIT: And never discount the widespread support. I can't find anything about meson being able to generate project files for KDevelop (the IDE I use, which interestingly uses CMake as it's build format, I occasionally use CLion at work, it also uses CMake), where almost every IDE out (even Visual Studio!) can open CMake projects natively now (in addition to CMake being able to generate projects for most IDE's directly if you so wish).

[bruxisma]

@OvermindDL1

It is not non-standard CMake in the absolute slightest. The code sample targets CMake 3.0, a version in which CXX_STANDARD does not exist. You can easily see from here that you use target_compile_features with cxx_std_ and the number of the standard. This is the new and modern cmake way to set the standard version. set_target_properties is an older per-target approach. In fact, the exact code taken from CMake's documentation is

target_compile_features(mylib PUBLIC cxx_std_11)

However, CXX_STANDARD does not set libc++ support for clang as clang uses libstdc++ by default on linux unless configured (and it's never configured to use libc++ by default on most linux distributions, as this would cause linker errors and ABI issues). The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression. There is nothing involving unexposed variables regarding this.

Furthermore, I do not appreciate you dropping a comment to make corrections from a comment I made months ago, especially given that I will be targeting CMake over Meson in my next attempt. It lacks tact, and in my opinion adds nothing to the conversation.

[OvermindDL1]

CMake 3.0 is old enough to no longer be supported (June 10th, 2014). No one should be running anything less than CMake 3.10 currently, though staying up to date with CMake 3.12 is preferred at the current moment.

And yes, the example you gave is entirely recommended against (I think even CGold mentions not doing that).

And yes, I did mean target_compile_features. ^.^;

However, CXX_STANDARD does not set libc++ support for clang as clang uses libstdc++ by default on linux unless configured (and it's never configured to use libc++ by default on most linux distributions, as this would cause linker errors and ABI issues). The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression. There is nothing involving unexposed variables regarding this.

The current toolchains all do that for clang just fine, at least with C++14 as that is what I compile with (and a couple C++17's).

Furthermore, I do not appreciate you dropping a comment to make corrections from a comment I made months ago, especially given that I will be targeting CMake over Meson in my next attempt. It lacks tact, and in my opinion adds nothing to the conversation.

I apologize, did not notice the date as I was reading the backlog (and there is a lot of backlog here). I just noticed something that appeared woefully wrong for any kind of modern suggestions from what I've seen for a few years now and did not see any corrections in posts shortly after thus wanted the incorrect information to not propagate. :-)

The only way to do it at this time is to check if the compiler supports -stdlib=libc++ and pass it to the compiler via a generator expression.

However yes, that is the job of the toolchain file, that should absolutely not ever appear in a build file, ever.

There is nothing involving unexposed variables regarding this.

I was just referencing the variables that are set then used elsewhere in only single locations, that is recommended against in CGold among others for a variety of reasons.

[bruxisma]

CMake 3.0 is old enough to no longer be supported (June 10th, 2014). No one should be running anything less than CMake 3.10 currently, though staying up to date with CMake 3.12 is preferred at the current moment.

They shouldn't, and yet some places are. For a game engine that tries to stay up to date and with the latest technology, staying on latest CMake is fine. However I am more than well aware of extremely active projects stuck on older versions such as 3.5 and in some cases 3.4. In fact, some well known C libraries still target CMake 2.8 (See: SDL2, libssh2, libgit2, etc.) It is a huge pain in the ass to be sure.

However yes, that is the job of the toolchain file, that should absolutely not ever appear in a build file, ever.

Making this assumption (that toolchain files are well written and cover all possible systems and all versions of CMake) is asking for trouble in the current general C++ ecosystem. When using clang, the option of compiling for libstdc++ or libc++ (especially when used via a FetchContent call) needs to be provided by the library. CheckIncludeCXX and CheckCXXCompileFlag are still necessary for even some flags that don't exist under GCC but do exist under Clang (and vice versa). The primary issue here is that compiler interfaces have diverged a large amount and instead of vendors spending time to facilitate compatibility options or even discussing a common interface, that behavior is thrust onto us, the developer when all we want to do is write code. Trust me, no one on this planet is more upset about the state of C and C++ build systems and dependency management than me, but pragmatism for existing libraries on CMake is needed. The nice thing about moving from a build system to CMake is that we can start out the gate with the latest and greatest features. The problem becomes stagnation of the build system because no one wants to touch it (except for that brave soul who kicked the code generation build scripts into separate scripts), but some work would be needed to make sure it doesn't get out of hand (which it will because... I mean it's a build system. They always do)

[isaachier]

I must be the one guy who actually enjoys writing build scripts especially in CMake ;).

[OvermindDL1]

Making this assumption (that toolchain files are well written and cover all possible systems and all versions of CMake) is asking for trouble in the current general C++ ecosystem. When using clang, the option of compiling for libstdc++ or libc++ (especially when used via a FetchContent call) needs to be provided by the library.

Such things should be properties passed to the toolchain that the toolchain can then act on.

As well as if you have very unique setup it is entirely reasonable to include toolchain files, like say for a very custom android build or so (though the extended polly set of cmake toolchains has covered almost everything I've needed thus far).

The problem becomes stagnation of the build system because no one wants to touch it (except for that brave soul who kicked the code generation build scripts into separate scripts), but some work would be needed to make sure it doesn't get out of hand (which it will because... I mean it's a build system. They always do)

This is actually a great point for choosing CMake now, it has a well defined set of standards to follow that work really well now after dealing with more than a decade of prior experience, both good and bad.

I must be the one guy who actually enjoys writing build scripts especially in CMake ;).

I enjoy it in comparison to alternatives for sure, though I've written in it long enough that it makes sense to me, though that definitely doesn't mean it would for everyone. ^.^;

[spongeboburu]

I've used a lot of different build systems over the years. The one that made most sense for me recently is CMake, but I can definately see the appeal of SCons and how flexible it is.

Even if SCons isn't perfect (what build system is?) it may be good enough.

• Does it provide the required features?
• How much overhead does it create?

If SCons scores "good enough" for both points then keep it.

[reduz]

Has any of you proposing the replacement of SCons by something else actually taken the effort to learn how Godot build system works?

As a hint, it's incredibly complex and I don't think there are many (or any) projects using CMake building for this many amount of platforms and allowing to have several builds targets configured/compiled at the same time the way we do.

All efforts by those who said "I will port Godot build system to somehting else" failed miserably so far when realizing the complexity of what SCons is taking care of.

To me, it's hands down the best tool for the job by far. Nothing is even close to it. If you guys complain about base compilation time (which is just 4/5 seconds on a medium to high end system with SSD), you really should first try to understand everything Godot does when building and see how it would perform in CMake or anything else.

[isaachier]

I did take a look. I am not convinced it is undoable in CMake or elsewhere. But I'm also lazy ;). We'll see what happens.

[fire]

As someone who did the bazel port. I was able to get to the point where the Godot editor started without icons.

I'm pretty sure CMake can reach that point.

PS. The closest competitor project with large amounts of platforms is Urho3d.

[fire]

I reproduced my bazel result on cmake.

https://github.com/fire/godot/tree/cmake

Assume visual studio 2017 is installed.

git clone https://github.com/fire/godot.git -b cmake
scons p=windows
Modify platform/register_platform_apis.gen.cpp
#include "register_platform_apis.h"

void register_platform_apis() {
}

void unregister_platform_apis() {
}

Install cmake

choco install cmake ninja -y
# Open visual studio command prompt amd 64 2017 native
# Go to godot source directory
cd ..
mkdir build
cd build
cmake ../godot -GNinja
ninja

Please play with it. The editor has the same problems as the bazel build (no icons), but you may find this sample helpful.

Note

• Icons work as of 71175b45f819e7cc5e4368dbf3e42abdd19af542
• Visual Script and GDscript work

[bruxisma]

@fire Thanks for this. I see a lot of places for improvement that would make maintenance of these CMake files easier (and time spent adding new features or compiler flags). I might tackle this next weekend however, schedule pending.

[isaachier]

Awesome work @fire. Glad to see someone here can produce something in more than just one build system :smile:.

[bruxisma]

Slight update. I had some downtime today to tackle building off of @fire's work. Unfortunately, the code generation scripts are still a little too embedded with the current Scons system and some work is needed to fully extricate them (this is of course assuming I'm reading the code correctly). That said, it'll probably be a separate issue + pull request. I did identify a few areas where an external template file (or series of template files) could be used. Python has support for template strings and this might make life a bit easier, instead of having to process strings, add them to a list, and then process more strings, and add them to a list, etc. etc.

Ideally, the code generation scripts should be able to be run as if they were executables without knowledge of the build. As long as the proper files were passed to them, we'd be good to go for fully porting to any other build system.

[bdbaddog]

@slurps-mad-rips Extricating that from your SCons system is a good idea even in SCons. You can hang up the GIL by having too much logic in process and hurt your build performance. Occasionally it's lead to some file close/open race conditions when building in parallel (though often resolved by opening the file with in a context).

[bruxisma]

Hello all,

I'm going to use this space to keep people updated on the port to CMake. If requested, I'll make a separate issue and keep track of changes there, though this is a nice area given it's got a massive backlog of discussion and I think it'd be a waste to lose so much context.

I digress. I'm going to have to take a break from the CMake port for the next week while I attend a conference, however you can see the efforts of my initial porting attempt here. Please keep in mind that I'm going to be shooting to stay 1:1 with the current master branch, and as such I'll be force pushing changes as they come in so I can keep a rebase. As I approach stability, I'll file a pull request to keep rebases easier to work with, as well as to make it easier to comment on changes or ask questions. I think this port to CMake is not only doable, but at the very least it'll make one more person (me) well versed in how Godot is actually built and constructed and I might be able to put that into words so others can get a bird's eye view of the build system. This port also might possibly make everyone's workflow easier in the long run, especially with releases. I'm not gonna lie, having the CONFIGURE_DEPENDS globbing of files mixed with Ninja is actually quite nice.

One last thing, I highly suggest you all take a peak at the helper CMake Modules I've written just because if you are writing other CMake project's, these modules might come in handy.

[fire]

Were you able to update to the latest 3.1 alpha? The alpha could be a good target. Currently trying the build.

[zaniar]

This issue's title is Consider moving the buildsystem to Meson. I think it is inappropriate to have CMake discussion here. Discussion about CMake should have it's own issue. Comparison of Meson vs anything else is fine.

[bruxisma]

@zaniar There's been a lot of discussion in this thread regarding build systems in general. Just because the title hasn't been updated doesn't mean we're going to suddenly create a new issue and lose all that context for discussion and start all over again. I think asking us to suddenly throw away several months worth of discussions is inappropriate.

@fire I've been busy with work the last few weeks, I've got a large deadline for Friday, and then after that I might have some time to get my fork up to date. I've got a few changes locally I forgot to push up but I don't have the time right now. (Between this build system port, my fork of gnu m4, 15 proposals I'm writing for the C++ standards San Diego meeting and CppCon in the last week of the month, I'm absolutely swamped, and I have no one to blame but myself)

[RiverMesa]

Yeah, Meson was the original discussion target, but I think it's kind of natural that other build systems have been brought up and discussed since, and it probably makes sense to keep the discussion within this issue here instead of spreading it out, like @slurps-mad-rips said.

[chanon]

I agree with moving to CMake. It has a lot of support with its toolchains. Android builds for example become very easy compared to what Godot is doing in its SCsub file which looks 'unmaintainable' to me.

As an open source project, Godot should go with solutions that are easy to maintain and that contributors are more likely to know and be able to contribute to.

Also, I think this is important for GDNative adoption. Whichever build tool is used, there needs to be an example or standard project with easily modifiable build script that supports building for all platforms, including iOS, Android, etc.

Also normal modules should be able to be compiled as shared libraries. This is related to #19486 so that the engine can be separated into multiple downloadable components.

[bruxisma]

I'm actually tackling these. I haven't pushed my changes up in a while because I've been busy with CppCon, the upcoming C++ standards meeting, and we're doing a release next week at work. Suffice to say, I've not had much time to finish up the port, but most of my time is going to be spent ripping the python scripts that generate C++ code out going forward. I'm hoping to resume work on my port this weekend. I'm currently evaluating whether to automatically acquire third party dependencies and apply patches stored in the repo or not. One question I do have for the core developers is how open they would be to breaking all the third_party repos out of the main repo and into their own repositories on the organization. I'm relying fairly heavily on FetchContent in my own projects, and I'm currently wishing I could do that for these third party dependencies that have been slightly forked. This would also let me work on each one in turn to clean up their cmake files while also doing incremental changes to the core repo once each third party library is ported.

Additionally should note, is that I'm trying to also modularize the engine itself as well, so these components could technically be updated separate from each other if someone wanted it. This might knock out the issue that @chanon mentioned.

Lastly, I'm adding support for a few additional build tools for the core developers if they're found on the system. Things like clang-format, clang-tidy, ccache or sccache, distcc, clang-check, address sanitizer, ubsanitizer, thread sanitizer, etc. These are mostly linters (and might make some people's heads explode with the warnings and various errors they've been detecting) but they'll be optional to enable. At the very least using sccache or ccache will give some build improvement when switching between builds and vice versa. Waiting for CMake 3.13 to be released would be nice as this would solve some of the kludge-work I've had to rely on, but the important thing is that I'm setting it up so that the current "getting started" workflow is to still run pip install via Python 3.5 or later, however it'll be inside of a virtualenv so people don't have to mess with their system's environment. (This would also allow others to experiment with using additional python libraries as dependencies for various code generation scripts and not have to remember to uninstall them later)

Anyways, sorry for the update sudden info dump y'all. I'm dedicated to getting this port finished, but life has gotten in the way recently 😅

[OvermindDL1]

I'm relying fairly heavily on FetchContent in my own projects, and I'm currently wishing I could do that for these third party dependencies that have been slightly forked.

You could also use Hunter and make a godot-specific code repository (a special git repo, like on github for example) and access all the dependencies that way as it handles the building, cache'ing, etc... as necessary and properly (or submit all the dependencies to the 'main' Hunter package repo and just use it straight). Hunter is just a single cmake file (HunterGate.cmake) and a call to it to acquire the package repository information (whether the official one or a custom one).

[voithos]

A humble request: this issue's comment thread has gotten huge - would someone who's been following it be able to summarize the general discussion-so-far in a comment (and maybe it could be linked to from @RiverMesa's initial post)?

[germandiago]

Hello everyone. I am planning to use Godot as my next engine for making a small game.

I saw this conversation and as a person who has experience in both CMake and Meson (and autotools, tup, plain make, waf, SCons, trying full projects in all of these tools...) I wanted to give my feedback.

The systems I used more intensively have been Autotools, CMake and Meson. I have been using Meson and CMake for the past years, since I already discarded all the other build systems in production setups.

Pros of cmake

• Mature project generators if you care about Visual Studio (I think Meson does support it well nowadays but did not try lately) and especially XCode (pretty sure it is more lacking here).
• Wider adoption, more support in IDEs
• Wider adoption -> easier to get contributions (though Meson is a breeze to learn I must say)

Pros of Meson

• many useful targets by default: sanitizers, unity builds for free, precompiled headers for free, coverage for free...
• the documentation kicks the ass out of CMake documentation: http://mesonbuild.com/
• I found cross compilation to be much easier
• Meson has an obvious way to do each thing

Which one would I choose? I would choose Meson if cross-compilation is going to be intensive and you care about fast compile-times (unity builds and precompiled headers are supported by default). I would consider CMake since it is the standard and contributions maybe are important for the team. I would choose myself Meson personally, but that is just a personal opinion, since I find the tool just better and that saves my own time (CMake scripting, space handling, generator expressions, etc... I am looking at you).

Sorry for this shameless self-promotion below , but I think that the context is begging for it.

I have a small series of articles about more or less basic Meson here if anyone is curious (4 articles):

• https://medium.com/@germandiagogomez/getting-started-with-meson-build-system-and-c-83270f444bee
• https://medium.com/@germandiagogomez/getting-started-with-meson-in-c-part-2-58150354ff17
• https://medium.com/@germandiagogomez/getting-started-with-meson-in-c-part-3-70b9bc419957
• https://medium.com/@germandiagogomez/getting-started-with-meson-part-4-8bceec6149e1

I have a (bit outdated) reply on build systems in StackOverflow:

https://stackoverflow.com/questions/5837764/autotools-vs-cmake-for-both-windows-and-linux-compilation/24953691#24953691

Just my two cents :)

[dorkbox]

@germandiago That is a great writeup -- I've been following this issue for a while and it would be wonderful if I could work with godot + CLion, which is sadly CMake only.

[OvermindDL1]

Yeah most modern IDE's are either CMake-only, and/or some Custom Build System, and/or 'raw calls' that lose a lot of features. CMake really has become standard, whether it is merited or not.