search

godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
91.41k stars 21.26k forks source link

Consider moving the buildsystem to Meson or another build system #16014

Closed RiverMesa closed 4 years ago

RiverMesa commented 6 years ago

Meson's website

I know, this is a bit of a hot topic, or at least a "nobody really wants to discuss it because nobody wants to actually do it" topic, so I'm putting massive emphasis here on the "consider" part. If the end result of this discussion is that Godot's build system shall stay the way it is, so be it.

So currently, Godot uses SCons, and I get why that is - it's a very powerful build system (since it's basically using Python to write build scripts), and the way it's setup in Godot makes it extremely easy to use. However, it's not without its downsides - primarily, that it's quite slow (being basically that SCons build files are Python scripts, and most Python interpreters do not particularly excel at execution speed), and because of its relative lack of popularity, difficult to find many resources or documentation about it. Plus, I don't personally know how maintainable or easy-to-work-with the buildsystem code, but in my opinion it shouldn't be the sort of situation of "don't touch this code ever, or you risk breaking it and having to do boring and painful work just to get the project working again".

Now, I realize that Godot's project structure is not simple and that's a huge reason why SCons is what was chosen as the build system (and why there has been massive reluctance to move to another system), but I don't really believe that it's the only build system that can possibly satisify Godot's requirements. Feel free to prove me wrong, though.

So what could a migration to Meson offer?

This is all in addition to supporting (almost?) everything that Godot already uses and requires - cross-compilation support, source generation, running external commands, et cetera. It's evidently a buildsystem that is capable of handling both small and big and complex projects - complex projects like systemd, GNOME, elementaryOS, Xorg, Mesa.

No promises, but I may be able to try and work on a proof-of-concept of this, to see how it fares. Also if there are any Meson experts/enthusiasts reading this, please feel free to chime in, as I probably got a few things wrong (both wrt Meson and SCons).

Cheers!

vnen commented 6 years ago

For what I understand, the buildsystem tooling must provide:

I don't know Meson, but if it provides everything above it should be fine (if I didn't forget something). Of course, someone has to go the excruciating pain of rewriting the buildsystem in Meson and then show the tooling and build times are effectively better for Godot than SCons.

Note that a few third-party dependencies requires a patch to be compiled with Godot. I noticed Wrap has a patch support, but it looks like the patch must be available somewhere to download, while I think it's easier to manage if they are still part of the repository. Also, some of them require special compilation flags that should not be applied to the whole build.

mhilbrunner commented 6 years ago

I think such a proof-of-concept would be very valuable because

For the second point, this means we can see the difference in build speed for Godot, we can see how much easier (or more complex) our build scripts become, and we can compare the differences more easily to make a choice instead of making educated guesses based on what whe think it could offer.

--

Now, personally, I have to say I'm neither for nor against this, but I think there are more low-hanging fruit to chase instead of (or before?) changing the entire build system. Especially at this critical point in time, where as many hands as possible are needed to make sure 3.0 and the first patches are in the best shape possible. But if we want to improve the build system, I'd look at the CI build times and the hassle of the Mono build situation. (I think these are on akien's TODO/list of aggrievances?)

Also, the bugged parallel builds on Windows suck, but that is more on a personal would-be-nice-if-it-was-fixed basis.

TL;DR:

My opinions. :)

RiverMesa commented 6 years ago

I absolutely do believe that getting 3.0 out there and focusing on the following releases (3.0.1, 3.1) should take priority over this, for the record. I know why people don't like to work on buildsystems (since it's really only for the benefit of other developers/particularly tech-savvy end users - which isn't unimportant, it's worthwhile to make the devs' and contributors' lives easier, but ultimately the end users' happiness is the goal) and there is this whole "opportunity cost" thing, ie. when the time spent (or lost/wasted, depending on who you ask) on improving the build system could be spent on improving the project itself. (Unless you're like me and can't really work on the C++ side of things anyway.)

And ultimately, SCons really does "just work" which for most developers out there will probably be good enough, so while this is something that I hope does eventually happen, I'm not holding out my breath for it for 3.1 or even 3.2, if it even garners enough support to actually end up being worked on to begin with.

All this said, I might kickstart the POC later this week in my fork, for those who are willing to follow the progress on that (and likely help me out, as I don't have a huge amount of experience writing build systems), but I won't be opening a pull request for this for a while most likely.

reduz commented 6 years ago

I dont really find SCons to be slow, and its flexibility (which we take a ton of advantage of) is probably unmatched by anything else. SCons is also extremely proven, i find it that it never screws up a build.

Honestly, unless someone volunteers to rewrite the whole build system (not just a proof of concept), and shows that it's either simpler (a lot less code) or much faster, I would not even consider anything else, not even a bit.

vnen commented 6 years ago

Now, personally, I have to say I'm neither for nor against this, but I think there are more low-hanging fruit to chase instead of (or before?) changing the entire build system.

That's not really detrimental for this change. I mean, most of Godot contributors work on free time and work on what they want. For @NullConstant's case it seems is either "do this or do nothing" (not saying you don't want to fix things, just that it's harder to work with the actual C++ code base and bug hunt on something unfamiliar), so doing this is a good thing in general.

Also, the "there are better things to do" argument is the way to have nothing done IMO. Of course nothing will be merged until we are confident that it won't break stuff and won't be done close to a release. I agree there are more important stuff, but it doesn't mean this can't be started.

The thing is that someone working on this will likely have to do this alone. Of course specific questions can be answered by the core devs, but anymore than that won't happen. That's why SCons was never considered to be replaced: no one so far wanted to get the hands dirty. Like I said about Bullet before: someone has to get it to work and prove it's possible and better, otherwise we will stay with the status quo.

vnen commented 6 years ago

I do think SCons is slow though. Not much slow, but things like the progress status and the cache make it slower for rebuilds (using disk I/O to read from the cache makes it slow), so I disable those when rebuilding. It does take a bit to start doing stuff, after that it's quite fast.

Don't know if Meson will be significantly faster, but if someone is willing to try I'm not against it.

mhilbrunner commented 6 years ago

Oh sure, if someone takes the time to do it and shows that its an improvement big enough to warrant switching, I'm all for it :)

RiverMesa commented 6 years ago

Like I said, I'm not really equipped to deal with hunting C++ bugs or adding new features on that front, so I'm okay with doing this instead.

I suppose that this gives me (and/or anyone else working on this in the future) a pretty clean, two-fold goal to reach for:

And I suppose that SCons' speed is not terrible-terrible (the bulk of the time of the build process is from the actual linking and compiling anyway, so the slowdown from the overhead of the build system is likely somewhat negligible), but it certainly could be improved, and offering Meson as an alternative might just be that improvement. We'll see.

Zephilinox commented 6 years ago

The biggest issue is probably replacing the python build scripts for autogenerating code like the shaders, it doesn't seem like meson has a feature like that.

reduz commented 6 years ago

and offering Meson as an alternative might just be that improvement. We'll see.

It should not be an alternative, either we move to it if it's clearly better, or we don't even offer it. Having to support two build systems is crazy.

RiverMesa commented 6 years ago

Right, that's fair.

I'm gonna start hacking away at it over the next few days/weeks, see if this goes anywhere and whether it's an effort truly worth pursuing (even if I end up being the only person working on it that whole time).

I'll report back if I get anything presentable (like a benchmark).

fire commented 6 years ago

Although, not meson, I'll make a test in the coming weeks to see what it takes to build godot under bazel for windows 64 bit under msvc.

Bazel is another system whose advantage is a shorter build description and faster builds. Also it's maintained by Google so it's not going anywhere.

groud commented 6 years ago

Anyway, I can't verify that, but it's very likely that 99% of the computational time spend to build Godot is spent in the compiler. So even if the overhead of the building software was none, this would lead to a 1% decrease of the build time. Even a 5% decrease would not be worth, unless it allows less code and more reliability (or if I'm proven wrong :) ).

vnen commented 6 years ago

For raw build speed of a clean clone, I doubt it'll have any significant difference. But it can be significant if you consider rebuild speeds. SCons always rebuilds a lot of stuff that weren't changed when I pull a new commit (especially OpenSSL and Bullet, which are both big libraries and take a while to build).

As I said before, SCons is slow to start (takes a few seconds before really start compiling). If you're building the whole source that's negligible, but if you only changed a line and are compiling to test, it can significantly improve the workflow (which IMO is what most matter in the buildsystem: help the people who work with the code). So 5s decreased in a 10min build is irrelevant, but 5s decreased in a 10s rebuild is a major improvement.

But I agree that any change in this regard should be tested a lot to prove it works. As @reduz stated, SCons never messed up a build, and that's something to consider.

mhilbrunner commented 6 years ago

Bazel is another system whose advantage is a shorter build description and faster builds. Also it's maintained by Google so it's not going anywhere.

There is even a growing Wikipedia category of "things Google discontinued". :)

Geequlim commented 6 years ago

@mhilbrunner Nope it is still maintained https://github.com/bazelbuild/bazel

Zephilinox commented 6 years ago

Their point was that just because google made it, doesn't mean it'll be around for a long time.

rraallvv commented 6 years ago

Since we have embraced C# for game development, we might as well try something like Cake. I haven't tried it myself though, so I'm interested in what are your thoughts on that one. For one thing, at least, scripts wold run faster than with Python.

Calinou commented 6 years ago

Since we have embraced C# for game development, we might as well try something like Cake.

I'm not sure if requiring a Mono installation to compile Godot is a good idea, considering that Godot is mostly a C++ project and can be compiled without C# support.

The level of performance offered by Meson or CMake (with the Ninja generator) should be more than enough for Godot's purposes.

rraallvv commented 6 years ago

@Calinou you are probably right, although it seems everybody has jumped on the C# bandwagon, personally I'm using it for game scripting with Godot and productivity apps with Xamarin.Forms, also if I really want to squeeze performance I can try something like an IL2cpp converter or jump right into C++. I'm also thinking to use the mono/csharp REPL and ASP.net for things that I'd normally use JavaScript, Python or PHP for, like for instance web development related stuff or shell scripting. The thing is that I wouldn't mind to have a build system based on C# either, since I'd be already using it for pretty much everything. At least for my it would be the end of "dammit... another language I need to learn... why don't they stick with C++ and -insert decent dynamic/managed language here-"

Edit:

To clarify, requiring the Mono Framework to be installed in order to build Godot doesn't sound as bad to me, since the so called AAA game engines require way more disk space:

Besides, we could hardcode Mono support in the editor by default and have some logic to detect whether to support C# scripting or not at runtime, and ask for the Mono framework or the appropriate export templates.

isaachier commented 6 years ago

I love CMake, and meson looks cool, but I tend to agree with this article: http://www.rojtberg.net/1481/do-not-use-meson.

bruxisma commented 6 years ago

I'm very interested in doing the footwork to move Godot to Meson. The only thing currently stopping me from contributing to Godot (and using it) is Scons. I had to use it years ago, and I've never had a more painful incremental development experience in my entire life.

The article linked above has a lot of strawman arguments, arguing that CMake already exists and we should just use that. Meson's build files are extremely easy to read, and use a syntax close to that of Python. Conversion can be quick. What I'm posting in this thread for is, if the footwork was done, would a pull request even be considered for merging?

vnen commented 6 years ago

What I'm posting in this thread for is, if the footwork was done, would a pull request even be considered for merging?

It will be considered, but it has to be proven advantageous over SCons.

isaachier commented 6 years ago

Once you consider CMake just another build system and not the de facto standard, you end up with too many options. Why meson over SCons? Or bazel/buck are decent tools as well? But the biggest argument I have in favor of CMake is the tooling built to support that ecosystem. Clang compilation databases, support for packaging to multiple operating systems, etc. The only valid downsides I have heard is bad syntax and docs, but that isn't really strong enough for me to change my position on this.

bruxisma commented 6 years ago

Why meson over SCons?

Meson generates ninja files, which is much faster for incremental development. Additionally, on windows, you don't need to do any sort of visual studio path manipulation. Meson works out of the box. Using meson over scons would reduce a lot of the overhead of targeting windows, especially for those wanting to use visual studio (of which, I should note, meson can generate projects just as CMake)

Or bazel/buck are decent tools as well?

Given that bazel and buck are both written in Java (a dependency godot has only if you'll be targeting android, and even then one could technically use just the NDK), I would argue they're right out as a primary dependency for all users. Python is already in the dependency list, and you can easily install meson (and ninja) through a quick pip install meson

Clang compilation databases

Meson supports this by default, as this is a default feature built into ninja

support for packaging to multiple operating systems

Meson also has support for this, and relies heavily on pkg-config where appropriate

The only valid downsides I have heard is bad syntax and docs

Meson has a very python-like syntax, relying on variables, objects, and builtin functions. In my opinion, writing

my_meson_list = ['x', 'y', 'z']
message(my_meson_list)

vs

set(MY_CMAKE_LIST "x;y;z")
message(STATUS ${MY_CMAKE_LIST})

is a lot easier, especially given how cmake is sometimes case sensitive, and other times not. When you start to get into generator expressions, it can get extremely hairy. 

# In one file
set(USE_STD_CXX11 "-std=c++11")
set(USE_STDLIB_LIBCXX "-stdlib=libc++")

# Possibly elsewhere
set(LIBCXX $<BOOL:${CAN_USE_STDLIB_LIBCXX}>,$<BOOL:${BUILD_WITH_LIBCXX}>)
set(NO_RTTI $<BOOL:${CAN_USE_NO_RTTI}>,$<BOOL:${DISABLE_RTTI}>)

# Later on...
target_compile_options(my-exe
  PUBLIC
  $<$<BOOL:${CAN_USE_STD_CXX11}>:${USE_STD_CXX11}>
  $<$<AND:${LIBCXX}>:${USE_STDLIB_LIBCXX}>
  $<$<AND:${NO_RTTI}>:${USE_NO_RTTI}>
  # Oh yeah, you're gonna want more of these, because you can't trust compiler interfaces
)

# Generator expressions mean you can't follow the flow of the build to see what is called and defined where. This is a completely valid use of CMake.
check_cxx_compiler_flag(${USE_STDLIB_LIBCXX} CAN_USE_STDLIB_LIBCXX)
check_cxx_compiler_flag(${USE_NO_RTTI} CAN_USE_NO_RTTI)

The equivalent meson looks like:

cxx = meson.get_compiler('cpp')
# This is an extremely simplified approach. One can do a different option when dealing with MSVC and gcc support.
args = compiler.get_supported_arguments('-std=c++11', '-stdlib=libc++')
add_project_arguments(args)

I think I should note, by the way, that of all the C++ build systems out there, Meson is the least terrible. Arguments against its use tend to be a picture of a web comic, some hand waving about how CMake is the most used, so you should just deal with it, and "hey at least CMake isn't autotools".

But above all (and I think this is a really important thing, especially for those using godot as for gdnative), Meson supports precompiled headers natively. CMake does not (and using cotire to hack the build can cause more problems than its worth).

I cannot tell you how much this could speed up the build for gdnative users, but it will be a significant boost on gcc and msvc (at the very least, a performance increase on msvc is inevitable). Add in the use of extern template for the very little amounts of templated code found in godot, and you could see decent build time improvements, which are important for iterative and incremental builds.

Calinou commented 6 years ago

Meson generates ninja files, which is much faster for incremental development.

Note that CMake can also generate Ninja build files instead of traditional Makefiles; this is done by passing -G Ninja to its command line. It works quite well on most projects I've tried and is slightly faster overall (plus, it'll use all CPU threads by default).

Meson still wins in the raw performance comparisons I've seen, but this definitely helps close the gap between Meson and CMake performance.

bruxisma commented 6 years ago

Yes, however, CMake does not automatically install ninja for you, and it is not the default generator. Most of CMake's time is spent configuring the project, in my experience, and while recent versions of CMake have vastly improved performance, not everyone will have the latest CMake.

isaachier commented 6 years ago

OK that is quite thorough! I'm glad you have a good idea of the features you value in a build system. I still think there ought to be a standard build system in the C++ world that projects can rely on. I just wish meson were a CMake transpiler as opposed to its own thing.

bruxisma commented 6 years ago

I still think there ought to be a standard build system in the C++ world that projects can rely on.

Trust me, no one wants this more than me. I've become sort of a build system trash goblin and even gave a talk on them at CppCon. I unfortunately (or fortunately?) care a lot about build systems, almost to the point of obsession. :)

PJB3005 commented 6 years ago

I just want to say this, but every time I've did a long build of Godot on Windows, I've noted that Python consistently took >15% of my CPU. I doubt it's only slow to start.

fire commented 6 years ago

Who's going to make a small prototype godot engine built with meson? I never got to the point of creating a small test for bazel.

A 2d only build of godot should be sufficient to test with all the options disabled.

fire commented 6 years ago

The first roadblock I faced was core/make_binders.py. This is the autogenerated code that godot uses.

fire commented 6 years ago

I made an initial prototype in the build system Bazel but I'll discuss the setup on another issue. https://github.com/godotengine/godot/issues/18518

Don't have time over the week, but others can expand on the Bazel BUILD.

isaachier commented 6 years ago

Lol how did bazel get involved?

rraallvv commented 6 years ago

Since there might be a one time opportunity to switch to a different build system, shouldn't we looking for the best possible options? I for one wold be interested in seeing a Cake vs Bazel vs Scons comparison chart, although I have to admit I have zero experience with Cake to do the footwork for those tests.

isaachier commented 6 years ago

@rraallvv do you mean Cake or CMake?

rraallvv commented 6 years ago

@isaachier My bad, I was talking about www.cakebuild.net, I should have been more specific, sorry about that.

isaachier commented 6 years ago

Isn't that just for C#?

rraallvv commented 6 years ago

@isaachier I really don't know, all this time I was assuming Cake was capable of that, just like SCons being written in Python and having support for C++. Nevertheless, I just asked at their gitter chat, I'll re-post here the answer.

isaachier commented 6 years ago

@rraallvv for the record I think the idea of a side-by-side comparison of major build systems is a great idea. Here is what I found online that I think are fairly unbiased (not written by the build system developers): https://carlosvin.github.io/posts/choosing-modern-cpp-stack, https://www.reddit.com/r/cpp/comments/6euc7b/build_systems_bazel_buck/die6g1y/, https://stackoverflow.com/a/12022652/1930331.

My personal opinion:

I try pushing CMake whenever I get a chance because I want a de facto standard for C++ and CMake has the biggest "market-share" right now.

rraallvv commented 6 years ago

@isaachier Thanks for sharing that information, I'm pushing C# since there isn't a build system that supports scripting in C++, it might sound silly but I don't know why someone hasn't done that yet, for instance this C++ REPL could be used for a build system if one wants to go to the next level and end the debate on which one is better, faster, etc.

isaachier commented 6 years ago

I don't understand. C++ would probably be an awful build language. Slowness has more to do with the internal build system core AFAIK not with the details of the scripting language used.

rraallvv commented 6 years ago

@isaachier My guess is that SCons is so slow because it was written in Python, but if someone were to take the task of porting the entire build system to C++ and also make it JIT compile C++ scripts to run those builds, it would be much faster. Maybe the same could be true for every build system out there.

isaachier commented 6 years ago

It's a nice idea but meson is not considered slow by most accounts, yet it is written in Python.

bruxisma commented 6 years ago

I doubt meson will be a flavor of the week as the GNOME project is currently abandoning autotools in favor of meson.

Additionally, Scons is slow not because it is written in python, but because it also executes the build. It is a direct build system, like make or ninja, but also uses massive amounts of XML for definitions and a large amount of its code is actually just XSLT conversions.

Meson, on the other hand, executes its build scripts to generate a dependency tree, and then dumps this out to a ninja file. Meson is fast, because ninja is fast. Additionally, not everything is a string in Meson, like in CMake.

Lastly, I should note, there are several in development build systems where you will be able to script in C++ (I"m actually writing one of them). However, asking godot to move to a build system that doesn't exist yet (and worse, that I myself am writing) is a bit arrogant, in my not so humble opinion ;)

I also want to reiterate, under the current development workflow python is a necessity. So, too, is meson. And that makes moving to meson as simple as a call to python's pip tool.

akanewf commented 6 years ago

I completely agree @reduz that I dont really find SCons to be slow either.

As @slurps-mad-rips said:

Scons is slow not because it is written in python, but because it also executes the build

man scons:

scons supports building multiple targets in parallel via a -j option that takes, as its argument, the number of simultaneous tasks that may be spawned: scons -j 4

I have a CPU with 8c16t, so use "scons p=x11 -j 18" is very fast for me. It's 10x faster than use "scons p=x11" as the default setting "-j 1". Do someone have a single core CPU at year 2018? Let your CPU working fully. please give it a try.

bruxisma commented 6 years ago

Not everyone has these large CPUs, but more importantly, meson (well, ninja) is much faster at incremental recompilations. Scons, by default, performs an MD5 hash sum to see if a file has changed. Godot uses the MD5-timestamp setting, which means it will only hash a file if a timestamp has changed. It's still ridiculous.

isaachier commented 6 years ago

I guess the best way to prove what is better is to write a new one and see :)

Zireael07 commented 6 years ago

Scons isn't fast for many people, especially windows users running into problem with anything above -j 1.

isaachier commented 6 years ago

@slurps-mad-rips it depends on the project. ninja seems to hype its speed a little too much. See this article: http://david.rothlis.net/ninja-benchmark/. Seems counting source files is your best bet here. If <20K won't make much of a difference.