Consideration of using unity build system for Godot

[msmshazan]

Consideration of using unity build system for compiling Godot engine. The unity build system compiles all source code as a single translation unity and therefore might decrease compile times significantly. More info : http://buffered.io/posts/the-magic-of-unity-builds/ Also if you want modularity you can compile chunks of translation units like one for sound ,UI , render, editor .. etc. The main focus of doing this is to reduce compile times and increase productivity of developers.

[bojidar-bg]

Not to be confused with the game engine of the same name, to which it seems to be completely unrelated.

[RoyBerube]

It would have to still involve SCons due to the amount of logic required for support of multiple platforms with one build system. I don't think it is impossible, just that it would take a lot of work. All of the SCons files would need work.

[mhilbrunner]

To quote the docs: Godot uses Scons to build. We love it, we are not changing it for anything else.

And it seems Unity builds (or really: one big header file including everything) would only be useful as an addition, not a replacement. But still: Is this really a pain point for people?

Honest question, for me the engine compiles quite fast (or at least, it never bothered me). Maybe I'm too used to UE4's compile times.

[rraallvv]

I'm trying out some things for this and the build time is reduced by about 30%, Here are some tests. The environment variable UNITY_BUILD is used to set the build type. Windows is still failing because the header windows.h pollutes the master files with macros like CreateDialog and FILE_OPEN and I didn't bother with those. macOS and iOS are backlogged.

For unity builds to work there are some things that need to be refactored first:

Many third party libraries and modules don't have include guards like this:

#ifndef MY_HEADER_H
#define MY_HEADER_H
...
#endif //MY_HEADER_H

So, I needed to have a blacklist with the sources that won't added to the master files.

There are declarations of the same static functions in multiple places, like this one:

static bool _is_number(CharType c) {
    return (c >= '0' && c <= '9');
}

For those I needed to move them to a single header with include guards, like #include core/char_utils.h for instance.

In 2.1 the same enum is defined for both the Godot 3 exporter and the resource binary formater:

enum {

    //numbering must be different from variant, in case new variant types are added (variant must be always contiguous for jumptable optimization)
    VARIANT_NIL = 1,
    VARIANT_BOOL = 2,
    VARIANT_INT = 3,
    VARIANT_REAL = 4,
    VARIANT_STRING = 5,
    VARIANT_VECTOR2 = 10,
        ...

So it needs a namespace or enum class or something like that.

I think those are minor changes, and the thing about include guards, and avoiding redefinition of static functions everywhere is probably a good thing to have anyway.

Also, since SCons doesn't support unity builds the scripts need to be changed to store all the sources for each target in lists instead of calling directly env.Object, env.Library, env.SharedLibrary and env.Program. Then in a final stage each list should be checked to see which sources are blacklisted, etc. Then the rest can be added to the master files like so:

drives/master.gen.cpp

#include "drivers/alsa/audio_driver_alsa.cpp"
#include "drivers/gl_context/context_gl.cpp"
#include "drivers/gles2/rasterizer_gles2.cpp"
...

drives/master_2.gen.cpp drives/master_3.gen.cpp drives/master_4.gen.c drives/master_5.gen.c

[lawnjelly]

Late to the party here, but I had a little go yesterday afternoon at converting some of the latest master to use unity build / single compilation unit techniques. I just did a little low hanging fruit to assess how difficult it would be, what kind of differences we could expect. My method was a little different to rraallvv's suggestion as I only found this thread after investigating this myself independently.

Method

First of all I wrote a batch file to put all the cpp files in a folder into a list of #includes for a single compilation unit. My linux batch-fu isn't very good and the ignore seems broken but it was something as simple as this:

#!/bin/bash
ls *.cpp --ignore="SCU.cpp" > output.txt
echo "// Single compilation unit" > SCU.cpp
cat output.txt | while read LINE; do echo "#include \"${LINE}\"" >> SCU.cpp; done

And there is a recursive version which searches subfolders too:

#!/bin/bash
ls *.cpp */*.cpp */*/*.cpp --ignore="SCU.cpp" > output.txt
echo "// Single compilation unit" > SCU.cpp
cat output.txt | while read LINE; do echo "#include \"${LINE}\"" >> SCU.cpp; done

However in practice I mostly used the former.

Most of the SCsub files for scons contain a line such as this:

env.add_source_files(env.main_sources, "*.cpp")

Adding all the cpp files in the folder to the build. So in order to get a unity build, essentially all we have to do is run the batch file to generate a single compilation file (I've named SCU.cpp but it could be anything). E.g.

// Single compilation unit
#include "default_controller_mappings.gen.cpp"
#include "input_default.cpp"
#include "main.cpp"
#include "main_timer_sync.cpp"
#include "performance.cpp"

and then change the line in the SCsub file from *.cpp to SCU.cpp. There is I'm sure a way of changing the source files in the SCsub automatically according to a user switch - I know next to nothing about scons but just note that it is possible very easily to have both a normal build and unity build available.

Changes

Depending on how 'strict' the codebase is, particularly concerning the global namespace determines how easy it is to get things to work. Here's a couple of links which mention some of the issues:

http://onqtam.com/programming/2018-07-07-unity-builds/ https://medium.com/@Jacob_Bell/compile-times-of-single-translation-unit-builds-4f3223b7f031

Essentially some ye olde and hacky techniques which pollute the global namespace can lead to conflicts. Luckily these were rare in most of the sections of Godot I got working.

The most common problem was use of a static 'global' function within the cpp to localize the use to the particular file. This pollutes the global namespace but you can normally get away with it because it only pollutes the compilation unit.. except when you start using unity builds.

2 easy example ways to fix this:

1) Exclude that particular file from the unity build (just a temporary solution) 2) Change the static function to a private member function

As an aside this also showed up a code smell .. in some files particularly in the visual script module certain static functions were repeated copy and paste in several files. It would seem to make more sense in this case to have one version of the function and reuse it. However this is just a side point, not necessary for the purposes of the unity build.

Another issue that can occur is with #defines. You occasionally get more than 1 cpp in a SCU file trying to define the same symbols. The compiler will complain and it is normally pretty easy to figure out. This is something that care should be taken over though because of the potential for subtle bugs, particularly if you maintain both a normal and unity build. A solution could be something as simple as systematically #undef the symbols at the bottom of cpp files that define them.

Results

Some of the folders I managed to convert fine so far:

main/
scene/
servers/
editor/

and some other misc folders like bullet. Note that I could have converted more but wanted to limit how much time I spent on it, this was enough for some comparisons.

Full rebuild times

Normal: 6 mins 35 Unity: 3 mins 05

Touching engine.h (included in a lot of places)

Normal: 2 mins 57 Unity: 2 mins 06

Incremental build times (touching one cpp file)

Normal: 0 mins 22 Unity: 0 mins 17

Binary size (godot.x11.tools.64)

Normal: 278 megs Unity: 212 megs

Discussion

The conversion was actually rather simple. I spent the largest amount of time googling how to write a batch file. Fixing any compilation errors was not very involved, about half the folders that I tried worked straight away with no changes.

From only changing maybe a third(?) of the folders we already have a halving of full build times. At a rough guesstimate I think we could probably get a full rebuild down to something like a minute with nearly everything done as unity build. Full rebuilds are not quite as important as the other two incremental timings in terms of developer iteration, but are certainly very important in terms of compiling different branches, and the travis etc builds (should they be done with unity build), and 6x quicker is not a bad improvement.

Amazingly even touching a single cpp file incremental build seemed to go quicker with the unity build. Sometimes this can be slightly slower when converting a unity build. This may have been faster because the linking stage was easier.

Finally the binary size was smaller. I haven't thoroughly tested this yet, and this may be purely debugging symbols, but any decrease we can get in the binary size for the engine would be welcome, especially on e.g. mobile.

Speedwise, using a unity build means you get link time optimization essentially for free, afaik. So I would expect the binary to be, if anything, faster at runtime than a normal build.

Some downsides

1) The build process might use more RAM. This is usually not a concern, unless e.g. compiling on a raspberry PI or something (in which case you could use a normal build?) 2) Developers either need to automate or manually add / remove cpps from the build. 3) Possible subtle issues with the order of compilation (consider for example multiple #defines of the same symbol). Developers have to be less sloppy with global namespace. 4) Once a unity build is used, developers will start relying on #includes from earlier cpp files. They shouldn't, but typically the only way a developer knows an include is needed is through a compile error. This means that order of #includes in a unity build may become important.

In my experience 3 and 4 are the biggest considerations.

There is also the issue of if you incorporate unity build techniques, whether to maintain a 'normal' build and a unity build. If you do decide to offer both you have the problem of ensuring PRs do not break either build. This is doable, and some shops do this, however my personal preference is to only maintain a unity build where possible, as it is far simpler.

Should we start incorporating unity build techniques?

In my experience this can be quite a revolutionary change. There are some downsides, but if you can e.g. half the iteration time, you can theoretically double the productivity of each developer. A developer sitting around waiting for a compilation to complete is a wasted developer. Not only that, they are likely to lose their train of thought and go off and do something else if changing a header file results in a 3 minute recompilation.

There may well be naysayers who have not used unity builds before. I would suggest therefore that it might be an idea to have a gradual introduction of these techniques, to test the water.

An example is the bullet source code included in third party. They actually INCLUDE a unity build for bullet in the source code, but we don't use it.

If you look in thirdparty/bullet you will see:

btBulletCollisionAll.cpp
btBulletDynamicsAll.cpp
btBulletLinearMathAll.cpp

These are, tada, single compilation unit files. So the first thing we should probably do is be using these.

I think there is also a good argument for creating SCU builds for many / most of the third party source code to get us started. This will give us some increase in full rebuild speeds with very few downsides.

Anyway these are just my thoughts on the subject, for further discussion.

If anyone wants to try this for themselves, they can download and compile my test fork from yesterday's master:

https://github.com/lawnjelly/godot/tree/scu

[rraallvv]

@lawnjelly since the build cache was improved builds are considerable faster now and work for collaborative environments too. Unity builds might have a detrimental effect for incremental builds and collaborative environments, but I still think those would be a nice addition for solo developers and could bring other benefits like smaller binary files. Unfortunately I don't see that happening anytime soon.

[lawnjelly]

@lawnjelly since the build cache was improved builds are considerable faster now and work for collaborative environments too.

I've only been compiling the source for a few short weeks, and I'm sure improvements have been made in the past (scons seems very convenient and adaptable), but surely that shouldn't be an argument against making the build faster still, if this can be done relatively painlessly? And the collaborative builds sound interesting, are there any docs to explain how to do these (am I missing out, is everyone else using these :flushed: )?

Unity builds might have a detrimental effect for incremental builds and collaborative environments

I suspect we would need empirical testing for this (like most optimization), I'm always finding the results can be quite counter-intuitive. For instance the tests I've done so far with incremental building are actually faster with unity build (the linking becomes a greater proportion of the time taken (link optimization is also an exciting area)). Maybe try the fork I linked and we can see if it is faster for you?

And I certainly don't want to come across all 'we must convert everything to use unity builds', I'm not trying to push that - there are pros and cons in different situations .. the potential pitfalls of symbol conflicts might be too much of a risk in some areas, we don't want to create bugs. It is interesting to try different approaches though and it can be fun researching this kind of thing. :smile:

If nothing else, it might be beneficial to build some of the third party libraries in this way (if they are compatible).

For myself I found that the build times could be a bottleneck, particularly when working on things like android builds, where I had to build repeatedly for different chipsets according to the test devices. Certainly no one goes to their grave wishing they'd spent longer in front of their PC waiting for it to compile. :grin:

[rraallvv]

@lawnjelly I haven't been active in the community lately, but if I remember correctly there are a few environment variables that have to be set in order to enable the build cache and builds from different participants in a collaborative environment have to be queued so that two builds don't modify the cache at the same time, you might want to take a look at some of the scripts in this repository https://github.com/GodotBuilder/godot-builds/tree/master/scripts

[lawnjelly]

I've done a bit more work on this in an experimental build. I've now got the main first party Godot code compiling very quickly, however aside from bullet and assimp, quite a bit of the thirdparty code is messy c, and has a lot of static global namespace violations, so I have mostly given up on converting the rest of third party.

Results

As a result the full rebuild time is mainly bottlenecked by thirdparty, however I've had continual improvements in the other incremental builds (timings are mins : seconds) :

Full Rebuild Normal 6:35 Unity 2:40

Touch engine.h Normal 2:57 Unity 0:35

Touch engine.cpp Normal 0:22 Unity 0:06

_File sizes (x11, tools, debugrelease, fully stripped) Normal 62.7 megs Unity 61.4 megs

So roughly speaking if you are developing on a compile / change / test / compile cycle you might expect a 4-6x speedup, and full builds (for example for different platforms) should be more than 2x speed.

Aims

Basically my aims have been to allow unity builds (especially for devs), but to have effectively zero impact on everything else. At the moment I have added a switch 'unity' to the Scons parameters that can be set to True.

For everything to do with the unity build, I have currently isolated it into one directory: misc/scu (The location and naming are just something which seemed sensible, I'm all open to ideas. I should really standardize on whether to use the term 'scu' (single compilation unit) or 'unity' (confusion with unity engine!).)

In the affected SCsub files there is a simple if .. else:

if env['unity']:
    env.add_source_files(env.main_sources, "#misc/scu/SCU_main.cc")
    env.Depends("#misc/scu/SCU_main.cc", "#main/app_icon.gen.h")
    env.Depends("#misc/scu/SCU_main.cc", "#main/splash.gen.h")
    env.Depends("#misc/scu/SCU_main.cc", "#main/splash_editor.gen.h")
else:
    env.add_source_files(env.main_sources, "*.cpp")

In fact I'll try and move all the logic in the unity section out into the misc/scu folder, using an 'include' or whatever the python-esque equivalent is, that way the SCsub files should never need to be touched after initial setup.

In the misc/scu directory you start with one file, SCU_Build.py. This will automatically create (in the blink of an eye) unity build files for every SCsub file we are interested in. At the moment I just run this manually, but it could be added to the main build process if desired.

The SCsub files add the relevant SCU_~~~~.cc source file instead of the .cpps, and with luck, it all just works (TM).

Some problems I had to overcome

Build order

One nasty problem I had to work around yesterday is that the unity builds exposes a bug whereby sometimes the cpp files are compiled by Scons BEFORE the generated files (gen.h) have been created. This is either similar or the same issue as #5042 .

I figured out that this was caused by Scons not having the proper dependency information, and therefore the build order being incorrect. So the solution (at least in my case) was to manually specify the dependencies for the unity files, such that the gen.h files were generated first.

E.g.:

    env.add_source_files(env.main_sources, "#misc/scu/SCU_main.cc")
    env.Depends("#misc/scu/SCU_main.cc", "#main/app_icon.gen.h")
    env.Depends("#misc/scu/SCU_main.cc", "#main/splash.gen.h")
    env.Depends("#misc/scu/SCU_main.cc", "#main/splash_editor.gen.h")

This tells Scons that it must generate app_icon, splash, and splash_editor BEFORE it tries to compile SCU_main.cc.

Excluding files from the unity build

On occasions, particular files or folders caused problems with the unity build. The simple solution was to exclude them. Inside the SCU_Build.py auto-generator it was easy to put files into an 'ignore list', and then they could be added manually after the unity build file to Scons.

_Here is an excerpt from SCUBuild.py:

process("scene/2d/", "cpp", "scene_2d.cc")
process("scene/3d/", "cpp", "scene_3d.cc")
process("scene/animation/", "cpp", "scene_animation.cc")
process_ignore("scene/gui/", "cpp", "scene_gui.cc", "line_edit.cpp")
process("scene/main/", "cpp", "scene_main.cc")
process("scene/resources/", "cpp", "scene_resources.cc")

The arguments are the source folder, the file type we are interested in, and the output file name. With process_ignore the extra argument is a file to ignore from the list.

core/make_binders.py

This file seemed to be doing some 'magic', and one of the generated files needed a #pragma once because of a multiple inclusion problem, but only on the unity build. I need to investigate this a little more for the best solution.

Integration

I brought up the unity build in #godotengine-devel on irc the other day to discuss, and mostly the responses were positive, in particular Akien was open to the idea. My view is that if it can all be added painlessly in (almost entirely) a separate folder from everything else, can be totally switched off, and does not complicate / affect the current build, it is well worth trying, considering the benefits.

As I say my aims are to produce something which is totally reversible, with next to zero impact on everything else in the build, which can be simply activated by developers by adding a 'unity' flag to the scons parameters. If we decide in practice it is more hassle than it is worth, we can simply remove the folder from git and revert the small SCsub differences.

The advantage of maintaining everything to do with the unity build in one folder, is that if any changes in git that are needed to maintain the unity build (which will happen occasionally, for instance changes to generated files) they can be done in one place isolated from the rest of Godot, and the policy on changing these files can potentially be a lot more lax than with standard PRs, as they don't affect the normal build. I'm a beginner with git, but if we can get the main build compiling completely fine with the unity folder as optional, it could even potentially be maintained off the main git repository?

So once I have a good solution for the couple remaining issues I will remake the changes as a potential PR. So please let me know any preferences for naming (scu versus unity) and location for the folder.

Unofficial version

It has also occurred to me that initially for a test I can make a totally separate system, with a different repository, with the unity folder, and add a python script which makes the necessary modifications to your SCsub files automatically. This will allow any of us devs to try it out (as long as you remember not to check in your modified SCsub files to git afterwards in a PR lol! :smile: ).

In fact thinking about it, maybe we would be better (at least for now) implementing this as a separate repository that links to the main build. Watch this space! :+1:

[msmshazan]

As for naming preferences Google calls it jumbo builds https://chromium.googlesource.com/chromium/src/+/lkgr/docs/jumbo.md

[lawnjelly]

OK, after a day slaving around writing a patching system, my unofficial version seems touch wood to be working, and is ready for testing by anyone foolhardy enough: https://github.com/lawnjelly/godot_SCU

Note that it is only so far aligned to the current master (or a recent-ish version). If it proves useful we can easily make modifications for the other forks.

The install instructions are in the repository. In short:

1) Create a folder as a sibling to your godot source code, and name it godot_SCU. 2) Then clone / download the repository in there, and run GO_PatchGodot.py. 3) Now just add unity=True to your Scons command line for building. 4) Profit!

Important

The script patches (modifies) many of the build files. As such it is essential you try it (at least until it is stable, and you are familiar with how it works) on a temporary version of the godot source and not on your current build. You have been warned! :smile:

Incidentally, if you want to compare compilation timings between normal / unity builds, for a rebuild all, the following might kind of work: 1) Run Scons -c 2) Delete all the .o object files in the godot_SCU folder 3) Build as normal

I've been meaning to figure out how to get Scons to delete the object files as part of the clean. It probably needs an SCsub file in godot_SCU, something like that.

[lawnjelly]

On the suggestion of @bdbaddog I changed the includes in the unity build files from using macros to being straight paths, as it seems scons can't deal with the macros. This is the reason why I was having to put in explicit dependencies to the generated files (gen.h).

Old:

// Single Compilation Unit
#define SCU_IDENT(x) x
#define SCU_XSTR(x) #x
#define SCU_STR(x) SCU_XSTR(x)
#define SCU_PATH(x,y) SCU_STR(SCU_IDENT(x)SCU_IDENT(y))
#define SCU_DIR main/

#include SCU_PATH(SCU_DIR,input_default.cpp)
#include SCU_PATH(SCU_DIR,main.cpp)
#include SCU_PATH(SCU_DIR,main_timer_sync.cpp)
#include SCU_PATH(SCU_DIR,performance.cpp)

New:

// Single Compilation Unit
#include "main/input_default.cpp"
#include "main/main.cpp"
#include "main/main_timer_sync.cpp"
#include "main/performance.cpp"

While it did have the desired effect of no longer needing the explicit env.Depends call, it had an unexpected side effect of slowing the build from 6 seconds to 28 seconds for a touched .cpp file :open_mouth: !

For this reason I have reverted to the macro approach. Perhaps scons parsing the source for dependencies is actually a bottleneck in such scenarios? Maybe it is only an issue in unity builds. It will be interesting to do some profiling on this versus the normal build if I have time.

[bdbaddog]

That's to be expected. SCons can actually find the files and then finds files those files include, etc. Without it you're only depending on the first level include files, which means you WON'T rebuild when any files those include files include are changed. Note that we have some performance improvements in SCons 3.1.1 and some further coming in the next release which should help a bit this this type of issue.

It's entirely possible this is only with Unity builds if the header files include lots of other header files. For correctness you really should keep the "New" version.

[lawnjelly]

Without it you're only depending on the first level include files, which means you WON'T rebuild when any files those include files include are changed.

For correctness you really should keep the "New" version.

Arg you are absolutely right. This is wiping out a lot of the lead on the incremental builds (and indeed the touch of a single cpp is slower).

Figures now are:

Full Rebuild Normal 6:35 Unity 2:40 Touch engine.h Normal 2:57 Unity 2:00 Touch engine.cpp Normal 0:22 Unity 0:28

This makes things a lot less clearcut, considering the pitfalls. :slightly_frowning_face:

I'll continue to do a bit more work on this and see if I can increase the margin. Maybe some more traditional other techniques would be needed to further decrease the build times (forward declaration, more modular linking etc), however that kind of thing is harder to retrofit.

[bdbaddog]

@lawnjelly - What version of SCons are you currently running?

[lawnjelly]

2.4.1 it says using scons -version. Probably far out of date, good point, if I'm trying to time these things. I'm presuming it's just the one provided on my linux mint, I'll try out the most recent version tomorrow. :+1:

[bdbaddog]

@lawnjelly - That's ancient.

RELEASE 2.4.1 - Mon, 07 Nov 2015 10:37:21 -0700

Try setting up a virtualenv and install "pip install scons". 3.1.1 is the newest. There have been many improvements since 2.4.1

[Anutrix]

I've been meaning to figure out how to get Scons to delete the object files as part of the clean.

A simple hack for this that I use is git clean -fx. (Caution note for new users: Try git clean -fnx to check what it'll delete or modify before running above). The best way for a clean build.

[akien-mga]

I've been meaning to figure out how to get Scons to delete the object files as part of the clean.

A simple hack for this that I use is git clean -fx.

Or scons -c, but you need to use the same build options as for your builds so that scons knows which build objects you want removed (e.g. scons p=x11 -c if you build with scons p=x11).

[Calinou]

Note: Before opening a proposal, see the discussion above for possible technical difficulties.

---

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.

The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.

If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!