Improve godot-cpp build times by adding a way to exclude certain cpp files from compilation

[dementive]

Describe the project you are working on

A game using godot-cpp

Describe the problem or limitation you are having in your project

Not really a problem but I noticed that when building my gdextension a majority of the build time was being spent compiling cpp files that will never be used in my project.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

For example my game will never have XR, multiplayer, or use any of the visual shader nodes. Removing the cpp files from these unused features greatly improves build time for me since im able to exclude literally hundreds (375 in my project) of cpp files from ever getting compiled.

I tested and on my laptop the time taken for a clean build was halved from 2 minutes to 1 minute, on CI/CD where you don't have 20 cores like my laptop I imagine this improvement might be even bigger. Interestingly this also made my incremental builds faster as well, compiling a 1 character change to my register_types.cpp file (where all my gdextension headers are included) was reduced from 4-3.8 seconds to 3.3-3.1 seconds. This also makes the compilation database much smaller (10000 lines before down to 3000 lines) so a lot of the clang tooling is probably able to run faster as well (although I didn't test this).

The only drawback I found was that if you exclude cpp files from compilation that actually do need to be used at runtime the compiler doesn't warn you about it and you don't find out it's missing some classes until you run the editor and the linker fails. This is pretty easy to debug and fix though since the linker errors tell you exactly what classes are missing, so I just re added any of the cpp files I actually needed and it was no problem.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The solution I came up with totally sucks and is definitely a hack but the results I got from it were good so I feel like if something similar could be implemented that isn't totally jank it would be a nice enhancement. The editor has a similar feature that shows you what classes you don't use so you can compile godot without them and it autogenerates the command iirc, something like that might be nice.

All I did was add the following python code to the end of the get_file_list() function in bindings_generator.py

    excluded_file_prefixes = [
        "animated_",
        "animation",
        "csg_",
        "e_net_",
        "editor_export",
        "editor_resource",
        "editor_scene",
        "fbx_",
        "gltf_",
        "gpu_particles",
        "http_",
        "input_event_screen",
        "java_",
        "lightmap",
        "missing_",
        "multiplayer_",
        "navigation_",
        "ogg_",
        "open_xr",
        "packet_peer",
        "parallax",
        "physical_",
        "physics_direct_space",
        "physics_test",
        "placeholder_",
        "resource_importer_",
        "script",
        "skeleton",
        "stream_peer",
        "text_server",
        "tile_",
        "video_stream",
        "visual_shader_",
        "voxel_",
        "web_",
        "xr_",
    ]

    filtered_files = []
    for f in files:
        split_point = "gen/src/classes/"
        if split_point not in f:
            filtered_files.append(f)
            continue

        filename = f.split(split_point)[1]
        if any(filename.startswith(prefix) for prefix in excluded_file_prefixes):
            continue

        filtered_files.append(f)

    return filtered_files

Clearly this is an awful way to do this, no doubt there is a way to mess with the build system to make it a cleaner interface. An idea I had for this was to maybe expose a regex that gets run here so gdextension user's could add some regex like: (animated_|animation|etc...) and then godot-cpp in the bindings generator (or somewhere else that might make more sense) would run through this regex to exclude certain files from compilation. Idk what would be best, im certainly no Scons expert but I think adding something like this to godot-cpp would benefit a lot of people and would be nice to have so I don't have to use this super hacky solution in my project.

If this enhancement will not be used often, can it be worked around with a few lines of script?

N/A

Is there a reason why this should be core and not an add-on in the asset library?

N/A

[dsnopek]

Thanks!

This is already basically possible via scons build_profile=profile.json, except it's "inclusive" (you list which classes you use) rather than "exclusive" (listing classes that you don't want to use).

Is there a reason why that wouldn't work in your case? Or, did you just not know about that option? :-)

[dementive]

Didn't know it was possible already my bad haha, thanks! I see it here now https://github.com/godotengine/godot-cpp/blob/master/test/build_profile.json

Perhaps it would be nice to kinda explain what it does in the Scons help text, from the description it gave I totally had no idea what it was meant to do and I don't think anyone else does either unless they go through the bindings generator python file to find it. From quickly looking at other projects on GitHub I don't see a single one that is using this build profile json file even though it's probably something a lot of people would want to use I'd imagine. Just a comment like "use enabled_classes list to explicitly enable classes for compilation and disabled_classes list to explicitly remove certain classes from the build" would make it much more clear I'd say.

Also I'm wondering if it would be possible to generate 🤔 I saw Scons has cpp scanner classes that might be able to figure out what classes are used so it could get generated automatically kind of like the compilation database does. Ima spend some time trying to figure this out because explicitly enabling/disabling each class either way will require me to manually keep track of hundreds of classes. For plugins this probably isn't too relevant but with my game there is a ton of code churn so the json method alone could be a drag.

EDIT: I gave up, tried for about an hour to figure it out, accurately parsing C++ source files is impossible and reading scons documentation nearly made me lose my mind since most Scons API functions just have no description, are dynamically typed so it's a pain to even figure out what to pass it, and have no example usage anywhere. There is definitely a way to get scons to tell you the dependencies but I can't figure it out. Just gonna do it manually with the json file.

[dementive]

I played around with it a bit more and figured it out. Adding this code to my SConstruct file and then calling the create_build_profile() function before the SConscript function is called on the godot-cpp environment got it working :)

def extract_class_names():
    with open('godot-cpp/gdextension/extension_api.json', 'r') as file:
        classes = json.load(file).get('classes', [])

    return {cls.get('name') for cls in classes if cls.get('name')}

def search_in_cpp_files(class_names):
    found_classes = set()

    for root, _, files in os.walk('src'):
        for file in files:
            if file.endswith(('.cpp', '.hpp')):
                file_path = os.path.join(root, file)
                with open(file_path, 'r') as f:
                    content = f.read()
                    for class_name in class_names:
                        if content.count(class_name) > 0:
                            found_classes.add(class_name)

    return found_classes

def create_json_output(enabled_classes):
    output_data = {
        "enabled_classes": enabled_classes
    }
    output_path = 'build/build_profile.json'

    with open(output_path, 'w') as file:
        json.dump(output_data, file, indent=2)

def create_build_profile():
    class_names = extract_class_names()
    found_in_src = search_in_cpp_files(class_names)
    create_json_output(list(class_names & found_in_src))

This got my clean builds down to only 20 seconds and my incremental builds down to only 2.5 seconds which is pretty sweet, and it automatically updates the build_profile.json so I don't have to worry about maintaining it as I change my source code.