Scons performance for rebuilds is slow

RevoluPowered commented 4 years ago

Issue description: Rebuilding a single file on high-end systems takes half of a full rebuild time.

Steps to reproduce: time scons -j16 make change to some cpp file time scons -j16

I want to do some research into this, so gonna use this issue and document it.

RevoluPowered commented 4 years ago

jfons: it's computing a bunch of MD5 checksums that it shouldn't really need
11:13 PM for some reason the timestamp check is not doing its job

RevoluPowered commented 4 years ago

Normal MD5 timestamp

time scons -j16
[Initial build] scons: done building targets.
scons -j16  2283.42s user 180.40s system 1030% cpu 3:58.98 total

RevoluPowered commented 4 years ago

using Decider('timestamp-match') scons -j16 2345.50s user 184.19s system 1095% cpu 3:50.99 total

marstaik commented 4 years ago

scons -j1 --profile=sconsprofiling.log target=release_debug with zero changes sconsprofiling.log

ghost commented 4 years ago

i'm pretty sure that's just scons being crappy and you can't really do anything about it, beyond just, getting rid of scons

i mean we could improve scons but... i wish we could just get rid of it to be honest

i personally hate all build systems, i think they are mostly crap, i wish there was a "push a button and have it do all the hard work for you, without manually having to type ten thousand files by hand" thingy

people have claimed that meson would be faster, which i find a bit perplexing, since you'd think cmake would be a trllion times faster than everything else... but cmake just sucks donkey kong, plus the whole thing is just so tedious and boring to use, so either the people who make build systems are really incompetent, or i don't understand computers anymore

RevoluPowered commented 4 years ago

@RaTcHeT302 we're profiling and fixing the problems, we already got OSX build times down to 14s for single file. So please try to keep on topic for the actual fixing part, we aren't focusing on 'scons is crappy' we want to work out what the problem is and fix it.

Findings from research

I measured build times and am consistently getting everything under 30s for single file rebuilds (which is not bad):

you must disable windows defender (shaved 1m) off build times for full rebuild and got scons -j16 to be under 30s on windows.
we need to possibly swap to /DEBUG:fastlink (10s vs 15s)
linker commands with MSVC appear to be running out of order? (TBC, reproduction steps: delete exe and files in the bin/ dir and build with verbose=yes, look at the commands for linking they're weirdly ordered)
the actual commands to the compiler are slow not the scons invokation
the only slow part of scons is generating and not caching the -D defines from all the profiling files. (proven by loading scons profiling information from python and checking in RunSnakeRun)
we should be able to shave a lot of time off builds if we cache the defines.

Right now manually linking with the direct commands is yeilding these results, so our link times are 10 or 15 seconds without /DEBUG:fastlink

Changing MD5 decider to timestamp-match etc only shaves off 50ms

RevoluPowered commented 4 years ago

Here is an example of the out of order linking flags https://gist.github.com/RevoluPowered/dfc7c914eee09c94824b14c851519a58

RevoluPowered commented 4 years ago

Seems only assimp is the problem, i am testing linkage times without modules/assimp and thirdparty/assimp for now, might be a bug.

Calinou commented 4 years ago

@RaTcHeT302 Per the Godot Engine Code of Conduct, please stop derailing issues with unrelated conversations. This is your last warning.

Feedback is always welcome but keep your criticism constructive. We encourage you to open discussions, proposals, issues, and bug reports. Use the community platforms to discuss improvements, not to vent out frustration. Similarly, when other users offer you feedback please accept it gracefully.

(emphasis mine)

RevoluPowered commented 4 years ago

Here are some results from more tests with antivirus off in group policy.

# /INCREMENTAL /DEBUG:fastlink
C:\Users\Gordon\Projects\godot>timecmd.bat link /nologo /DEBUG:fastlink /INCREMENTAL /SUBSYSTEM:WINDOWS winmm.lib dsound.lib kernel32.lib ole32.lib oleaut32.lib user32.lib gdi32.lib IPHLPAPI.lib Shlwapi.lib wsock32.lib Ws2_32.lib shell32.lib advapi32.lib dinput8.lib dxguid.lib imm32.lib bcrypt.lib Avrt.lib dwmapi.lib cfgmgr32.lib opengl32.lib /STACK:8388608 /OUT:bin\godot.windows.tools.64.exe main\main.windows.tools.64.lib modules\modules.windows.tools.64.lib modules\module_xatlas_unwrap.windows.tools.64.lib modules\module_websocket.windows.tools.64.lib modules\module_webrtc.windows.tools.64.lib modules\module_webp.windows.tools.64.lib modules\module_webm.windows.tools.64.lib modules\module_vorbis.windows.tools.64.lib modules\module_visual_script.windows.tools.64.lib modules\module_vhacd.windows.tools.64.lib modules\module_upnp.windows.tools.64.lib modules\module_tinyexr.windows.tools.64.lib modules\module_theora.windows.tools.64.lib modules\module_tga.windows.tools.64.lib modules\module_svg.windows.tools.64.lib modules\module_stb_vorbis.windows.tools.64.lib modules\module_squish.windows.tools.64.lib modules\module_regex.windows.tools.64.lib modules\module_pvr.windows.tools.64.lib modules\module_opus.windows.tools.64.lib modules\module_opensimplex.windows.tools.64.lib modules\module_ogg.windows.tools.64.lib modules\module_mobile_vr.windows.tools.64.lib modules\module_mbedtls.windows.tools.64.lib modules\module_lightmapper_rd.windows.tools.64.lib modules\module_jsonrpc.windows.tools.64.lib modules\module_jpg.windows.tools.64.lib modules\module_hdr.windows.tools.64.lib modules\module_gridmap.windows.tools.64.lib modules\module_glslang.windows.tools.64.lib modules\module_gdscript.windows.tools.64.lib modules\module_gdnavigation.windows.tools.64.lib modules\module_gdnative.windows.tools.64.lib modules\module_freetype.windows.tools.64.lib modules\module_etc.windows.tools.64.lib modules\module_enet.windows.tools.64.lib modules\module_denoise.windows.tools.64.lib modules\module_dds.windows.tools.64.lib modules\module_cvtt.windows.tools.64.lib modules\module_csg.windows.tools.64.lib modules\module_camera.windows.tools.64.lib modules\module_bullet.windows.tools.64.lib modules\module_bmp.windows.tools.64.lib modules\module_basis_universal.windows.tools.64.lib modules\module_assimp.windows.tools.64.lib platform\platform.windows.tools.64.lib drivers\drivers.windows.tools.64.lib editor\editor.windows.tools.64.lib scene\scene.windows.tools.64.lib servers\servers.windows.tools.64.lib core\core.windows.tools.64.lib modules\freetype\freetype_builtin.windows.tools.64.lib platform\windows\godot_windows.windows.tools.64.obj platform\windows\crash_handler_windows.windows.tools.64.obj platform\windows\os_windows.windows.tools.64.obj platform\windows\display_server_windows.windows.tools.64.obj platform\windows\key_mapping_windows.windows.tools.64.obj platform\windows\joypad_windows.windows.tools.64.obj platform\windows\windows_terminal_logger.windows.tools.64.obj platform\windows\vulkan_context_win.windows.tools.64.obj platform\windows\context_gl_windows.windows.tools.64.obj platform\windows\godot_res.windows.tools.64.obj
   Creating library bin\godot.windows.tools.64.lib and object bin\godot.windows.tools.64.exp
command took 0:0:12.14 (12.14s total)
# /DEBUG (current default)
C:\Users\Gordon\Projects\godot>timecmd.bat link /nologo /DEBUG /SUBSYSTEM:WINDOWS winmm.lib dsound.lib kernel32.lib ole32.lib oleaut32.lib user32.lib gdi32.lib IPHLPAPI.lib Shlwapi.lib wsock32.lib Ws2_32.lib shell32.lib advapi32.lib dinput8.lib dxguid.lib imm32.lib bcrypt.lib Avrt.lib dwmapi.lib cfgmgr32.lib opengl32.lib /STACK:8388608 /OUT:bin\godot.windows.tools.64.exe main\main.windows.tools.64.lib modules\modules.windows.tools.64.lib modules\module_xatlas_unwrap.windows.tools.64.lib modules\module_websocket.windows.tools.64.lib modules\module_webrtc.windows.tools.64.lib modules\module_webp.windows.tools.64.lib modules\module_webm.windows.tools.64.lib modules\module_vorbis.windows.tools.64.lib modules\module_visual_script.windows.tools.64.lib modules\module_vhacd.windows.tools.64.lib modules\module_upnp.windows.tools.64.lib modules\module_tinyexr.windows.tools.64.lib modules\module_theora.windows.tools.64.lib modules\module_tga.windows.tools.64.lib modules\module_svg.windows.tools.64.lib modules\module_stb_vorbis.windows.tools.64.lib modules\module_squish.windows.tools.64.lib modules\module_regex.windows.tools.64.lib modules\module_pvr.windows.tools.64.lib modules\module_opus.windows.tools.64.lib modules\module_opensimplex.windows.tools.64.lib modules\module_ogg.windows.tools.64.lib modules\module_mobile_vr.windows.tools.64.lib modules\module_mbedtls.windows.tools.64.lib modules\module_lightmapper_rd.windows.tools.64.lib modules\module_jsonrpc.windows.tools.64.lib modules\module_jpg.windows.tools.64.lib modules\module_hdr.windows.tools.64.lib modules\module_gridmap.windows.tools.64.lib modules\module_glslang.windows.tools.64.lib modules\module_gdscript.windows.tools.64.lib modules\module_gdnavigation.windows.tools.64.lib modules\module_gdnative.windows.tools.64.lib modules\module_freetype.windows.tools.64.lib modules\module_etc.windows.tools.64.lib modules\module_enet.windows.tools.64.lib modules\module_denoise.windows.tools.64.lib modules\module_dds.windows.tools.64.lib modules\module_cvtt.windows.tools.64.lib modules\module_csg.windows.tools.64.lib modules\module_camera.windows.tools.64.lib modules\module_bullet.windows.tools.64.lib modules\module_bmp.windows.tools.64.lib modules\module_basis_universal.windows.tools.64.lib modules\module_assimp.windows.tools.64.lib platform\platform.windows.tools.64.lib drivers\drivers.windows.tools.64.lib editor\editor.windows.tools.64.lib scene\scene.windows.tools.64.lib servers\servers.windows.tools.64.lib core\core.windows.tools.64.lib modules\freetype\freetype_builtin.windows.tools.64.lib platform\windows\godot_windows.windows.tools.64.obj platform\windows\crash_handler_windows.windows.tools.64.obj platform\windows\os_windows.windows.tools.64.obj platform\windows\display_server_windows.windows.tools.64.obj platform\windows\key_mapping_windows.windows.tools.64.obj platform\windows\joypad_windows.windows.tools.64.obj platform\windows\windows_terminal_logger.windows.tools.64.obj platform\windows\vulkan_context_win.windows.tools.64.obj platform\windows\context_gl_windows.windows.tools.64.obj platform\windows\godot_res.windows.tools.64.obj
   Creating library bin\godot.windows.tools.64.lib and object bin\godot.windows.tools.64.exp
command took 0:0:17.39 (17.39s total)
# /DEBUG:fastlink (suggested default for debug)
:\Users\Gordon\Projects\godot>timecmd.bat link /nologo /DEBUG:fastlink /SUBSYSTEM:WINDOWS winmm.lib dsound.lib kernel32.lib ole32.lib oleaut32.lib user32.lib gdi32.lib IPHLPAPI.lib Shlwapi.lib wsock32.lib Ws2_32.lib shell32.lib advapi32.lib dinput8.lib dxguid.lib imm32.lib bcrypt.lib Avrt.lib dwmapi.lib cfgmgr32.lib opengl32.lib /STACK:8388608 /OUT:bin\godot.windows.tools.64.exe main\main.windows.tools.64.lib modules\modules.windows.tools.64.lib modules\module_xatlas_unwrap.windows.tools.64.lib modules\module_websocket.windows.tools.64.lib modules\module_webrtc.windows.tools.64.lib modules\module_webp.windows.tools.64.lib modules\module_webm.windows.tools.64.lib modules\module_vorbis.windows.tools.64.lib modules\module_visual_script.windows.tools.64.lib modules\module_vhacd.windows.tools.64.lib modules\module_upnp.windows.tools.64.lib modules\module_tinyexr.windows.tools.64.lib modules\module_theora.windows.tools.64.lib modules\module_tga.windows.tools.64.lib modules\module_svg.windows.tools.64.lib modules\module_stb_vorbis.windows.tools.64.lib modules\module_squish.windows.tools.64.lib modules\module_regex.windows.tools.64.lib modules\module_pvr.windows.tools.64.lib modules\module_opus.windows.tools.64.lib modules\module_opensimplex.windows.tools.64.lib modules\module_ogg.windows.tools.64.lib modules\module_mobile_vr.windows.tools.64.lib modules\module_mbedtls.windows.tools.64.lib modules\module_lightmapper_rd.windows.tools.64.lib modules\module_jsonrpc.windows.tools.64.lib modules\module_jpg.windows.tools.64.lib modules\module_hdr.windows.tools.64.lib modules\module_gridmap.windows.tools.64.lib modules\module_glslang.windows.tools.64.lib modules\module_gdscript.windows.tools.64.lib modules\module_gdnavigation.windows.tools.64.lib modules\module_gdnative.windows.tools.64.lib modules\module_freetype.windows.tools.64.lib modules\module_etc.windows.tools.64.lib modules\module_enet.windows.tools.64.lib modules\module_denoise.windows.tools.64.lib modules\module_dds.windows.tools.64.lib modules\module_cvtt.windows.tools.64.lib modules\module_csg.windows.tools.64.lib modules\module_camera.windows.tools.64.lib modules\module_bullet.windows.tools.64.lib modules\module_bmp.windows.tools.64.lib modules\module_basis_universal.windows.tools.64.lib modules\module_assimp.windows.tools.64.lib platform\platform.windows.tools.64.lib drivers\drivers.windows.tools.64.lib editor\editor.windows.tools.64.lib scene\scene.windows.tools.64.lib servers\servers.windows.tools.64.lib core\core.windows.tools.64.lib modules\freetype\freetype_builtin.windows.tools.64.lib platform\windows\godot_windows.windows.tools.64.obj platform\windows\crash_handler_windows.windows.tools.64.obj platform\windows\os_windows.windows.tools.64.obj platform\windows\display_server_windows.windows.tools.64.obj platform\windows\key_mapping_windows.windows.tools.64.obj platform\windows\joypad_windows.windows.tools.64.obj platform\windows\windows_terminal_logger.windows.tools.64.obj platform\windows\vulkan_context_win.windows.tools.64.obj platform\windows\context_gl_windows.windows.tools.64.obj platform\windows\godot_res.windows.tools.64.obj
   Creating library bin\godot.windows.tools.64.lib and object bin\godot.windows.tools.64.exp
command took 0:0:12.28 (12.28s total)

RevoluPowered commented 4 years ago

Testing in go_faster_mk2_testing

I have got single file rebuild time to: 13.092s with MSVC on Windows 10. 🎆

This CPU is a Ryzen 7 3700X for now it 'forces' /MT16 since I have 16 threads, later I will try and fix the warning in LINK.

$ time scons -j16
scons: Reading SConscript files ...
Automatically detected platform: windows
Configuring for Windows: target=debug, bits=default
Found MSVC version 14.2, arch amd64, bits=64
YASM is necessary for WebM SIMD optimizations.
WebM SIMD optimizations are disabled. Check if your CPU architecture, CPU bits or platform are supported!
Checking for C header file mntent.h... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
Compiling ==> core\image.cpp
image.cpp
progress_finish(["progress_finish"], [])
Linking Static Library ==> core\core.windows.tools.64.lib
Linking Program        ==> bin\godot.windows.tools.64.exe
LINK : warning LNK4044: unrecognized option '/MT16'; ignored
scons: done building targets.

real    0m13.092s
user    0m0.000s
sys     0m0.015s

Entire build time is down from 3m to 2m 15s.

$ time scons -j16
scons: Reading SConscript files ...
Automatically detected platform: windows
Configuring for Windows: target=debug, bits=default
Found MSVC version 14.2, arch amd64, bits=64
YASM is necessary for WebM SIMD optimizations.
WebM SIMD optimizations are disabled. Check if your CPU architecture, CPU bits or platform are supported!
Checking for C header file mntent.h... no
scons: done reading SConscript files.
scons: Building targets ...
Generating method binders.
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\canvas.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\canvas_occlusion.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\scene_high_end.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\giprobe_sdf.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\gi.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\giprobe.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\giprobe_debug.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_debug.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_debug_probes.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_direct_light.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_integrate.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_preprocess.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sky.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\volumetric_fog.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\bokeh_dof.glsl.gen.h"
Compiling ==> platform\windows\godot_windows.cpp
Compiling ==> platform\windows\crash_handler_windows.cpp
godot_windows.cpp
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\copy.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\copy_to_fb.glsl.gen.h"
crash_handler_windows.cpp
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\cubemap_downsampler.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\cubemap_filter.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\cubemap_roughness.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\cube_to_dp.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\luminance_reduce.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\resolve.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\roughness_limiter.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\screen_space_reflection.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\screen_space_reflection_filter.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\screen_space_reflection_scale.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\shadow_reduce.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\specular_merge.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\ssao.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\ssao_blur.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\ssao_minify.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\subsurface_scattering.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\tonemap.glsl.gen.h"
Compiling ==> platform\windows\key_mapping_windows.cpp
Compiling ==> platform\windows\joypad_windows.cpp
key_mapping_windows.cpp
Compiling ==> platform\windows\windows_terminal_logger.cpp
Compiling ==> platform\windows\vulkan_context_win.cpp
joypad_windows.cpp
....
pe1\type1.c
Compiling ==> thirdparty\freetype\src\type42\type42.c
type1.c
Compiling ==> thirdparty\freetype\src\winfonts\winfnt.c
type42.c
winfnt.c
Compiling ==> thirdparty\freetype\src\sfnt\sfnt.c
Building compilation database compile_commands.json
sfnt.c
progress_finish(["progress_finish"], [])
Linking Static Library ==> modules\freetype\freetype_builtin.windows.tools.64.lib
Linking Static Library ==> core\core.windows.tools.64.lib
Linking Program        ==> bin\godot.windows.tools.64.exe
LINK : warning LNK4044: unrecognized option '/MT16'; ignored
   Creating library bin\godot.windows.tools.64.lib and object bin\godot.windows.tools.64.exp
scons: done building targets.

real    2m16.477s
user    0m0.000s
sys     0m0.015s

LinuxUserGD commented 4 years ago

Does scons use ccache by default? Should make rebuilds faster too than just skipping already compiled files/targets I think.

Calinou commented 4 years ago

Does scons use ccache by default? Should make rebuilds faster too than just skipping already compiled files/targets I think.

No, but our SCons setup is configured to use ccache automatically as long as it's installed. You can run watch ccache -s while building Godot to confirm this.

suncaller commented 3 years ago

Adding another entry point in case it's useful. Incremental build with no change:

Windows 10, i5-6600k CPU, 3.50GHz, 16GB RAM

> scons target=debug -j16 --debug=time

[vcvarsall.bat] Environment initialized for: 'x64'
scons: Reading SConscript files ...
Automatically detected platform: windows
Configuring for Windows: target=debug, bits=default
Found MSVC compiler: amd64
Compiled program architecture will be a 64 bit executable (forcing bits=64).
...
scons: done reading SConscript files.
scons: Building targets ...
[ 99%] progress_finish(["progress_finish"], [])
[100%] Command execution time: progress_finish: 0.001998 seconds
[100%] Command execution time: .: 0.000000 seconds
scons: done building targets.
Total build time: 17.022759 seconds
Total SConscript file execution time: 6.119761 seconds
Total SCons execution time: 10.834999 seconds
Total command execution time: 0.067999 seconds

Calinou commented 3 years ago

Edit: This is now documented at https://github.com/godotengine/godot-docs/pull/5593.

For those on Linux, Pyston is worth a try. It's a JIT-enabled implementation of Python compatible with SCons.

If anyone wants to try this, download a release from https://github.com/pyston/pyston/releases, extract the portable .tar.gz and run ./pyston -m pip install scons to install SCons within Pyston. Then run the usr/bin/scons script that is located within the folder where you extracted Pyston while in the Godot root folder.

I've also tried Nuitka but it didn't provide any performance improvement after compiling SCons with it. PyPy isn't compatible with SCons and most likely won't be for a while (if ever).

To make calling SCons via Pyston easier, save this script as pyston-scons somewhere in your $PATH:

#!/bin/sh

# This assumes you installed Pyston to $HOME/.local/opt/pyston. Change if necessary.
"$HOME/.local/opt/pyston/usr/bin/scons" "$@"

Then make it executable by running chmod +x /path/to/pyston-scons.

Benchmark

CPU: Intel Core i7-6700K @ 4.4 GHz
SSD: Samsung 850 EVO (1 TB)
OS: Fedora 34

The CPython version used for comparison is 3.9.0 compiled with pyenv/pyenv-installer with optimizations enabled. I'm performing a null build (no files changed between builds) and 1 warmup run before each comparison:

❯ hyperfine -w1 "scons -j10 use_llvm=yes use_lld=yes werror=no tests=no" "~/Downloads/pyston_2.3_portable-v2/usr/bin/scons -j10 use_llvm=yes use_lld=yes werror=no tests=no"
Benchmark #1: scons -j10 use_llvm=yes use_lld=yes werror=no tests=no
  Time (mean ± σ):      7.261 s ±  0.061 s    [User: 7.007 s, System: 0.286 s]
  Range (min … max):    7.193 s …  7.410 s    10 runs

Benchmark #2: ~/Downloads/pyston_2.3_portable-v2/usr/bin/scons -j10 use_llvm=yes use_lld=yes werror=no tests=no
  Time (mean ± σ):      5.176 s ±  0.024 s    [User: 4.962 s, System: 0.250 s]
  Range (min … max):    5.132 s …  5.214 s    10 runs

Summary
  '~/Downloads/pyston_2.3_portable-v2/usr/bin/scons -j10 use_llvm=yes use_lld=yes werror=no tests=no' ran
    1.40 ± 0.01 times faster than 'scons -j10 use_llvm=yes use_lld=yes werror=no tests=no'

dmoody256 commented 3 years ago

I have a scons branch that builds godot with ninja: https://github.com/dmoody256/scons/tree/ninja_scons_daemon

(venv) Sep.09 04:16 ubuntu[godot]: nproc 
16
(venv) Sep.09 04:16 ubuntu[godot]: python ../scons/scripts/scons.py platform=linuxbsd
scons: Reading SConscript files ...
Enabling ALSA
Enabling PulseAudio
Enabling D-Bus
Enabling udev support
Note: Building a debug binary (which will run slowly). Use `target=release_debug` to build an optimized release binary.
Checking for C header file mntent.h... yes
scons: done reading SConscript files.
scons: Building targets ...
[Initial build] Generating: build.ninja
[Initial build] scons: done building targets.
[Time elapsed: 00:00:05.777]
(venv) Sep.09 04:17 ubuntu[godot]: time ninja 2>&1 > /dev/null

real    5m24.044s
user    64m57.609s
sys 4m58.330s
(venv) Sep.09 04:25 ubuntu[godot]: touch core/core_bind.cpp 
(venv) Sep.09 04:25 ubuntu[godot]: time ninja core/core_bind.linuxbsd.tools.64.o 
[2/2] Compiling core/core_bind.linuxbsd.tools.64.o

real    0m5.172s
user    0m5.294s
sys 0m0.266s
(venv) Sep.09 04:25 ubuntu[godot]: git diff
diff --git a/SConstruct b/SConstruct
index 8feb9e61bb..a51f76dbba 100644
--- a/SConstruct
+++ b/SConstruct
@@ -69,6 +69,11 @@ elif platform_arg == "javascript":
 # want to have to pull in manually.
 # Then we prepend PATH to make it take precedence, while preserving SCons' own entries.
 env_base = Environment(tools=custom_tools)
+
+env_base.SetOption('experimental', ['ninja'])
+env_base['NINJA_DISABLE_AUTO_RUN'] = True
+env_base.Tool('ninja')
+
 env_base.PrependENVPath("PATH", os.getenv("PATH"))
 env_base.PrependENVPath("PKG_CONFIG_PATH", os.getenv("PKG_CONFIG_PATH"))
 if "TERM" in os.environ:  # Used for colored output.

The tool from my scons branch has been localized as an isolated tool for use in godot in https://github.com/godotengine/godot/pull/52793

fire commented 2 years ago

@akien-mga Changes made to the build system made scons faster?

dmoody256 commented 2 years ago

just wanted to drop a note that ninja tool work I had been working on and testing integration with godot build previously has been merged into SCons master and is ready to be released in SCons 4.4.

dmoody256 commented 2 years ago

I just tested godot master with scons master and ninja build worked. However there is still an issue with source code being generated during SConscript reading phase. Source code should be generated during the building phase forming proper scons nodes. This will cause unnecessary rebuilds when using ninja because ninja is only looking mtime so the files will be regenerated because they happen during sconscript reading, and ninja will detect the mtime change. More details outlined in this comment: https://github.com/godotengine/godot/pull/52793#issuecomment-1034515410

akien-mga commented 1 year ago

We've done a number of improvements recently which significantly improved the rebuild times, notably:

57806
57820
63087
77170
80482
80612

For me on a Linux laptop, with the following command: scons -j7 p=linuxbsd dev_build=yes dev_mode=yes linker=mold scu_build=all, I get these metrics:

"Null" rebuild (no change): 4.8s Minor incremental rebuild (adding empty line in main.cpp): 12.4s

I'm using Pyston to speed up the SCons parsing a bit, and mold to speed up linking. That brings the SCons overhead to around 5s as shown above. The rest is Godot's normal build + link time.

We'll keep looking for ways to improve things further, but I think this issue can be considered resolved.

godotengine / godot

Scons performance for rebuilds is slow #41271

Findings from research

Benchmark

57806

57820

63087

77170

80482

80612