Describe the problem or limitation you are having in your project
Engine binary size keeps steadily increasing over time, and we need to keep this under control to ensure the engine remains fast to download, particularly on mobile/web platforms.
We don't know in advance which PRs increase binary size the most unless someone goes and compiles both master and the PR rebased against master, with the same SCons options used for both builds (and both binaries stripped). This can lead to surprises when it's time to make a new release and notice the editor binary size grew by several MBs.
Godot Benchmarks tracks binary sizes for each build (not displayed in graphs yet), but this is only a daily build of master and it does not track pull requests. Pull request tracking is eventually planned, but it will be scoped to specific PRs on-demand as benchmarking every single PR would take too much time on the benchmarking server.
Describe the feature / enhancement and how it helps to overcome the problem or limitation
Compare pull requests' binary sizes with master on CI and display the difference in the output log.
This will compare binary sizes with the latest successful build of master on CI, which is assumed to use the same SCons options. Binaries are stripped from their debug symbols before comparing their sizes.
Compared to the benchmarking server, this will compare builds that use different SCons options (typically debug builds without LTO), so the exact differences won't be the same. This is still a useful metric as most of the time, a significant size increase in a debug build will translate to a significant size increase in a release build.
Question: Why are official builds not used as a reference?
Official builds are compiled with a different set of SCons options, with a different toolchain. This means we can't compare their size with the binaries built on CI, which use options tailored for fast builds (as opposed to low binary sizes).
Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams
Describe the project you are working on
The Godot editor :slightly_smiling_face:
Describe the problem or limitation you are having in your project
Engine binary size keeps steadily increasing over time, and we need to keep this under control to ensure the engine remains fast to download, particularly on mobile/web platforms.
We don't know in advance which PRs increase binary size the most unless someone goes and compiles both
masterand the PR rebased againstmaster, with the same SCons options used for both builds (and both binaries stripped). This can lead to surprises when it's time to make a new release and notice the editor binary size grew by several MBs.Godot Benchmarks tracks binary sizes for each build (not displayed in graphs yet), but this is only a daily build of
masterand it does not track pull requests. Pull request tracking is eventually planned, but it will be scoped to specific PRs on-demand as benchmarking every single PR would take too much time on the benchmarking server.Describe the feature / enhancement and how it helps to overcome the problem or limitation
Compare pull requests' binary sizes with
masteron CI and display the difference in the output log.This will compare binary sizes with the latest successful build of
masteron CI, which is assumed to use the same SCons options. Binaries are stripped from their debug symbols before comparing their sizes.Compared to the benchmarking server, this will compare builds that use different SCons options (typically debug builds without LTO), so the exact differences won't be the same. This is still a useful metric as most of the time, a significant size increase in a debug build will translate to a significant size increase in a release build.
Question: Why are official builds not used as a reference?
Official builds are compiled with a different set of SCons options, with a different toolchain. This means we can't compare their size with the binaries built on CI, which use options tailored for fast builds (as opposed to low binary sizes).
Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams
I made a proof-of-concept implementation which was completed by @fire: https://github.com/godotengine/godot/pull/91715
If this enhancement will not be used often, can it be worked around with a few lines of script?
No.
Is there a reason why this should be core and not an add-on in the asset library?
This is about the engine's continuous integration process.