search

landley / toybox

toybox
http://landley.net/toybox
BSD Zero Clause License
2.4k stars 335 forks source link

On incremental builds' reproducibility #502

Open garandria opened 4 months ago

garandria commented 4 months ago

The Toybox binary obtained by an incremental build may not be reproducible depending on some configuration options (while a traditional, clean build consistently leads to the same binary). It would be helpful to understand why this is occurring.

I'm describing and providing a minimal example hereafter.

Let us say that I have an initial config file 1.config that I have just built. Then, I want few more options in my Toybox binary so I run make menuconfig and I enable for instance CONFIG_CKSUM and CONFIG_DU which gives me 2.config. Instead of cleaning up the directory with a make distclean, I can just build 2.config on top of the artefact of 1.config assuming that Make will perform an incremental build and reuse artefacts.

For the example, I have done a clean build of 2.config to compare the build outputs and build time. First, we can notice that incremental build is indeed faster than clean build (see the graph below). In addition, the toybox binary obtained by a clean build is bit-by-bit identical to the toybox binary obtained with an incremental build (I am simply using GNU cmp to compare two binaries). So everything seems fine: (1) a clean build of 2.config is the same as an incremental build of 2.config; (2) we reduce build time.

However, If in addition to CONFIG_CKSUM and CONFIG_DU I enable CONFIG_TOYBOX_HELP and CONFIG_TOYBOX_HELP_DASHDASH (which gives me say 3.config), the build time is still faster with incremental build but the output is not reproducible anymore (i.e., the binary is not bit-by-bit identical). Some other experiments suggest that CONFIG_TOYBOX_HELP is causing specifically the issue.

I tried to dig a little bit into the binary and checked the output of the diffoscope but is not enough informative to understand the specific reasons why some options are causing the issue.

It has been demonstrated that configuration options can have an impact on the reproducibility of a build, as described in the paper Options Matter: Documenting and Fixing Non-Reproducible Builds in Highly-Configurable Systems for instance. However, for 2000 builds of Toybox's distinct configurations, all of them were reproducible when clean builds are performed.

So, it seems that the non-reproducibility of the builds here happens because of the incremental build. Do you have any insights on why is it happening?

I am building Toybox v0.8.5. The configuration files {1,2,3}.config as well as the output of the diffoscope for 3.config are available at https://gist.github.com/garandria/f8443b5fde20bf7fd1a66ca881721e46

Note: I could share other cases with problematic options causing non-reproducibility on some configurations. But this case has the merit of being simple for investigating the root causes.

landley commented 4 months ago

I've een working on reproducible build stuff for 20 years (https://landley.net/aboriginal/about.html and https://lists.j-core.org/pipermail/j-core/2016-August/000314.html and so on) but "we changed the config and did an incremental build" is not a use case I've particularly been targeting.

The top level makefile is mostly just a wrapper, the build is done by scripts/make.sh which should reuse artifacts (ala generated/obj/*.o). You can set V=1 to see what the build is doing (same as the kernel build). The "distclean" target deletes .config, you probably want the "clean" target (again, same as the kernel build).

The build checks which commands need to be rebuilt, but the toybox global symbols (CONFIGTOYBOX*) are not checked, and can theoretically vary if you don't do a clean build. The reason I don't check if they changed is I don't (reliably) have a copy of the previous build's .config to compare it with, and considered doing that "pilot error".

The 0.8.5 release was 3 years ago, I don't remember specifically what's changed since then. (Rather a lot, including a significant chunk of the build plumbing. I note that the https://github.com/landley/toybox/blob/master/scripts/recreate-prereq.sh and https://github.com/landley/toybox/tree/master/scripts/prereq work was fairly recent, ala http://lists.landley.net/pipermail/toybox-landley.net/2024-April/030271.html)