ERL-1446: Repeated Erlang/OTP compilations from different source directories don't produce identical build results

[OTP-Maintainer]

Original reporter: jeromedebretagne Affected version: OTP-23.1 Component: Not Specified Migrated from: https://bugs.erlang.org/browse/ERL-1446

---

+Steps to reproduce+

$ cd otp_src1
 $ git checkout OTP-23.1.5
 $ export ERL_COMPILER_OPTIONS=deterministic
 $ git clean -xfdq
 $ ./otp_build autoconf
 $ ./otp_build configure
 $ make -j4
 $ make RELEASE_ROOT=/path/to/build/directory1 release
  
 $ cd otp_src2
 $ git checkout OTP-23.1.5
 $ export ERL_COMPILER_OPTIONS=deterministic
 $ git clean -xfdq
 $ ./otp_build autoconf
 $ ./otp_build configure
 $ make -j4
 $ make RELEASE_ROOT=/path/to/build/directory2 release

Comparing the 2 build directory outputs show differences in more than 600 files, in 29 .erl files, in more than 500 .beam files and in many of the libraries and executables.

Cf. the attached output showing all the non-identical files.

  
 +Expected result+
  
 Builds should produce deterministic results.
 Cf. [https://reproducible-builds.org/] for why it matters.

[OTP-Maintainer]

jeromedebretagne said:

Hello,

There was no comment to confirm why this is considered a duplicate. To be explicit, the 2 issues are slightly different: the other one is covering the case of re-using the same source directory, this one is covering the case of using two different source directories.

This issue is technically a superset of the other one, but not the other way around, as there will be more assumptions to fix to fully close this case.

I've opened two distinct bugs to propose a step-by-step approach, the other issue should be smaller and easier to start with.

Cheers, Jérôme

[JeromeDeBretagne]

I'm re-attaching the original output: non_deterministic_builds_from_different_dirs.git-status.txt

[TD5]

Potentially the fix in #5141 will have some on effect on the non-determinism seen when moving between compilations in different directories, by virtue of respecting the deterministic option more reliably for ?FILE macro usage.

Other sources of non-determinism due to a dependency on the particular compilation directory path being used are:

• asn1ct_gen.erl, which can bake in the cwd, outdir, etc. via the options it records in the -asn1_info attribute.
• leex/yecc write out the absolute path to via -file attribute.

In these cases, unless there's some subtle dependency on these values (they don't look particularly likely to be depended on by some later build step to me), it might be useful to prevent them writing full paths when deterministic is set.

[JeromeDeBretagne]

Thanks @TD5 for the comment, indeed https://github.com/erlang/otp/pull/5141 should improve the situation and should land in OTP 24.1 it seems. I will try to update this issue to list what is still missing to get fully reproducible builds from difference directories.

[TD5]

In https://github.com/erlang/otp/pull/5965 I've tackled codegen and beam-file compilation producing artefacts which bake in absolute paths of the build machine. With that, you could build OTP in deterministic mode to get a sense of the improvement:

./otp_build configure --enable-deterministic-build
./otp_build boot -a
./otp_build release -a

The PR does not tackle builds of native code (my motivation was around more efficiently caching Dialyzer's analysis), and I am not sure how much of the non-determinism came from native code, but at least no .beam files seem to contain anything that's dependent of the absolute location of the otp/ checkout anymore when --enable-deterministic-build is set.

[JeromeDeBretagne]

Thanks @TD5 for your #5965 PR, we are getting closer and closer to fully deterministic builds!

Building the maint branch in a cross-compilation case, I can only detect absolute paths in native code now, and not in beam files anymore.

Here is the list of remaining files containing absolute paths after a build with --enable-deterministic-build:

grep: ./bin/erl_call: binary file matches
grep: ./bin/erlc: binary file matches
grep: ./bin/to_erl: binary file matches
grep: ./bin/ct_run: binary file matches
grep: ./bin/run_erl: binary file matches
grep: ./bin/dialyzer: binary file matches
grep: ./bin/typer: binary file matches
grep: ./bin/epmd: binary file matches
grep: ./bin/escript: binary file matches
grep: ./usr/lib/libei.a: binary file matches
grep: ./usr/lib/libei_st.a: binary file matches
grep: ./lib/megaco-4.4/priv/lib/megaco_flex_scanner_drv_mt.so: binary file matches
grep: ./lib/megaco-4.4/priv/lib/megaco_flex_scanner_drv.so: binary file matches
grep: ./lib/os_mon-2.7.1/priv/bin/memsup: binary file matches
grep: ./lib/os_mon-2.7.1/priv/bin/cpu_sup: binary file matches
grep: ./lib/crypto-5.1.1/priv/lib/otp_test_engine.so: binary file matches
grep: ./lib/crypto-5.1.1/priv/lib/crypto.so: binary file matches
grep: ./lib/asn1-5.0.19/priv/lib/asn1rt_nif.so: binary file matches
grep: ./lib/erl_interface-5.3/bin/erl_call: binary file matches
grep: ./lib/erl_interface-5.3/lib/libei.a: binary file matches
grep: ./lib/erl_interface-5.3/lib/libei_st.a: binary file matches
grep: ./lib/runtime_tools-1.19/priv/lib/trace_ip_drv.so: binary file matches
grep: ./lib/runtime_tools-1.19/priv/lib/dyntrace.so: binary file matches
grep: ./lib/runtime_tools-1.19/priv/lib/trace_file_drv.so: binary file matches
grep: ./erts-13.0.4/bin/inet_gethost: binary file matches
grep: ./erts-13.0.4/bin/dyn_erl: binary file matches
grep: ./erts-13.0.4/bin/erl_call: binary file matches
grep: ./erts-13.0.4/bin/erlexec: binary file matches
grep: ./erts-13.0.4/bin/erlc: binary file matches
grep: ./erts-13.0.4/bin/yielding_c_fun: binary file matches
grep: ./erts-13.0.4/bin/to_erl: binary file matches
grep: ./erts-13.0.4/bin/beam.smp: binary file matches
grep: ./erts-13.0.4/bin/ct_run: binary file matches
grep: ./erts-13.0.4/bin/run_erl: binary file matches
grep: ./erts-13.0.4/bin/dialyzer: binary file matches
grep: ./erts-13.0.4/bin/typer: binary file matches
grep: ./erts-13.0.4/bin/erl_child_setup: binary file matches
grep: ./erts-13.0.4/bin/heart: binary file matches
grep: ./erts-13.0.4/bin/epmd: binary file matches
grep: ./erts-13.0.4/bin/escript: binary file matches
grep: ./erts-13.0.4/lib/internal/libethread.a: binary file matches
grep: ./erts-13.0.4/lib/internal/liberts_internal.a: binary file matches
grep: ./erts-13.0.4/lib/internal/liberts_internal_r.a: binary file matches