Automatic verification of reproducibility

[danielfullmer]

As of 2020-05-17, the target files for crosshatch and marlin with the vanilla flavor can be built bit-for-bit reproducibly. Additionally, the PR for LineageOS https://github.com/danielfullmer/robotnix/pull/9 also builds reproducible target files for marlin. This was verified using the --check option for nix-build.

It would be valuable to automatically create and publish a "reproducibility report" like r13y.com does for NixOS.

[mschwaig]

Does what you observed mean bit-for-bit reproducibly on the same machine or on a completely different one?

From what I've read about r13y.com on their website they produce the two builds on different hardware running different kernels (see current website source).

From what I have read they use --check as well but the only do the second build locally, since the first one should get substituted for the binary that Hydra built for cache.nixos.org.

[danielfullmer]

My previous reproducibility tests have been entirely on my own machine, so using the same kernel, filesystem, (probably) day of year, etc. Building again on another machine in a different environment would definitely help give some assurance that robotnix builds are actually reproducible.

[samueldr]

Ambient impurities to be aware of:

• filesystem the build runs on (tmpfs for /tmp, ext4, zfs, etc)
• CPU feature set for running CPU (less likely to be an issue for cross-compilation)
• anything under /proc and /dev
• the impure-ish /bin/sh from Nix's sandbox

[mschwaig]

I have done some work looking at the reproducibility of Robotnix.

Unfortunately I was not able to fully reproduce the target files even with the Robotnix code from around 2020-05-17.

I have opened https://github.com/danielfullmer/robotnix/issues/209 with my analysis based on current master.