jfsutils: not reproducible

raboof commented 9 months ago

Building this package multiple times does not yield bit-by-bit identical results, complicating the detection of Continuous Integration (CI) breaches. For more information on this issue, visit reproducible-builds.org.

Fixing bit-by-bit reproducibility also has additional advantages, such as avoiding hard-to-reproduce bugs, making content-addressed storage more effective and reducing rebuilds in such systems.

Steps To Reproduce

1. Build the package

This step will build the package. Specific arguments are passed to the command to keep the build artifacts so we can compare them in case of differences.

Execute the following command:

nix-build '<nixpkgs>' -A jfsutils && nix-build '<nixpkgs>' -A jfsutils --check --keep-failed

Or using the new command line style:

nix build nixpkgs#jfsutils && nix build nixpkgs#jfsutils --rebuild --keep-failed

2. Compare the build artifacts

If the previous command completes successfully, no differences were found and there's nothing to do, builds are reproducible. If it terminates with the error message error: derivation '<X>' may not be deterministic: output '<Y>' differs from '<Z>', use diffoscope to investigate the discrepancies between the two build outputs. You may need to add the --exclude-directory-metadata recursive option to ignore files and directories metadata (e.g. timestamp) differences.

nix run nixpkgs#diffoscopeMinimal -- --exclude-directory-metadata recursive <Y> <Z>

3. Examine the build log

To examine the build log, use:

nix-store --read-log $(nix-instantiate '<nixpkgs>' -A jfsutils)

Or with the new command line style:

nix log $(nix path-info --derivation nixpkgs#jfsutils)

Additional context

The difference is in the jfs_mkfs binary: diffoscope output at https://reproducible.nixos.org/nixos-iso-minimal-runtime/diff/f6324ec1d2bd8e7eb94aa170221be10b9f59702ddd8c6eda0190dcb70f82e850-bb3de609e45cb1278589282568f7b92cf577c1a69c6aad2efc16f013a94dcceb.html

Add a :+1: reaction to issues you find important.

raboof commented 9 months ago

this one seems difficult to reproduce, perhaps some fluke?

raboof commented 9 months ago

this one seems difficult to reproduce, perhaps some fluke?

Hmm, I did just see it again once but not again on subsequent attempts - so not entirely a fluke, but tricky to reproduce it seems

Kamillaova commented 7 months ago

Successfully reproduced on 23.11, in less than 10 minutes

error: derivation '/nix/store/k4rj6a31jhwby9b63r8ldf5pjvb7c3z1-jfsutils-1.1.15.drv' may not be deterministic: output '/nix/store/8k8k7bzwbq32876qfv07dqwxdzi89khg-jfsutils-1.1.15' differs from '/nix/store/8k8k7bzwbq32876qfv07dqwxdzi89khg-jfsutils-1.1.15.check'

Kamillaova commented 7 months ago

Diff:

(WTF)

Kamillaova commented 7 months ago

This problem is bit to bit reproducible on other machines

raboof commented 6 months ago

Interesting! I'm afraid it's still rare on my laptop.

If you want to dig deeper it might be interesting to understand better where in the build process the nondeterminism is introduced. For example, you could add:

  postBuild = ''
    echo 'XXXX'
    md5sum mkfs/mkfs.o
    strings mkfs/mkfs.o | grep main.cold
  '';

to pkgs/tools/filesystems/jfsutils/default.nix and see if that file is created inconsistently, or is post-processed inconsistently somehow.

Kamillaova commented 6 months ago

Reproduced with gcc+binutils built with -ftrivial-auto-var-init=zero, and also with graphene-hardened-malloc & libfaketime

I think this is also reproduces with llvmPackages_{15,16,17}.stdenv

Kamillaova commented 6 months ago

The problem is always jfs fsck

objdump -t: A (correct): B (incorrect):

Looks like really broken toolchain (UB?), but... It reproduces with LLVM toolchain, with GNU toolchain built with -ftrivial-auto-var-init=zero..

I'll try dontStrip = true; now

Kamillaova commented 6 months ago

After 4 hours of recompiling jfsutils with dontStrip I got this:

Interesting

Kamillaova commented 6 months ago

https://sources.debian.org/patches/haproxy/2.6.12-1%2Bdeb12u1/reproducible.patch/ 🤨 What is this

Kamillaova commented 6 months ago

@raboof what do you think about enabling dontStrip for jfsutils? This generally fixes the problem, but it actually seems like a bug in binutils and llvm

raboof commented 6 months ago

@raboof what do you think about enabling dontStrip for jfsutils?

Interesting, normally you'd expect stripping to maybe remove nondeterminisms, not add them.

Does the package become much bigger by not stripping?

it actually seems like a bug in binutils and llvm

perhaps we can file an upstream bug and link to that when we enable dontStrip?

NixOS / nixpkgs