extra-utils/util-linux: not reproducible

[raboof]

Building this package twice does not produce the bit-by-bit identical result each time, making it harder to detect CI breaches. You can read more about this at https://reproducible-builds.org/ .

Fixing bit-by-bit reproducibility also has additional advantages, such as avoiding hard-to-reproduce bugs, making content-addressed storage more effective and reducing rebuilds in such systems.

Steps To Reproduce

nix-build /nix/store/5dir0fa7zd6ap0w4gckslsj20fq33xcw-extra-utils.drv --check --keep-failed

You can use diffoscope to analyze the differences in the output of the two builds.

To view the build log of the build that produced the artifact in the binary cache:

nix-store --read-log /nix/store/5dir0fa7zd6ap0w4gckslsj20fq33xcw-extra-utils.drv

Additional context

It seems this may somehow have something to do with dcgettext?

https://arnout.engelen.eu/nixos-iso-minimal-r13y/diff/30eab6bb4206e4f50a90769408a563c681ead7c4c78fa31ffe6ff0f4f29d4a94-39849b71113e7e124f53fbc6c57478d05e8c8ecd3dbcc134080201e507e3420f.html

[wentasah]

Which package are you talking about? I don't see any package named extra-utils in nixpkgs.

[wentasah]

So I can reproduce this with the following commands:

nix-build --expr '(import ./nixos { system =  "x86_64-linux"; configuration=./nixos/modules/installer/cd-dvd/installation-cd-minimal.nix; }).config.system.build.toplevel'
extrautils=$(nix-store -q --requisites ./result|grep extra-utils)
nix-build --check --keep-failed "$(nix-store -q --deriver "$extrautils")"

I run the above on different commits and found that the following commit builds reproducibly:

636051e3534 ("linux: avoid NO_HZ_FULL on i686-linux", 2022-11-02)

The first commit after it, which I detected as non-reproducible is:

5bdb380ee7f ("tbls: 1.56.3 -> 1.56.4", 2022-10-26)

There is 2192 commits in between. For some of them, extra-utils is not available in the resulting closure, which would complicate bisection. I'll try to run it to see if it finds something interesting.

[wentasah]

As expected, bisect was not much useful. It haven't complete yet, because many builds are very slow.

Perhaps, faster command to reproduce this is to build just extraUtils rather than the toplevel attribute:

nix-build ./nixos --arg configuration '{
  imports = [ ./nixos/modules/installer/cd-dvd/installation-cd-minimal.nix ];
}' -A config.system.build.extraUtils
nix-build --check --keep-failed "$(nix-store -q --deriver ./result)"

When running this with commit 636051e3534, it is reproducible.

However, reproducibilty is lost when running this with minimalist configuration:

nix-build ./nixos --arg configuration '{
  fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
  boot.loader.grub.devices = [ "/dev/sda" ];
}' -A config.system.build.extraUtils
nix-build --check --keep-failed "$(nix-store -q --deriver ./result)"

With current nixos-unstable 85d6b3990def7eef45f4502a82496de02a02b6e8, both commands are not reproducible.

Will continue later.

[wentasah]

Bisect with building extraUtils only was more successful. f58fea6249f0a31e3787dd4be539b917adfdfb7f is the first bad commit (Merge pull request #184266, libaio: 0.3.112 -> 0.3.113). With this commit

nix-build ./nixos --arg configuration '{
  imports = [ ./nixos/modules/installer/cd-dvd/installation-cd-minimal.nix ];
}' -A config.system.build.extraUtils
nix-build --check --keep-failed "$(nix-store -q --deriver ./result)"

fails, while both parents succeed. So far, I don't see why. More investigation needed.

[raboof]

Are you sure f58fea6249f0a31e3787dd4be539b917adfdfb7f fails for you? It seems to succeed for me (comparing the version from the binary cache with my local version).

On 3928cfa27d9, when I get /nix/store/mblyf06gzf7qxrpy79mwx88gx1mf4y6v-extra-utils from the binary cache, it fails. However, when I avoid the binary cache it does build reproducibly (nix-store --delete and adding --no-substitute to the nix-build ./nixos).

On 3928cfa27d9, strings /nix/store/mblyf06gzf7qxrpy79mwx88gx1mf4y6v-extra-utils/lib/libblkid.so.1 (from the cache) shows dcgettext, while my locally-built one does not. On f58fea6249f0a31e3787dd4be539b917adfdfb7f, strings /nix/store/r3vphxah73v2vlgq39nj9r9p9swb8hjb-extra-utils/lib/libblkid.so.1 (from the cache) does not show dcgettext, which is consistent with my local build.

When you build locally on f58fea6249f0a31e3787dd4be539b917adfdfb7f, does libblkid.so.1 have dcgettext or not?

[raboof]

if we focus on libblkid.so.1 for a while, it seems that one comes from util-linuxMinimal.lib - does strings $(nix-build . -A util-linuxMinimal.lib)/lib/libblkid.so | grep dcget show dcgettext for either of those commits for you? For me it's never there.

[wentasah]

Are you sure f58fea6249f0a31e3787dd4be539b917adfdfb7f fails for you? It seems to succeed for me (comparing the version from the binary cache with my local version).

Yes. It still fails for me. Since my last comment I upgraded my NixOS from 22.05 to 22.11. So the outcome is at least deterministic across NixOS (pre)releases if not across different machines.

On 3928cfa27d9, when I get /nix/store/mblyf06gzf7qxrpy79mwx88gx1mf4y6v-extra-utils from the binary cache, it fails. However, when I avoid the binary cache it does build reproducibly (nix-store --delete and adding --no-substitute to the nix-build ./nixos).

For me 3928cfa27d9 does not fail, i.e. the version built by me is the same as from the binary cache.

However with f58fea6249f, I can confirm that the subsequent local builds are reproducible.

On 3928cfa27d9, strings /nix/store/mblyf06gzf7qxrpy79mwx88gx1mf4y6v-extra-utils/lib/libblkid.so.1 (from the cache) shows dcgettext, while my locally-built one does not.

Yes, the version from the cache shows dcgettext.

On f58fea6249f0a31e3787dd4be539b917adfdfb7f, strings /nix/store/r3vphxah73v2vlgq39nj9r9p9swb8hjb-extra-utils/lib/libblkid.so.1 (from the cache) does not show dcgettext, which is consistent with my local build.

When you build locally on f58fea6249f0a31e3787dd4be539b917adfdfb7f, does libblkid.so.1 have dcgettext or not?

Here, my local build shows dcgettext.

if we focus on libblkid.so.1 for a while, it seems that one comes from util-linuxMinimal.lib - does strings $(nix-build . -A util-linuxMinimal.lib)/lib/libblkid.so | grep dcget show dcgettext for either of those commits for you? For me it's never there.

I also don't see it with both commits. And

nix-build . -A util-linuxMinimal.lib --check

succeeds. So the version with dcgettext probably comes from somewhere else.

[raboof]

Interesting, so:

• local builds of extra-utils for me consistently don't have dcgettext
• local builds of extra-utils for you consistently do have dcgettext
• in the binary cache f58fea6249f0a31e3787dd4be539b917adfdfb7f doesn't have it
• in the binary cache 3928cfa27d9925f9fbd1d211cf2549f723546a81 does have it
• for both of us, nix-build . -A util-linuxMinimal.lib --check consistently doesn't have it.

For me, on f58fea6249f0a31e3787dd4be539b917adfdfb7f:

$ nix-build . -A util-linuxMinimal.lib
/nix/store/b630nmmdy0firqak9jxgiiyar4m9haxr-util-linux-minimal-2.38.1-lib
$ nix-build ./nixos --arg configuration '{
  imports = [ ./nixos/modules/installer/cd-dvd/installation-cd-minimal.nix ];
}' -A config.system.build.extraUtils 
this path will be fetched (7.45 MiB download, 23.42 MiB unpacked):
  /nix/store/r3vphxah73v2vlgq39nj9r9p9swb8hjb-extra-utils
copying path '/nix/store/r3vphxah73v2vlgq39nj9r9p9swb8hjb-extra-utils' from 'https://cache.nixos.org'...
/nix/store/r3vphxah73v2vlgq39nj9r9p9swb8hjb-extra-utils
$ nix-build --check --keep-failed "$(nix-store -q --deriver ./result)" &>/dev/stdout | grep libblkid.so.1
'/nix/store/b630nmmdy0firqak9jxgiiyar4m9haxr-util-linux-minimal-2.38.1-lib/lib/libblkid.so.1.1.0' -> '/nix/store/r3vphxah73v2vlgq39nj9r9p9swb8hjb-extra-utils/lib/libblkid.so.1'
patching /nix/store/r3vphxah73v2vlgq39nj9r9p9swb8hjb-extra-utils/lib/libblkid.so.1...

... so for me it does seem to use util-linuxMinimal.lib to populate extra-utils (and indeed it doesn't have dcgettext in both places) - what does this look like for you?

[raboof]

I haven't seen this problem pop up in reports for a while, closing for now, will re-open when we see it again.