Excess rebuilding: fileset library undermined by git commit hashes

[Ericson2314]

It is very nice that we use @infinisil's fileset work to filter sources. Unfortunately, this only helps between "revision-dirty" builds. Otherwise, we don't get much benefit because git-commit-hash-based versions still change when only excluded files changed.

(This especially annoyed me when testing the perl bindings full build, e.g. for #10378)

Not sure what we should do instead. Git (or nar) hashes of post-filtered files are an option, but those will not be immediately obvious to people. Another option is trying to bake in the regular commit hash as a final post step with something like gettext.

CC @roberth @fricklerhandwerk

[fricklerhandwerk]

I have hard time parsing this. Do you mean that the version attribute changes even when files excluded by Git change? Is this because how flake.nix is processed? We could obtain that information purely by reading Git blobs, right?

[roberth]

hercules-ci-agent solves this problem by

• not referring to the rev anywhere in the derivations
• adding the rev as a package attribute (passthru)
• adding the rev to the services.hercules-ci-agent.labels option in NixOS and nix-darwin, so that only the config files need to be rebuilt, which is quick, and need to happen anyway.

Basically all unreleased software faces this problem, so do we want to solve this just for Nix (and put a build-problem label on this issue), or do we want to have a Nix feature that manages this information? This isn't the only instance of wanting to put evaluation-time data onto derivations without affecting their hashes. Basically all meta attributes suffer from having no persistence, or even bad propagation without persisting anything.

[Ericson2314]

Do you mean that the version attribute changes even when files excluded by Git change?

Yes

Is this because how flake.nix is processed? We could obtain that information purely by reading Git blobs, right?

Don't think it is anything that complex thankfully. Just our choice of how version is set.

---

@roberth Totally agree there is a larger problem, and ideally we do finally solve that once and for all (and then maybe drop the name parth from store paths because the other metadata solution is better). But I suppose I opened this issue wondering if we could find something quick-and-dirty for Nix itself in the meantime. Can easily be convinced either way, though.

[roberth]

If we split out the libraries into separate derivations, and not put a rev in those, that'd probably reduce rebuilds even more, even if the derivation(s) for the executables still have a rev. So that's without inventing a scheme to insert rev at the last second.

[Ericson2314]

Currently the version ends up in the config.h which all the libraries get; I think untangling that might get us most the way to a gettext-style thing too.

[roberth]

Untangle seems to overstate it :rocket:

Btw I did prove out https://github.com/NixOS/nix/pull/8543 for splitting libraries, but I'd prefer something more principled, and something that uses the meson work.