search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
11.59k stars 1.45k forks source link

Have our cake and eat it too derivation metadata #10780

Open roberth opened 1 month ago

roberth commented 1 month ago

Is your feature request related to a problem? Please describe.

meta fields are not included in derivations because they would cause unnecessary rebuilds.

Yet, we want users to be able to analyze this information, which isn't exposed reliably.

This lets us solve the problem of lost meta.timeout values in the store layer.

Describe the solution you'd like

Change derivation output hashing so that a section of the .drv is excluded from the computation, as well as being made unobservable from the derivation's environment. This causes outputs of equivalent derivations to still happily collide onto the same output, so that output caching remains as effective as it is today.

This extra info only needs to be unobservable under what we might call "equivalence under validity", an idea that we already exploit in other areas. For example, it is fundamental to fixed output derivations. An example here is meta.timeout: if we don't pass it to the build environment, it can not cause successful builds to have different outputs. We might even call it "equivalence under weak validity", because if a different timeout causes one derivation to fail, that's also ok.

The idea of making output hashes not depend on the entire derivation is also not new; again fixed output derivations already apply this trick very effectively.

Describe alternatives you've considered

Store such information in EvalState, so that we don't have to change the output hashing. This causes a problem when the build happens with nix eval + nix-store -r, so we'd have to persist this info. However, then we can't discern between stale entries from previous evaluations and the entries we actually need. Also it does not work without forwarding this info to remote builders somehow.

Additional context

Another output hashing change:

Priorities

Add :+1: to issues you find important.

fricklerhandwerk commented 1 month ago

Triaged in Nix maintainer meeting:

nixos-discourse commented 1 month ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-05-29-nix-team-meeting-minutes-148/46195/1

roberth commented 1 month ago

Added breaking label, because this won't be hashed correctly by any existing Nix version, which is a big deal considering our limited forward compatibility mechanisms. Best incremental change I can think of is to do something similar to what the dynamic derivations did: intentionally breaking the first token.

flokli commented 1 month ago

I'd still prefer to be this auxillary metadata of some sort, rather than changing the ATerm and hashing scheme just for the sake of adding metadata in there.

Sure, care needs to be taken to make sure it properly traverses the places it needs to (like remote building scenarios etc), but this feels less hacky than what's proposed here.

roberth commented 1 month ago

I don't think this is proposing any new "hackiness" that we don't already have in the form of fixed output derivations, which also excludes almost everything from the path hash. I can understand that it feels that way though, because adding something feels safer than changing something. The main benefit I see in this approach is that we don't complicate the data model, but rather double down on an aspect that's not new but underutilized (aforementioned FOD-inspired hashing).

I've started collecting derivation design issues in the derivation design label. I've called it "derivation v2" briefly, but renamed it not to bias it towards a solution that may well be too heavy handed, but I do have to note that we have a number of issues that require breaking changes. An overview of related issues should help inform such decisions.