Open lheckemann opened 6 years ago
I think it would be an improvement even without no-signatures: it would help against unnoticed IP recycling, and you already need to enter a lot of data about remote builders anyway.
The thing with changing it to require signatures is that that would break existing setups: up to now, using remote builders hasn't required setting up signing keys. I don't see how IP recycling comes into this either.
I would say that adding normal substituters for remote builders in parallel with the old style is the most likely way just from the testing point of view, and then addition/default/deprecation/removal-of-old have to go in this order with large enough waiting periods anyway…
Well, if you look at this step:
alice
makescharlotte
buildc.drv
(no-op) and copies the output path over
the builder charlotte
was already effectively used as a substitutor as no building happened (and charlotte
might even have obtained the output of c.drv
via substituting from some other place, never building it itself). So security-wise there's no difference since substitution can already happen.
What this would also allow (though not imply directly) is lazy copying — so in the strongest case where only one builder is available for all the derivations that need to be realised as part of a build plan, everything is built only on that builder and only the end result and its reference closure are copied to the build initiator. Since this is a change in behaviour, I'm not sure if that's universally desirable — but I can definitely see some nice use cases for it, like nixos-rebuild on devices with small storage.
I would say it is very annoying that it is complicated to do nix copy
when the source host has all dependencies of the path you need but not the path itself, or to grab all the build dependencies of a path from a cache that can give you the path itself and its runtime dependencies instead — so a unification and an option to choose which dependency-walking logic to use would be nice…
While I do agree that that would really be nice to have, I'd say that's out of scope for this issue as it is quite a bit more complex than just "treat remote builders as binary caches".
I use short-lived remote builders (usually ec2 spot instances). That only live for a few hours. If we would stop copying closures back from the remote builder (into my local cache), I would potentially end up rebuilding everything every time I spin up a spot instance. Costs aside, it would result in unnecessary wait time for potentially cached artifacts.
Since this use case exists, I suppose it would make sense not to universally treat remote builders as substituters, and this should be done manually by putting them in the substituters
option. This could still do with some improvement though:
Err I think it's easier to optionally require fetching closures during substitution. More generally, I think all 4 of these make sense:
One can few that as feature sets "generated" by combining remote builders and substitutors into a single configurable concept.
remote builders and substitutors into a single configurable concept.
Which they are in fact already, in principle, since 2.0 — they're all remote stores, just used in different ways. Very much +1 on this idea! Although the "always get closure" thing I think would be more of a per-build thing?
Good point about 2.0. I thought this was the case but I forgot.
Yeah I suppose the closure option has no security implications, so it can be purely per-build. (The rest also make sense per-build, but for untrusted users would need to be within the daemon's whitelist.)
I marked this as stale due to inactivity. → More info
I closed this issue due to inactivity. → More info
Still relevant, the current rules making managing multiple systems annoying.
I have perpared a patch a few weeks ago, but I didn't manage to properly test it. Hang on for a bti please :)
I can see some potential advantages to treating remote builders as substituters before trying to build any derivations — this would allow the build plan to be executed more efficiently in some cases, with less copying around of paths and potentially less overall building as well.
Consider:
b.drv
, a derivation which produces the output path/nix/store/b
c.drv
, a derivation which produces the output path/nix/store/c
a.drv
, which produces the output/nix/store/a
and depends on the outputs ofb.drv
andc.drv
./nix/store/a
has a reference to/nix/store/b
but none to/nix/store/c
.alice
, havingbob
andcharlotte
configured as remote builders, doesn't have any of the output paths available, nor can it build any of the drvs (incompatible system or something)bob
doesn't have any of the paths available, but can build themcharlotte
has all the paths built.Now running
nix build /nix/store/a.drv
onalice
can result in something like the following:alice
makesbob
buildb.drv
and copies the output path overalice
makescharlotte
buildc.drv
(no-op) and copies the output path overalice
copies/nix/store/c
tobob
and makesbob
builda.drv
, then copies the output path overEnd result being that all three derivation's output paths have been copied to all three systems, and
b.drv
anda.drv
have been unnecessarily rebuilt.If
alice
hasbob
andcharlotte
configured as substituters additionally,alice
will first query them for/nix/store/a
, and upon determining thatcharlotte
already has the path, will only copy/nix/store/a
and/nix/store/b
to itself, resulting in no unnecessary building and no superfluous space usage.Are there any drawbacks to universally treating builders as (absolutely trusted, i.e. no sigs required) substituters as well?