Is your feature request related to a problem? Please describe.
There is a clear desire (links in context) to have some kind of cache awareness to the build step. In other words, like how Nix substitutes derivations if possible instead of building, what if all we want to build for is to popullate the cache? There is no point to fetch the result to our local machine if it already exists in the desired location in that case, and its just a waste of bandwidth.
Describe the solution you'd like
A couple of proposals have come up each with their merits. This one simply builds off the direction the new cli has been taking with the --store and --eval-store flags, and suggests we also simply add a --target-store.
Then even if the derivation doesn't exist on --store, if it exists on the --target-store we skip the build. Additionally, once the build has finished, we can automatically copy the result to the target in a single step that is also clear to the reader semantically (imo at least).
You could have some auxilary settings to control for things like:
consider all configured caches, or just the target itself. We could even reuse the use-substitutes flag for this, as it sounds about right
whether we want to copy build dependencies to the target as well. The nix-store manual refers to this as a "cache deployment", so perhaps we can use that terminology? --deploy-type cache. We could maybe reuse this logic for nix copy itself as well.
Need to determine, for scheduling purposes, whether a CI task needs to be scheduled for a build? Simply slap a --dry-run on your build command (including --target-store) and be 100% sure of the results. We could maybe have different exit codes for whether a build will be run or not, for scripting purposes.
To me this is the nicest solution I've considered so far since it takes out any manual processing, and allows the user to effortlessly achieve the desired result. It also builds off the current cli implementation so I was hoping it'd be in the same spirit as we are already moving. We could also change the nix copy arguments --to and --from to --target-store and --store respectively, to remain consistent.
Describe alternatives you've considered
I thought #7587 would be a decent solution, but its a bit more error prone than I would have liked and ultimately this information needs to parsed to decide whether or not to build anyway, so we might as well just incooperate that into the build command itself.
3946 might actually be a duplicate of this, but perhaps the proposed design here is a little different. There we are just simply looking to build package that are lacking substitutes to decide what to do with them at a later step, whereas here we incooperate the desired result directly into the command invocation (to copy the build result to some target). To me, its different enough to warrant its own issue.
To try and address this problem, but we dediced to actually stop using it. It does not do exactly what we want for various reasons that I will not elaborate here, but essentially we just fall back to parsing nix-build --dry-run for now, which has its own set of difficulties, that I feel can only be addressed suitably by this proposal.
--target-store is the same as --store, but adding a layering mechanism. You want the layering mechanism separate from which store it is being used for.
Is your feature request related to a problem? Please describe.
There is a clear desire (links in context) to have some kind of cache awareness to the build step. In other words, like how Nix substitutes derivations if possible instead of building, what if all we want to build for is to popullate the cache? There is no point to fetch the result to our local machine if it already exists in the desired location in that case, and its just a waste of bandwidth.
Describe the solution you'd like A couple of proposals have come up each with their merits. This one simply builds off the direction the new cli has been taking with the
--store
and--eval-store
flags, and suggests we also simply add a--target-store
.Then even if the derivation doesn't exist on
--store
, if it exists on the--target-store
we skip the build. Additionally, once the build has finished, we can automatically copy the result to the target in a single step that is also clear to the reader semantically (imo at least).You could have some auxilary settings to control for things like:
target
itself. We could even reuse theuse-substitutes
flag for this, as it sounds about rightnix-store
manual refers to this as a "cache deployment", so perhaps we can use that terminology?--deploy-type cache
. We could maybe reuse this logic fornix copy
itself as well.Need to determine, for scheduling purposes, whether a CI task needs to be scheduled for a build? Simply slap a
--dry-run
on your build command (including--target-store
) and be 100% sure of the results. We could maybe have different exit codes for whether a build will be run or not, for scripting purposes.To me this is the nicest solution I've considered so far since it takes out any manual processing, and allows the user to effortlessly achieve the desired result. It also builds off the current cli implementation so I was hoping it'd be in the same spirit as we are already moving. We could also change the
nix copy
arguments--to
and--from
to--target-store
and--store
respectively, to remain consistent.Describe alternatives you've considered I thought #7587 would be a decent solution, but its a bit more error prone than I would have liked and ultimately this information needs to parsed to decide whether or not to build anyway, so we might as well just incooperate that into the build command itself.
3946 might actually be a duplicate of this, but perhaps the proposed design here is a little different. There we are just simply looking to build package that are lacking substitutes to decide what to do with them at a later step, whereas here we incooperate the desired result directly into the command invocation (to copy the build result to some target). To me, its different enough to warrant its own issue.
Additional context I put together: https://github.com/divnix/nix-uncached
To try and address this problem, but we dediced to actually stop using it. It does not do exactly what we want for various reasons that I will not elaborate here, but essentially we just fall back to parsing
nix-build --dry-run
for now, which has its own set of difficulties, that I feel can only be addressed suitably by this proposal.