support building only drvs that lack substitutes (aka `nix-build-uncached`)

[colemickens]

Is your feature request related to a problem? Please describe. Not necessarily, since nix-build-uncached exists, but it seems like a nice-to-have in nix itself. Also, I think the implementation in nix itself would likely be much more simple than nix-build-uncached itself, just due to how it works.

Describe the solution you'd like nix build --lacking-substitutes .#packages would build only the packages which can not be substituted from a known store or trusted binary cache.

Also, a more-eloquent flag name would work, but I can't think of any :).

Describe alternatives you've considered Just using nix-build-uncached.

Additional context This makes using Nix in CI scenarios a bit easier. For example, nixpkgs-wayland cuts down on wasted resources significantly by using nix-build-uncached. Otherwise the builder will download hundreds of MBs of store paths just to fulfill the build, even though in reality we only want to build binary artifacts that aren't already in our CI's binary cache.

Alternatively

It seems like nix-build-uncached has to resolve unsubstitutable-derivations by parsing the output of nix-build supposedly because nix build's dry-run doesn't work; maybe that could be fixed.

cc: @mic92

[Mic92]

If I correctly recall nix build --dry-run does not allow import-from-derivation, which is why I had to use nix-build instead. nix build without dry-run allows that.

[Ericson2314]

Here's an idea:

nix-build --store $remote_store --builders auto

This flips things around so that the goal is just making paths exist in the remote store, but the local store (auto means directly or via daemon) is allowed to help build missing pieces.

I think that should do exactly what you want. it probably doesn't yet not yet work, but we should fix that.

[matthewbauer]

I'm not sure if we want to limit ourselves to one remote store here though. nix-build-uncached will look for any substitution, so we don't end up duplicating things in https://cache.nixos.org in or another cache too. nix-build --store will guarantee that everything exists in the binary cache, even if some other binary cache already has it.

I think a better way to do this is by augmenting Nix copy to be more specific in what is copied over:

nix copy --realise-mode outputs-if-missing --upstream-substituter https://cache.nixos.org --substitute-on-destination --to http://my-cache .#packages

It would add two new options:

• --realise-mode outputs-if-missing: Only realise paths that aren't already in dstStore.
• --upstream-substituter: Skip paths that already exist in this substituter (https://blog.cachix.org/posts/2020-07-28-upstream-caches-avoiding-pushing-paths-in-cache-nixos-org/).
• --substitute-on-destination: This already exists, but allows you to skip copying things that the remote store can substitute. I think we could extend it so that anything in upstream-substituter is counted toward this.

---

Here's a command to get missing paths for a cache that works with flakes (at least in flake-compat mode):

$ nix-store -q --outputs $(nix-instantiate --no-gc-warning -E '(import (builtins.fetchTarball https://github.com/edolstra/flake-compat/archive/master.tar.gz) { src = ./.; }).defaultNix.checks.${builtins.currentSystem}') | while read storePath; do nix path-info --store https://nixiosk.cachix.org $storePath >/dev/null 2>&1 || echo $storePath; done

[Ericson2314]

@matthewbauer to solve the "don't duplicate stores" issue, I rather just be able to layer stores:

nix-build --store "layered://?layers=$remote_store, https://cache.nixos.org" --builders auto

As that I think that is a generally useful concept, especially once Nix tries to deal with secrets in the store.

[domenkozar]

Refs https://github.com/NixOS/nix/issues/3428

[colemickens]

@matthewbauer What if I'm trying to just get a list of store paths and not actually do a copy (my final destination store is not supported natively by cachix/nix)?

[Ericson2314]

I'm starting on this. I think if it I put off splitting up derivation-goal.cc until after, I'll be able to keep my sanity. I got it to build and not regress with Store instead of LocalStore and (surprisingly few!) downcasts; now I'm trying to make some new fucuntional come of that ugly change.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[cole-h]

Still desirable.

[domenkozar]

@Ericson2314 see #5025

[Ericson2314]

Per #4364, which is closed as a duplicate of this, one should be able to do

nix-build --store my-remote-store --builders 'auto - - 1 1'

to achieve this.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[endgame]

Still important, and stalebot is still a jerk.

[nrdxp]

nix-build --store my-remote-store --builders 'auto - - 1 1'

That doesn't work in the case of an s3 cache, which has no daemon

[SuperSandro2000]

Still important, and stalebot is still a jerk.

stale bot in NixOS is configured to be purely informational and to help identify old and stale issues. It will not close issues and you do not need to comment on the issue just for the comment.

[nrdxp]

Just realized that nix path-info --json --store <remote-uri> almost already does this. It contains a boolean "valid" marking whether or not the given path is valid. The only thing is that it will error if it doesn't know about the drv file at all. If we could just have it simply return false instead and continue we could pass multiple paths at once.

Perhaps one way we could do this and still ensure the derivation is a valid artifact would be to teach path-info to use the --eval-store as a backup, if the derivation exists in the eval-store but not the store, consider it uncached (`"valid": false) in the current schema instead of erroring.

If we did that with the output of nix-store --query --requisites --include-outputs we would have our uncached build closure in two commands and without even having to explicitly build it (assuming we at least have the drv).

As a further optimization, we could easily guarantee we can do this in a single invocation with something like: #7437

[nrdxp]

I'd like to correct my previous comment a bit, I must have been misremembering or working with a different tool but there is no such "valid" attribute after all, however we can still do this. Basically we could implement something like --keep-going for path-info and it will simply store the path as invalid if the --json flag is also passed and that would just about do it.

Maybe an additional flag --all-caches could be added to check all caches and not just a single one, but that would just about do it.

[Ericson2314]

nix-build --store my-remote-store --builders 'auto - - 1 1'

is still the right idea conceptually, and works. Rather than thinking of new work-arounds, we should just investigate what if anything about it doesn't work well and fix them.

For example, it will fail with a read-only store because it doesn't know how to upload things. Fine. Then we should come up with a new solution like an "overlay store", or a way for people to tell nix how to do uploads for their substitutor store.

[nrdxp]

I like the idea of querying the caches independant of building personally, but I don't see a problem with doing both either. In std-action, for example, we are doing all evaluation before hand in its own run, and produce a json that is used to build various job matrices. The runners in these matrices don't do any evaluation but use the json to know about the derivation paths ahead of time so they can skip any extra nix evaluation and get straight to work.

If we had a simple way to query all caches during the eval phase, we could conditionally skip certain builds from even starting if they are already cached. That's why I suggest augmenting nix path-info. Right now we still do that, but the task builder has to start up, install nix, then we parse the output of nix-build --dry-run (basically exactly what nix-build-uncached does), to see if there is actually anything to build or not. It would be better to skip all that setup and just not spawn a build job at all, but in order to make that feasible we need a more efficient way to query if a package is cached than calling nix path-info or nix-build --dry-run n times sequentially.

[colemickens]

Not to mix issues but in a CI scenario with a "clean" builder, can you not rely on nix-eval-jobs's --check-cache-status?

[nrdxp]

Not to mix issues but in a CI scenario with a "clean" builder, can you not rely on nix-eval-jobs's --check-cache-status?

It depends on the scenario, but if there is any sort of IFD, even a small amount, nix-eval-jobs can actually be exponentially more expensive, both in space and time, since each thread pays that cost again and again. In my own personal projects, nix-eval-jobs worked really well in my initial testing, but at work haskell.nix relies of IFD for dependancy resolution so nix-eval-jobs is a definite no-go there.

I wanted a solution that is generally applicable. I definitely didn't want some inexperienced user accidentally adding some IFD to their project due to lack of experience and all of a sudden their CI runner is OOMing.

Actually the eval is really not all that expensive even with IFD if basically all of the derivations that are produced during the eval (from IFD) are already reified before it starts. The difference is fairly drastic, from about 25m (no cache) to only 2m, for everything that will be run in CI in my current work project. If we could get the eval-cache working for nix eval that cost might come down to a few seconds, ideally: #4279

Also, just in principal, I'd rather support fixing and augmenting upstream than working from a fork that may or may not be maintained in the long term. If it would be possible, I'd rather the official Nix evalautor itself learn how to use threads, and in a more efficient manner (some kind of shared state) than nix-eval-jobs seems to, at least from my own testing.

[nrdxp]

Just pinging here that you can use either #7587 or #7526 to resolve this. The latter may never be merged which is why I decided to split them up into separate PRs. With nix path-info --query-substitutes --json you can trivally pass the derivations of unbuilt outputs to be built by passing outputs (or flake refs) and then grabbing the derivations from the json and sending them to your build command.

[Mic92]

I expanded the documentation off nix-eval-jobs also a bit w.r.t. to concurrency and memory usage: https://github.com/nix-community/nix-eval-jobs/pull/202/files

[Mic92]

https://github.com/Mic92/nix-fast-build is the spiritual successor of nix-build-uncached. Check the --skip-cached flag: https://github.com/Mic92/nix-fast-build#avoiding-redundant-package-downloads

[tomberek]

nix-build --store $remote_store --builders auto

@Ericson2314: This almost works now, but I run into this:

querying info about '/nix/store/pa10z4ngm0g83kx9mssrqzz30s84vq7k-hello-2.12.1.tar.gz' on 'https://cache.nixos.org'...
downloading 'https://cache.nixos.org/pa10z4ngm0g83kx9mssrqzz30s84vq7k.narinfo'...
warning: ignoring substitute for '/nix/store/pa10z4ngm0g83kx9mssrqzz30s84vq7k-hello-2.12.1.tar.gz' from 'https://cache.nixos.org', as it's not signed by any of the keys in 'trusted-public-keys'

when attempting:

nix build --store s3://my-store nixpkgs#hello --builders auto -vv

I guess my local settings are not passing along trusted-public-keys? And I've not been able to convince the client/daemon to accept those keys via any combination of attempts so far.

(@Mic92: while this isn't quite the UX we want, it is conceptually quite consistent with other notions. I'd like to ensure the underlying mechanism is good and then we can expose it in a more intuitive manner.)

Edit: This seems to be due to the remote store not trusting pathInfo

diff --git a/src/libstore/store-api.hh b/src/libstore/store-api.hh
index 8b6bf9aed..985c6dfaf 100644
--- a/src/libstore/store-api.hh
+++ b/src/libstore/store-api.hh
@@ -400,7 +400,7 @@ public:
      */
     virtual bool pathInfoIsUntrusted(const ValidPathInfo &)
     {
-        return true;
+        return false;
     }

     virtual bool realisationIsUntrusted(const Realisation & )

makes things work. Likely we want this to either use global settings, or better would be for each store to have trust+policy settings apply to them.

[arianvp]

I tried the trick above but it doesn't seem to work with flakes:

nix build \
    --extra-substituters 's3://cache20240603113122461500000001?region=eu-central-1' \
    --extra-trusted-public-keys 'nixos-village:6yZ26qopUY/jErtM/YtjB0jFshjx4jgUYxEmhqKEcQE=' \
    --store "s3://cache20240603113122461500000001?region=eu-central-1&secret-key=$(realpath ./cache-secret-key)" \
    --builders 'auto' \
    .#nixosConfigurations.web-push.config.system.build.toplevel 
uploaded 's3://cache20240603113122461500000001/nar/05063mx0m30fkb17h7hjnj7yl4kb9k5a88xnym00aj0bniig8201.nar.xz' (19856 bytes) in 293 ms
uploaded 's3://cache20240603113122461500000001/17g3406b7014sk07w9khd4lsylw3pldg.narinfo' (510 bytes) in 204 ms
error: path '/nix/store/17g3406b7014sk07w9khd4lsylw3pldg-source/flake.nix' does not exist

[arianvp]

Doing a nix flake archive && nix flake archive --store 's3://' beforehand seems to solve the issue. But now running into the same issue as Tomberek where the path info is not trusted