NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.63k stars 1.51k forks source link

A command to add a gc root #7138

Open roberth opened 2 years ago

roberth commented 2 years ago

Is your feature request related to a problem? Please describe.

I'm generating a script that needs create an indirect gc root to a store path that's in the script's closure. E.g.

writeScript "foo" ''
  # wrong (missing gc root)
  ln -s ${pkgs.hello} ./program
''

Describe the solution you'd like

A new command as follows. (I didn't find a suitable command in the unstable manual or the nix-store manpage) Here's a fictional doc I hope we can turn into reality.


nix store add-root path storepath

Create an indirect garbage collection root at location path to refer to the storepath. Note that most commands that produce store paths have an option to produce a garbage collection root, so that the store path can not be garbage collected before a root is created. A valid use case for nix store add-root is in scripts built with Nix. In this case the store path is already referenced by the script's store path, and nix store add-root is used to prevent collection of the store path after the script exits.

writeScript "foo" ''
  nix store add-root ./program ${pkgs.hello}
''

Describe alternatives you've considered

Abusing some other command to create the root and then replace the symlink. This is bad because it's complicated and may leave the symlink in a bad state.

Additional context

RonnyPfannschmidt commented 2 years ago

@roberth i believe the command is nix-store --add-root ... -r ... where the config file is added behind r

roberth commented 2 years ago

That only works for derivations after -r (--realise). In a readily built script, you basically can't refer to the derivation, because that would pull in all the transitive build dependencies as output (runtime) dependencies.

chriswarbo commented 1 year ago

@roberth The -r option of nix-store can accept either a derivation or an output. For example, I'm using shellHook to symlink to a config file (.pre-commit-config.json); I can turn this into a GC root using nix-store --add-root and it seems to work:

# Check existing symlink (not a GC root yet)
$ ls -lA | grep pre-commit
lrwxr-xr-x  1 chrisw staff   66 Nov  9 22:33 .pre-commit-config.yaml -> /nix/store/q9gq4gasykzawdgamgsq06gsw9pyr56c-pre-commit-config.json

# Check whether we can add a GC root using nix-store
$ nix-store --add-root PRECOMMIT-TEST-ROOT -r $(readlink -f .pre-commit-config.yaml)
/Users/chrisw/TEST/PRECOMMIT-TEST-ROOT

# Check it points at the right thing
$ ls -lA | grep pre-commit
lrwxr-xr-x  1 chrisw staff   66 Nov  9 22:33 .pre-commit-config.yaml -> /nix/store/q9gq4gasykzawdgamgsq06gsw9pyr56c-pre-commit-config.json
lrwxr-xr-x  1 chrisw staff   66 Nov 10 12:16 PRECOMMIT-TEST-ROOT -> /nix/store/q9gq4gasykzawdgamgsq06gsw9pyr56c-pre-commit-config.json

# Check the GC root was created
$ ls -lA /nix/var/nix/gcroots/auto/ | grep PRECOMMIT
lrwxr-xr-x 1 root nixbld 51 Nov 10 12:16 g8wvba0kpaysc69vhlhs0jn1ymv1qwg8 -> /Users/chrisw/TEST/PRECOMMIT-TEST-ROOT
thufschmitt commented 1 year ago

That could indeed be useful as a plumbing command for the new CLI.

In the meantime, nix build --out-path program ${pkgs.hello} should do that job all right for your use-case I think. The only caveat is if you want the .drv file and not the output path. But I don't think that's your case.

roberth commented 1 year ago

Thanks! Now I wonder what broke my initial attempt. Your examples do work, so all that remains for this issue is the .drv case, which might become more important with RFC 92 Dynamic derivations.

Gabriella439 commented 1 year ago

I think this is still necessary. I believe there should be a way to create a GC root for a derivation without realising that derivation.

The motivation for this is so that it can be used in conjunction with keep-outputs = true. If you create a GC root for the derivation for a build before initiating the build then that protects all build-time dependencies even if the build fails and (this is key) it would protect build-time dependencies even after the build fails.

Ericson2314 commented 1 year ago

https://github.com/NixOS/nix/issues/7261 would make it easier to do things to drv files themselves in general.

kjeremy commented 1 year ago

I would really like something like this for a remote store. We launch our builds from docker containers that use the host's socket to do building but need a way to generate roots on the host.

Abdillah commented 9 months ago

Instead of using cachix, I use one of my machine for cache. Build the whole systems on GitHub action and upload it. Exactly what @kjeremy expected, the caches need to be retained, at least until the next successful build.

Using nix develop --profile /path/to/gc/root <expr> turns out only supported for eval, not for resulting store path. I wonder if we link manually the store to a "root link", then link the "root link" to a file inside /nix/var/nix/gcroots/per-user/<username>/ what will happened.

Anyway, an official feature is definitely expected.

Abdillah commented 9 months ago

I can use this command to create a gc root (I think):

nix --add-root <link> --indirect --realize <existing-store-path>
layus commented 1 month ago

I found a way :magic_wand: !

indirect_root=test
path_to_root=/nix/store/...
nix build --expr 'builtins.toFile "placeholder" "" ' --out-link $indirect_root
ln -sfn $path_to_root $indirect_root
# confirm it was not deleted in the meantime 
nix-store -q --roots $path_to_root | grep $indirect_root || { echo failed; exit 1;}

This is obviously not a very good solution, but if it can help anyone in the meantime... It sure helped me :-)

layus commented 1 month ago

Gentle ping for ppl interested in this issue. Could you have a look at #11505 and #11506 to see if they would address your needs ?