Design and enable creation of remote gc roots

layus commented 2 weeks ago

Is your feature request related to a problem? Please describe.

It is impossible to get an atomic gc root on remote store paths. The creation of gc roots is explicitly disabled in nix build if we do not have a LocalFSStore:

https://github.com/NixOS/nix/blob/26c3fc11eada3fa7df0284190095868a947fefe2/src/nix/build.cc#L141-L143

Describe the solution you'd like

Ideally, there should be a way to create remote gc roots for paths that are build remotely using --store ssh-ng:... and other remote stores and operations. Otherwise there is no way to enforce atomicity.

The biggest challenge is to create roots that are owned by the right user, and pick a place to put them.

I suggest we might discover the user from the ssh connection, and put the indirect root in a well-known location in side it's home. I need to check if there is an obvious place to do that. If there is a user process on the remote machine that proxies the requests to the local nix daemon (so the remote nix daemon from the point of view of the user) then it is the right place to do it. If connection are forwarded to the remote nix daemon directly, it may become tricky.

As for the location, I assume .local/nix/gcroots would be good enough. .cache is probably not good.

And for the name, it is really tricky. I do not think we should trust the name provided on the command line, as there is a high potential for conflicts and race conditions. Using the nix store path name itself might work. Alternatively, trusting the name passed by the user provides lots of flexibility. They can decide on their own how to organize their remote roots.

As a note, I think these roots should be updated on each access, so as to allow a basic LRU gc to happen if needed.

Describe alternatives you've considered

Direct gc roots are not good, as they are owned by root. The user who created them cannot remove them.
If we had /nix/var/nix/gcroots/per-user/ around we could craft something with it, but it has been recently removed.

Additional context

Some hints in the code:

https://github.com/NixOS/nix/blob/26c3fc11eada3fa7df0284190095868a947fefe2/src/libstore/gc-store.hh#L88-L92

https://github.com/NixOS/nix/blob/26c3fc11eada3fa7df0284190095868a947fefe2/src/libstore/indirect-root-store.hh#L24-L29

/cc @kjeremy @kevincox for you have showed interest in this topic before.

Priorities

Add :+1: to issues you find important.

Ericson2314 commented 1 week ago

I think you want the mounted-ssh-ng://?

The main reason it is experimental is that it should be a configure flag on ssh-ng:// instead. https://github.com/NixOS/nix/pull/11139 helps with that.

edolstra commented 1 week ago

As for the location, I assume .local/nix/gcroots would be good enough. .cache is probably not good.

An issue with this is how/when those GC roots get cleaned up. We don't want roots to accumulate forever without an obvious way to get rid of them.

As a note, I think these roots should be updated on each access, so as to allow a basic LRU gc to happen if needed.

What is meant with updating a root? And what counts as an access?

I think you want the mounted-ssh-ng://?

Probably we want SSHStore to create GC roots via SSH's sftp protocol. That way, whether it's permitted is a configuration policy of the SSH server.

layus commented 1 week ago

Le 13 novembre 2024 21:46:25 GMT+01:00, Eelco Dolstra @.***> a écrit :

As for the location, I assume .local/nix/gcroots would be good enough. .cache is probably not good.

An issue with this is how/when those GC roots get cleaned up. We don't want roots to accumulate forever without an obvious way to get rid of them.

Agreed. But if they are opt-in, it is not a problem. Users will have to deal with that on their own. They could for example setup a periodic task, just like they do with nix-collect-garbage.

I do not see why accumulating roots is bad per se. The nix store accumulates paths without limit. Cleanup is a manual action there too.

What bugs me is that building with a remote store gives no guarantee, not even the guarantee that the path will be available for download just after the build.

Given that it was so difficult to get nix build to print the output path because indirect roots were safer, I have trouble understanding why remote builds on a remote store do not get the same rooting and safe access guarantees.

If I use a remote SSH store it means I have an account on the machine. Admins can shout if I use too many resources, or just delete my roots. I can also GC them using my account. Is it that different than a local machine ?

As a note, I think these roots should be updated on each access, so as to allow a basic LRU gc to happen if needed.

What is meant with updating a root? And what counts as an access?

Updating would mean bumping the timestamp of the symlink. An access is whatever operation that would create the symlink. So technically there is nothing to do, except create the symlink unconditionally.

I think you want the mounted-ssh-ng://?

Yes and no. I want the features, but I do not want to be restricted arbitrarily to SSH, or this SSH implementation.

Probably we want SSHStore to create GC roots via SSH's sftp protocol. That way, whether it's permitted is a configuration policy of the SSH server.

That makes sense, and also gives a hint that this issue is different from just the SSH store.

nixos-discourse commented 4 days ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-11-13-nix-team-meeting-minutes-194/56211/1

NixOS / nix

Design and enable creation of remote gc roots #11812