Rootless Nix: Path Rewriting (would like to contribute)

[timjrd]

Hi :) .

I have been using and playing with Nix for quite some time now, and as I find it a very clever piece of software, I am looking for a way to contribute.

Recently I had to install Nix on a university workstation where I couldn't create /nix. As I still wanted to use the official binary cache, I tried the PRoot workaround - even though it degrades performances - but it didn't work well (obscure bugs that don't happen on my normal installation like failing mallocs).

Anyway, it seems that a lot of users would be very interested in a Nix feature allowing a rootless installation compatible with the official binary cache.

@edolstra proposed the solution to rewrite store prefixes by patching pre-built binaries when exchanging paths between stores with different prefixes.

To be compatible with the existing pre-built packages, the canonical store prefix (as stated by Eelco in the above comment) could be /nix/store/.

If the user's store prefix is shorter - say /a/b/ - we can rewrite /nix/store/ to /a/b///////. Both strings must have the exact same length so we use multiple successive slashes as padding as allowed by POSIX.

If the user's store prefix is longer - say /home/user/nix/store/ - we can rewrite /nix/store/ to say /tmp/x0z3i/ and then symlink /tmp/x0z3i/ to /home/user/nix/store/. As stated in the Nix manual, having the Nix store directory being a symlink is a bad idea because a builder could dereference that symlink, potentially resulting in /home/user/nix/store/ being present in the result of local builds. The solution would be to rewrite prefixes after each build, from /home/user/nix/store/ to /tmp/x0z3i/////////// in our example.

When exporting or sending paths/closures, we can rewrite /tmp/x0z3i/ to /nix/store/, with as many slashes after the prefix as needed.

If I have the consent of the core Nix developers, I would be very happy to start working on this feature during my summer break (I'm a computer science student). I'm writing this issue because I would like to discuss the relevance and the details of this possible feature with the developers ahead of time.

So, what do you think of this ?

[dtzWill]

If your university machines have new enough kernels (mine don't :P) you can use --store=/path/to/store' and Nix will use some bind-mount and mount namespace goodness to act as if that path was /nix/store from the perspective of builders and such. Particularly neat is that you can usenix runto enter a shell where/nix/store` is that path... which has worked pretty well for me on my own machines (as compared to the similar edge-case problems you describe with using proot).

[timjrd]

I was not aware of the new mount namespace feature, I just tested it on my machine and it works fine, but I see two drawbacks here: the installer still needs root access so you have to compile from source which is not very convenient, and more importantly some distributions including Debian disables user namespaces by default.

[edolstra]

The first issue (the installer needing to be installed in /nix) could be solved by creating a statically linked Nix.

Other than that, using chroot stores via --store ... seems a lot cleaner than doing path rewriting, which as you point out only works for paths shorter than /nix/store. Using indirections like /tmp/x0z3i opens a can of security worms since such symlinks could be deleted (e.g. on reboot) and recreated by another user.

[veprbl]

Both proot and --store require that you need to always be in chroot (run proot or nix-shell in all interactive shells, in crontab, in batch job system, all other users on a multiuser system would not be able to use anything without it too). proot is a bottleneck for most parallel tasks. I also had some stack allocation problems with it too, but they were solved by https://github.com/NixOS/nixpkgs/commit/10cc95e154d32c096155649d54f6774dee50889e . The main problem with --store is that it requires CLONE_NEWUSER and CLONE_NEWNS. They are disabled and/or limited to root in many distributions (e.g. RHEL7, Debian). There is a nice utility unshare(1) that allows to poke these.

I still think that it would be nice to have this feature. One could set up a private hydra cache to build in /xxxxxxxxxxxxxxxxxxxxxxxxxx/store/ (as many "x" as they need) and then it would be possible to rewrite resulting NAR's to many other different locations. Perhaps the first step would be to just check how well rewriting works using some external tool without trying to modify nix itself.

[7c6f434c]

A limitation of path rewriting regardless of initial path length (and correspondingly a recommendation for choice of packages to test): if the path gets embedded into a JAR, or another ZIP-with-semantics, it can be more complicated to locate and rewrite it. I am not sure if this actually happens.

[cstrahan]

if the path gets embedded into a JAR, or another ZIP-with-semantics, it can be more complicated to locate and rewrite it

I do know that this happened with Bazel: https://github.com/NixOS/nixpkgs/commit/c69d90b88867b0d8ebcf526ce02f9b33763b1c84

[timjrd]

Following the suggestion of @veprbl I wrote a small tool to rewrite store prefixes in a binary stream (and in a NAR dump of a closure). I tested it on firefox and it seems to work well (firefox starts and seems to work normally). You can grab it here and test it that way :

make
./pack $(nix-build '<nixpkgs>' -A firefox) firefox.nar firefox.dep
./unpack /tmp/store firefox.nar firefox.dep 

[veprbl]

https://github.com/spack/spack supports binary package relocation:

• http://spack.readthedocs.io/en/latest/binary_caches.html#relocation
• https://github.com/spack/spack/blob/f5467957bca49ca612cfc32710ed2ca8a943583d/lib/spack/spack/relocate.py

They are even more optimistic and just use patchelf and install_name_tool on object files and plain text replace on text files.

[timjrd]

@veprbl Does this work for embedded constant strings like resources paths ? It seems to me that raw binary patching would be more robust and generic while being easier to implement (with the limitation of the path length of course).

@7c6f434c @cstrahan We could dive recursively into compressed archives and reconstruct them. Using something like libarchive it shouldn't be that hard. This could even be reused to improve Nix's automatic runtime dependencies detection ;-) .

[edolstra]

Compressed archives are not supported by Nix. They hide dependencies and don't work with hash rewriting. (I've been thinking about adding an option to enable hash rewriting for all builds, to catch builds that do hide dependencies.)

[copumpkin]

We've noodled in the past about pluggable rewriting though, to allow rewriting through more opaque formats. Might be fun to start exploring that, although I don't really see too much use for it outside of e.g., java JAR files (which will rarely contain paths anyway)

[shlevy]

:100: for pluggable rewriting (and pluggable reference registration...)

[edolstra]

Hell no. That means derivations will work on some Nix installations and fail in subtle ways on others, depending on their plugin configuration.

[shlevy]

I meant pluggable by the derivation. So a derivation can say "hey, if you need to rewrite paths, this is how you do it for me"

[copumpkin]

What @shlevy said 😄

[timjrd]

Anyone interested in testing this ?

Maybe I could properly rewrite this and add support for nested compressed archives with libarchive in a reasonably modular way to allow future extensions for other opaque formats. This could later be used in Nix to rewrite hashes in order to catch potentially hidden dependencies, or to rewrite store prefixes, or to improve automatic runtime dependencies detection...

Would that be useful ?

[Ekleog]

Related issue (same objective, different way to do it): https://github.com/NixOS/nix/issues/2107

[veprbl]

I was previously mentioning that "spack" supports some relocation. Today I found out that Homebrew has some limited support for relocation too. It seems like it works only in cases when it can be done without rewriting: https://github.com/Homebrew/brew/blob/60428d4dcb5a0b31fd48f69356282f4855b15404/Library/Homebrew/dev-cmd/bottle.rb#L357-L370 Makes me wonder how often it is the case.

[matthewbauer]

I have a PR for a statically built Nix here: https://github.com/NixOS/nixpkgs/pull/56281

It requires the user namespace feature to be available, like with nix run. But it makes it easier to do this in embedded environments.

[tomberek]

@timjrd Tested and have used nixrewrite several times. I think it could be sped up, but functions well.

[adrian-gierakowski]

I wonder if this could help with: https://github.com/NixOS/nix/issues/2925

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/idea-for-very-minimally-changing-nix-cache-nixos-org-to-allow-custom-store-paths-without-rebuilding-the-world/8153/4

[stale[bot]]

I marked this as stale due to inactivity. → More info

[veprbl]

Still important to me

[stale[bot]]

I marked this as stale due to inactivity. → More info

[veprbl]

Still important to me

[Ericson2314]

We have more path rewriting functionality from the CA stuff, but we should keep on the heat on distros to allow unprivileged users to sandbox processes too I think!

[veprbl]

The unprivileged user namespaces are enabled in RHEL8 by default, so many sites should get the feature in the following 5 years. The sandboxing is not a complete solution, though. It only works when already running in the sandbox, but not when things are to be shared with other systems (e.g. crontab, job submission, other users).

[nomeata]

It doesn’t rewrite nix paths, but still worth linking from here: https://github.com/DavHau/nix-portable simulate the /nix/store directory, and may work in some cases.

[cosmojg]

How's the progress on this?

I would deeply appreciate the ability to use the nixpkgs binary cache as an unprivileged user on locked-down systems like HPC clusters.

[Ericson2314]

@cosmojg do these HPC clusters not give you mount namespacing?

[veprbl]

@Ericson2314 Most of the computing centers still don't allow unprivileged mount namespaces. They just rely on suid bit to enable Singularity and nothing beyond that. Also, namespaces are not a nice solution for sharing software between users.

[Thesola10]

Anyone on macOS want to try out fakedir? It's a more thorough version of libredirect which also resolves symbolic links and parses+preloads absolute dependencies.

The running assumption is that the Nix binary itself doesn't have Nix Store dependencies, so it has to be built from source atop the Xcode SDK (or be quasi-static as defined in https://github.com/NixOS/nixpkgs/issues/214611)

It also likely clobbers libredirect itself, but that's likely not an issue since it only appears to be required for OpenSSH unit tests.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/built-with-custom-store-every-package-install-fails-with-error-path-is-not-in-the-nix-store/8143/4