search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.16k stars 1.47k forks source link

Copy local flakes to the store lazily #3121

Open edolstra opened 4 years ago

edolstra commented 4 years ago

Currently flakes are evaluated from the Nix store, so when using a local flake, it's first copied to the store. This means that

$ cd /path/to/nixpkgs
$ nix build .#hello

is a lot slower than the non-flake alternative

$ nix build -f . hello

Ideally, we would copy the flake to the store only when its outPath attribute is evaluated. However, we also need to ensure that it's not possible to access untracked files (i.e. we need to check every file against git ls-files).

yajo commented 10 months ago

is there any way to make Nix do the copying with hardlinks instead?

https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-auto-optimise-store does that with files already found in the nix store. So, it won't hardlink them with those in your CWD, but if it has to copy the dirty flake once and again, at least that won't mean triplicated storage. "Only" ~duplicated.

AleXoundOS commented 10 months ago

In newer version of Linux, you can reflink between vfs barriers as long as it's the same superblock. Btrfs for example supports this but ZFS does not.

On my machine (NixOS 23.05, linux 6.5.5) if source directory and /nix/store are on the same Btrfs filesystem, data is not shared among copies after running nix develop, thus taking space. btrfs filesystem du shows exclusive data for each copy.

It would be nice if it worked, lowering the severity of the problem at least for Btrfs users (even though metadata of such copies still takes up space).

Atemu commented 10 months ago

@AleXoundOS note that reflinking is something an application must choose to do; it is not the default. (IMO it really should be but that's a topic for a different project.)

I don't think it's worth spending time on this at this point though as it's much more important to push https://github.com/NixOS/nix/pull/6530 forward. After that's done, reflinking would be an extremely minor optimisation; probably not signficant enough to be worth considering.

nixos-discourse commented 8 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-nix-shells-without-polluting-repositories/37362/6

wmertens commented 7 months ago

Looks like this is not getting anywhere for the moment, how about allowing listing necessary files in the flake?

E.g. the attribute files would be an optional list of relative paths, and if it exists, only those paths get copied. Directories get recursively copied.

To make it more dev-friendly, the list could be globs instead of paths, so that you can include or exclude.

axelkar commented 5 months ago

If you don't use IFD, can you just first load all nix files imported by flake.nix into memory and eval them then?

Can someone tell me why you'd need to copy directories flakes reside in into the store if self is not used as a path? E.g. not used at all or just an attr like self.packages.x86_64-linux.default. It could be alright for self + /pkgs/foo.nix but that doesn't justify copying every single file, does it?

What about simple devshell flakes? Why are their directories copied into the store? It's just one nix file that doesn't use ./. or self.

I personally use path:///home/axel/.nix-config just so I don't have to commit every single time I try a new change. path:// doesn't copy anything into the store, right? Could there be a better way to bypass Git?

Edit: nevermind the last part, I thought path:// was going away. Anyways, could .git be copied or just read while evaluating? It ignores files from .gitignore and only contains committed files do it's perfectly pure. Can someone explain why it needs copying?

Edit 2: If something like a files attribute or .flakeignore were to be added could derivations use something like bind mounts or links to not have to copy anything to the store?

2xsaiko commented 5 months ago

I personally use path:///home/axel/.nix-config just so I don't have to commit every single time I try a new change. path:// doesn't copy anything into the store, right? Could there be a better way to bypass Git?

You don't have to commit every time you make a change with the git fetcher ("tree is dirty" is just a warning) and flakes are always copied to the store, even with path URLs. If that doesn't work for you, have a shell.nix until #6530 is complete.