Flakes: Not including untracked files is confusing for unaware users.

[ElvishJerricco]

As flakes become more and more common, so too is the issue that untracked files aren't included when your flake is copied to the nix store. Personally, I consider this behavior a good thing, as it prevents me from accidentally copying result symlinks or nixos.qcow2 images into the store unnecessarily. But in my experience assisting users on Matrix and Discourse, it's a common error to not track a necessary file with git. This manifests with users being very confused why their shiny new code seems to not exist at all.

I'm not sure what the answer is here. Again, I personally prefer the current behavior, but clearly it's a problem for people who are unaware of the cause. I figured there should be an open issue to at least discuss the matter.

[silky]

Thanks for making this issue :)

I'd honestly be happy/expect to see this pop up when running adding some verbosity to the commands; it's what I tried to do to debug the issue, and was left confused when additional v's didn't yield me any particularly relevant hints ....

[Kha]

Relevant: https://github.com/NixOS/nix/pull/6858#issuecomment-1203872793 and following comments

[squalus]

For anyone else running into this, here's how to include untracked files: nix build path:///path/to/myflake#mypackage.

The entire flake directory gets copied to the Nix store in this case.

[Atemu]

With https://github.com/NixOS/nix/pull/6530, the problem of accidentally copying large untracked files should no longer exist.

I think it'd make sense to disable the unintuitive git integration then.

I'm all for integrating different tools with another but that should always be an optional feature. A build tool should not care about which VCS I'm using or what its state is; especially not by default. (Being able to fetch a build expression via git is a great feature though.)

[andrewhamon]

I too was pretty surprised at the tight git integration, and it makes certain things impossible, like conditionally importing an untracked file that may or may not exists (i.e. poor mans secret management, or a way to let folks customize a nix shell in a large shared repo).

[silky]

Depending on your setup @andrewhamon you might be interested in something like this - https://github.com/hasura/graphql-engine/blob/master/.envrc#L59 - to source a local .envrc file.

[bergkvist]

Even being familiar with this behavior, I find it cumbersome/unintuitive, and it is one of my biggest barriers to moving from traditional nix over to flakes.

What if instead of using the git staging area, there was a .flakeignore, similar to .dockerignore/.gitignore? I think this would be a lot more intuitive to new users, and could provide similar benefits in a more flexible way.

It might also be a step towards decoupling nix flakes and git - such that companies that use alternatives to git for version control would actually be able to adopt flakes as well.

[outfoxxed]

It is also extremely frustrating on nixos with direnv for development environments, where the project does not need or want the flake in version control. I have to remove the nix files from git, commit, and re-add them. The only reason for these flakes to exist is to set up my local dev env.

[tejing1]

Could nix by default use path:. instead of git+file:. when flake.nix is untracked and/or gitignored? It would at least streamline the local-only flake.nix usecase.

[ryanalta]

Being newish to nix and flakes I'm definitely in the user group of of 'unaware users'. One problem I run into is that I might want a .env in the development shell, but definitely don't want to track in git (it's in my .gitignore so I don't accidentally add, commit and push it).

I'm not sure what the best practice is for this with flakes, but my first guess of having an .env file in the directory with flake.nix appears to be wrong.

[mwotton]

I think the behaviour is right, but some breadcrumbs in the error message about it being potentially related to source control would be very much appreciated.

[1oglop1]

It would be great to have a warning or hint if the local file is untracked. It is my second day of learning nix and this behaviour is quite confusing. Because the problem with untracked files appears only inside the git repo which I have not initialized for small learning projects, then after progressing towards a more real project I kept getting the error No such file or directory and it took me some time to realize that it may be an untracked file.

[agraven]

Another factor that exacerbated the confusion for me is that this behavior also aplloes wheb you are in an ignored subfolder of a repo with a root higher up the tree. I sync my dotfiles by making my $HOME a git repo, so when i went into my projects directory and attempted to create a flake with nix flake init, it failed to build with an extremely unhelpful "No such file or directory" error that provided zero indication what the actual problem was.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/weird-missing-nix-store-when-running-a-home-manager-rebuild/37898/2

[fwcd]

My mental model is that Nix, like most other package managers, should operate independently from any particular source control and, if needed, implement its own ignore mechanism, as suggested by @bergkvist. Having to add/commit files to avoid having the packages silently be left in an inconsistent state has tripped me up several times now.

[eclairevoyant]

Having to commit files

Reminder that you do not need to commit anything. git add -N is sufficient if you don't want to stage the changes yet.

edit: thumbs down notwithstanding, my statement is still correct.

[codingCoffee]

I'm using an .env.nix file for storing variables which I do not want committed into my repo. And I'm importing this file into configuration.nix to use it. I've gitignored the .env.nix file. Now when running nixos-rebuild switch the nix store does not copy / symlink my .env.nix file into the nix store, thereby raising the error No such file or directory.

If I'm not using a git repo, things work flawlessly. Having everything in a git repo causes issues.

Any solution to this pattern of usage!?

[mightyiam]

I'm using an .env.nix file for storing variables which I do not want committed into my repo. And I'm importing this file into configuration.nix to use it.

This might be an impurity.

[tejing1]

Any solution to this pattern of usage!?

If you refer to path:.#output instead of .#output, I believe it should consider the directory itself, without regard to git. That has its own problems however, such as that the entire directory contents, including the .git directory, will be copied to the nix store, which could get rather inconvenient.

At a basic level, what you're trying to do is somewhat contrary to the intention of flakes. Flakes are supposed to contain and lock every input to a given result, without exception, and you're trying to carve out an exception.

[codingCoffee]

This might be an impurity.

Flakes are supposed to contain and lock every input to a given result, without exception, and you're trying to carve out an exception.

Duly noted. Will figure out a way to better manage secrets. sops-nix or agenix come to mind.

[DaniD3v]

Wanted to comment that I just spent like 3 hours wondering why my modules folder including my home-manager configuration didn't get copied into the nix store. The lack of documentation doesn't exactly help that. Maybe add a tip to the error message of nix not finding files in store paths.

Also this is kinda annoying for my secret management but the workaround with path:// mentioned above fixes this.

I think we should have a .flakeignore or something. This behavior is counterintuitive and can throw really confusing erros

[JeffLabonte]

Could nix by default use path:. instead of git+file:. when flake.nix is untracked and/or gitignored? It would at least streamline the local-only flake.nix usecase.

Well, using :path. instead of . fixed my problem. Thank you!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/path-nix-store-does-not-exist/55413/4