Cleaning is broken for stack projects

[quetz]

Using project.shellFor results in multiple (one for each package) copies of whole project directory (including dist-newstyle) in nix store. It happens during clean-source-with-nix evaluation that is part of project {..} evaluation.

Is there any way to avoid it? Growing nix store by several GB on each nix-shell invocation is pretty harsh.

Looks like it is related to sources being needed for cabal setup (but I'm not sure here). In this case, is it possible to manually assemble pkgset and avoid this copying steps?

[hamishmack]

We have changed stuff recently in the way cabal components are clean, so this might be a bug.

Are you using explicit hs-source-dirs for each component in the .cabal file (not the default which is .)? That way the cleaning of the cabal component will know which directories are needed. This will also reduce the number of times haskell.nix will need to rebuild the components (since it will only rebuild if the source for the component changes). Perhaps use hs-source-dirs: src and the component name instead of src for each of the other components.

I would also recommend clean the project src directory before passing it to project. If you use git you can use the src = haskell-nix.haskellLib.cleanGit { name = "project-name"; src = ./.; } which will filter out anything not in the git index. This does mean you have to git add your source files before they will be seen by haskell.nix.

[quetz]

Are you using explicit hs-source-dirs for each component in the .cabal file (not the default which is .)? Yes.

Using cleanGit helps (in a sense that it does not include full project directory in a nix store) but still there's nix store path for each of project package with exactly same contents (cleanGit results of project directory). Also, noop changes in source files (adding blank line) result in creating fresh new set of those nix store paths on each nix-shell invocation (no project building even required).

Why is it neeed to have full copy of source tree in nix store upon entering nix shell?

[michaelpj]

The source for each package should be filtered further to just be the source of that package.

But it is a bit odd that you have the any source being copied in for each nix-shell invocation. That shouldn't be happening - it only builds the dependencies anyway!

[ocharles]

Is it surprising that the source is in the store? It's not surprising to me - it has to run cabal configure, and ot can only do that on things in the store, no? (This only applies to non-materialised projects)

[michaelpj]

For the configure step we filter it down to only contain cabal.project and the .cabal files. So a) it shouldn't change on changes to source files, and b) it should be really small.

[michaelpj]

Although the way that source filtering step got done was changed recently. @hamishmack do you think we could conceivably be copying the pre-filtering source as well?

[ocharles]

I'm pretty certain this is a regression that came in here:

https://github.com/input-output-hk/haskell.nix/pull/843/files#diff-5176cec122f219c11fae32aa032f2a89ccb02f417f682e169cb52f5b64ec6307R43

This change means that every package now has includeSiblings = true, which means that it ends up having src set to the entire project source tree, rather than just its relative sub-directory. I see @michaelpj said

I do find it tempting to say you should just Not Do That...

and I'm inclined to agree.

[quetz]

What gets created on every nix-shell invocation on noop changes in sources (added empty line):

- <project>-root-<package directory name> (for each package in project)
- <project>-root-<package dir>-lib-<package name>-root (for package that gets its source file modified)
- ... -config.drv (for each executable / library for each package in project)
- ... -ghc-<ghc version>-env.drv (for each executable / library for each package in project)
- ... .drv (for each executable / library for each package in project)

Looks like it all boils down to -root-<dir> store paths that contain whole (filtered) source. And even noop changes are detected by nix producing different hash each time. And then everything else refers to this new path producing different hash also, and so on...

[ocharles]

Nix shouldn't (can't) produce a different hash time, so you must have something changing. If you can find the path (I used inotifytools to watch /nix/store), you can do a recursive diff on them to see what's changing. The hash of a source path is the product of its contents hash + name, and this is stable (if those inputs are stable).

@quetz I suggest reverting the line I linked to in #843, though. Just to reiterate, that's changing true to false here:

https://github.com/input-output-hk/haskell.nix/blob/22ad57c7e78a6bf409738efdd87a2c23d13bfabb/lib/import-and-filter-project.nix#L55

[quetz]

Nix shouldn't (can't) produce a different hash time, so you must have something changing.

Yeah, but source code is changing (I insert empty line, so this is noop change). It should lead to different hash and if everything else refers to this new source it gets new hash too...

@quetz I suggest reverting the line I linked to in #843, though. Just to reiterate, that's changing true to false here

Will experiment with that, thanks.

[quetz]

@quetz I suggest reverting the line I linked to in #843, though. Just to reiterate, that's changing true to false here:

https://github.com/input-output-hk/haskell.nix/blob/22ad57c7e78a6bf409738efdd87a2c23d13bfabb/lib/import-and-filter-project.nix#L55

It is much better with this change. Some store paths are still created on each nix-shell invocations, but source code gets reduced to only one component that is changed.

But still, is it possible to get shell in haskell.nix behave as advertised in nix?

The command nix-shell will build the dependencies of the specified derivation, but not the derivation itself.

I want to be dropped into shell where all dependencies of my project are built and registered, but project itself is not touched (do not configure my packages, do not make derivations that can be used to build these packages).

[hamishmack]

How can I reproduce this issue?

[ocharles]

@hamishmack With the following directory structure:

- foo
- bar
|-- bar.cabal
|-- src/.../.hs
- shell.nix
- cabal.project

That is, a directory structure where your .cabal packages are in sub directories. Now make foo a very large file that's not referenced from any .cabal file (i.e., foo is not part of the Haskell build) and observe that it gets copied into the Nix store. If you make multiple .cabal subdirectories, notice that the .drv for building the library component will have a src that contains foo (so foo is being copied for all projects).

[hamishmack]

I can't seem reproduce it that way. See https://github.com/hamishmack/test-1013.

git clone https://github.com/hamishmack/test-1013.git
cd test-1013
nix-shell --run 'echo ok'
echo test >> foo                                                                 
nix-shell --run 'echo ok'

Seems to work as expected for me. No derivations are rebuilt. It works with or without the cleanGit call.

One thing that would not work is if something was triggering the source to be copied to the store before the haskell.nix project function sees it. For instance if the example project was in subDir and we used:

src = haskellLib.cleanGit { src = ./.; } + "/subDir";

Nix will be forced to copy the all of haskellLib.cleanGit { src = ./.; } to the store in order to make a path for src. To avoid this we would have to use:

src = haskellLib.cleanSourceWith { src = haskellLib.cleanGit { src = ./.; }; subDir = "subDir"; };

[ocharles]

@hamishmack I'm not able to check that out right, now but I'd suggest instantiating the derivation for bar.components.library (nix-instantiate -A bar.components.library) and looking at what the input source is.

This isn't about things being rebuild, it's about a huge amount of stuff being copied into the store. It's naturally not noticable with a small test case, which is why I suggest making foo big (e.g., use dd to make a multiple GB file).

In all of my tests with master at the moment, some components have their src point to the entire directory tree, but then have a prePatch phase that just copies a subset during the build. You wouldn't see this in nix-shell, because in a shell no components are actually built.

[ocharles]

Ok, I had a chance to look at indeed hsPkgs.bar.components.exes.bar doesn't include the foo file from the source root. So something else is up. I'll try and get a minimal repro

[phile314]

I think I have also encountered this error just now. We are using callStackToNix/mkStackPkgSet. On Hydra, the builds are working with current master of haskell.nix. However, when using a local checkout it breaks because we have some sockets in the same directory which trip up nix:

error: file 'REPO_ROOT/temp_files/mydata/mysock.sock' has an unsupported type

This socket file is being filtered out using the nixpkgs cleanSourceWith function, so I have no idea why it should be copied. If I put the src in quotes, it works again.

Broken:

  stack-nix-build = with tiko.pkgs-2003.haskell-nix; callStackToNix {
    src = cleanSourceWith { src = ./.; ...};
  };

Works:

  stack-nix-build = with tiko.pkgs-2003.haskell-nix; callStackToNix {
    src = "${ cleanSourceWith { src = ./.; ...} }";
  };

[hamishmack]

I did not realise you were using callStackToNix and mkStackPkgSet directly. It might be easier to use stackProject' or stackProject. If you want to stick to callStackToNix and mkStackPkgSet, make sure you are calling importAndFilterProject as well, like stackProject' does.

[quetz]

I can't seem reproduce it that way. See https://github.com/hamishmack/test-1013.

Try this PR - https://github.com/hamishmack/test-1013/pull/1

[hamishmack]

Doh. Still seems to work correctly.

[hamishmack]

I wonder if it might only affect stack projects?

[hamishmack]

Switching to a stack based project does it! https://github.com/hamishmack/test-1013/commit/f003490548313d26f3d7cc7de763abf3ed72f4a2

We must be doing something wrong in handling of stack projects.

[hamishmack]

I think the problem is that we never got around to adding code to clean the source to callStackToNix.

In callCabalProjectToNix we:

• identify the sub directory relative to the original source
• filter just the files that are needed (.cabal and package.yaml)
• make a temp dir with just those files
• cd to the sub dir if necessary

I think we can do something similar in callStackToNix, but we might need to parse the stack.yaml file to see if it is a path to a file (since we will need to include that file too).

[quetz]

Doh. Still seems to work correctly.

How do you test it? If I take your code and then invoke nix-shell several times while changing source code in between I get these store paths created:

/nix/store/0612xilw8k2a3ardnd1gczkahs4rg25g-bar-exe-bar-0.1.0.0.drv
/nix/store/1f4h280516wb0lqzb1glrdzsqfl0pd33-test-1013-root-lib
/nix/store/1wa6k73zk6wj63p32m9xy7qaffiq3kl1-test-1013-root-bar-exe-bar-root
/nix/store/6j3vmyy9aymzrd7idl5ja1760lvvsfa2-test-1013-root-bar
/nix/store/7i3ynrdmx6ni44n7c33jy9a48slxzzhp-test-1013-root-bar
/nix/store/ddlbk32vhq3mjs391iwr4mx462li7s0n-bar-exe-bar-0.1.0.0.drv
/nix/store/dqjih4jz2zy4nplcliyxgml607hz2ky9-git-ls-files.drv
/nix/store/g5xkc116b3xki8c9si73xl0p2x9wnkaq-test-1013-root-bar-exe-bar-root
/nix/store/hfcxlb9g2karm36bz3i6cl0icy5qfq5s-git-ls-files
/nix/store/id0ihnlly9mar1hp2cf8pla92dmahbcq-test-1013-root-lib
/nix/store/j439sgd2s3wb9y9ynfy3pw6xqy5i7c0d-test-1013-root-bar
/nix/store/jqwywdn41nzs63p7sslz3758dkfp5igb-test-1013-root-lib
/nix/store/r0jqijacdswkr5nv8bfgcjf756cl05wj-bar-exe-bar-0.1.0.0.drv
/nix/store/r6918pjdgrr2xiw0jr89v95jjz0mxq5z-test-1013-root-lib
/nix/store/sn2n92jaq5gx93n18s3z7vgdlzsjf64z-test-1013-gitFiles
/nix/store/w4wn1k1l79msadgk0wwwig6ab7a2v4vs-test-1013-root-bar-exe-bar-root
/nix/store/yjnj3wg1zfzzzya7a4gazl0ihb83cpvd-ghc-shell-for-bar.drv
/nix/store/zpsc81kb9zi4wak2xj9pw4x4kb828kp5-test-1013-root-bar

I do not ever invoke nix-build or cabal build from within shell, just wait till I get dropped into shell.

[hamishmack]

How do you test it?

I am testing with:

hamish@Hamishs-MacBook-Air test-1013 % ls -l /nix/store | wc -l                                                         
   19501
hamish@Hamishs-MacBook-Air test-1013 % echo test >> foo                                                                               
hamish@Hamishs-MacBook-Air test-1013 % NIV_OVERRIDE_haskell_nix=/Users/hamish/iohk/haskell.nix nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok
hamish@Hamishs-MacBook-Air test-1013 % ls -l /nix/store | wc -l                                                         
   19501

[ocharles]

It's definitely not Stack only as this effects us, but we don't use stack! I have a to-do to try and investigate more

[hamishmack]

Ah think I understand now. Its not foo that is the problem. The question is why does this copy to the store:

hamish@Hamishs-MacBook-Air test-1013 % ls -l /nix/store | wc -l                                                         
   19501
hamish@Hamishs-MacBook-Air test-1013 % echo >> bar/Main.hs     
hamish@Hamishs-MacBook-Air test-1013 % NIV_OVERRIDE_haskell_nix=/Users/hamish/iohk/haskell.nix nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok
hamish@Hamishs-MacBook-Air test-1013 % ls -l /nix/store | wc -l                                                         
   19503

[hamishmack]

That is odd.

[hamishmack]

Hmm maybe not, probably just .drv for bar exe and its setup drv.

[quetz]

That's not only .drv, but source code is copied into store upon entering nix-shell (see, my previous message with example of nix store paths that are actually added).

[hamishmack]

That's not only .drv, but source code is copied into store upon entering nix-shell (see, my previous message with example of nix store paths that are actually added).

I suspect there might be something different in the nix configuration. It is very odd that even git-ls-files is being recalculated. That should only happen when files in the .git dir changes. I wonder if builtins.filterSource or builtins.hasContext are not working as expected for some reason.

I went back to cabalProject and I tested on linux (I have pushed the code I used to the repo https://github.com/hamishmack/test-1013/tree/fd3d095ab6808ee817cca52d105e0ed2e71d4b0f).

Touching the source for lib and bar triggers build of lib as expected (since only bar is listed in the shellFor packages arg). No sign of git-ls-files though.

Then I checked just touching bar and it did copy the bar source to the store, but it was filtered correctly and foo is not included.

Here is what I tried:

[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > a.txt

[hamish@x86_64-builder:~/iohk/test-1013]$ echo >> lib/src/MyLib.hs 

[hamish@x86_64-builder:~/iohk/test-1013]$ echo >> bar/Main.hs 

[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
warning: don't know how to open Nix store 'iohk.cachix.org'
these derivations will be built:
  /nix/store/s6jhj4vx6lim0as0rb0hy08lglf7k2j8-lib-lib-lib-0.1.0.0.drv
  /nix/store/xj3jc30iigcb5rd196cvwcpk86r0g05g-lib-lib-lib-0.1.0.0-haddock.drv
  /nix/store/5zkzdi0100ixbp1wzb9i0j1gvfq1r0z8-hoogle-local-0.1.drv
  /nix/store/ws9k1bkhs2i8cv0f1isam5gsxf1lqn2y-ghc-shell-for-bar-config.drv
  /nix/store/bd8mz39qyinh5fqhjr3kn4ppi75r7iqp-ghc-shell-for-bar-ghc-8.10.3-env.drv
building '/nix/store/s6jhj4vx6lim0as0rb0hy08lglf7k2j8-lib-lib-lib-0.1.0.0.drv'...
unpacking sources
unpacking source archive /nix/store/yznvkb5hx5m5n761j5vmhd7d1ngwbsbi-test-1013-root-lib-lib-lib-root
source root is test-1013-root-lib-lib-lib-root
patching sources
configuring
Configure flags:
--prefix=/nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0 lib:lib --package-db=clear --package-db=/nix/store/azgvznk2r2jv3irsqi00w4xjl7kjysqk-lib-lib-lib-0.1.0.0-config/lib/ghc-8.10.3/package.conf.d --exact-configuration --dependency=rts=rts --dependency=ghc-heap=ghc-heap-8.10.3 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=integer-gmp=integer-gmp-1.0.3.0 --dependency=base=base-4.14.1.0 --dependency=deepseq=deepseq-1.4.4.0 --dependency=array=array-0.5.4.0 --dependency=ghc-boot-th=ghc-boot-th-8.10.3 --dependency=pretty=pretty-1.1.3.6 --dependency=template-haskell=template-haskell-2.16.0.0 --dependency=ghc-boot=ghc-boot-8.10.3 --dependency=ghc=ghc-8.10.3 --dependency=Cabal=Cabal-3.2.1.0 --dependency=array=array-0.5.4.0 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.2.1 --dependency=directory=directory-1.3.6.0 --dependency=filepath=filepath-1.4.2.1 --dependency=ghc-boot=ghc-boot-8.10.3 --dependency=ghc-compact=ghc-compact-0.1.0.0 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=hpc=hpc-0.6.1.0 --dependency=mtl=mtl-2.2.2 --dependency=parsec=parsec-3.1.14.0 --dependency=process=process-1.6.9.0 --dependency=text=text-1.2.4.1 --dependency=time=time-1.9.3 --dependency=transformers=transformers-0.5.6.2 --dependency=unix=unix-2.7.2.2 --dependency=xhtml=xhtml-3000.2.2.1 --dependency=terminfo=terminfo-0.4.1.4 --dependency=text=text-1.2.4.1 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.2.1 --with-ghc=ghc --with-ghc-pkg=ghc-pkg --with-hsc2hs=hsc2hs --with-gcc=cc --with-ld=ld.gold --ghc-option=-optl-fuse-ld=gold --ld-option=-fuse-ld=gold --with-ar=ar --with-strip=strip --disable-executable-stripping --disable-library-stripping --disable-library-profiling --disable-executable-profiling --enable-static --enable-shared --disable-coverage --enable-library-for-ghci --enable-split-sections
Configuring library for lib-0.1.0.0..
Warning: The flag --disable-executable-profiling is deprecated. Please use
--disable-profiling instead.
building
Preprocessing library for lib-0.1.0.0..
Building library for lib-0.1.0.0..
[1 of 1] Compiling MyLib            ( MyLib.hs, dist/build/MyLib.o, dist/build/MyLib.dyn_o )
installing
Installing library in /nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0/lib/x86_64-linux-ghc-8.10.3/lib-0.1.0.0-1K833QoZdbwhP5y6tYtVq
post-installation fixup
patching script interpreter paths in /nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0
checking for references to /build/ in /nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0...
patchelf: wrong ELF type
building '/nix/store/ws9k1bkhs2i8cv0f1isam5gsxf1lqn2y-ghc-shell-for-bar-config.drv'...
building '/nix/store/xj3jc30iigcb5rd196cvwcpk86r0g05g-lib-lib-lib-0.1.0.0-haddock.drv'...
unpacking sources
unpacking source archive /nix/store/yznvkb5hx5m5n761j5vmhd7d1ngwbsbi-test-1013-root-lib-lib-lib-root
source root is test-1013-root-lib-lib-lib-root
patching sources
configuring
Configure flags:
--prefix=/nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0 lib:lib --package-db=clear --package-db=/nix/store/xcvjsvkl6vxh7y6sk7q2vs8mpwhmy498-lib-lib-lib-0.1.0.0-haddock-config/lib/ghc-8.10.3/package.conf.d --exact-configuration --dependency=rts=rts --dependency=ghc-heap=ghc-heap-8.10.3 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=integer-gmp=integer-gmp-1.0.3.0 --dependency=base=base-4.14.1.0 --dependency=deepseq=deepseq-1.4.4.0 --dependency=array=array-0.5.4.0 --dependency=ghc-boot-th=ghc-boot-th-8.10.3 --dependency=pretty=pretty-1.1.3.6 --dependency=template-haskell=template-haskell-2.16.0.0 --dependency=ghc-boot=ghc-boot-8.10.3 --dependency=ghc=ghc-8.10.3 --dependency=Cabal=Cabal-3.2.1.0 --dependency=array=array-0.5.4.0 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.2.1 --dependency=directory=directory-1.3.6.0 --dependency=filepath=filepath-1.4.2.1 --dependency=ghc-boot=ghc-boot-8.10.3 --dependency=ghc-compact=ghc-compact-0.1.0.0 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=hpc=hpc-0.6.1.0 --dependency=mtl=mtl-2.2.2 --dependency=parsec=parsec-3.1.14.0 --dependency=process=process-1.6.9.0 --dependency=text=text-1.2.4.1 --dependency=time=time-1.9.3 --dependency=transformers=transformers-0.5.6.2 --dependency=unix=unix-2.7.2.2 --dependency=xhtml=xhtml-3000.2.2.1 --dependency=terminfo=terminfo-0.4.1.4 --dependency=text=text-1.2.4.1 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.2.1 --with-ghc=ghc --with-ghc-pkg=ghc-pkg --with-hsc2hs=hsc2hs --with-gcc=cc --with-ld=ld.gold --ghc-option=-optl-fuse-ld=gold --ld-option=-fuse-ld=gold --with-ar=ar --with-strip=strip --disable-executable-stripping --disable-library-stripping --disable-library-profiling --disable-executable-profiling --enable-static --enable-shared --disable-coverage --enable-library-for-ghci --enable-split-sections --docdir=/nix/store/hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc/share/doc/lib
Configuring library for lib-0.1.0.0..
Warning: The flag --disable-executable-profiling is deprecated. Please use
--disable-profiling instead.
building
Preprocessing library for lib-0.1.0.0..
Running Haddock on library for lib-0.1.0.0..
Warning: The documentation for the following packages are not installed. No
links will be generated to these packages: base-4.14.1.0, ghc-prim-0.6.1,
integer-gmp-1.0.3.0
Warning: --source-* options are ignored when --hyperlinked-source is enabled.
Haddock coverage:
   0% (  0 /  2) in 'MyLib'
  Missing documentation for:
    Module header
    someFunc (MyLib.hs:3)
Warning: MyLib: could not find link destinations for:
    IO
Documentation created: dist/doc/html/lib/index.html, dist/doc/html/lib/lib.txt
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/72hy3lxz34196111fr911yvzy255xiic-lib-lib-lib-0.1.0.0-haddock
strip is /nix/store/rqfgki7ck1bxqhk3hd7wziqhahjadfbj-binutils-2.35.1/bin/strip
patching script interpreter paths in /nix/store/72hy3lxz34196111fr911yvzy255xiic-lib-lib-lib-0.1.0.0-haddock
checking for references to /build/ in /nix/store/72hy3lxz34196111fr911yvzy255xiic-lib-lib-lib-0.1.0.0-haddock...
shrinking RPATHs of ELF executables and libraries in /nix/store/hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc
strip is /nix/store/rqfgki7ck1bxqhk3hd7wziqhahjadfbj-binutils-2.35.1/bin/strip
patching script interpreter paths in /nix/store/hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc
checking for references to /build/ in /nix/store/hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc...
building '/nix/store/5zkzdi0100ixbp1wzb9i0j1gvfq1r0z8-hoogle-local-0.1.drv'...
importing builtin packages
importing other packages
building hoogle database
Starting generate
[3/34] base... 1.11s 
[11/34] ghc... 3.06s      
[28/34] template-haskell... 0.22s
[34/34] xhtml... 0.06s       
Found 28 warnings when processing items

Reordering items... 0.03s
Writing tags... 0.18s
Writing names... 0.08s
Writing types... 0.90s
Took 8.63s
building haddock index
finishing up
building '/nix/store/bd8mz39qyinh5fqhjr3kn4ppi75r7iqp-ghc-shell-for-bar-ghc-8.10.3-env.drv'...
ok

[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > b.txt

[hamish@x86_64-builder:~/iohk/test-1013]$ diff a.txt b.txt 
1c1
< total 23713556
---
> total 23713644
14089a14090
> -r--r--r--   1 root    root         4369 Jan  1  1970 1jpr0ph3pxdmx6c14hhgp9w4fj0swwzc-bar-exe-bar-0.1.0.0-config.drv
53405a53407
> -r--r--r--   1 root    root         3321 Jan  1  1970 5zkzdi0100ixbp1wzb9i0j1gvfq1r0z8-hoogle-local-0.1.drv
63117a63120
> dr-xr-xr-x   3 root    root            5 Jan  1  1970 72hy3lxz34196111fr911yvzy255xiic-lib-lib-lib-0.1.0.0-haddock
101504a101508
> dr-xr-xr-x   4 root    root            4 Jan  1  1970 bc29hk4q7kw7yy08y2vkcvwn4759bab1-hoogle-local-0.1
101826a101831
> -r--r--r--   1 root    root         5124 Jan  1  1970 bd8mz39qyinh5fqhjr3kn4ppi75r7iqp-ghc-shell-for-bar-ghc-8.10.3-env.drv
143533a143539
> dr-xr-xr-x   3 root    root            6 Jan  1  1970 h2cxg5cfydy10i9b11h0hgipgkvrc5ng-ghc-shell-for-bar-config
147608a147615
> dr-xr-xr-x   3 root    root            3 Jan  1  1970 hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc
150550a150558
> -r--r--r--   1 root    root         2253 Jan  1  1970 hvdq530187jxdlmgl0nkniya3gw3pz4b-ghc-shell-for-bar.drv
179988a179997
> dr-xr-xr-x   3 root    root            3 Jan  1  1970 l58h0jxa3z1q8bwn9sv0dmm1kr3hgcjf-test-1013-root-bar-exe-bar-root
193515a193525
> dr-xr-xr-x   8 root    root            8 Jan  1  1970 mm8wn9nnzyz5gvl0ds4fjy232ir0w5dn-ghc-shell-for-bar-ghc-8.10.3-env
197350a197361
> -r--r--r--   1 root    root         4969 Jan  1  1970 n2xf830psj3vfpdqa3awx1vnq3z7g9vn-bar-exe-bar-0.1.0.0-ghc-8.10.3-env.drv
215420a215432
> dr-xr-xr-x   6 root    root            7 Jan  1  1970 q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0
234125a234138
> -r--r--r--   1 root    root         4952 Jan  1  1970 s6jhj4vx6lim0as0rb0hy08lglf7k2j8-lib-lib-lib-0.1.0.0.drv
257789a257803
> -r--r--r--   1 root    root         3575 Jan  1  1970 ws9k1bkhs2i8cv0f1isam5gsxf1lqn2y-ghc-shell-for-bar-config.drv
264357a264372
> -r--r--r--   1 root    root         5943 Jan  1  1970 xj3jc30iigcb5rd196cvwcpk86r0g05g-lib-lib-lib-0.1.0.0-haddock.drv
270934a270950
> -r--r--r--   1 root    root         4048 Jan  1  1970 y92vdq5rm57djmrj7x630z7lqd9ibhhc-bar-exe-bar-0.1.0.0.drv
277283a277300
> dr-xr-xr-x   3 root    root            3 Jan  1  1970 yznvkb5hx5m5n761j5vmhd7d1ngwbsbi-test-1013-root-lib-lib-lib-root

[hamish@x86_64-builder:~/iohk/test-1013]$ echo >> bar/Main.hs 

[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok

[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > c.txt

[hamish@x86_64-builder:~/iohk/test-1013]$ diff b.txt c.txt 
1c1
< total 23713644
---
> total 23713649
172835a172836
> -r--r--r--   1 root    root         4048 Jan  1  1970 kbzly2g84v444rcdgrzqbgs76f2zv7ds-bar-exe-bar-0.1.0.0.drv
266254a266256
> dr-xr-xr-x   3 root    root            3 Jan  1  1970 xqrljn0d5n7ffs1xxpph83sn5kpf0gaa-test-1013-root-bar-exe-bar-root

[hamish@x86_64-builder:~/iohk/test-1013]$ tree /nix/store/xqrljn0d5n7ffs1xxpph83sn5kpf0gaa-test-1013-root-bar-exe-bar-root
/nix/store/xqrljn0d5n7ffs1xxpph83sn5kpf0gaa-test-1013-root-bar-exe-bar-root
└── bar
    ├── bar.cabal
    ├── CHANGELOG.md
    ├── Main.hs
    └── Setup.hs

1 directory, 4 files

[hamishmack]

Just on the off chance I checked that git add works ok too (it does only the git file list was updated):

[hamish@x86_64-builder:~/iohk/test-1013]$ touch new-file

[hamish@x86_64-builder:~/iohk/test-1013]$ git add new-file

[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --pure --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
warning: don't know how to open Nix store 'iohk.cachix.org'
building '/nix/store/y3hvls2dnm8ljkify5mlw1k3nfysaj3v-git-ls-files.drv'...
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok

[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > d.txt

[hamish@x86_64-builder:~/iohk/test-1013]$ diff c.txt d.txt 
1c1
< total 23713649
---
> total 23713659
38444a38445
> -r--r--r--   1 root    root          245 Jan  1  1970 49x5nzzyx771sjspxd1vilg13rly5hqd-git-ls-files
263487a263489
> dr-xr-xr-x   3 root    root            3 Jan  1  1970 xfx052kcbcjl3zx3z8l8yj32whalp2ck-test-1013-gitFiles
269368a269371
> -r--r--r--   1 root    root         1809 Jan  1  1970 y3hvls2dnm8ljkify5mlw1k3nfysaj3v-git-ls-files.drv

[quetz]

I'm using version with project.shellFor (fffc08c20e6b30e139e27463dfa9a471bc0f00c7). After 3 nix-shell invocations and echo >> lib/src/MyLib.hs in between I get these store paths (grouped by name):

    3 bar-exe-bar-0.1.0.0-config.drv
    3 bar-exe-bar-0.1.0.0-ghc-8.10.3-env.drv
    3 bar-exe-bar-0.1.0.0.drv
    3 ghc-shell-for-bar-config
    3 ghc-shell-for-bar-config.drv
    3 ghc-shell-for-bar-ghc-8.10.3-env
    3 ghc-shell-for-bar-ghc-8.10.3-env.drv
    3 ghc-shell-for-bar.drv
    3 hoogle-local-0.1
    3 hoogle-local-0.1.drv
    3 lib-lib-lib-0.1.0.0
    3 lib-lib-lib-0.1.0.0-haddock
    3 lib-lib-lib-0.1.0.0-haddock-doc
    3 lib-lib-lib-0.1.0.0-haddock.drv
    3 lib-lib-lib-0.1.0.0.drv
    3 test-1013-root-bar
    3 test-1013-root-lib

So it's clear that on each nix-shell invocation a new set of these paths is generated. My understanding is that if project is not built and nix-shell is supposed to configure and prepare only dependencies of the project (and not project itself) it should not add extra stuff to store if dependencies are not changed.

[hamishmack]

So it's clear that on each nix-shell invocation a new set of these paths is generated. My understanding is that if project is not built and nix-shell is supposed to configure and prepare only dependencies of the project (and not project itself) it should not add extra stuff to store if dependencies are not changed.

The packages argument passed to shellFor specifies which packages not to build (since we are going to build them in the shell). In this case it is packages = ps: [ ps.bar ];. Since lib is a build dependency of bar it is included in the shell's ghc-pkg list and needs to be rebuilt when the source changes.

Changing the packages argument in shell.nix to packages = ps: [ ps.bar ps.lib ]; avoids the rebuild. Not passing the packages arg at all should also work since the default is to include all the "local" packages in the cabal.project file (in this case bar and lib):

[hamish@x86_64-builder:~/iohk/test-1013]$ cat shell.nix
let project = import ./.;
in project.shellFor { packages = ps: [ ps.bar ps.lib ]; exactDeps = true; }

[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > a.txt

[hamish@x86_64-builder:~/iohk/test-1013]$ echo >> lib/src/MyLib.hs

[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok

[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > b.txt

[hamish@x86_64-builder:~/iohk/test-1013]$ diff a.txt b.txt
1c1
< total 23871193
---
> total 23871223
54229a54230
> -r--r--r--   1 root    root         4048 Jan  1  1970 5yap5i3s3hv1rclb6f5mmvcidpgxrw6x-bar-exe-bar-0.1.0.0.drv
68778a68780
> -r--r--r--   1 root    root         4969 Jan  1  1970 7i4739h8s5x3lcxlbzajaqkzi6fpia14-bar-exe-bar-0.1.0.0-ghc-8.10.3-env.drv
159109a159112
> -r--r--r--   1 root    root         4952 Jan  1  1970 idzc3bkaj9l7sbgzn9qwbg6wrs29bhv5-lib-lib-lib-0.1.0.0.drv
196247a196251
> dr-xr-xr-x   3 root    root            3 Jan  1  1970 mgndx5v702i2rbj86v31nqdy0zwg11fs-test-1013-root-lib-lib-lib-root
237372a237377
> -r--r--r--   1 root    root         4369 Jan  1  1970 rzpqgq6bkqhpsn3q0zmx0g8nsfxdczrj-bar-exe-bar-0.1.0.0-config.drv

[hamish@x86_64-builder:~/iohk/test-1013]$ tree mgndx5v702i2rbj86v31nqdy0zwg11fs-test-1013-root-lib-lib-lib-root
mgndx5v702i2rbj86v31nqdy0zwg11fs-test-1013-root-lib-lib-lib-root [error opening dir]

0 directories, 0 files

[hamishmack]

It's definitely not Stack only as this effects us, but we don't use stack! I have a to-do to try and investigate more

There is a definitely a problem with callStackToNix (and there for stackProject and stackProject'). There might also be a problem with cabal projects, but so far I can't find it.

[hamishmack]

@phile314 It does look like your issue is caused by the same problem with callStackToNix not filtering like callCabalProjectToNix. I am curious to know if you get the same issue if you build your project with cabalProject { src = cleanSourceWith { src = ./.; ...}; }.

I did a test where I deleted the foo file in the test-1013 repo and replaced it with a socket (created by running nc -l -U foo in another terminal). With stackProject in the default.nix I was able to reproduce the error, but changing it back to cabalProject made it go away (log bellow).

The work around (src = "${ cleanSourceWith { src = ./.; ...} }";) is definitely going to copy everything in your git repo to the store on changes to any file. It does not matter too much since it callStackToNix would currently do that anyway, but don't use it with cabalProject.

[hamish@x86_64-builder:~/iohk/test-1013]$ cat default.nix 
let
  sources = import nix/sources.nix {};
  haskellNix = import sources."haskell.nix" {};
  pkgs = import haskellNix.sources.nixpkgs-unstable haskellNix.nixpkgsArgs;
in pkgs.haskell-nix.stackProject {
    # compiler-nix-name = "ghc8103";
    # src = ./.;
    src = pkgs.haskell-nix.haskellLib.cleanGit { name = "test-1013"; src = ./.; };
  }

[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: To make project.stack-nix for test-1013 a fixed-output derivation but not materialized, set `stack-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.stack-nix for test-1013 entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
error: file '/home/hamish/iohk/test-1013/foo' has an unsupported type
(use '--show-trace' to show detailed location information)

[hamish@x86_64-builder:~/iohk/test-1013]$ vim default.nix 

[hamish@x86_64-builder:~/iohk/test-1013]$ cat default.nix 
let
  sources = import nix/sources.nix {};
  haskellNix = import sources."haskell.nix" {};
  pkgs = import haskellNix.sources.nixpkgs-unstable haskellNix.nixpkgsArgs;
in pkgs.haskell-nix.cabalProject {
    compiler-nix-name = "ghc8103";
    # src = ./.;
    src = pkgs.haskell-nix.haskellLib.cleanGit { name = "test-1013"; src = ./.; };
  }

[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok

[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l foo.sock 
srwxr-xr-x 1 hamish users 0 Feb  4 04:05 foo.sock

[phile314]

Calling callStackToNix and mkPkgSet manually:

nix-build nix/tiko.nix -A tiko-apps.tiko-systems
trace: To make project.stack-nix for tikoHsSrc a fixed-output derivation but not materialized, set `stack-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.stack-nix for tikoHsSrc entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
building '/nix/store/7spbpxyhj42582r399lxq6nrm2z7vr10-tikoHsSrc-stack-to-nix-pkgs.drv'...
substituteStream(): WARNING: pattern '/nix/store/nw2igscbaa9f26irawxaikdxxcqz695v-tikoHsSrc' doesn't match anything in file '/nix/store/vqp59h05ql6ykpf831qmfqilf2r50gr1-tikoHsSrc-stack-to-nix-pkgs/default.nix'
substituteStream(): WARNING: pattern '/nix/store/nw2igscbaa9f26irawxaikdxxcqz695v-tikoHsSrc' doesn't match anything in file '/nix/store/vqp59h05ql6ykpf831qmfqilf2r50gr1-tikoHsSrc-stack-to-nix-pkgs/pkgs.nix'
    unpacking 'https://github.com/input-output-hk/stackage.nix/archive/273aed7b946fd83a8e83d4d8fa9e4f6318cfae62.tar.gz'...
warning: dumping very large path (> 256 MiB); this may run out of memory
error: writing to file: Broken pipe
(use '--show-trace' to show detailed location information)

Using stackProject':

nix-build nix/tiko.nix -A tiko-apps.tiko-systems --show-trace
trace: To make project.stack-nix for tikoHsSrc a fixed-output derivation but not materialized, set `stack-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.stack-nix for tikoHsSrc entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
building '/nix/store/4m7sj4nhc63fcs5z1kqbnfpbzpr9756a-stack-repos.drv'...
substituteStream(): WARNING: pattern '# nix-sha256:' doesn't match anything in file 'stack.yaml'
building '/nix/store/zkqjxbvvy14zaq57l4m5nvdqp552n00v-cache-file.drv'...
building '/nix/store/sb1ls548qbfk8bzn3zfzss6xx0wg3kk3-tikoHsSrc-stack-to-nix-pkgs.drv'...
substituteStream(): WARNING: pattern '/nix/store/nw2igscbaa9f26irawxaikdxxcqz695v-tikoHsSrc' doesn't match anything in file '/nix/store/pixyhknryys8l4his30jydcp1qrg1ar3-tikoHsSrc-stack-to-nix-pkgs/default.nix'
substituteStream(): WARNING: pattern '/nix/store/nw2igscbaa9f26irawxaikdxxcqz695v-tikoHsSrc' doesn't match anything in file '/nix/store/pixyhknryys8l4his30jydcp1qrg1ar3-tikoHsSrc-stack-to-nix-pkgs/pkgs.nix'
warning: dumping very large path (> 256 MiB); this may run out of memory
error: while evaluating the attribute 'src' of the derivation 'tiko-systems-lib-tiko-systems-0.0.0.0' at /nix/store/v2q65pfhcy79iarkm415srahigqic0zw-nixos-20.03/pkgs/stdenv/generic/make-derivation.nix:191:11:
file '/home/philipp/Projects/tiko/tiko/out/mydata/mysock.sock' has an unsupported type

Using cabalProject:

nix-build nix/tiko.nix -A tiko-apps.tiko-systems --show-trace
trace: No index state specified for tikoHsSrc, using the latest index state that we know about (2021-02-01T00:00:00Z)!
building '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv'...
...
No .cabal file found, running hpack on ./package.yaml
WARNING: Specified file "README.md" for extra-source-files does not exist
generated ./tiko-systems.cabal
Using index-state 2021-02-01T00:00:00Z
Warning: The package list for 'hackage.haskell.org-at-2021-02-01T000000Z' is
18662 days old.
Run 'cabal update' to get the latest list of available packages.
Warning: Requested index-state2021-02-01T00:00:00Z is newer than
'hackage.haskell.org-at-2021-02-01T000000Z'! Falling back to older state
(2021-01-31T23:42:25Z).
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: tiko-systems-0.0.0.0 (user goal)
[__1] unknown package: tiko-XXXX (dependency of tiko-systems)
[__1] fail (backjumping, conflict set: tiko-XXXX, tiko-systems)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: tiko-systems, tiko-XXXX
...
builder for '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv' failed with exit code 1
building '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv'...
...
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: tiko-systems-0.0.0.0 (user goal)
[__1] unknown package: tiko-XXXX (dependency of tiko-systems)
[__1] fail (backjumping, conflict set: tiko-XXXX, tiko-systems)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: tiko-systems, tiko-XXXX

builder for '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv' failed with exit code 1
error: while evaluating the attribute 'hsPkgs.tiko-systems' at /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:536:15:
while evaluating the attribute 'hsPkgs' at /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:516:43:
while evaluating 'mkCabalProjectPkgSet' at /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:152:13, called from /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:501:30:
while evaluating the attribute 'buildPackages.haskell-nix.compiler."${(((plan-pkgs).pkgs  hackage)).compiler.nix-name}".version' at /nix/store/v2q65pfhcy79iarkm415srahigqic0zw-nixos-20.03/lib/attrsets.nix:344:7:
while evaluating anonymous function at /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/lib/import-and-filter-project.nix:5:1, called from /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:503:31:
build of '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv' failed

For the cabalProject try I just changed the top-level nix-call, so it's not surprising that it fails because I am pretty sure the cabal files would need to be modified to match our stack overrides. However, I guess the fact that it calls cabal shows that with cabalProject the source filtering works?

[lspitzner]

Just a quick debugging suggestion: If you create a file without read permissions in your project dir (but not check it into git) then your nix expression will immediately break if the cleaning step fails (like it does here). Even though it breaks the nix expression immediately you might prefer that over silently filling up your nix store with overly large source derivations.

(Could you please rename this issue to something along "cleaning is broken for multiple-package projects"? Thanks!)

[hamishmack]

Could you please rename this issue to something along "cleaning is broken for multiple-package projects"?

I think it is broken for stack projects. I could be wrong, but so far I have not seen an example of a cabal.project based project where it is broken in haskell.nix. I also don't think it would work for a "single-package" stack project (files in the package dir that should be filtered would be copied to the store as well).

Once you are in a nix-shell, cabal build will use the ghc-pkg list packages (as far as I know stack does not). For this reason I recommend using cabal.project files and haskell-nix.cabalProject for your nix-shell.

[hamishmack]

It's definitely not Stack only as this effects us, but we don't use stack! I have a to-do to try and investigate more

@ocharles do you have a package in the root of your project? It looks like that was broken. The test case had both packages in subdirs. I came across filtering issues while updating Leksah to the latest haskell.nix. There were two bugs, one resulted in all the files being copied and the other one (that was masked by the first) caused filtering to when it ran to leave files out. #1041 fixes it for Leksah.

[ocharles]

Do you mean a .cabal file in the root directory? I don't think so, but I will check out #1041

[hamishmack]

A .cabal file in the root of the project (same dir as cabal.project). With packages: . in the cabal.project.

[ocharles]

I'm pretty sure we don't have that but I'll double check tomorrow

[purefn]

Has there been any attempt to fix this issue? We've just tried upgrading haskell.nix on our large, monorepo stack project in order to use a newer LTS and GHC 8.10.4, but copying our entire project source tree for every component means it takes > 30 minutes before anything even gets built.

[purefn]

Update: nevermind. I just figured out that it is the includeSiblings = true here https://github.com/input-output-hk/haskell.nix/blob/22ad57c7e78a6bf409738efdd87a2c23d13bfabb/lib/import-and-filter-project.nix#L55 that is causing ths problem for us. Is there a way to override that? For a large monorepo, this is not a good default.

[hamishmack]

Please try out #1074 as it might help.

The includeSiblings = true in import-and-filter-project.nix should only be a problem if per component level filtering is not working (or if you had something like extra-source-files: ../../**.txt in your .cabal file).

Are you using hpack? That currently breaks the component level filtering. We should probably filter just the package dir when that happens, but I think we might include everything (just in case hpack wants to reference files outside the package dir).

[purefn]

The issue I was having was after callStackToNix. Because of the includeSiblings = true in importAndFilter function, every component that was built was causing the entire source tree to be copied into the nix store, once for every component. We have hundreds (maybe 1000+) components in our project, so that meant we were getting that many copies of our entire source tree in the nix store. I created a fork to test disabling that and it is now much much faster and the build takes up a fraction of the disk space.

[hamishmack]

That indicates that the component filter is failing completely (the files should be filtered when the component builds to just the ones referenced in the .cabal file). I would really like to get to the bottom of why that is not working. It could be a problem with the output of callStackToNix (please do try #1074) or it could be something else. If you are using hpack it could be to do with that.

Unfortunately just changing to includeSiblings = false would break the ability to use relative directories in .cabal files to reference files in other packages.

Perhaps try the #1074 branch with this trace enabled. It should tell log for every file in the component the reason it was not filtered (if it prints nothing then the filter is not even running).

[hamishmack]

I have checked #1074 issue I was able to reproduce before (a file foo in the root of the project with no read access does not fail). The filtering also seems to exclude the file correctly (but it did before too).

I used pkgs.srcOnly to extract the source files for the component derivation like this:

git clone https://github.com/hamishmack/test-1013.git
cd test-1013
find $(nix-build -E 'let p = import ./default.nix; in p.pkgs.srcOnly p.lib.components.library' --no-out-link)

Only the files for the component are were included.

[purefn]

@hamishmack I tried it and got the same result. I think the issue is that there are two source trees copied to the nix store, one is filtered down to just the component source, while the other contains all of our source. I build a single package and looked for the source afterwards, and found this

$ ls /nix/store/80ww5zhf9xjj2f0f3v49aqmiiic8s4b8-range-root-devtools-simlint-root /nix/store/qln3qdfyx9nmkl4gilrwr5igqis0xpgh-range-root-devtools-simlint
/nix/store/80ww5zhf9xjj2f0f3v49aqmiiic8s4b8-range-root-devtools-simlint-root:
devtools

/nix/store/qln3qdfyx9nmkl4gilrwr5igqis0xpgh-range-root-devtools-simlint:
administration-client  defender-logs          jwt-config-simspace      portal-services-client    simspace-generics
aeson-simspace        deployment-target-agent     jwt-middleware-simspace  project           simspace-h5p-drupal
apitest           devtools            jwt-test-simspace    PULL_REQUEST_TEMPLATE.md  simspace-ids
attack-designer        doc            keycloak         range-query           simspace-immortal
attack-integrations    docker             lariat-core          range-server      simspace-pagination
authz             docker-compose.yml      learning         range-server-api      simspace-servant-docs
bors.toml         documentation-generation    lib              range-server-ts-gen       simspace-snapshot.yaml
brittany          document-server         logger-handle    README.md             simspace-squeal
BSD3-LICENSE          event-controller        Makefile         reps          simspace-test
build.sbt         event-historian         mattermost           request-logger        simspace-tracing
caching           event-template          MIT-LICENSE          request-logger-simspace   simtosca
campaign-status        except-simspace        monad-logger-extra       resource-manager      simyaml
client            hardhat-expansion   mto              s3-simspace           simyamltest
common-package.yaml    hardhat-server         nagios           SAMLSetup.md      stack.yaml
common-testlib        hardhat-traffic-config      network-impact       sample-data           team-assessment
common-utilities       hardhat-ts-gen         nix              scoring           time-simspace
content-repo          hardhat-versioning-library  notification         servant-convert       tools
control-point         hspec-extra         notification-queue       servant-range-proxy       tracker-client
datagen           http-error          package.json         servant-simjson       ts-gen
data-server       human-readable          perf-testing         servant-zoo           ttx
db-infra          Jenkinsfile         persistent-runners       shared            uri-utils
db-migration-test      json-logger        portal-client    simjson           user-emulation
db-schema         json-query-string   portal-runner    simspace-amazonka         warden
default.nix       json-websocket-client   portal-server    simspace-db-seed-user     yarn.lock

When I use my fork with includeSiblings = false, both range-root-devtools-simlint-root and range-root-devtools-simlint contain only the package/component source.