Closed quetz closed 1 year ago
We have changed stuff recently in the way cabal components are clean, so this might be a bug.
Are you using explicit hs-source-dirs
for each component in the .cabal
file (not the default which is .
)? That way the cleaning of the cabal component will know which directories are needed. This will also reduce the number of times haskell.nix will need to rebuild the components (since it will only rebuild if the source for the component changes). Perhaps use hs-source-dirs: src
and the component name instead of src
for each of the other components.
I would also recommend clean the project src
directory before passing it to project
. If you use git
you can use the src = haskell-nix.haskellLib.cleanGit { name = "project-name"; src = ./.; }
which will filter out anything not in the git index. This does mean you have to git add
your source files before they will be seen by haskell.nix.
Are you using explicit hs-source-dirs for each component in the .cabal file (not the default which is .)? Yes.
Using cleanGit
helps (in a sense that it does not include full project directory in a nix store) but still there's nix store path for each of project package with exactly same contents (cleanGit results of project directory). Also, noop changes in source files (adding blank line) result in creating fresh new set of those nix store paths on each nix-shell invocation (no project building even required).
Why is it neeed to have full copy of source tree in nix store upon entering nix shell?
The source for each package should be filtered further to just be the source of that package.
But it is a bit odd that you have the any source being copied in for each nix-shell
invocation. That shouldn't be happening - it only builds the dependencies anyway!
Is it surprising that the source is in the store? It's not surprising to me - it has to run cabal configure, and ot can only do that on things in the store, no? (This only applies to non-materialised projects)
For the configure step we filter it down to only contain cabal.project
and the .cabal
files. So a) it shouldn't change on changes to source files, and b) it should be really small.
Although the way that source filtering step got done was changed recently. @hamishmack do you think we could conceivably be copying the pre-filtering source as well?
I'm pretty certain this is a regression that came in here:
This change means that every package now has includeSiblings = true
, which means that it ends up having src
set to the entire project source tree, rather than just its relative sub-directory. I see @michaelpj said
I do find it tempting to say you should just Not Do That...
and I'm inclined to agree.
What gets created on every nix-shell
invocation on noop changes in sources (added empty line):
- <project>-root-<package directory name> (for each package in project)
- <project>-root-<package dir>-lib-<package name>-root (for package that gets its source file modified)
- ... -config.drv (for each executable / library for each package in project)
- ... -ghc-<ghc version>-env.drv (for each executable / library for each package in project)
- ... .drv (for each executable / library for each package in project)
Looks like it all boils down to -root-<dir>
store paths that contain whole (filtered) source. And even noop changes are detected by nix producing different hash each time. And then everything else refers to this new path producing different hash also, and so on...
Nix shouldn't (can't) produce a different hash time, so you must have something changing. If you can find the path (I used inotifytools
to watch /nix/store
), you can do a recursive diff
on them to see what's changing. The hash of a source path is the product of its contents hash + name, and this is stable (if those inputs are stable).
@quetz I suggest reverting the line I linked to in #843, though. Just to reiterate, that's changing true
to false
here:
Nix shouldn't (can't) produce a different hash time, so you must have something changing.
Yeah, but source code is changing (I insert empty line, so this is noop change). It should lead to different hash and if everything else refers to this new source it gets new hash too...
@quetz I suggest reverting the line I linked to in #843, though. Just to reiterate, that's changing true to false here
Will experiment with that, thanks.
@quetz I suggest reverting the line I linked to in #843, though. Just to reiterate, that's changing
true
tofalse
here:
It is much better with this change. Some store paths are still created on each nix-shell
invocations, but source code gets reduced to only one component that is changed.
But still, is it possible to get shell in haskell.nix behave as advertised in nix?
The command nix-shell will build the dependencies of the specified derivation, but not the derivation itself.
I want to be dropped into shell where all dependencies of my project are built and registered, but project itself is not touched (do not configure my packages, do not make derivations that can be used to build these packages).
How can I reproduce this issue?
@hamishmack With the following directory structure:
- foo
- bar
|-- bar.cabal
|-- src/.../.hs
- shell.nix
- cabal.project
That is, a directory structure where your .cabal
packages are in sub directories. Now make foo
a very large file that's not referenced from any .cabal
file (i.e., foo
is not part of the Haskell build) and observe that it gets copied into the Nix store. If you make multiple .cabal
subdirectories, notice that the .drv
for building the library component will have a src
that contains foo
(so foo
is being copied for all projects).
I can't seem reproduce it that way. See https://github.com/hamishmack/test-1013.
git clone https://github.com/hamishmack/test-1013.git
cd test-1013
nix-shell --run 'echo ok'
echo test >> foo
nix-shell --run 'echo ok'
Seems to work as expected for me. No derivations are rebuilt. It works with or without the cleanGit
call.
One thing that would not work is if something was triggering the source to be copied to the store before the haskell.nix project function sees it. For instance if the example project was in subDir
and we used:
src = haskellLib.cleanGit { src = ./.; } + "/subDir";
Nix will be forced to copy the all of haskellLib.cleanGit { src = ./.; }
to the store in order to make a path for src
. To avoid this we would have to use:
src = haskellLib.cleanSourceWith { src = haskellLib.cleanGit { src = ./.; }; subDir = "subDir"; };
@hamishmack I'm not able to check that out right, now but I'd suggest instantiating the derivation for bar.components.library
(nix-instantiate -A bar.components.library
) and looking at what the input source is.
This isn't about things being rebuild, it's about a huge amount of stuff being copied into the store. It's naturally not noticable with a small test case, which is why I suggest making foo
big (e.g., use dd
to make a multiple GB file).
In all of my tests with master
at the moment, some components have their src
point to the entire directory tree, but then have a prePatch
phase that just copies a subset during the build. You wouldn't see this in nix-shell
, because in a shell no components are actually built.
Ok, I had a chance to look at indeed hsPkgs.bar.components.exes.bar
doesn't include the foo
file from the source root. So something else is up. I'll try and get a minimal repro
I think I have also encountered this error just now. We are using callStackToNix
/mkStackPkgSet
. On Hydra, the builds are working with current master of haskell.nix. However, when using a local checkout it breaks because we have some sockets in the same directory which trip up nix:
error: file 'REPO_ROOT/temp_files/mydata/mysock.sock' has an unsupported type
This socket file is being filtered out using the nixpkgs cleanSourceWith
function, so I have no idea why it should be copied. If I put the src in quotes, it works again.
Broken:
stack-nix-build = with tiko.pkgs-2003.haskell-nix; callStackToNix {
src = cleanSourceWith { src = ./.; ...};
};
Works:
stack-nix-build = with tiko.pkgs-2003.haskell-nix; callStackToNix {
src = "${ cleanSourceWith { src = ./.; ...} }";
};
I did not realise you were using callStackToNix
and mkStackPkgSet
directly. It might be easier to use stackProject'
or stackProject
. If you want to stick to callStackToNix
and mkStackPkgSet
, make sure you are calling importAndFilterProject
as well, like stackProject'
does.
I can't seem reproduce it that way. See https://github.com/hamishmack/test-1013.
Try this PR - https://github.com/hamishmack/test-1013/pull/1
Doh. Still seems to work correctly.
I wonder if it might only affect stack projects?
Switching to a stack based project does it! https://github.com/hamishmack/test-1013/commit/f003490548313d26f3d7cc7de763abf3ed72f4a2
We must be doing something wrong in handling of stack projects.
I think the problem is that we never got around to adding code to clean the source to callStackToNix
.
In callCabalProjectToNix
we:
I think we can do something similar in callStackToNix
, but we might need to parse the stack.yaml
file to see if it is a path to a file (since we will need to include that file too).
Doh. Still seems to work correctly.
How do you test it?
If I take your code and then invoke nix-shell
several times while changing source code in between I get these store paths created:
/nix/store/0612xilw8k2a3ardnd1gczkahs4rg25g-bar-exe-bar-0.1.0.0.drv
/nix/store/1f4h280516wb0lqzb1glrdzsqfl0pd33-test-1013-root-lib
/nix/store/1wa6k73zk6wj63p32m9xy7qaffiq3kl1-test-1013-root-bar-exe-bar-root
/nix/store/6j3vmyy9aymzrd7idl5ja1760lvvsfa2-test-1013-root-bar
/nix/store/7i3ynrdmx6ni44n7c33jy9a48slxzzhp-test-1013-root-bar
/nix/store/ddlbk32vhq3mjs391iwr4mx462li7s0n-bar-exe-bar-0.1.0.0.drv
/nix/store/dqjih4jz2zy4nplcliyxgml607hz2ky9-git-ls-files.drv
/nix/store/g5xkc116b3xki8c9si73xl0p2x9wnkaq-test-1013-root-bar-exe-bar-root
/nix/store/hfcxlb9g2karm36bz3i6cl0icy5qfq5s-git-ls-files
/nix/store/id0ihnlly9mar1hp2cf8pla92dmahbcq-test-1013-root-lib
/nix/store/j439sgd2s3wb9y9ynfy3pw6xqy5i7c0d-test-1013-root-bar
/nix/store/jqwywdn41nzs63p7sslz3758dkfp5igb-test-1013-root-lib
/nix/store/r0jqijacdswkr5nv8bfgcjf756cl05wj-bar-exe-bar-0.1.0.0.drv
/nix/store/r6918pjdgrr2xiw0jr89v95jjz0mxq5z-test-1013-root-lib
/nix/store/sn2n92jaq5gx93n18s3z7vgdlzsjf64z-test-1013-gitFiles
/nix/store/w4wn1k1l79msadgk0wwwig6ab7a2v4vs-test-1013-root-bar-exe-bar-root
/nix/store/yjnj3wg1zfzzzya7a4gazl0ihb83cpvd-ghc-shell-for-bar.drv
/nix/store/zpsc81kb9zi4wak2xj9pw4x4kb828kp5-test-1013-root-bar
I do not ever invoke nix-build
or cabal build
from within shell, just wait till I get dropped into shell.
How do you test it?
I am testing with:
hamish@Hamishs-MacBook-Air test-1013 % ls -l /nix/store | wc -l
19501
hamish@Hamishs-MacBook-Air test-1013 % echo test >> foo
hamish@Hamishs-MacBook-Air test-1013 % NIV_OVERRIDE_haskell_nix=/Users/hamish/iohk/haskell.nix nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok
hamish@Hamishs-MacBook-Air test-1013 % ls -l /nix/store | wc -l
19501
It's definitely not Stack only as this effects us, but we don't use stack! I have a to-do to try and investigate more
Ah think I understand now. Its not foo that is the problem. The question is why does this copy to the store:
hamish@Hamishs-MacBook-Air test-1013 % ls -l /nix/store | wc -l
19501
hamish@Hamishs-MacBook-Air test-1013 % echo >> bar/Main.hs
hamish@Hamishs-MacBook-Air test-1013 % NIV_OVERRIDE_haskell_nix=/Users/hamish/iohk/haskell.nix nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok
hamish@Hamishs-MacBook-Air test-1013 % ls -l /nix/store | wc -l
19503
That is odd.
Hmm maybe not, probably just .drv for bar exe and its setup drv.
That's not only .drv, but source code is copied into store upon entering nix-shell (see, my previous message with example of nix store paths that are actually added).
That's not only .drv, but source code is copied into store upon entering nix-shell (see, my previous message with example of nix store paths that are actually added).
I suspect there might be something different in the nix configuration. It is very odd that even git-ls-files
is being recalculated. That should only happen when files in the .git dir changes. I wonder if builtins.filterSource
or builtins.hasContext
are not working as expected for some reason.
I went back to cabalProject
and I tested on linux (I have pushed the code I used to the repo https://github.com/hamishmack/test-1013/tree/fd3d095ab6808ee817cca52d105e0ed2e71d4b0f).
Touching the source for lib
and bar
triggers build of lib as expected (since only bar
is listed in the shellFor
packages
arg). No sign of git-ls-files
though.
Then I checked just touching bar
and it did copy the bar
source to the store, but it was filtered correctly and foo
is not included.
Here is what I tried:
[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > a.txt
[hamish@x86_64-builder:~/iohk/test-1013]$ echo >> lib/src/MyLib.hs
[hamish@x86_64-builder:~/iohk/test-1013]$ echo >> bar/Main.hs
[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
warning: don't know how to open Nix store 'iohk.cachix.org'
these derivations will be built:
/nix/store/s6jhj4vx6lim0as0rb0hy08lglf7k2j8-lib-lib-lib-0.1.0.0.drv
/nix/store/xj3jc30iigcb5rd196cvwcpk86r0g05g-lib-lib-lib-0.1.0.0-haddock.drv
/nix/store/5zkzdi0100ixbp1wzb9i0j1gvfq1r0z8-hoogle-local-0.1.drv
/nix/store/ws9k1bkhs2i8cv0f1isam5gsxf1lqn2y-ghc-shell-for-bar-config.drv
/nix/store/bd8mz39qyinh5fqhjr3kn4ppi75r7iqp-ghc-shell-for-bar-ghc-8.10.3-env.drv
building '/nix/store/s6jhj4vx6lim0as0rb0hy08lglf7k2j8-lib-lib-lib-0.1.0.0.drv'...
unpacking sources
unpacking source archive /nix/store/yznvkb5hx5m5n761j5vmhd7d1ngwbsbi-test-1013-root-lib-lib-lib-root
source root is test-1013-root-lib-lib-lib-root
patching sources
configuring
Configure flags:
--prefix=/nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0 lib:lib --package-db=clear --package-db=/nix/store/azgvznk2r2jv3irsqi00w4xjl7kjysqk-lib-lib-lib-0.1.0.0-config/lib/ghc-8.10.3/package.conf.d --exact-configuration --dependency=rts=rts --dependency=ghc-heap=ghc-heap-8.10.3 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=integer-gmp=integer-gmp-1.0.3.0 --dependency=base=base-4.14.1.0 --dependency=deepseq=deepseq-1.4.4.0 --dependency=array=array-0.5.4.0 --dependency=ghc-boot-th=ghc-boot-th-8.10.3 --dependency=pretty=pretty-1.1.3.6 --dependency=template-haskell=template-haskell-2.16.0.0 --dependency=ghc-boot=ghc-boot-8.10.3 --dependency=ghc=ghc-8.10.3 --dependency=Cabal=Cabal-3.2.1.0 --dependency=array=array-0.5.4.0 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.2.1 --dependency=directory=directory-1.3.6.0 --dependency=filepath=filepath-1.4.2.1 --dependency=ghc-boot=ghc-boot-8.10.3 --dependency=ghc-compact=ghc-compact-0.1.0.0 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=hpc=hpc-0.6.1.0 --dependency=mtl=mtl-2.2.2 --dependency=parsec=parsec-3.1.14.0 --dependency=process=process-1.6.9.0 --dependency=text=text-1.2.4.1 --dependency=time=time-1.9.3 --dependency=transformers=transformers-0.5.6.2 --dependency=unix=unix-2.7.2.2 --dependency=xhtml=xhtml-3000.2.2.1 --dependency=terminfo=terminfo-0.4.1.4 --dependency=text=text-1.2.4.1 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.2.1 --with-ghc=ghc --with-ghc-pkg=ghc-pkg --with-hsc2hs=hsc2hs --with-gcc=cc --with-ld=ld.gold --ghc-option=-optl-fuse-ld=gold --ld-option=-fuse-ld=gold --with-ar=ar --with-strip=strip --disable-executable-stripping --disable-library-stripping --disable-library-profiling --disable-executable-profiling --enable-static --enable-shared --disable-coverage --enable-library-for-ghci --enable-split-sections
Configuring library for lib-0.1.0.0..
Warning: The flag --disable-executable-profiling is deprecated. Please use
--disable-profiling instead.
building
Preprocessing library for lib-0.1.0.0..
Building library for lib-0.1.0.0..
[1 of 1] Compiling MyLib ( MyLib.hs, dist/build/MyLib.o, dist/build/MyLib.dyn_o )
installing
Installing library in /nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0/lib/x86_64-linux-ghc-8.10.3/lib-0.1.0.0-1K833QoZdbwhP5y6tYtVq
post-installation fixup
patching script interpreter paths in /nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0
checking for references to /build/ in /nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0...
patchelf: wrong ELF type
building '/nix/store/ws9k1bkhs2i8cv0f1isam5gsxf1lqn2y-ghc-shell-for-bar-config.drv'...
building '/nix/store/xj3jc30iigcb5rd196cvwcpk86r0g05g-lib-lib-lib-0.1.0.0-haddock.drv'...
unpacking sources
unpacking source archive /nix/store/yznvkb5hx5m5n761j5vmhd7d1ngwbsbi-test-1013-root-lib-lib-lib-root
source root is test-1013-root-lib-lib-lib-root
patching sources
configuring
Configure flags:
--prefix=/nix/store/q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0 lib:lib --package-db=clear --package-db=/nix/store/xcvjsvkl6vxh7y6sk7q2vs8mpwhmy498-lib-lib-lib-0.1.0.0-haddock-config/lib/ghc-8.10.3/package.conf.d --exact-configuration --dependency=rts=rts --dependency=ghc-heap=ghc-heap-8.10.3 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=integer-gmp=integer-gmp-1.0.3.0 --dependency=base=base-4.14.1.0 --dependency=deepseq=deepseq-1.4.4.0 --dependency=array=array-0.5.4.0 --dependency=ghc-boot-th=ghc-boot-th-8.10.3 --dependency=pretty=pretty-1.1.3.6 --dependency=template-haskell=template-haskell-2.16.0.0 --dependency=ghc-boot=ghc-boot-8.10.3 --dependency=ghc=ghc-8.10.3 --dependency=Cabal=Cabal-3.2.1.0 --dependency=array=array-0.5.4.0 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.2.1 --dependency=directory=directory-1.3.6.0 --dependency=filepath=filepath-1.4.2.1 --dependency=ghc-boot=ghc-boot-8.10.3 --dependency=ghc-compact=ghc-compact-0.1.0.0 --dependency=ghc-prim=ghc-prim-0.6.1 --dependency=hpc=hpc-0.6.1.0 --dependency=mtl=mtl-2.2.2 --dependency=parsec=parsec-3.1.14.0 --dependency=process=process-1.6.9.0 --dependency=text=text-1.2.4.1 --dependency=time=time-1.9.3 --dependency=transformers=transformers-0.5.6.2 --dependency=unix=unix-2.7.2.2 --dependency=xhtml=xhtml-3000.2.2.1 --dependency=terminfo=terminfo-0.4.1.4 --dependency=text=text-1.2.4.1 --dependency=binary=binary-0.8.8.0 --dependency=bytestring=bytestring-0.10.12.0 --dependency=containers=containers-0.6.2.1 --with-ghc=ghc --with-ghc-pkg=ghc-pkg --with-hsc2hs=hsc2hs --with-gcc=cc --with-ld=ld.gold --ghc-option=-optl-fuse-ld=gold --ld-option=-fuse-ld=gold --with-ar=ar --with-strip=strip --disable-executable-stripping --disable-library-stripping --disable-library-profiling --disable-executable-profiling --enable-static --enable-shared --disable-coverage --enable-library-for-ghci --enable-split-sections --docdir=/nix/store/hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc/share/doc/lib
Configuring library for lib-0.1.0.0..
Warning: The flag --disable-executable-profiling is deprecated. Please use
--disable-profiling instead.
building
Preprocessing library for lib-0.1.0.0..
Running Haddock on library for lib-0.1.0.0..
Warning: The documentation for the following packages are not installed. No
links will be generated to these packages: base-4.14.1.0, ghc-prim-0.6.1,
integer-gmp-1.0.3.0
Warning: --source-* options are ignored when --hyperlinked-source is enabled.
Haddock coverage:
0% ( 0 / 2) in 'MyLib'
Missing documentation for:
Module header
someFunc (MyLib.hs:3)
Warning: MyLib: could not find link destinations for:
IO
Documentation created: dist/doc/html/lib/index.html, dist/doc/html/lib/lib.txt
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/72hy3lxz34196111fr911yvzy255xiic-lib-lib-lib-0.1.0.0-haddock
strip is /nix/store/rqfgki7ck1bxqhk3hd7wziqhahjadfbj-binutils-2.35.1/bin/strip
patching script interpreter paths in /nix/store/72hy3lxz34196111fr911yvzy255xiic-lib-lib-lib-0.1.0.0-haddock
checking for references to /build/ in /nix/store/72hy3lxz34196111fr911yvzy255xiic-lib-lib-lib-0.1.0.0-haddock...
shrinking RPATHs of ELF executables and libraries in /nix/store/hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc
strip is /nix/store/rqfgki7ck1bxqhk3hd7wziqhahjadfbj-binutils-2.35.1/bin/strip
patching script interpreter paths in /nix/store/hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc
checking for references to /build/ in /nix/store/hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc...
building '/nix/store/5zkzdi0100ixbp1wzb9i0j1gvfq1r0z8-hoogle-local-0.1.drv'...
importing builtin packages
importing other packages
building hoogle database
Starting generate
[3/34] base... 1.11s
[11/34] ghc... 3.06s
[28/34] template-haskell... 0.22s
[34/34] xhtml... 0.06s
Found 28 warnings when processing items
Reordering items... 0.03s
Writing tags... 0.18s
Writing names... 0.08s
Writing types... 0.90s
Took 8.63s
building haddock index
finishing up
building '/nix/store/bd8mz39qyinh5fqhjr3kn4ppi75r7iqp-ghc-shell-for-bar-ghc-8.10.3-env.drv'...
ok
[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > b.txt
[hamish@x86_64-builder:~/iohk/test-1013]$ diff a.txt b.txt
1c1
< total 23713556
---
> total 23713644
14089a14090
> -r--r--r-- 1 root root 4369 Jan 1 1970 1jpr0ph3pxdmx6c14hhgp9w4fj0swwzc-bar-exe-bar-0.1.0.0-config.drv
53405a53407
> -r--r--r-- 1 root root 3321 Jan 1 1970 5zkzdi0100ixbp1wzb9i0j1gvfq1r0z8-hoogle-local-0.1.drv
63117a63120
> dr-xr-xr-x 3 root root 5 Jan 1 1970 72hy3lxz34196111fr911yvzy255xiic-lib-lib-lib-0.1.0.0-haddock
101504a101508
> dr-xr-xr-x 4 root root 4 Jan 1 1970 bc29hk4q7kw7yy08y2vkcvwn4759bab1-hoogle-local-0.1
101826a101831
> -r--r--r-- 1 root root 5124 Jan 1 1970 bd8mz39qyinh5fqhjr3kn4ppi75r7iqp-ghc-shell-for-bar-ghc-8.10.3-env.drv
143533a143539
> dr-xr-xr-x 3 root root 6 Jan 1 1970 h2cxg5cfydy10i9b11h0hgipgkvrc5ng-ghc-shell-for-bar-config
147608a147615
> dr-xr-xr-x 3 root root 3 Jan 1 1970 hhnr1p5z5lb36nq0v4rbmz7irg0ixr4y-lib-lib-lib-0.1.0.0-haddock-doc
150550a150558
> -r--r--r-- 1 root root 2253 Jan 1 1970 hvdq530187jxdlmgl0nkniya3gw3pz4b-ghc-shell-for-bar.drv
179988a179997
> dr-xr-xr-x 3 root root 3 Jan 1 1970 l58h0jxa3z1q8bwn9sv0dmm1kr3hgcjf-test-1013-root-bar-exe-bar-root
193515a193525
> dr-xr-xr-x 8 root root 8 Jan 1 1970 mm8wn9nnzyz5gvl0ds4fjy232ir0w5dn-ghc-shell-for-bar-ghc-8.10.3-env
197350a197361
> -r--r--r-- 1 root root 4969 Jan 1 1970 n2xf830psj3vfpdqa3awx1vnq3z7g9vn-bar-exe-bar-0.1.0.0-ghc-8.10.3-env.drv
215420a215432
> dr-xr-xr-x 6 root root 7 Jan 1 1970 q3splvkbd7f6n3w03zqdphg0z3xgy36f-lib-lib-lib-0.1.0.0
234125a234138
> -r--r--r-- 1 root root 4952 Jan 1 1970 s6jhj4vx6lim0as0rb0hy08lglf7k2j8-lib-lib-lib-0.1.0.0.drv
257789a257803
> -r--r--r-- 1 root root 3575 Jan 1 1970 ws9k1bkhs2i8cv0f1isam5gsxf1lqn2y-ghc-shell-for-bar-config.drv
264357a264372
> -r--r--r-- 1 root root 5943 Jan 1 1970 xj3jc30iigcb5rd196cvwcpk86r0g05g-lib-lib-lib-0.1.0.0-haddock.drv
270934a270950
> -r--r--r-- 1 root root 4048 Jan 1 1970 y92vdq5rm57djmrj7x630z7lqd9ibhhc-bar-exe-bar-0.1.0.0.drv
277283a277300
> dr-xr-xr-x 3 root root 3 Jan 1 1970 yznvkb5hx5m5n761j5vmhd7d1ngwbsbi-test-1013-root-lib-lib-lib-root
[hamish@x86_64-builder:~/iohk/test-1013]$ echo >> bar/Main.hs
[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok
[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > c.txt
[hamish@x86_64-builder:~/iohk/test-1013]$ diff b.txt c.txt
1c1
< total 23713644
---
> total 23713649
172835a172836
> -r--r--r-- 1 root root 4048 Jan 1 1970 kbzly2g84v444rcdgrzqbgs76f2zv7ds-bar-exe-bar-0.1.0.0.drv
266254a266256
> dr-xr-xr-x 3 root root 3 Jan 1 1970 xqrljn0d5n7ffs1xxpph83sn5kpf0gaa-test-1013-root-bar-exe-bar-root
[hamish@x86_64-builder:~/iohk/test-1013]$ tree /nix/store/xqrljn0d5n7ffs1xxpph83sn5kpf0gaa-test-1013-root-bar-exe-bar-root
/nix/store/xqrljn0d5n7ffs1xxpph83sn5kpf0gaa-test-1013-root-bar-exe-bar-root
└── bar
├── bar.cabal
├── CHANGELOG.md
├── Main.hs
└── Setup.hs
1 directory, 4 files
Just on the off chance I checked that git add
works ok too (it does only the git file list was updated):
[hamish@x86_64-builder:~/iohk/test-1013]$ touch new-file
[hamish@x86_64-builder:~/iohk/test-1013]$ git add new-file
[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --pure --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
warning: don't know how to open Nix store 'iohk.cachix.org'
building '/nix/store/y3hvls2dnm8ljkify5mlw1k3nfysaj3v-git-ls-files.drv'...
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok
[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > d.txt
[hamish@x86_64-builder:~/iohk/test-1013]$ diff c.txt d.txt
1c1
< total 23713649
---
> total 23713659
38444a38445
> -r--r--r-- 1 root root 245 Jan 1 1970 49x5nzzyx771sjspxd1vilg13rly5hqd-git-ls-files
263487a263489
> dr-xr-xr-x 3 root root 3 Jan 1 1970 xfx052kcbcjl3zx3z8l8yj32whalp2ck-test-1013-gitFiles
269368a269371
> -r--r--r-- 1 root root 1809 Jan 1 1970 y3hvls2dnm8ljkify5mlw1k3nfysaj3v-git-ls-files.drv
I'm using version with project.shellFor
(fffc08c20e6b30e139e27463dfa9a471bc0f00c7). After 3 nix-shell
invocations and echo >> lib/src/MyLib.hs
in between I get these store paths (grouped by name):
3 bar-exe-bar-0.1.0.0-config.drv
3 bar-exe-bar-0.1.0.0-ghc-8.10.3-env.drv
3 bar-exe-bar-0.1.0.0.drv
3 ghc-shell-for-bar-config
3 ghc-shell-for-bar-config.drv
3 ghc-shell-for-bar-ghc-8.10.3-env
3 ghc-shell-for-bar-ghc-8.10.3-env.drv
3 ghc-shell-for-bar.drv
3 hoogle-local-0.1
3 hoogle-local-0.1.drv
3 lib-lib-lib-0.1.0.0
3 lib-lib-lib-0.1.0.0-haddock
3 lib-lib-lib-0.1.0.0-haddock-doc
3 lib-lib-lib-0.1.0.0-haddock.drv
3 lib-lib-lib-0.1.0.0.drv
3 test-1013-root-bar
3 test-1013-root-lib
So it's clear that on each nix-shell invocation a new set of these paths is generated. My understanding is that if project is not built and nix-shell
is supposed to configure and prepare only dependencies of the project (and not project itself) it should not add extra stuff to store if dependencies are not changed.
So it's clear that on each nix-shell invocation a new set of these paths is generated. My understanding is that if project is not built and nix-shell is supposed to configure and prepare only dependencies of the project (and not project itself) it should not add extra stuff to store if dependencies are not changed.
The packages
argument passed to shellFor
specifies which packages not to build (since we are going to build them in the shell). In this case it is packages = ps: [ ps.bar ];
. Since lib
is a build dependency of bar
it is included in the shell's ghc-pkg list
and needs to be rebuilt when the source changes.
Changing the packages
argument in shell.nix
to packages = ps: [ ps.bar ps.lib ];
avoids the rebuild. Not passing the packages
arg at all should also work since the default is to include all the "local" packages in the cabal.project
file (in this case bar
and lib
):
[hamish@x86_64-builder:~/iohk/test-1013]$ cat shell.nix
let project = import ./.;
in project.shellFor { packages = ps: [ ps.bar ps.lib ]; exactDeps = true; }
[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > a.txt
[hamish@x86_64-builder:~/iohk/test-1013]$ echo >> lib/src/MyLib.hs
[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok
[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l /nix/store > b.txt
[hamish@x86_64-builder:~/iohk/test-1013]$ diff a.txt b.txt
1c1
< total 23871193
---
> total 23871223
54229a54230
> -r--r--r-- 1 root root 4048 Jan 1 1970 5yap5i3s3hv1rclb6f5mmvcidpgxrw6x-bar-exe-bar-0.1.0.0.drv
68778a68780
> -r--r--r-- 1 root root 4969 Jan 1 1970 7i4739h8s5x3lcxlbzajaqkzi6fpia14-bar-exe-bar-0.1.0.0-ghc-8.10.3-env.drv
159109a159112
> -r--r--r-- 1 root root 4952 Jan 1 1970 idzc3bkaj9l7sbgzn9qwbg6wrs29bhv5-lib-lib-lib-0.1.0.0.drv
196247a196251
> dr-xr-xr-x 3 root root 3 Jan 1 1970 mgndx5v702i2rbj86v31nqdy0zwg11fs-test-1013-root-lib-lib-lib-root
237372a237377
> -r--r--r-- 1 root root 4369 Jan 1 1970 rzpqgq6bkqhpsn3q0zmx0g8nsfxdczrj-bar-exe-bar-0.1.0.0-config.drv
[hamish@x86_64-builder:~/iohk/test-1013]$ tree mgndx5v702i2rbj86v31nqdy0zwg11fs-test-1013-root-lib-lib-lib-root
mgndx5v702i2rbj86v31nqdy0zwg11fs-test-1013-root-lib-lib-lib-root [error opening dir]
0 directories, 0 files
It's definitely not Stack only as this effects us, but we don't use stack! I have a to-do to try and investigate more
There is a definitely a problem with callStackToNix
(and there for stackProject
and stackProject'
). There might also be a problem with cabal projects, but so far I can't find it.
@phile314 It does look like your issue is caused by the same problem with callStackToNix
not filtering like callCabalProjectToNix
. I am curious to know if you get the same issue if you build your project with cabalProject { src = cleanSourceWith { src = ./.; ...}; }
.
I did a test where I deleted the foo
file in the test-1013
repo and replaced it with a socket (created by running nc -l -U foo
in another terminal). With stackProject
in the default.nix
I was able to reproduce the error, but changing it back to cabalProject
made it go away (log bellow).
The work around (src = "${ cleanSourceWith { src = ./.; ...} }";
) is definitely going to copy everything in your git repo to the store on changes to any file. It does not matter too much since it callStackToNix
would currently do that anyway, but don't use it with cabalProject
.
[hamish@x86_64-builder:~/iohk/test-1013]$ cat default.nix
let
sources = import nix/sources.nix {};
haskellNix = import sources."haskell.nix" {};
pkgs = import haskellNix.sources.nixpkgs-unstable haskellNix.nixpkgsArgs;
in pkgs.haskell-nix.stackProject {
# compiler-nix-name = "ghc8103";
# src = ./.;
src = pkgs.haskell-nix.haskellLib.cleanGit { name = "test-1013"; src = ./.; };
}
[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: To make project.stack-nix for test-1013 a fixed-output derivation but not materialized, set `stack-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.stack-nix for test-1013 entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
error: file '/home/hamish/iohk/test-1013/foo' has an unsupported type
(use '--show-trace' to show detailed location information)
[hamish@x86_64-builder:~/iohk/test-1013]$ vim default.nix
[hamish@x86_64-builder:~/iohk/test-1013]$ cat default.nix
let
sources = import nix/sources.nix {};
haskellNix = import sources."haskell.nix" {};
pkgs = import haskellNix.sources.nixpkgs-unstable haskellNix.nixpkgsArgs;
in pkgs.haskell-nix.cabalProject {
compiler-nix-name = "ghc8103";
# src = ./.;
src = pkgs.haskell-nix.haskellLib.cleanGit { name = "test-1013"; src = ./.; };
}
[hamish@x86_64-builder:~/iohk/test-1013]$ nix-shell --run 'echo ok'
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: No index state specified for test-1013, using the latest index state that we know about (2021-02-03T00:00:00Z)!
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: Warning: `stdenv.lib` is deprecated and will be removed in the next release. Please use `pkgs.lib` instead. For more information see https://github.com/NixOS/nixpkgs/issues/108938
trace: To make project.plan-nix for hoogle a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for hoogle entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
ok
[hamish@x86_64-builder:~/iohk/test-1013]$ ls -l foo.sock
srwxr-xr-x 1 hamish users 0 Feb 4 04:05 foo.sock
Calling callStackToNix
and mkPkgSet
manually:
nix-build nix/tiko.nix -A tiko-apps.tiko-systems
trace: To make project.stack-nix for tikoHsSrc a fixed-output derivation but not materialized, set `stack-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.stack-nix for tikoHsSrc entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
building '/nix/store/7spbpxyhj42582r399lxq6nrm2z7vr10-tikoHsSrc-stack-to-nix-pkgs.drv'...
substituteStream(): WARNING: pattern '/nix/store/nw2igscbaa9f26irawxaikdxxcqz695v-tikoHsSrc' doesn't match anything in file '/nix/store/vqp59h05ql6ykpf831qmfqilf2r50gr1-tikoHsSrc-stack-to-nix-pkgs/default.nix'
substituteStream(): WARNING: pattern '/nix/store/nw2igscbaa9f26irawxaikdxxcqz695v-tikoHsSrc' doesn't match anything in file '/nix/store/vqp59h05ql6ykpf831qmfqilf2r50gr1-tikoHsSrc-stack-to-nix-pkgs/pkgs.nix'
unpacking 'https://github.com/input-output-hk/stackage.nix/archive/273aed7b946fd83a8e83d4d8fa9e4f6318cfae62.tar.gz'...
warning: dumping very large path (> 256 MiB); this may run out of memory
error: writing to file: Broken pipe
(use '--show-trace' to show detailed location information)
Using stackProject'
:
nix-build nix/tiko.nix -A tiko-apps.tiko-systems --show-trace
trace: To make project.stack-nix for tikoHsSrc a fixed-output derivation but not materialized, set `stack-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.stack-nix for tikoHsSrc entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
building '/nix/store/4m7sj4nhc63fcs5z1kqbnfpbzpr9756a-stack-repos.drv'...
substituteStream(): WARNING: pattern '# nix-sha256:' doesn't match anything in file 'stack.yaml'
building '/nix/store/zkqjxbvvy14zaq57l4m5nvdqp552n00v-cache-file.drv'...
building '/nix/store/sb1ls548qbfk8bzn3zfzss6xx0wg3kk3-tikoHsSrc-stack-to-nix-pkgs.drv'...
substituteStream(): WARNING: pattern '/nix/store/nw2igscbaa9f26irawxaikdxxcqz695v-tikoHsSrc' doesn't match anything in file '/nix/store/pixyhknryys8l4his30jydcp1qrg1ar3-tikoHsSrc-stack-to-nix-pkgs/default.nix'
substituteStream(): WARNING: pattern '/nix/store/nw2igscbaa9f26irawxaikdxxcqz695v-tikoHsSrc' doesn't match anything in file '/nix/store/pixyhknryys8l4his30jydcp1qrg1ar3-tikoHsSrc-stack-to-nix-pkgs/pkgs.nix'
warning: dumping very large path (> 256 MiB); this may run out of memory
error: while evaluating the attribute 'src' of the derivation 'tiko-systems-lib-tiko-systems-0.0.0.0' at /nix/store/v2q65pfhcy79iarkm415srahigqic0zw-nixos-20.03/pkgs/stdenv/generic/make-derivation.nix:191:11:
file '/home/philipp/Projects/tiko/tiko/out/mydata/mysock.sock' has an unsupported type
Using cabalProject
:
nix-build nix/tiko.nix -A tiko-apps.tiko-systems --show-trace
trace: No index state specified for tikoHsSrc, using the latest index state that we know about (2021-02-01T00:00:00Z)!
building '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv'...
...
No .cabal file found, running hpack on ./package.yaml
WARNING: Specified file "README.md" for extra-source-files does not exist
generated ./tiko-systems.cabal
Using index-state 2021-02-01T00:00:00Z
Warning: The package list for 'hackage.haskell.org-at-2021-02-01T000000Z' is
18662 days old.
Run 'cabal update' to get the latest list of available packages.
Warning: Requested index-state2021-02-01T00:00:00Z is newer than
'hackage.haskell.org-at-2021-02-01T000000Z'! Falling back to older state
(2021-01-31T23:42:25Z).
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: tiko-systems-0.0.0.0 (user goal)
[__1] unknown package: tiko-XXXX (dependency of tiko-systems)
[__1] fail (backjumping, conflict set: tiko-XXXX, tiko-systems)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: tiko-systems, tiko-XXXX
...
builder for '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv' failed with exit code 1
building '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv'...
...
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: tiko-systems-0.0.0.0 (user goal)
[__1] unknown package: tiko-XXXX (dependency of tiko-systems)
[__1] fail (backjumping, conflict set: tiko-XXXX, tiko-systems)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: tiko-systems, tiko-XXXX
builder for '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv' failed with exit code 1
error: while evaluating the attribute 'hsPkgs.tiko-systems' at /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:536:15:
while evaluating the attribute 'hsPkgs' at /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:516:43:
while evaluating 'mkCabalProjectPkgSet' at /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:152:13, called from /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:501:30:
while evaluating the attribute 'buildPackages.haskell-nix.compiler."${(((plan-pkgs).pkgs hackage)).compiler.nix-name}".version' at /nix/store/v2q65pfhcy79iarkm415srahigqic0zw-nixos-20.03/lib/attrsets.nix:344:7:
while evaluating anonymous function at /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/lib/import-and-filter-project.nix:5:1, called from /nix/store/hhjl2zsr3anjvrgway9zms0af00j4za7-haskell.nix_2021-01-06/overlays/haskell.nix:503:31:
build of '/nix/store/l5lx3lhb60k870pghig5y6bndgl2c8k7-tikoHsSrc-plan-to-nix-pkgs.drv' failed
For the cabalProject
try I just changed the top-level nix-call, so it's not surprising that it fails because I am pretty sure the cabal files would need to be modified to match our stack overrides. However, I guess the fact that it calls cabal shows that with cabalProject the source filtering works?
Just a quick debugging suggestion: If you create a file without read permissions in your project dir (but not check it into git) then your nix expression will immediately break if the cleaning step fails (like it does here). Even though it breaks the nix expression immediately you might prefer that over silently filling up your nix store with overly large source derivations.
(Could you please rename this issue to something along "cleaning is broken for multiple-package projects"? Thanks!)
Could you please rename this issue to something along "cleaning is broken for multiple-package projects"?
I think it is broken for stack projects. I could be wrong, but so far I have not seen an example of a cabal.project
based project where it is broken in haskell.nix. I also don't think it would work for a "single-package" stack project (files in the package dir that should be filtered would be copied to the store as well).
Once you are in a nix-shell
, cabal build
will use the ghc-pkg list
packages (as far as I know stack does not). For this reason I recommend using cabal.project
files and haskell-nix.cabalProject
for your nix-shell
.
It's definitely not Stack only as this effects us, but we don't use stack! I have a to-do to try and investigate more
@ocharles do you have a package in the root of your project? It looks like that was broken. The test case had both packages in subdirs. I came across filtering issues while updating Leksah to the latest haskell.nix. There were two bugs, one resulted in all the files being copied and the other one (that was masked by the first) caused filtering to when it ran to leave files out. #1041 fixes it for Leksah.
Do you mean a .cabal
file in the root directory? I don't think so, but I will check out #1041
A .cabal
file in the root of the project (same dir as cabal.project
). With packages: .
in the cabal.project
.
I'm pretty sure we don't have that but I'll double check tomorrow
Has there been any attempt to fix this issue? We've just tried upgrading haskell.nix on our large, monorepo stack project in order to use a newer LTS and GHC 8.10.4, but copying our entire project source tree for every component means it takes > 30 minutes before anything even gets built.
Update: nevermind. I just figured out that it is the includeSiblings = true
here https://github.com/input-output-hk/haskell.nix/blob/22ad57c7e78a6bf409738efdd87a2c23d13bfabb/lib/import-and-filter-project.nix#L55 that is causing ths problem for us. Is there a way to override that? For a large monorepo, this is not a good default.
Please try out #1074 as it might help.
The includeSiblings = true
in import-and-filter-project.nix
should only be a problem if per component level filtering is not working (or if you had something like extra-source-files: ../../**.txt
in your .cabal
file).
Are you using hpack
? That currently breaks the component level filtering. We should probably filter just the package dir when that happens, but I think we might include everything (just in case hpack wants to reference files outside the package dir).
The issue I was having was after callStackToNix
. Because of the includeSiblings = true
in importAndFilter
function, every component that was built was causing the entire source tree to be copied into the nix store, once for every component. We have hundreds (maybe 1000+) components in our project, so that meant we were getting that many copies of our entire source tree in the nix store. I created a fork to test disabling that and it is now much much faster and the build takes up a fraction of the disk space.
That indicates that the component filter is failing completely (the files should be filtered when the component builds to just the ones referenced in the .cabal
file). I would really like to get to the bottom of why that is not working. It could be a problem with the output of callStackToNix
(please do try #1074) or it could be something else. If you are using hpack
it could be to do with that.
Unfortunately just changing to includeSiblings = false
would break the ability to use relative directories in .cabal
files to reference files in other packages.
Perhaps try the #1074 branch with this trace enabled. It should tell log for every file in the component the reason it was not filtered (if it prints nothing then the filter is not even running).
I have checked #1074 issue I was able to reproduce before (a file foo
in the root of the project with no read access does not fail). The filtering also seems to exclude the file correctly (but it did before too).
I used pkgs.srcOnly
to extract the source files for the component derivation like this:
git clone https://github.com/hamishmack/test-1013.git
cd test-1013
find $(nix-build -E 'let p = import ./default.nix; in p.pkgs.srcOnly p.lib.components.library' --no-out-link)
Only the files for the component are were included.
@hamishmack I tried it and got the same result. I think the issue is that there are two source trees copied to the nix store, one is filtered down to just the component source, while the other contains all of our source. I build a single package and looked for the source afterwards, and found this
$ ls /nix/store/80ww5zhf9xjj2f0f3v49aqmiiic8s4b8-range-root-devtools-simlint-root /nix/store/qln3qdfyx9nmkl4gilrwr5igqis0xpgh-range-root-devtools-simlint
/nix/store/80ww5zhf9xjj2f0f3v49aqmiiic8s4b8-range-root-devtools-simlint-root:
devtools
/nix/store/qln3qdfyx9nmkl4gilrwr5igqis0xpgh-range-root-devtools-simlint:
administration-client defender-logs jwt-config-simspace portal-services-client simspace-generics
aeson-simspace deployment-target-agent jwt-middleware-simspace project simspace-h5p-drupal
apitest devtools jwt-test-simspace PULL_REQUEST_TEMPLATE.md simspace-ids
attack-designer doc keycloak range-query simspace-immortal
attack-integrations docker lariat-core range-server simspace-pagination
authz docker-compose.yml learning range-server-api simspace-servant-docs
bors.toml documentation-generation lib range-server-ts-gen simspace-snapshot.yaml
brittany document-server logger-handle README.md simspace-squeal
BSD3-LICENSE event-controller Makefile reps simspace-test
build.sbt event-historian mattermost request-logger simspace-tracing
caching event-template MIT-LICENSE request-logger-simspace simtosca
campaign-status except-simspace monad-logger-extra resource-manager simyaml
client hardhat-expansion mto s3-simspace simyamltest
common-package.yaml hardhat-server nagios SAMLSetup.md stack.yaml
common-testlib hardhat-traffic-config network-impact sample-data team-assessment
common-utilities hardhat-ts-gen nix scoring time-simspace
content-repo hardhat-versioning-library notification servant-convert tools
control-point hspec-extra notification-queue servant-range-proxy tracker-client
datagen http-error package.json servant-simjson ts-gen
data-server human-readable perf-testing servant-zoo ttx
db-infra Jenkinsfile persistent-runners shared uri-utils
db-migration-test json-logger portal-client simjson user-emulation
db-schema json-query-string portal-runner simspace-amazonka warden
default.nix json-websocket-client portal-server simspace-db-seed-user yarn.lock
When I use my fork with includeSiblings = false
, both range-root-devtools-simlint-root
and range-root-devtools-simlint
contain only the package/component source.
Using
project.shellFor
results in multiple (one for each package) copies of whole project directory (including dist-newstyle) in nix store. It happens duringclean-source-with-nix
evaluation that is part ofproject {..}
evaluation.Is there any way to avoid it? Growing nix store by several GB on each
nix-shell
invocation is pretty harsh.Looks like it is related to sources being needed for cabal setup (but I'm not sure here). In this case, is it possible to manually assemble pkgset and avoid this copying steps?