Closed quetz closed 1 year ago
I have merged #1074 into master as it at least stops callStackToNix
from depending on more than it should. Try chmod -r
on a directory that it should not need. Then run nix-build
with --show-trace
. That might give us a clue as to what is trying to copy the files in that directory to the store (or if we are unlucky it will just skip them).
Actually chmod -r
on a directory won't work (callStackToNix
tries to include all directories). chmod -r
on a file instead might work.
I think something must be accessing the package src
without filtering it for a component first. I can't see it in haskell.nix, but it is hard to check this sort of thing without a type system.
I think something must be accessing the package
src
without filtering it for a component first. I can't see it in haskell.nix, but it is hard to check this sort of thing without a type system.
I haven't been able to track it down either. But having that includeSiblings = true
makes haskell.nix unusable for us. So I guess we'll be sticking with our fork that sets it to false for now.
I changed includeSiblings
in importAndFilterProject
to throw an exception. Here's the stack trace
error: while evaluating the attribute 'prePatch' of the derivation 'simlint-lib-simlint-0.2.0.0' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/pkgs/stdenv/generic/make-derivation.nix:191:11:
while evaluating the attribute 'prePatch' at /home/rwallace/haskell.nix/builder/comp-builder.nix:251:7:
while evaluating 'optionalString' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/strings.nix:180:5, called from /home/rwallace/haskell.nix/builder/comp-builder.nix:259:10:
while evaluating the attribute 'subDir' at /home/rwallace/haskell.nix/lib/default.nix:321:5:
while evaluating anonymous function at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:84:45, called from undefined position:
while evaluating the attribute 'value' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:379:9:
while evaluating the option `packages.simlint.src':
while evaluating the attribute 'mergedValue' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:411:5:
while evaluating anonymous function at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:413:22, called from /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:413:9:
while evaluating 'merge' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/types.nix:567:20, called from /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:416:12:
while evaluating 'mergeEqualOption' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/options.nix:108:27, called from /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/types.nix:572:21:
while evaluating anonymous function at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/options.nix:110:23, called from /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/options.nix:110:10:
while evaluating 'filter'' at /home/rwallace/haskell.nix/lib/clean-source-with.nix:67:23, called from undefined position:
So the entire source tree is being copied for each component because of the prePatch
, which evaluates cleanSrc'.subDir
, which evaluates packages.simlint.src
. I figured as much as that, so this didn't prove too helpful. So I started adding trace
s all over the source filtering code. Eventually I added a trace to cleanSourceWith
outPath = lib.traceValFn (x: "${toString name} ${name'} outPath ${x}") ((builtins.path { filter = filter'; path = origSrc; name = name'; }) + origSubDir);
Now I'm able to see that each of the nix store paths that have a full copy of our repo each have a name = null
, and that is the only difference between them. So I looked into why that is null
, and it turns out that's because appendSubDir
doesn't pass a name to cleanSourceWith
. I tried adding one, just using inherit (src) name;
for a quick test.
Now I can see that all the nix store paths produced by appendSubDir
in importAndFilterProject
are pointing to the same place
trace: range range outPath /nix/store/n38d83dnga3gk9s6sblzaf57ijpk3yf0-range
trace: range outPath /nix/store/34ivvl2fny4rs6c07lky828la2ky3rip-range
trace: range-root range-root outPath /nix/store/vcq6hfby2ynyfzhcpvm3j380ds30ybpa-range-root/except-simspace
trace: range-root-root range-root-root outPath /nix/store/6kqrv6xl4jb8cnid3s18z3chjdw3qggx-range-root-root
trace: range-root range-root outPath /nix/store/vcq6hfby2ynyfzhcpvm3j380ds30ybpa-range-root/lib/simspace-prelude
trace: range-root-root range-root-root outPath /nix/store/y4cj5c8agp0wijvx9inj14fkzinhlayy-range-root-root
trace: range-root range-root outPath /nix/store/vcq6hfby2ynyfzhcpvm3j380ds30ybpa-range-root/simjson
trace: range-root-root range-root-root outPath /nix/store/j8z901jmzf1x93kgj966i3dc139r3097-range-root-root
trace: range-root range-root outPath /nix/store/vcq6hfby2ynyfzhcpvm3j380ds30ybpa-range-root/time-simspace
trace: range-root-root range-root-root outPath /nix/store/v65731b92yca3fjzbgkxr5bfikbbwy07-range-root-root
so they are not creating n
duplicates of our source code in the nix store. Now I only have two copies, the one produced by cleanGit
and then the one produced by importAndFilterProject
. All the other nix store paths, the range-root-root
variety (I suspect we can fix them up to include the sub dir path in the name but haven't done that yet), only contain the source for the component being built. :tada:
Unfortunately, this is still painfully slow and unusable. :sob: I don't have any insights as to why that might be.
Using https://github.com/hercules-ci/gitignore.nix things work again using the latest master. Thanks!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using
project.shellFor
results in multiple (one for each package) copies of whole project directory (including dist-newstyle) in nix store. It happens duringclean-source-with-nix
evaluation that is part ofproject {..}
evaluation.Is there any way to avoid it? Growing nix store by several GB on each
nix-shell
invocation is pretty harsh.Looks like it is related to sources being needed for cabal setup (but I'm not sure here). In this case, is it possible to manually assemble pkgset and avoid this copying steps?