Cleaning is broken for stack projects

[quetz]

Using project.shellFor results in multiple (one for each package) copies of whole project directory (including dist-newstyle) in nix store. It happens during clean-source-with-nix evaluation that is part of project {..} evaluation.

Is there any way to avoid it? Growing nix store by several GB on each nix-shell invocation is pretty harsh.

Looks like it is related to sources being needed for cabal setup (but I'm not sure here). In this case, is it possible to manually assemble pkgset and avoid this copying steps?

[hamishmack]

I have merged #1074 into master as it at least stops callStackToNix from depending on more than it should. Try chmod -r on a directory that it should not need. Then run nix-build with --show-trace. That might give us a clue as to what is trying to copy the files in that directory to the store (or if we are unlucky it will just skip them).

[hamishmack]

Actually chmod -r on a directory won't work (callStackToNix tries to include all directories). chmod -r on a file instead might work.

[hamishmack]

I think something must be accessing the package src without filtering it for a component first. I can't see it in haskell.nix, but it is hard to check this sort of thing without a type system.

[purefn]

I think something must be accessing the package src without filtering it for a component first. I can't see it in haskell.nix, but it is hard to check this sort of thing without a type system.

I haven't been able to track it down either. But having that includeSiblings = true makes haskell.nix unusable for us. So I guess we'll be sticking with our fork that sets it to false for now.

[purefn]

I changed includeSiblings in importAndFilterProject to throw an exception. Here's the stack trace

error: while evaluating the attribute 'prePatch' of the derivation 'simlint-lib-simlint-0.2.0.0' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/pkgs/stdenv/generic/make-derivation.nix:191:11:
while evaluating the attribute 'prePatch' at /home/rwallace/haskell.nix/builder/comp-builder.nix:251:7:
while evaluating 'optionalString' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/strings.nix:180:5, called from /home/rwallace/haskell.nix/builder/comp-builder.nix:259:10:
while evaluating the attribute 'subDir' at /home/rwallace/haskell.nix/lib/default.nix:321:5:
while evaluating anonymous function at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:84:45, called from undefined position:
while evaluating the attribute 'value' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:379:9:
while evaluating the option `packages.simlint.src':
while evaluating the attribute 'mergedValue' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:411:5:
while evaluating anonymous function at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:413:22, called from /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:413:9:
while evaluating 'merge' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/types.nix:567:20, called from /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/modules.nix:416:12:
while evaluating 'mergeEqualOption' at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/options.nix:108:27, called from /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/types.nix:572:21:
while evaluating anonymous function at /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/options.nix:110:23, called from /nix/store/8iyvwzjld11rspy3bjq7d3ra50z55agz-nixpkgs-2003-src/lib/options.nix:110:10:
while evaluating 'filter'' at /home/rwallace/haskell.nix/lib/clean-source-with.nix:67:23, called from undefined position:

So the entire source tree is being copied for each component because of the prePatch, which evaluates cleanSrc'.subDir, which evaluates packages.simlint.src. I figured as much as that, so this didn't prove too helpful. So I started adding traces all over the source filtering code. Eventually I added a trace to cleanSourceWith

      outPath = lib.traceValFn (x: "${toString name} ${name'} outPath ${x}") ((builtins.path { filter = filter'; path = origSrc; name = name'; }) + origSubDir);

Now I'm able to see that each of the nix store paths that have a full copy of our repo each have a name = null, and that is the only difference between them. So I looked into why that is null, and it turns out that's because appendSubDir doesn't pass a name to cleanSourceWith. I tried adding one, just using inherit (src) name; for a quick test.

Now I can see that all the nix store paths produced by appendSubDir in importAndFilterProject are pointing to the same place

trace: range range outPath /nix/store/n38d83dnga3gk9s6sblzaf57ijpk3yf0-range
trace:  range outPath /nix/store/34ivvl2fny4rs6c07lky828la2ky3rip-range
trace: range-root range-root outPath /nix/store/vcq6hfby2ynyfzhcpvm3j380ds30ybpa-range-root/except-simspace
trace: range-root-root range-root-root outPath /nix/store/6kqrv6xl4jb8cnid3s18z3chjdw3qggx-range-root-root
trace: range-root range-root outPath /nix/store/vcq6hfby2ynyfzhcpvm3j380ds30ybpa-range-root/lib/simspace-prelude
trace: range-root-root range-root-root outPath /nix/store/y4cj5c8agp0wijvx9inj14fkzinhlayy-range-root-root
trace: range-root range-root outPath /nix/store/vcq6hfby2ynyfzhcpvm3j380ds30ybpa-range-root/simjson
trace: range-root-root range-root-root outPath /nix/store/j8z901jmzf1x93kgj966i3dc139r3097-range-root-root
trace: range-root range-root outPath /nix/store/vcq6hfby2ynyfzhcpvm3j380ds30ybpa-range-root/time-simspace
trace: range-root-root range-root-root outPath /nix/store/v65731b92yca3fjzbgkxr5bfikbbwy07-range-root-root

so they are not creating n duplicates of our source code in the nix store. Now I only have two copies, the one produced by cleanGit and then the one produced by importAndFilterProject. All the other nix store paths, the range-root-root variety (I suspect we can fix them up to include the sub dir path in the name but haven't done that yet), only contain the source for the component being built. :tada:

Unfortunately, this is still painfully slow and unusable. :sob: I don't have any insights as to why that might be.

[domenkozar]

Using https://github.com/hercules-ci/gitignore.nix things work again using the latest master. Thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.