Referencing source dependencies lead to shadowed dependency error

dmalikov commented 6 years ago

According to https://github.com/Gabriel439/haskell-nix/blob/master/project1/README.md#source-dependencies it's possible to reference a local source dependency via release.nix.

However, sometimes it leads to dependency shadowing (if I'm getting this error right):

libb >> nix-shell --command zsh
these derivations will be built:
  /nix/store/2ddm8i6my675a7lncm4frkcasprf5gxl-liba-0.1.0.0.drv
  /nix/store/7wmsgz49759pff7ikky2nksxn8cckr63-ghc-8.4.3-with-packages.drv
building '/nix/store/2ddm8i6my675a7lncm4frkcasprf5gxl-liba-0.1.0.0.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/nv4h00ma4lmwj9bnc8dhf9nzyj28d6ds-ghc-8.4.3.
ignoring (possibly broken) abi-depends field for packages
ignoring (possibly broken) abi-depends field for packages
unpacking sources
unpacking source archive /nix/store/nf0yf8xsi7l92ark5q08ddhrkh2syy9b-hi
source root is hi
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/private/tmp/nix-build-liba-0.1.0.0.drv-0/setup-package.conf.d -j1 -threaded
Loaded package environment from /private/tmp/nix-build-liba-0.1.0.0.drv-0/hi/.ghc.environment.x86_64-darwin-8.4.3
<command line>: cannot satisfy -package-id liba-0.1.0.0-inplace:
    liba-0.1.0.0-inplace is unusable due to missing dependencies:
      aeson-1.2.4.0-KPiiXcK6dELrtdAOauWhV
    (use -v for more information)
builder for '/nix/store/2ddm8i6my675a7lncm4frkcasprf5gxl-liba-0.1.0.0.drv' failed with exit code 1
cannot build derivation '/nix/store/7wmsgz49759pff7ikky2nksxn8cckr63-ghc-8.4.3-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/7wmsgz49759pff7ikky2nksxn8cckr63-ghc-8.4.3-with-packages.drv' failed

So, there is libb depends on liba, and in libb/release.nix there is a reference:

{ compiler }:

let
  config = {
    packageOverrides = pkgs: rec {
      haskell = pkgs.haskell // {
        packages = pkgs.haskell.packages // {
          "${compiler}" = pkgs.haskell.packages."${compiler}".override {
            overrides = haskellPackagesNew: haskellPackagesOld: rec {
              liba = haskellPackagesNew.callPackage ../default.nix { };
              libb = haskellPackagesNew.callPackage ./default.nix { };
            };
          };
        };
      };
    };
  };

  pkgs = import <nixpkgs> { inherit config; };

in
  { project = pkgs.haskell.packages.${compiler}.libb;
  }

Minimal reproducible example could be found there - https://github.com/dmalikov/repro-shadowed-deps/commit/7aa926d7458b02ee6d15f7342433620de2077383

Gabriella439 commented 6 years ago

@dmalikov: I think you should open an issue against Nixpkgs because this should work and this is one of the use cases documented in the Nixpkgs manual here:

https://nixos.org/nixpkgs/manual/#how-to-build-projects-that-depend-on-each-other

Gabriella439 commented 6 years ago

Also, I have an older version of nixpkgs installed, so I had to make some tweaks to your example to run it (like using an older version of GHC), but I wasn't able to reproduce the failure:

$ nix-shell --command zsh
...
building path(s) ‘/nix/store/a4m6d9vfjvaiygi9nk7cwm5r7d6c46x6-liba-0.1.0.0-doc’, ‘/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0’
setupCompilerEnvironmentPhase
Build with /nix/store/h4g79r7nn469mskvl1a0ij7ngj5ky9s4-ghc-8.0.2.
unpacking sources
unpacking source archive /nix/store/i0837ni9k63vaqp5ff7nds5sp09zmyfy-repro-shadowed-deps
source root is repro-shadowed-deps
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/private/tmp/nix-build-liba-0.1.0.0.drv-0/package.conf.d -j8 -threaded
[1 of 1] Compiling Main             ( Setup.hs, /private/tmp/nix-build-liba-0.1.0.0.drv-0/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --docdir=/nix/store/a4m6d9vfjvaiygi9nk7cwm5r7d6c46x6-liba-0.1.0.0-doc/share/doc --with-gcc=clang --package-db=/private/tmp/nix-build-liba-0.1.0.0.drv-0/package.conf.d --ghc-option=-optl=-Wl,-headerpad_max_install_names --ghc-option=-j8 --disable-split-objs --disable-library-profiling --disable-profiling --enable-shared --disable-coverage --enable-library-vanilla --enable-executable-dynamic --enable-tests --extra-include-dirs=/nix/store/ifqjyzz2paal834dj3kaxairjkhcnfg6-libc++-4.0.1/include --extra-lib-dirs=/nix/store/ifqjyzz2paal834dj3kaxairjkhcnfg6-libc++-4.0.1/lib
Configuring liba-0.1.0.0...
Dependency aeson -any: using aeson-1.1.2.0
Dependency base >=4.9 && <5: using base-4.9.1.0
Using Cabal-1.24.2.0 compiled by ghc-8.0
Using compiler: ghc-8.0.2
Using install prefix: /nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0
Binaries installed in:
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/bin
Libraries installed in:
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/lib/ghc-8.0.2/liba-0.1.0.0
Dynamic libraries installed in:
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/lib/ghc-8.0.2/x86_64-osx-ghc-8.0.2
Private binaries installed in:
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/libexec
Data files installed in:
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/share/x86_64-osx-ghc-8.0.2/liba-0.1.0.0
Documentation installed in:
/nix/store/a4m6d9vfjvaiygi9nk7cwm5r7d6c46x6-liba-0.1.0.0-doc/share/doc
Configuration files installed in:
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/etc
No alex found
Using ar found on system at:
/nix/store/hx4n5nzjy21482vmfg43ykhjy8wvz4af-cctools-binutils-darwin/bin/ar
No c2hs found
No cpphs found
Using gcc version 4.2.1 given by user at:
/nix/store/kn7dqpg4xig6h9sqj51qn1lczzma0p93-clang-wrapper-4.0.1/bin/clang
Using ghc version 8.0.2 found on system at:
/nix/store/h4g79r7nn469mskvl1a0ij7ngj5ky9s4-ghc-8.0.2/bin/ghc
Using ghc-pkg version 8.0.2 found on system at:
/nix/store/h4g79r7nn469mskvl1a0ij7ngj5ky9s4-ghc-8.0.2/bin/ghc-pkg
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.17.3 found on system at:
/nix/store/h4g79r7nn469mskvl1a0ij7ngj5ky9s4-ghc-8.0.2/bin/haddock
No happy found
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.67 found on system at:
/nix/store/h4g79r7nn469mskvl1a0ij7ngj5ky9s4-ghc-8.0.2/bin/hpc
Using hsc2hs version 0.68.1 found on system at:
/nix/store/h4g79r7nn469mskvl1a0ij7ngj5ky9s4-ghc-8.0.2/bin/hsc2hs
Using hscolour version 1.24 found on system at:
/nix/store/63rk94hdqy94kgrrlyjf5iv6dcmvsynr-hscolour-1.24.2/bin/HsColour
No jhc found
Using ld found on system at:
/nix/store/kn7dqpg4xig6h9sqj51qn1lczzma0p93-clang-wrapper-4.0.1/bin/ld
No lhc found
No lhc-pkg found
No pkg-config found
Using strip found on system at:
/nix/store/hx4n5nzjy21482vmfg43ykhjy8wvz4af-cctools-binutils-darwin/bin/strip
Using tar found on system at:
/nix/store/1d61rvl2vz19xrf6h5dazxks1wnpb2rg-gnutar-1.29/bin/tar
No uhc found
building
Building liba-0.1.0.0...
Preprocessing library liba-0.1.0.0...
[1 of 1] Compiling LibA             ( src/LibA.hs, dist/build/LibA.o )
running tests
Package has no test suites.
haddockPhase
Running Haddock for liba-0.1.0.0...
Running hscolour for liba-0.1.0.0...
Preprocessing library liba-0.1.0.0...
Preprocessing library liba-0.1.0.0...
Haddock coverage:
   0% (  0 /  2) in 'LibA'
  Missing documentation for:
    Module header
    f (src/LibA.hs:5)
Documentation created: dist/doc/html/liba/index.html,
dist/doc/html/liba/liba.txt
installing
Installing library in
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/lib/ghc-8.0.2/liba-0.1.0.0
Creating package registration file:
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/lib/ghc-8.0.2/package.conf.d/liba-0.1.0.0.conf
post-installation fixup
stripping (with flags -S) in /nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/lib
patching script interpreter paths in /nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0
patching script interpreter paths in /nix/store/a4m6d9vfjvaiygi9nk7cwm5r7d6c46x6-liba-0.1.0.0-doc
building path(s) ‘/nix/store/72mxqrjl4d2s1jfvv0i7wq9a1g155iax-ghc-8.0.2-with-packages’
/nix/store/r3v9airk5avfmq8qwy2mvs32mvzx3prf-random-1.1/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/9yd6myqb4sh3km06ydn1vayrbsnxj98r-uuid-types-1.0.3/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/mljwwdxb4krrzx6z50snz8l65acczasl-unordered-containers-0.2.8.0/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/daaqkfy3b6sb7vl4qd15ji540v78cxql-time-locale-compat-0.1.1.3/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/ac5zmcj8mvhcczcznxxf5adiyazy5k0m-transformers-compat-0.5.1.4/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/3fmh45v33p7xhnz6nd5mjif0nvjnc43m-tagged-0.8.5/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/7hq9dcl52d59r368zq94g7gvks9j1ww6-dlist-0.8.0.3/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/cgcy4pbj62csjanx9yjxs9g2hwsvrkss-base-compat-0.9.3/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/b5i9xym3c289l6k6hk3j303l7v1nmwj9-primitive-0.6.2.0/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/786llhijwdp3ny8y6w72905d631mrx5z-integer-logarithms-1.0.2/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/v4c1cq2v1jls5qf18dabznvvfgmm6kd4-text-1.2.2.2/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/nbwgf8alb80fy0j2zbm3ickfn9rs0r53-hashable-1.2.6.1/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/j3ligj2dxlha0igr4j5hj650b72x0y5y-scientific-0.3.5.2/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/h80dx8dzkf8jkijl9kvp82z53y8i471f-attoparsec-0.13.1.0/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/8zghq24j4any01z9rwvhp5bsx9i3y2g6-aeson-1.1.2.0/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
/nix/store/nrfh8p3g69ij4bfim1rzw7hw48n14w59-liba-0.1.0.0/nix-support:
propagated-native-build-inputs: /nix/store/n34c66g4c0akp7g69p934l5yg8z93ps4-vector-0.12.0.1/nix-support/propagated-native-build-inputs
%

One thing to check is to possibly delete your ~/.cabal or ~/.ghc directory. If you've built or installed anything there outside of Nix it will interfere with your Nix builds

ElvishJerricco commented 6 years ago

I also cannot reproduce this with a very new nixpkgs.

One thing to check is to possibly delete your ~/.cabal or ~/.ghc directory. If you've built or installed anything there outside of Nix it will interfere with your Nix builds

@Gabriel439 How so? Nix builds have HOME=/homeless-shelter, so they don't use your home files.

Gabriella439 commented 6 years ago

@ElvishJerricco: You're right. ~/.ghc or ~/.cabal wouldn't have an effect here

dmalikov commented 6 years ago

@Gabriel439

I think you should open an issue against Nixpkgs

Sure, will do.

but I wasn't able to reproduce the failure

Could you please try to run nix-shell --command zsh from the libb directory? Also, why is it built with ghc802? In the correspondent shell.nix there is ghc843 set.

Gabriella439 commented 6 years ago

@dmalikov: That is what I did. I ran nix-shell --command zsh from the libb subdirectory of the project, so it built liba along the way

The reason I built with ghc802 was that my current installed version of nixpkgs is old and I updated your example code to reference ghc802 instead of ghc843

Gabriella439 / haskell-nix

Referencing source dependencies lead to shadowed dependency error #56