search

snowleopard / hadrian

Hadrian: a new build system for the Glasgow Haskell Compiler. Now merged into the GHC tree!
https://gitlab.haskell.org/ghc/ghc/tree/master/hadrian
MIT License
208 stars 39 forks source link

shell.nix depends on build artifacts #532

Closed int-index closed 5 years ago

int-index commented 6 years ago

This issue has been migrated to Trac: https://ghc.haskell.org/trac/ghc/ticket/15794

Steps to reproduce:

  1. git clone --recursive git://git.haskell.org/ghc.git
  2. cd ghc/hadrian
  3. git pull https://github.com/snowleopard/hadrian && git checkout master
  4. nix-shell --pure

Expected result: Successful build.

Actual result:

hadrian(master)$ nix-shell --pure
these derivations will be built:
  /nix/store/lmpgkvswlkcgpcjdilbyh202vm8a5i5r-directory-1.3.1.5.drv
  /nix/store/rcg0mmq0bincjfjnzbabc8fq7307bylq-process-1.6.3.0.drv
  /nix/store/15alkpni823gkfp63c79962v032wxvjs-extra-1.6.4.drv
  /nix/store/1b54g6nv8rw0kq3i5xjpcbhy88jvdnbr-xml-1.3.14.drv
  /nix/store/89salv051l8yr2y2k52pzkqraw9lfib9-test-framework-0.8.1.1.drv
  /nix/store/iqq0xsvgf3hf551b3jj0x3lg661c5y4f-test-framework-hunit-0.3.0.2.drv
  /nix/store/cy7ynm7r9jb41kxb1gxbwdprhvl03ivk-parsec-3.1.13.0.drv
  /nix/store/rgnsvxfvmbhrkniy0c86v65jgvx8nsan-test-framework-quickcheck2-0.3.0.4.drv
  /nix/store/fyrj002axd2skdz370hi13fq6prjh5cz-network-uri-2.6.1.0.drv
  /nix/store/7qcfy5gy9sjcz7cijqghgj3cilm2pzps-HTTP-4000.3.8.drv
  /nix/store/gk0w3grh2hz008sp6ksy669zvl6pyznb-js-flot-0.8.3.drv
  /nix/store/h39chz7m8zvawlrzfjsz496vl2rbp921-hashable-1.2.6.1.drv
  /nix/store/zpv0xi25jq6p5kkpvbc11sq73dxi2g0a-unordered-containers-0.2.8.0.drv
  /nix/store/i7p5c3mv9p20z7kb2mrdlpx0fwmyz3f5-shake-0.16.2.drv
  /nix/store/pzfcvib6sbsaascsc7djzcfqzlw847gc-Cabal-2.3.0.0.drv
  /nix/store/j5w4i42pgfy03s7fjsj4dfkis26c3nnp-happy-1.19.8.drv
  /nix/store/pl2dxjvbqs36rh80q5i09zdhbh6hjl2r-alex-3.2.3.drv
  /nix/store/dck8p81zlij9qksglaw8hkp11msrm426-hadrian-0.1.0.0.drv
building '/nix/store/lmpgkvswlkcgpcjdilbyh202vm8a5i5r-directory-1.3.1.5.drv'...
building '/nix/store/1b54g6nv8rw0kq3i5xjpcbhy88jvdnbr-xml-1.3.14.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/d57p5gzyld7b3y4irzv70bfa3d8xxwgx-ghc-8.2.2.
setupCompilerEnvironmentPhase
Build with /nix/store/d57p5gzyld7b3y4irzv70bfa3d8xxwgx-ghc-8.2.2.
unpacking sources
unpacking source archive /nix/store/khjm3agcb1y91925r78h3chaq56w9fi7-6wsbrbb60dzbfqf62c6sskwcijc3arjj-directory
unpacking sources
unpacking source archive /nix/store/dnx7hpl7wz81r3av4l09pqqhdqi4qpg6-xml-1.3.14.tar.gz
source root is xml-1.3.14
source root is 6wsbrbb60dzbfqf62c6sskwcijc3arjj-directory
setting SOURCE_DATE_EPOCH to timestamp 1424727750 of file xml-1.3.14/xml.cabal
patching sources
patching sources
compileBuildDriverPhase
compileBuildDriverPhase
setupCompileFlags: -package-db=/tmp/nix-build-xml-1.3.14.drv-0/package.conf.d -j1 -threaded
setupCompileFlags: -package-db=/tmp/nix-build-directory-1.3.1.5.drv-0/package.conf.d -j1 -threaded
[1 of 1] Compiling Main             ( Setup.hs, /tmp/nix-build-directory-1.3.1.5.drv-0/Main.o )
[1 of 1] Compiling Main             ( Setup.hs, /tmp/nix-build-xml-1.3.14.drv-0/Main.o )
Linking Setup ...
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/v3yazy78gy6kkpvrcn3schm8lc8y5dgl-xml-1.3.14 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --docdir=/nix/store/zdzgar7mwysym9dbai87fann20pbw75n-xml-1.3.14-doc/share/doc --with-gcc=gcc --package-db=/tmp/nix-build-xml-1.3.14.drv-0/package.conf.d --ghc-option=-optl=-Wl,-rpath=/nix/store/v3yazy78gy6kkpvrcn3schm8lc8y5dgl-xml-1.3.14/lib/ghc-8.2.2/xml-1.3.14 --ghc-option=-j1 --disable-split-objs --disable-library-profiling --disable-profiling --enable-shared --disable-coverage --enable-library-vanilla --enable-executable-dynamic --enable-tests --ghc-option=-split-sections
configuring
configureFlags: --verbose --prefix=/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --docdir=/nix/store/w0gkzrf5q6ng40vzyvdl5m8pvw9v9dxy-directory-1.3.1.5-doc/share/doc --with-gcc=gcc --package-db=/tmp/nix-build-directory-1.3.1.5.drv-0/package.conf.d --ghc-option=-optl=-Wl,-rpath=/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5/lib/ghc-8.2.2/directory-1.3.1.5 --ghc-option=-j1 --disable-split-objs --disable-library-profiling --disable-profiling --enable-shared --disable-coverage --enable-library-vanilla --enable-executable-dynamic --enable-tests --ghc-option=-split-sections
Configuring directory-1.3.1.5...
Warning: The 'build-type' is 'Configure' but there is no 'configure' script.
You probably need to run 'autoreconf -i' to generate it.
Configuring xml-1.3.14...
Dependency base >=3 && <5: using base-4.10.1.0
Dependency bytestring -any: using bytestring-0.10.8.2
Dependency text -any: using text-1.2.3.0
Dependency base >=4.5 && <4.12: using base-4.10.1.0
Dependency directory -any: using directory-1.3.1.5
Dependency filepath >=1.3 && <1.5: using filepath-1.4.2
Dependency time >=1.4 && <1.9: using time-1.8.0.2
Dependency unix >=2.5.1 && <2.8: using unix-2.7.2.2
Source component graph:
    component lib
    component test:test dependency lib
Configured component graph:
    component directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK
        include base-4.10.1.0
        include time-1.8.0.2
        include filepath-1.4.2-DyDAQ5oOwBVDvLxMoNLDxx
        include unix-2.7.2.2
    component directory-1.3.1.5-8aJ3UecsjNUIxs4bq1vU8T-test
        include base-4.10.1.0
        include directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK
        include filepath-1.4.2-DyDAQ5oOwBVDvLxMoNLDxx
        include time-1.8.0.2
        include unix-2.7.2.2
Linked component graph:
    unit directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK
        include base-4.10.1.0
        include time-1.8.0.2
        include filepath-1.4.2-DyDAQ5oOwBVDvLxMoNLDxx
        include unix-2.7.2.2
        System.Directory=directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK:System.Directory,System.Directory.Internal=directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK:System.Directory.Internal,System.Directory.Internal.Prelude=directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK:System.Directory.Internal.Prelude
    unit directory-1.3.1.5-8aJ3UecsjNUIxs4bq1vU8T-test
        include base-4.10.1.0
        include directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK
        include filepath-1.4.2-DyDAQ5oOwBVDvLxMoNLDxx
        include time-1.8.0.2
        include unix-2.7.2.2
Ready component graph:
    definite directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK
        depends base-4.10.1.0
        depends time-1.8.0.2
        depends filepath-1.4.2-DyDAQ5oOwBVDvLxMoNLDxx
        depends unix-2.7.2.2
    definite directory-1.3.1.5-8aJ3UecsjNUIxs4bq1vU8T-test
        depends base-4.10.1.0
        depends directory-1.3.1.5-EFdDcGoPsV2JQnFPz1QOGK
        depends filepath-1.4.2-DyDAQ5oOwBVDvLxMoNLDxx
        depends time-1.8.0.2
        depends unix-2.7.2.2
Using Cabal-2.0.1.0 compiled by ghc-8.2
Using compiler: ghc-8.2.2
Using install prefix:
/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5
Executables installed in:
/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5/bin
Libraries installed in:
/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5/lib/ghc-8.2.2/directory-1.3.1.5
Dynamic Libraries installed in:
/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5/lib/ghc-8.2.2/x86_64-linux-ghc-8.2.2
Private executables installed in:
/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5/libexec/x86_64-linux-ghc-8.2.2/directory-1.3.1.5
Data files installed in:
/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5/share/x86_64-linux-ghc-8.2.2/directory-1.3.1.5
Documentation installed in:
/nix/store/w0gkzrf5q6ng40vzyvdl5m8pvw9v9dxy-directory-1.3.1.5-doc/share/doc
Configuration files installed in:
/nix/store/59r1c7sfgph9j2xxsc1k3j6dwkm32ybk-directory-1.3.1.5/etc
No alex found
Using ar found on system at:
/nix/store/aplvnhqdm6s9wj9r0jh46r46wvh65j86-binutils-2.28.1/bin/ar
No c2hs found
No cpphs found
No doctest found
Using gcc version 6.4.0 given by user at:
/nix/store/wriy1xis74fybcg3m1jnq5bd5myxvhm6-gcc-wrapper-6.4.0/bin/gcc
Using ghc version 8.2.2 found on system at:
/nix/store/d57p5gzyld7b3y4irzv70bfa3d8xxwgx-ghc-8.2.2/bin/ghc
Using ghc-pkg version 8.2.2 found on system at:
/nix/store/d57p5gzyld7b3y4irzv70bfa3d8xxwgx-ghc-8.2.2/bin/ghc-pkg
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.18.1 found on system at:
/nix/store/d57p5gzyld7b3y4irzv70bfa3d8xxwgx-ghc-8.2.2/bin/haddock
No happy found
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.67 found on system at:
/nix/store/d57p5gzyld7b3y4irzv70bfa3d8xxwgx-ghc-8.2.2/bin/hpc
Using hsc2hs version 0.68.2 found on system at:
/nix/store/d57p5gzyld7b3y4irzv70bfa3d8xxwgx-ghc-8.2.2/bin/hsc2hs
Using hscolour version 1.24 found on system at:
/nix/store/yf0bs1q9ph388jms8xlgcd5vh99amjm4-hscolour-1.24.2/bin/HsColour
No jhc found
Using ld found on system at:
/nix/store/wriy1xis74fybcg3m1jnq5bd5myxvhm6-gcc-wrapper-6.4.0/bin/ld
No lhc found
No lhc-pkg found
No pkg-config found
Using runghc version 8.2.2 found on system at:
/nix/store/d57p5gzyld7b3y4irzv70bfa3d8xxwgx-ghc-8.2.2/bin/runghc
Using strip version 2.28 found on system at:
/nix/store/aplvnhqdm6s9wj9r0jh46r46wvh65j86-binutils-2.28.1/bin/strip
Using tar found on system at:
/nix/store/vvq16kzwgx9yhkf0fwwms5xzgg0rwdpl-gnutar-1.29/bin/tar
No uhc found
Setup: configure script not found.
builder for '/nix/store/lmpgkvswlkcgpcjdilbyh202vm8a5i5r-directory-1.3.1.5.drv' failed with exit code 1
cannot build derivation '/nix/store/dck8p81zlij9qksglaw8hkp11msrm426-hadrian-0.1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/dck8p81zlij9qksglaw8hkp11msrm426-hadrian-0.1.0.0.drv' failed

Notice that the build fails for a library shipped with GHC. In shell.nix it's included as localPackage.

Workaround

First, we need to run configurePhase of the default Nix derivation for GHC HEAD:

  1. cd ghc
  2. nix-shell '<nixpkgs>' -A haskell.compiler.ghcHEAD
  3. configurePhase
  4. exit

Now we can cd into ghc/hadrian and proceed normally, the build won't fail.

The issue is that this step shouldn't be required, we must have a self-contained shell.nix.

snowleopard commented 6 years ago

Thanks @int-index!

Adding @mpickering.

mpickering commented 6 years ago

I don't really know how to fix this. What does hadrian do about this? As in order to build hadrian, I was under the impression you needed to build certain libraries bundled with GHC?

alpmestan commented 6 years ago

hadrian at the very least uses the Cabal that comes in the GHC source tree, the one that lives at libraries/Cabal/Cabal. So if you want to build a nix environment with hadrian compiled and ready to be used to develop GHC, you're always going to already build some of the code from the GHC source tree, and to rebuild hadrian when this code changes and you re-enter the environment/shell.

mpickering commented 6 years ago

The problem is that from a fresh clone, you can't build the libraries in the source tree? You have to run ./boot first in order to generate from cabal files?

int-index commented 6 years ago

Yes @mpickering, the problem is that shell.nix doesn't work on a fresh clone.

An additional problem is that sometimes after doing some work in a nix shell, then exiting it, and entering again, all libraries in the GHC source tree and Hadrian get rebuilt. It's probably nix not being smart enough and considering build artifacts when computing the hashes — this would be solved if we had srcdir / builddir separation, I suppose.

int-index commented 6 years ago

I don't understand why Hadrian is using libraries shipped with GHC to build itself, can't we use the ones from Hackage/nixpkgs?

mpickering commented 6 years ago

Alp seems to imply that only Cabal is necessary to build from the source tree. Perhaps we could try that? I don't actually use hadrian to do my builds though..

int-index commented 6 years ago

Does Hadrian need Cabal that is shipped with GHC specifically, or just a fresh enough Cabal that isn't on Hackage at the moment?

alpmestan commented 6 years ago

Well, it needs a Cabal that can process all the .cabal files from the GHC source tree. Using the in-tree one guarantees that we won't have any problem there, but you might want to try using the latest released version and seeing how that goes?

int-index commented 6 years ago

I won't be trying to do this any time soon, as I've got a shell.nix[1] that works fine for me with the old make-based build system.

[1] https://gist.github.com/int-index/244f1ea276afa7a611a3aff4edddd202

angerman commented 6 years ago

In general you likely want to build hadrian with ateast the cabal that comes with the ghc you want to build. That is because cabal and ghc are sometimes developed in tandem.

The Make based buildsystem built ghc-cabal against the Cabal that came with ghc die precisely that reason.

int-index commented 5 years ago

Closing in light of #705, created https://ghc.haskell.org/trac/ghc/ticket/15794 instead.