Failed to find package '"ghc-heap"'

[chitrak7]

On running the build, the following error is coming: Error when running Shake build system:

• OracleQ (CabalFile (Context {stage = Stage1, package = Package {pkgLanguage = Haskell, pkgType = Library, pkgName = "ghc", pkgPath = "compiler"}, way = v})) Failed to find package '"ghc-heap"' CallStack (from HasCallStack): error, called at src/Hadrian/Haskell/Cabal/Parse.hs:109:47 in main:Hadrian.Haskell.Cabal.Parse

Any idea how to tackle this, I tried to add ghc-heap to GHC Package and added it to stage 0. But error still persists.

[chitrak7]

https://github.com/ghc/ghc/commit/ec22f7ddc81b40a9dbcf140e5cf44730cb776d00 I believe the error is coming due to this commit. ghc-heap is added as a dependency

[snowleopard]

@chitrak7 Thanks for reporting! Let me have a look.

[snowleopard]

I think this commit should fix the issue: https://github.com/snowleopard/hadrian/commit/c0292ffcaee940cfd3a81a878a680e0e273c10bc

The build still fails due to llvm-passes missing, which will hopefully be resolved soon.

[chitrak7]

@snowleopard I did the same thing by resetting my HEAD to a commit previous to llvm-pass. But even there, some errors are popping up. I think we should not close this issue just yet.

[snowleopard]

@chitrak7 OK, let me reopen for now.

[angerman]

@chitrak7 what error s are you seeing, and on what OS are you building? GHC HEAD is pretty busted on macOS right now afaict. I'd suggest to work off of a known good (Hadrian, GHC) commit combination that works for you.

E.g. you don't need to try to keep tracking GHC as aggressive as possible. GHC HEAD can be broken, and even be broken for an extended period of time.

While hadrian is not the primary build/ci tool for GHC, and an integral part of GHC, we will inevitably run into situations where Hadrian and GHC diverge for some time until Hadrian fully catches up to what happend in GHC, and GHC is known to be buildable across all platforms.

Don't let this interfere with your progress, just use a hadrian/ghc commit combination you know is good. Create a branch off of that hadrian commit, and implement what you intend to implement. In most cases, I believe a merge will be trivial in the end.

[sighingnow]

resetting my HEAD to a commit previous to llvm-pass.

@chitrak7 Doing resetting will re-introduce the "ghc-heap" related errors. You should revert the "llvm-passes" commit (locally, with a newly created branch):

git revert 85f2506f87c890ba985825c74dea92c10e80152e

[chitrak7]

@sighingnow Thanks! Currently, I am working on a stable version of GHC which is 15 days behind HEAD.

[chitrak7]

Hi @angerman @snowleopard . Currently, make is able to build ghc. I tried to figure out the difference between the failing commands This is the Hadrians command to build dependencies. _build/stage0/bin/hsc2hs --cc=/usr/bin/gcc --ld=/usr/bin/gcc --cross-safe --cflag=-fno-stack-protector --cflag=-Iincludes --cflag=-I_build/generated --cflag=-I_build/stage0/libraries/ghc-heap/build --cflag=-I/usr/local/lib/ghc-8.2.1/base-4.10.0.0/include --cflag=-I/usr/local/lib/ghc-8.2.1/integer-gmp-1.0.1.0/include --cflag=-I/usr/local/lib/ghc-8.2.1/include --cflag=-Wall --cflag=-Werror=unused-but-set-variable --cflag=-Wno-error=inline --cflag=-include --cflag=_build/stage0/libraries/ghc-heap/build/autogen/cabal_macros.h --cflag=-Dx86_64_HOST_ARCH=1 --cflag=-Dlinux_HOST_OS=1 --cflag=-D__GLASGOW_HASKELL__=802 --template=/home/chitrak/ghc/_build/stage0/lib/template-hsc.h libraries/ghc-heap/GHC/Exts/Heap/InfoTableProf.hsc -o _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf.hs

This is the make's command "inplace/bin/hsc2hs" '--cc=gcc' '--ld=gcc' --cross-safe --cflag=-Wall --cflag=-fno-stack-protector --lflag=-fuse-ld=gold --cflag=-Dx86_64_HOST_ARCH --cflag=-Dlinux_HOST_OS --cflag=-D__GLASGOW_HASKELL__=805 '--cflag=-fno-stack-protector' '--cflag=-Wall' '--cflag=-Ilibraries/ghc-heap/dist-install/build/./autogen' '--cflag=-Ilibraries/ghc-heap/.' '--cflag=-I/home/chitrak/ghc/libraries/base/include' '--cflag=-I/home/chitrak/ghc/libraries/base/dist-install/build/include' '--cflag=-I/home/chitrak/ghc/libraries/base/dist-install/build/include' '--cflag=-I/home/chitrak/ghc/libraries/base/dist-install/build/dist-install/build/include' '--cflag=-I/home/chitrak/ghc/libraries/integer-gmp/include' '--cflag=-I/home/chitrak/ghc/libraries/integer-gmp/dist-install/build/include' '--cflag=-I/home/chitrak/ghc/libraries/integer-gmp/dist-install/build/include' '--cflag=-I/home/chitrak/ghc/libraries/integer-gmp/dist-install/build/dist-install/build/include' '--cflag=-I/home/chitrak/ghc/rts/dist/build' '--cflag=-I/home/chitrak/ghc/includes' '--cflag=-I/home/chitrak/ghc/includes/dist-derivedconstants/header' '--cflag=-Werror=unused-but-set-variable' '--cflag=-Wno-error=inline' --cflag=-Ilibraries/ghc-heap/dist-install/build/./autogen --cflag=-include --cflag=libraries/ghc-heap/dist-install/build/./autogen/cabal_macros.h libraries/ghc-heap/./GHC/Exts/Heap/InfoTable.hsc -o libraries/ghc-heap/dist-install/build/GHC/Exts/Heap/InfoTable.hs

[chitrak7]

The error is coming because hadrian is using local ghc-8.2.1, whereas make is using ghc-8.5.1 even in stage0. I changed --cflag=-D__GLASGOW_HASKELL=802 to --cflag=-D__GLASGOW_HASKELL=805 and the above command executed. Can you please look into it and help me what is happenig?

[snowleopard]

@chitrak7 You are showing the Make command for Stage1 -- see dist-install in the path:

libraries/ghc-heap/dist-install/build/GHC/Exts/Heap/InfoTable.hs

However, Hadrian fails in Stage0. Can you show which command Make uses in Stage0?

[angerman]

@chitrak7 chaning the__GLASGOW_HASKELL__ value is not a good idea, as it is used to determine the compiler and use in conditional branches. As such it can easily lead to completely unusable binaries.

In general I doubt that we are using ghc-8.5.1. (which is the current HEAD) to build GHC with make. Of course this may happen iif you build ghc HEAD, and then use the binary from that built to build GHC.

However building HEAD with HEAD is something that might or might not work, and I would advise against it, unless you really need to do this. (e.g. building some cross compilers that depend on changes to the code generator).

I'm currently a bit lost where our current issue is at. Are we still looking for the llvm-passes or something else? Most of the stuff should be committed in GHC HEAD, as we are approaching the GHC 8.6.1 release.

[snowleopard]

Are we still looking for the llvm-passes or something else?

@angerman I think it's something else, related specifically to adding ghc-heap to GHC.

[alpmestan]

I tried building GHC with the tip of hadrian's and ghc's master branches. I ran into this error:

| Run Hsc2Hs Stage0: libraries/ghc-heap/GHC/Exts/Heap/InfoTableProf.hsc => _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf.hs
In file included from /nix/store/j4q18y05wpnjin423wf69nfb7r7q0ml9-glibc-2.25-123-dev/include/_G_config.h:15:0,
                 from /nix/store/j4q18y05wpnjin423wf69nfb7r7q0ml9-glibc-2.25-123-dev/include/libio.h:31,
                 from /nix/store/j4q18y05wpnjin423wf69nfb7r7q0ml9-glibc-2.25-123-dev/include/stdio.h:75,
                 from includes/rts/Flags.h:16,
                 from includes/Rts.h:192,
                 from InfoTableProf.hsc:12:
InfoTableProf.hsc: Dans la fonction « main »:
/home/alp/WT/ghc-hadrian/_build/stage0/lib/template-hsc.h:75:24: erreur : « struct StgInfoTable_ » has no member named « srt_bitmap »
                 (long) offsetof (t, f));
                        ^
InfoTableProf.hsc:40:5: note : in expansion of macro « hsc_peek »
/home/alp/WT/ghc-hadrian/_build/stage0/lib/template-hsc.h:79:24: erreur : « StgInfoTable {alias struct StgInfoTable_} » has no member named « srt_bitmap »
                 (long) offsetof (t, f));
                        ^
InfoTableProf.hsc:62:5: note : in expansion of macro « hsc_poke »
compiling _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf_hsc_make.c failed (exit code 1)
command was: /nix/store/5l6b97jscljkvsq9y9amg71v5jfzddm4-gcc-wrapper-6.4.0/bin/cc -c _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf_hsc_make.c -o _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf_hsc_make.o -fno-stack-protector -Iincludes -I_build/generated -I_build/stage0/libraries/ghc-heap/build -I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/base-4.10.1.0/include -I/nix/store/biv9a4szj648s7d77a1r607ndvnwv8k6-gmp-6.1.2-dev/include -I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/integer-gmp-1.0.1.0/include -I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/include -Wall -Werror=unused-but-set-variable -Wno-error=inline -include _build/stage0/libraries/ghc-heap/build/autogen/cabal_macros.h -Dx86_64_HOST_ARCH=1 -Dlinux_HOST_OS=1 -D__GLASGOW_HASKELL__=802
shakeArgsWith   0.000s    0%                           
Function shake  0.251s    1%                           
Database read   0.001s    0%                           
With database   0.000s    0%                           
Running rules  19.241s   98%  =========================
Total          19.493s  100%                           
Error when running Shake build system:
* _build/stage0/bin/ghc
* OracleQ (PackageDataFile (Context {stage = Stage0, package = Package {pkgLanguage = Haskell, pkgType = Program, pkgName = "ghc-bin", pkgPath = "ghc"}, way = v}))
* _build/stage0/ghc/setup-config
* _build/stage0/lib/package.conf.d/ghc-8.5.conf
* OracleQ (PackageDataFile (Context {stage = Stage0, package = Package {pkgLanguage = Haskell, pkgType = Library, pkgName = "ghc", pkgPath = "compiler"}, way = v}))
* _build/stage0/compiler/setup-config
* _build/stage0/lib/package.conf.d/ghc-heap-8.5.conf
* _build/stage0/libraries/ghc-heap/build/HSghc-heap-8.5.o
* _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf.o
* _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf.o _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf.hi
* OracleQ (KeyValues ("_build/stage0/libraries/ghc-heap/.dependencies","_build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf.o"))
* _build/stage0/libraries/ghc-heap/.dependencies
* _build/stage0/libraries/ghc-heap/.dependencies.mk
* _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf.hs
user error (Development.Shake.cmd, system command failed
Command: _build/stage0/bin/hsc2hs --cc=/nix/store/5l6b97jscljkvsq9y9amg71v5jfzddm4-gcc-wrapper-6.4.0/bin/cc --ld=/nix/store/5l6b97jscljkvsq9y9amg71v5jfzddm4-gcc-wrapper-6.4.0/bin/cc --cross-safe --cflag=-fno-stack-protector --cflag=-Iincludes --cflag=-I_build/generated --cflag=-I_build/stage0/libraries/ghc-heap/build --cflag=-I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/base-4.10.1.0/include --cflag=-I/nix/store/biv9a4szj648s7d77a1r607ndvnwv8k6-gmp-6.1.2-dev/include --cflag=-I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/integer-gmp-1.0.1.0/include --cflag=-I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/include --cflag=-Wall --cflag=-Werror=unused-but-set-variable --cflag=-Wno-error=inline --cflag=-include --cflag=_build/stage0/libraries/ghc-heap/build/autogen/cabal_macros.h --cflag=-Dx86_64_HOST_ARCH=1 --cflag=-Dlinux_HOST_OS=1 --cflag=-D__GLASGOW_HASKELL__=802 --template=/home/alp/WT/ghc-hadrian/_build/stage0/lib/template-hsc.h libraries/ghc-heap/GHC/Exts/Heap/InfoTableProf.hsc -o _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf.hs
Exit code: 1
Stderr:
In file included from /nix/store/j4q18y05wpnjin423wf69nfb7r7q0ml9-glibc-2.25-123-dev/include/_G_config.h:15:0,
                 from /nix/store/j4q18y05wpnjin423wf69nfb7r7q0ml9-glibc-2.25-123-dev/include/libio.h:31,
                 from /nix/store/j4q18y05wpnjin423wf69nfb7r7q0ml9-glibc-2.25-123-dev/include/stdio.h:75,
                 from includes/rts/Flags.h:16,
                 from includes/Rts.h:192,
                 from InfoTableProf.hsc:12:
InfoTableProf.hsc: Dans la fonction « main »:
/home/alp/WT/ghc-hadrian/_build/stage0/lib/template-hsc.h:75:24: erreur : « struct StgInfoTable_ » has no member named « srt_bitmap »
                 (long) offsetof (t, f));
                        ^
InfoTableProf.hsc:40:5: note : in expansion of macro « hsc_peek »
/home/alp/WT/ghc-hadrian/_build/stage0/lib/template-hsc.h:79:24: erreur : « StgInfoTable {alias struct StgInfoTable_} » has no member named « srt_bitmap »
                 (long) offsetof (t, f));
                        ^
InfoTableProf.hsc:62:5: note : in expansion of macro « hsc_poke »
compiling _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf_hsc_make.c failed (exit code 1)
command was: /nix/store/5l6b97jscljkvsq9y9amg71v5jfzddm4-gcc-wrapper-6.4.0/bin/cc -c _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf_hsc_make.c -o _build/stage0/libraries/ghc-heap/build/GHC/Exts/Heap/InfoTableProf_hsc_make.o -fno-stack-protector -Iincludes -I_build/generated -I_build/stage0/libraries/ghc-heap/build -I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/base-4.10.1.0/include -I/nix/store/biv9a4szj648s7d77a1r607ndvnwv8k6-gmp-6.1.2-dev/include -I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/integer-gmp-1.0.1.0/include -I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/include -Wall -Werror=unused-but-set-variable -Wno-error=inline -include _build/stage0/libraries/ghc-heap/build/autogen/cabal_macros.h -Dx86_64_HOST_ARCH=1 -Dlinux_HOST_OS=1 -D__GLASGOW_HASKELL__=802
)

We can see in particular that we have -Iincludes and -I/nix/store/9af0m2zjvpml0b8646sxayn28sgr5nc9-ghc-8.2.2/lib/ghc-8.2.2/include (my bootstrap compiler's includes), in that order. So gcc just systematically picks up the includes from includes/, whereas here we would want it to use the bootstrap compiler's own include files, right?

[snowleopard]

@alpmestan Aha, sounds like a plausible explanation! Do you think we should swap the order of includes? I can give this a try.

[alpmestan]

@snowleopard I think a nicer solution would be to tweak the common settings that introduce -Iincludes, a bit like this:

diff --git a/src/Settings/Builders/Common.hs b/src/Settings/Builders/Common.hs
index 1995c666d5..67e4bceaf5 100644
--- a/src/Settings/Builders/Common.hs
+++ b/src/Settings/Builders/Common.hs
@@ -11,6 +11,7 @@ module Settings.Builders.Common (

 import Base
 import Expression
+import GHC.Packages
 import Hadrian.Haskell.Cabal.PackageData
 import Oracles.Flag
 import Oracles.Setting
@@ -27,7 +28,7 @@ cIncludeArgs = do
     iconvIncludeDir <- getSetting IconvIncludeDir
     gmpIncludeDir   <- getSetting GmpIncludeDir
     ffiIncludeDir   <- getSetting FfiIncludeDir
-    mconcat [ arg "-Iincludes"
+    mconcat [ notStage0 ? arg "-Iincludes"
             , arg $ "-I" ++ root -/- generatedDir
             , arg $ "-I" ++ path
             , pure . map ("-I"++) . filter (/= "") $ [iconvIncludeDir, gmpIncludeDir]

But instead of doing notStage0 (which gets us past this particular error but is not precise enough, as we fail to build the ghc library), something more precise which I'm in the process of figuring out.

[snowleopard]

@alpmestan Right, many thanks! Makes sense. Perhaps, notStage0 ||^ package ghc would work.

[alpmestan]

@snowleopard Yes, I'll see where this gets me. I was looking around for that ||^ operator, was sure there was something for that but could not find it, thanks!

[alpmestan]

PR up at #610.

[snowleopard]

I think we can close this now. Thanks @alpmestan!