Use Cabal directly in place of ghc-cabal + make build root configurable

[alpmestan]

This commit implements two significant changes (that were not easy to separate):

• Don't use ghc-cabal anymore for getting information about Haskell packages. We now instead directly use Cabal-the-library.

• Make the build root configurable. This effectively gets rid of the inplace logic and allows us to place all build artefacts in some directory of our choice, by passing '--build-root ' to hadrian. The GHCs we produce are now relocatable.

The code for this was mostly taken from #445. I have successfully built functional quick/prof/perf/devel2/quick-cross builds of GHC with this branch. I might commit another tweak or two as I'm now testing the test/docs rules. But I'm not expecting major changes there.

I'm also not clear on what we want to do with the install/wrapper rules, with relocatable GHC builds.

[snowleopard]

@alpmestan Hmm... Does it work for you locally?

Where is this error thrown?

dieVerbatim: user error (hadrian: Error Parsing: file "compiler/ghc.cabal" doesn't exist. Cannot
continue.

Perhaps, the corresponding code lacks the necessary need on the file being parsed?

[alpmestan]

@snowleopard The code that parses the cabal file certainly needs it, and it's then exposed through oracles. See this code (it comes from the textfile oracle module). And somehow the boot/configure rule(s) don't get a chance to fire before the oracles are read maybe?

(To be entirely honest, this is all convincing me more and more that having hadrian handle that is unecessary. The make build system doesn't run this for us either after all, and the technical complications that we're seeing are a bit of an annoying price to pay for saving a few keystrokes. Or maybe we want that -c option to trigger the execution of ./boot && ./configure before we hand anything off to Shake? Ugly, but trivial to implement, and same effect. I still think the simplest thing to do is to do like the make build system: require that the user runs the boot and configure scripts, especially given that very often we end up needing to pass a few configure options, in which case we can't juse use -c, if I'm not mistaken.)

[snowleopard]

and the technical complications that we're seeing are a bit of an annoying price to pay for saving a few keystrokes

@alpmestan I don't see any technical complications that are specific to running boot and configure! The current implementation simply fails to register a dependency, and also fails to produce an informative error message -- something that a good build system should be able to do.

If I understand you correctly, you are saying "we don't understand what's going on, so let's just remove this feature". I don't think this is a good argument, sorry :)

[snowleopard]

I still don't see the code that produced this unhelpful error message dieVerbatim: user error (hadrian: Error Parsing: file "compiler/ghc.cabal" doesn't exist. Cannot continue. -- where is it?

[snowleopard]

There can be only one reason we see this failure: some code reads the file compiler/ghc.cabal without first needing it. There is nothing magic about configureRules: it's just another build rule that says that it builds compiler/ghc.cabal among other things. If you ask it to do it by calling need ["compiler/ghc.cabal"], it will do it. So, the inescapable conclusion follows: a missing dependency.

[angerman]

So what is happening here?

• parseCabalPkgId :: FilePath -> IO String does not need the filepath that's input (which it actually should). Let us assume we do that for a moment, so we end up with
• parseCabalPkgId :: FilePath -> Action String.

parseCabalPkgId is used in a few Rules (), for example: library :: Context -> Rules (). This and other functions use the information from Context as well as the package id, to compute the path the rule applies to. We can not use action :: Action a -> Rules () here, because that doesn't give us the pkg id, but simply queue the action for execution.

Thus to make this work, we likely want some way to deconstruct a path into the build directory into some context and then have generic rules instead of rule generation. E.g. rather than enumerating all valid rules, build generic rules and fail in the body of the rule. I believe this is also the preferred approach @ndmitchell once suggested to me. Which at that time was the reason for the introduction of the other generic rule iirc.

[snowleopard]

@angerman Ah, I think I now understand what's going on -- if I understood you correctly, we now need to parse Cabal files in order to generate build rules. This is something we didn't have to do before.

Is this essential for this patch, or is it just an unrelated refactoring?

[angerman]

@snowleopard the logic is pretty integral in the Rules.Library module, as we use the cabal file as the truth source for any derived information now.

I'm tempted to just rework the whole Rules.Library module to turn it into generic rules. That would I beleive make them easier to understand, drop code, and potentially make the hadrian boot up time faster.

I don't feel like jamming this onto this PR through, as it pulls in a rather long tail. One we start deconstructing paths into Contexts, we need to make sure we have some easy path forward to keep the Context.Paths module in sync. I assume we'd want some bidirectional appraoch to the paths (we'd want to make sure we generate the same path that we parse).

[snowleopard]

@angerman I agree, it would be best not to mix this into this PR. Now that we know what causes the error when using -c, could you raise an issue, so we know what needs to be done? Then I'm fine for temporarily disabling -c in Travis and keeping this PR as is in this respect.

[alpmestan]

By the way, Simon (Marlow) just released a new version of alex (see https://github.com/simonmar/alex/issues/123#issuecomment-377324152), so this should "unlock" the CI scenarios that use 8.4 as stage0.

[angerman]

@snowleopard I've opened #538 for the rule replacement.

[snowleopard]

OK, I think we are mostly done with this PR. There are a few things that I'd like to look at after merging:

• Try to break import loops by adding more .Type modules (no need to worry about this now).
• Work on #538 to fix -c.
• Figure out where we can replace buildDir with contextDir.
• Finish the Settings/Packages.hs story.

@angerman There is one question that I think I haven't got an answer from you yet:

I don't understand why Hadrian is different from Make in terms of which GHC is used to build Haddock: Make uses ghc-stage2, i.e. the final GHC we build, whereas you propose for Hadrian to use ghc-stage1, i.e. the intermediate version of GHC. My understanding was that ghc-stage1 just didn't have enough features (or latest libraries) to build Haddock.

Could you clarify?

@alpmestan Do you still have any unfinished work on this PR? Anything I missed that needs further discussion?

[alpmestan]

I created an issue about reviving wrapper/install rules under a "modernised" form (in light of this PR, that is) at https://github.com/snowleopard/hadrian/issues/539.

@alpmestan Do you still have any unfinished work on this PR? Anything I missed that needs further discussion?

I still need to update the README as well as the user settings document. And address a couple of small things here and there. But nothing huge. Unless I'm missing something? Github's UI has certainly not made it easy for us to track what got addressed or not as I was pushing new commits. Please do let me know if you spot anything.

[alpmestan]

Updated {README, doc/user-settings}.md.

[alpmestan]

Look at this beautiful travis CI page: https://travis-ci.org/snowleopard/hadrian/builds/360321326 :tada:

[alpmestan]

Alright, I pushed another patch that addresses some minor things.

I also tried to get rid of buildDir/buildPath/getBuildPath, replacing them by their context counterparts, but ran into various problems that have to do with autogen paths I think. For reference, if anyone wants to pick this up, the WIP patch is available in this gist. @snowleopard Do you mind keeping things as they are for now? If you don't, but still would like to see this resolved eventually, please do feel free to create an issue linking to my patch, where I can tell you a bit more about the problems that I see with my patch.

@snowleopard Anything else that you see that I should take care of?

[snowleopard]

@alpmestan Thanks! I've added the buildDir/contextDir item to the list of things I'd like to have a look: https://github.com/snowleopard/hadrian/pull/531#issuecomment-377539369. After merging this PR, I'll copy this list into a separate issue.

I think I can merge this now. There are some unanswered questions, but discussing them here seems to be counterproductive. I suggest to merge and move on to discussing the leftovers in separate issues.

@alpmestan Waiting for a Good-To-Go signal from you :)

[alpmestan]

@alpmestan Waiting for a Good-To-Go signal from you :)

[snowleopard]

Done!

@alpmestan @angerman Thank you both for this amazing PR!