search

snowleopard / hadrian

Hadrian: a new build system for the Glasgow Haskell Compiler. Now merged into the GHC tree!
https://gitlab.haskell.org/ghc/ghc/tree/master/hadrian
MIT License
208 stars 39 forks source link

Reduce the number of times a Cabal file is parsed #671

Closed snowleopard closed 5 years ago

snowleopard commented 5 years ago

When I do zero build with Hadrian in verbose mode, I see the following:

| Executable found: bash => C:/msys/usr/bin/bash.exe
| KeyValue oracle: reading 'hadrian/cfg/system.config'...
| Building Stage1 libraries: binary, Cabal, ghc, ghc-boot, ghc-boot-th, ghc-heap, ghci, hpc, mtl, parsec, template-haskell, text, transformers, array, base, bytestring, containers, deepseq, directory, filepath, ghc-compact, ghc-prim, haskeline, integer-simple, pretty, process, rts, stm, time, xhtml, Win32
| Building Stage1 programs : ghc, ghc-pkg, hsc2hs, unlit, hpc, runghc, touchy

| Windows path mapping: /mingw64/bin/ => C:/msys/mingw64/bin/
| CabalFile oracle: reading 'libraries/binary/binary.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/Cabal/Cabal/Cabal.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'compiler/ghc.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/ghc-boot/ghc-boot.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/ghc-boot-th/ghc-boot-th.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/ghc-heap/ghc-heap.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/ghci/ghci.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/hpc/hpc.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/mtl/mtl.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/parsec/parsec.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/template-haskell/template-haskell.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/text/text.cabal' (Stage: stage0)...
Warning: libraries\text\text.cabal:4:1: The field "bug-reports" is specified
more than once at positions 4:1, 42:1
| CabalFile oracle: reading 'libraries/transformers/transformers.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/binary/binary.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/Cabal/Cabal/Cabal.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'compiler/ghc.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/ghc-boot/ghc-boot.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/ghc-boot-th/ghc-boot-th.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/ghc-heap/ghc-heap.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/ghci/ghci.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/hpc/hpc.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/mtl/mtl.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/parsec/parsec.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/template-haskell/template-haskell.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/text/text.cabal' (Stage: stage1)...
Warning: libraries\text\text.cabal:4:1: The field "bug-reports" is specified
more than once at positions 4:1, 42:1
| CabalFile oracle: reading 'libraries/transformers/transformers.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/array/array.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/base/base.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/bytestring/bytestring.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/containers/containers.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/deepseq/deepseq.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/directory/directory.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/filepath/filepath.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/ghc-compact/ghc-compact.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/ghc-prim/ghc-prim.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/haskeline/haskeline.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/integer-simple/integer-simple.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/pretty/pretty.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/process/process.cabal' (Stage: stage1)...
| Windows path mapping: C:/msys/home/nam83/ => C:/msys/home/nam83/
| CabalFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/stm/stm.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/time/time.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/xhtml/xhtml.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'libraries/Win32/Win32.cabal' (Stage: stage1)...
Warning: libraries\Win32\Win32.cabal:16:2: Tabs used as indentation at 16:2,
17:2
| CabalFile oracle: reading 'utils/compare_sizes/compareSizes.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'utils/genapply/genapply.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'utils/deriveConstants/deriveConstants.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'utils/haddock/haddock.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'utils/ghc-pkg/ghc-pkg.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'utils/genprimopcode/genprimopcode.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'utils/touchy/touchy.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'utils/touchy/touchy.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'utils/unlit/unlit.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'utils/ghctags/ghctags.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'utils/runghc/runghc.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'utils/ghc-pkg/ghc-pkg.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'utils/unlit/unlit.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'utils/hpc/hpc-bin.cabal' (Stage: stage1)...
Warning: utils\hpc\hpc-bin.cabal:11:2: Tabs used as indentation at 11:2
| CabalFile oracle: reading 'utils/hsc2hs/hsc2hs.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'utils/hsc2hs/hsc2hs.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/Cabal/Cabal/Cabal.cabal' (Stage: stage2)...
| Windows path mapping: C:/msys/home/nam83/ghc/inplace/perl/ => C:/msys/home/nam83/ghc/inplace/perl/
| Windows path mapping: C:/msys/mingw64/lib/../mingw/bin/ => C:/msys/mingw64/mingw/bin/
| Windows path mapping: /usr/local/bin/ => C:/msys/usr/local/bin/
| PackageDataFile oracle: reading 'utils/unlit/unlit.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'utils/genprimopcode/genprimopcode.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'utils/touchy/touchy.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'utils/deriveConstants/deriveConstants.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'utils/compare_sizes/compareSizes.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'utils/hsc2hs/hsc2hs.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'utils/genapply/genapply.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/base/base.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/array/array.cabal' (Stage: stage2)...
| KeyValues oracle: reading '_build/stage0/utils/genprimopcode/.dependencies'...
| KeyValues oracle: reading '_build/stage0/utils/compare_sizes/.dependencies'...
| KeyValues oracle: reading '_build/stage0/utils/genapply/.dependencies'...
| KeyValues oracle: reading '_build/stage0/utils/deriveConstants/.dependencies'...
| KeyValues oracle: reading '_build/stage0/utils/hsc2hs/.dependencies'...
| Windows path mapping: C:/msys/home/nam83/ghc/inplace/mingw/bin/ => C:/msys/home/nam83/ghc/inplace/mingw/bin/
| CabalFile oracle: reading 'libraries/containers/containers.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/bytestring/bytestring.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'ghc/ghc-bin.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'ghc/ghc-bin.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'libraries/binary/binary.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'libraries/ghc-boot-th/ghc-boot-th.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'libraries/transformers/transformers.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'libraries/ghc-heap/ghc-heap.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'libraries/hpc/hpc.cabal' (Stage: stage0)...
| KeyValues oracle: reading '_build/stage0/libraries/ghc-boot-th/.dependencies'...
| KeyValues oracle: reading '_build/stage0/libraries/ghc-heap/.dependencies'...
| KeyValues oracle: reading '_build/stage0/libraries/transformers/.dependencies'...
| PackageDataFile oracle: reading 'libraries/template-haskell/template-haskell.cabal' (Stage: stage0)...
| KeyValues oracle: reading '_build/stage0/libraries/binary/.dependencies'...
| PackageDataFile oracle: reading 'libraries/mtl/mtl.cabal' (Stage: stage0)...
| KeyValues oracle: reading '_build/stage0/libraries/hpc/.dependencies'...
| PackageDataFile oracle: reading 'libraries/text/text.cabal' (Stage: stage0)...
| PackageDataFile oracle: reading 'libraries/ghc-boot/ghc-boot.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'compiler/ghc.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/deepseq/deepseq.cabal' (Stage: stage2)...
| KeyValues oracle: reading '_build/stage0/libraries/template-haskell/.dependencies'...
| KeyValues oracle: reading '_build/stage0/libraries/mtl/.dependencies'...
| KeyValues oracle: reading '_build/stage0/libraries/ghc-boot/.dependencies'...
| CabalFile oracle: reading 'libraries/directory/directory.cabal' (Stage: stage2)...
| PackageDataFile oracle: reading 'libraries/ghci/ghci.cabal' (Stage: stage0)...
| KeyValues oracle: reading '_build/stage0/libraries/text/.dependencies'...
| CabalFile oracle: reading 'libraries/Win32/Win32.cabal' (Stage: stage2)...
Warning: libraries\Win32\Win32.cabal:16:2: Tabs used as indentation at 16:2,
17:2
| CabalFile oracle: reading 'libraries/filepath/filepath.cabal' (Stage: stage2)...
| PackageDataFile oracle: reading 'libraries/parsec/parsec.cabal' (Stage: stage0)...
| KeyValues oracle: reading '_build/stage0/libraries/ghci/.dependencies'...
| PackageDataFile oracle: reading 'compiler/ghc.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/binary/binary.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/ghc-boot/ghc-boot.cabal' (Stage: stage2)...
| KeyValues oracle: reading '_build/stage0/libraries/parsec/.dependencies'...
| PackageDataFile oracle: reading 'libraries/Cabal/Cabal/Cabal.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'libraries/parsec/parsec.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/ghc-boot-th/ghc-boot-th.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/ghc-heap/ghc-heap.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/text/text.cabal' (Stage: stage2)...
Warning: libraries\text\text.cabal:4:1: The field "bug-reports" is specified
more than once at positions 4:1, 42:1
| CabalFile oracle: reading 'libraries/ghc-prim/ghc-prim.cabal' (Stage: stage2)...
| KeyValues oracle: reading '_build/stage0/compiler/.dependencies'...
| CabalFile oracle: reading 'libraries/transformers/transformers.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/ghci/ghci.cabal' (Stage: stage2)...
| KeyValues oracle: reading '_build/stage0/libraries/Cabal/Cabal/.dependencies'...
| CabalFile oracle: reading 'libraries/hpc/hpc.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/xhtml/xhtml.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/integer-simple/integer-simple.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/mtl/mtl.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/pretty/pretty.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/process/process.cabal' (Stage: stage2)...
| PackageDataFile oracle: reading 'utils/ghc-pkg/ghc-pkg.cabal' (Stage: stage0)...
| CabalFile oracle: reading 'rts/rts.cabal' (Stage: stage2)...
| KeyValues oracle: reading '_build/stage0/utils/ghc-pkg/.dependencies'...
| CabalFile oracle: reading 'libraries/template-haskell/template-haskell.cabal' (Stage: stage2)...
| CabalFile oracle: reading 'libraries/time/time.cabal' (Stage: stage2)...
| PackageDataFile oracle: reading 'ghc/ghc-bin.cabal' (Stage: stage0)...
| KeyValues oracle: reading '_build/stage0/ghc/.dependencies'...
| PackageDataFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| CabalFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'rts/rts.cabal' (Stage: stage1)...
| Executable found: make => C:/msys/usr/bin/make.exe
| Windows path mapping: C:/msys/usr/bin/ => C:/msys/usr/bin/
| Windows path mapping: /usr/bin/ => C:/msys/usr/bin/
| PackageDataFile oracle: reading 'utils/touchy/touchy.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'utils/unlit/unlit.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/ghc-prim/ghc-prim.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/ghc-prim/.dependencies'...
| PackageDataFile oracle: reading 'libraries/integer-simple/integer-simple.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/integer-simple/.dependencies'...
| PackageDataFile oracle: reading 'libraries/base/base.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/base/.dependencies'...
| PackageDataFile oracle: reading 'libraries/ghc-boot-th/ghc-boot-th.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/transformers/transformers.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/array/array.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/ghc-heap/ghc-heap.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/xhtml/xhtml.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/filepath/filepath.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/xhtml/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/transformers/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/ghc-boot-th/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/filepath/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/ghc-heap/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/array/.dependencies'...
| PackageDataFile oracle: reading 'libraries/mtl/mtl.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/deepseq/deepseq.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/stm/stm.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/mtl/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/deepseq/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/stm/.dependencies'...
| PackageDataFile oracle: reading 'libraries/bytestring/bytestring.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/containers/containers.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/pretty/pretty.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/pretty/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/bytestring/.dependencies'...
| PackageDataFile oracle: reading 'libraries/template-haskell/template-haskell.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/containers/.dependencies'...
| PackageDataFile oracle: reading 'libraries/Win32/Win32.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/ghc-compact/ghc-compact.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/template-haskell/.dependencies'...
| PackageDataFile oracle: reading 'libraries/binary/binary.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/ghc-compact/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/binary/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/Win32/.dependencies'...
| PackageDataFile oracle: reading 'libraries/text/text.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/time/time.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/text/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/time/.dependencies'...
| PackageDataFile oracle: reading 'libraries/directory/directory.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/parsec/parsec.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/directory/.dependencies'...
| PackageDataFile oracle: reading 'libraries/process/process.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/hpc/hpc.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/ghc-boot/ghc-boot.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/parsec/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/hpc/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/ghc-boot/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/process/.dependencies'...
| PackageDataFile oracle: reading 'utils/hpc/hpc-bin.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/ghci/ghci.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'utils/runghc/runghc.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/haskeline/haskeline.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'utils/hsc2hs/hsc2hs.cabal' (Stage: stage1)...
| PackageDataFile oracle: reading 'libraries/Cabal/Cabal/Cabal.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/utils/hsc2hs/.dependencies'...
| KeyValues oracle: reading '_build/stage1/utils/hpc/.dependencies'...
| KeyValues oracle: reading '_build/stage1/utils/runghc/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/ghci/.dependencies'...
| KeyValues oracle: reading '_build/stage1/libraries/haskeline/.dependencies'...
| PackageDataFile oracle: reading 'compiler/ghc.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/libraries/Cabal/Cabal/.dependencies'...
| KeyValues oracle: reading '_build/stage1/compiler/.dependencies'...
| PackageDataFile oracle: reading 'utils/ghc-pkg/ghc-pkg.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/utils/ghc-pkg/.dependencies'...
| PackageDataFile oracle: reading 'ghc/ghc-bin.cabal' (Stage: stage1)...
| KeyValues oracle: reading '_build/stage1/ghc/.dependencies'...
| PackageDataFile oracle: reading 'utils/ghctags/ghctags.cabal' (Stage: stage2)...
| PackageDataFile oracle: reading 'utils/haddock/haddock.cabal' (Stage: stage2)...
| KeyValues oracle: reading '_build/stage2/utils/ghctags/.dependencies'...
| KeyValues oracle: reading '_build/stage2/utils/haddock/.dependencies'...

Many Cabal files are parsed multiple times:

For example, rts.cabal ends up being parsed 13 times! No wonder zero build now takes around 30s.

Can we parse a Cabal file only once?

alpmestan commented 5 years ago

I'm a bit fuzzy on what the oracles actually are, but I initially would have hoped that they would cache things better than that. Maybe it's possible and we're just misusing oracles or missing another layer to prevent those calls from happening, caching things earlier in the "pipeline"?

To answer your last question: in theory, I'm pretty sure we really just need to parse them once and from that point we just share the in-memory (or on-disk?) representation. They don't change during the build, and some of them don't even exist before ./configure.

snowleopard commented 5 years ago

I'm a bit fuzzy on what the oracles actually are, but I initially would have hoped that they would cache things better than that.

@alpmestan It's not the fault of oracles: they are given different queries -- different contexts (with varying stage and way for the very same package). They do cache things when called with the same query.

To answer your last question: in theory, I'm pretty sure we really just need to parse them once and from that point we just share the in-memory (or on-disk?) representation.

Yes, I agree. I'll see if I can optimise this by switching to having a Package as a key to these oracles instead of a Context. This should be possible.

alpmestan commented 5 years ago

Yes, that does sound a whooole lot better. Let me know how much of an improvement this is :-)

alpmestan commented 5 years ago

Have you made any progress on this? A patch that I can pick up perhaps? I'm really beginning to be bothehred by this, as hadrian takes 30+ seconds to boot and 5 to run the test I'm looking into at a given point.

snowleopard commented 5 years ago

@alpmestan I've started working on this, but hit a few non-obvious questions -- sometimes we do seem to need the information about the current stage (e.g. to pass the right compiler to Cabal configure) and way (to figure out the right configuration flags). My code is in a big mess at the moment, but there is nothing too valuable that can't be thrown away. I won't have much time to work on this in the next couple of days, so feel free to just start from scratch if you like.

alpmestan commented 5 years ago

Well, we also have to keep in mind that we have two ways to represent parsed .cabal files. One is just a straight Haskell representation of the content, and the other "instantiates" all the variable things (conditionals about OS, ghc, etc, but also cabal flags) in a given context (OS, compiler, choice of flags, ...).

snowleopard commented 5 years ago

Yes! I'm also tempted to get rid of this duplication. My current plan is:

Does this sound reasonable?

alpmestan commented 5 years ago

Yes. And we might want to have --trace emit messages about oracles/caching/parsing that are precise and let us see exactly when we're doing work that we're not supposed to do anymore. At the moment we have some hints but not all the information we would need, I think. It might even be worth taking care of this first, so that we can easily see the improvement there and confirm that we get the desired behaviour?

snowleopard commented 5 years ago

I actually think I have a pretty good idea what's going on (the amount of diagnostic information is sufficient for me), but sure, it would be nice to improve on this aspect!

alpmestan commented 5 years ago

May contextDependencies be to blame here? It even has a -- TODO: Cache the computation. comment.

snowleopard commented 5 years ago

@alpmestan I believe the comment refers to this issue: #550.

Apologies for taking so long with this, I've been swamped by other commitments, but I hope to send a PR for your review today/tomorrow!

alpmestan commented 5 years ago

OK. Because while investigating an issue for an upcoming PR (that fixes dynamic-enabled ways for good on my linux system) I realized that this is called a looooooot of times, and we seem to repeatedly be looking things up. Anyway, I'm looking forward to your patch!

snowleopard commented 5 years ago

Fixed in #692.