search

23Skidoo / ghc-parmake

A parallel wrapper for 'ghc --make'.
BSD 3-Clause "New" or "Revised" License
10 stars 2 forks source link

The ghc -M / -c approach is not sufficient for correct builds in all cases #9

Open nh2 opened 11 years ago

nh2 commented 11 years ago

See: http://ghc.haskell.org/trac/ghc/ticket/481

While there is a way to get around the external dependencies problem, the cases regarding TemplateHaskell described in that bug can probably not be implemented using ghc -M and ghc -c.

(Also, http://ghc.haskell.org/trac/ghc/ticket/7277 is not even handled properly by ghc --make itself.)

This is probably important for the GSoC project, because it means that any system based on ghc -M alone cannot produce always-correct builds.

The only workaround that I see for now is to call ghc --make after our parallel build, and live with the singlethreadedness that comes with that.

23Skidoo commented 11 years ago

Thanks, will look into this.

23Skidoo commented 11 years ago

I wonder if we can also use our persistent ghc server for deps analysis and extend it to support outputting information that is unavailable via -M.

nh2 commented 11 years ago

I guess it will be quite difficult, as GHC discovers these dependencies on the fly, and according to what they say in Trac, even down to the per-function level.

dcoutts commented 11 years ago

I don't understand why http://ghc.haskell.org/trac/ghc/ticket/481 is a problem for ghc -M and ghc -c. The ghc -c uses the recompilation check too to decide if it has to compile or if it can be skipped. More details?

nh2 commented 11 years ago

@dcoutts Because in that case TH forces you to do a recompilation that you would not consider necessary only at the -M dependencies and the time stamps of the files. Parmake does that. If it didn't, it would call ghc -c on every module, every time.

(If I'm wrong with this: I would love that, I really want this to work.)

nh2 commented 11 years ago

Another point:

https://github.com/ghc/ghc/commit/b994313a1f7b233ec5da31d004a5db92758b0836

The commit mentions No support for ghc -M.

It also looks like there is no way no support addDependentFile in any way: If dependencies can be chosen (e.g. conditionally) at compile time of a module, the dependencies for that module cannot be known before its compilation. They cannot be made part of the ghc -M output, because for this, actual compilation must already have run through and evaluated all TH code.

It looks like only a subset of Haskell can be compiled with ghc -M these days.

23Skidoo commented 11 years ago

@nh2

ghc/ghc@b994313

This is the DEPENDS pragma, I believe. Another related issue is http://ghc.haskell.org/trac/ghc/ticket/3588

nh2 commented 11 years ago

This is the DEPENDS pragma, I believe

addDependentFile and the DEPENDS pragma are different:

Being a pragma, DEPENDS carries static knowledge, could be evaluated by ghc -M and would not make a problem; addDependentFile however is Template Haskell and its output can depend on any IO knowledge, e.g. the current date or the weather; you can only get its value by actually compiling the module.

23Skidoo commented 11 years ago

Aha, thanks. Yeah, that's a problem. Perhaps I can make the persistent server output this info during compilation.

nh2 commented 11 years ago

Also relevant: http://ghc.haskell.org/trac/ghc/ticket/8144 and https://github.com/nh2/ghc-parmake/commit/8f2833aa6739bed13b9dc6ca34dc7b1e6c8465ba.

nh2 commented 11 years ago

Shall we close this issue?

We can probably simply keep the final GHC pass because your ghc-server will make it a no-op / very fast.

23Skidoo commented 11 years ago

@nh2 I'm still a bit unsatisfied with that approach, so I'd like to keep this open.

ezyang commented 8 years ago

Why not write some programs using the GHC API to help you get out the information you need? You can define a private interface between ghc-parmake and the GHC API program which is shared across versions of GHC, so you can have different versions of the GHC API program for different versions of GHC. GHC 8.0 is even coming with frontend plugins (https://ghc.haskell.org/trac/ghc/ticket/11194) which should make it even easier to write programs which accept GHC-style arguments.

Once you figure out exactly what you need, and what a good format is, you can submit the frontends to GHC for upstreaming.

23Skidoo commented 8 years ago

Thanks, I'll try to find some time to experiment with frontend plugins.

The master branch actually contains a GHC API-based reimplementation of ghc-parmake (which is not up to par with the 0.1 branch yet).

ezyang commented 8 years ago

Yeah, the GHC API can be really helpful. For example, in https://github.com/ezyang/ghc-shake I can run the preprocessor and parseHeader, get the dependencies, and then feed it directly into the build system. For this approach to scale, I need to separate the GHC API bits from the build system bits, but it works very nicely.

23Skidoo commented 8 years ago

For example, in https://github.com/ezyang/ghc-shake I can run the preprocessor and parseHeader, get the dependencies, and then feed it directly into the build system.

Looks pretty cool!