search

ndmitchell / shake

Shake build system
http://shakebuild.com
Other
771 stars 118 forks source link

How to debug failure to recompile? #446

Open ezyang opened 8 years ago

ezyang commented 8 years ago

I have a bug in my build system, where I am not recompiling a file when I think I should. What is the recommended way to diagnose this problem?

UPDATE: It turned out that the bug was from a buggy, old version of the build system, which wrote down incomplete dependency information causing Shake to skip a rule (and thus never getting the updated dependency information). What is an "easy" way to discover this is the case?

One answer is, "blow away the build directory and try again." But this is not very satisfactory. It could very well be a bug, and restarting from scratch is surely not going to induce the recompilation failure. Or perhaps Shake could have some idea about how "old" information in the database is, and if it knows that it was recompiled more recently suggest to try to refresh that information?

Mathnerd314 commented 8 years ago

There's a shakeVersion field in ShakeOptions; changing it is equivalent to blowing away the build directory. The general rule is to bump it any time a rule's list of dependencies or stored value could change; even more conservatively, any time a rule changes.

In fact, only that rule needs to rebuilt, so full rebuilds are overly conservative. Pluto has a per-rule timestamp for each .java/.class file, which could be adapted for Shake, but the syntax would be a bit onerous (ver 123 $ "foo" %> \out -> ...). Hashing the code of functions would work too; e.g. a Template Haskell solution like

  "foo" %> $(hashF [| \out ->
      ...
     |])
    where hashF f = [| ($(hashToLit $ hash f), $f) |]
ndmitchell commented 8 years ago

The equivalent to the Pluto approach (and the Redo approach) is to have each rule in a separate file and depend on the .o file, which you can do, but putting each rule in a separate file is both annoying (hard to edit all the rules) and insufficient (if you import a function that changes it doesn't help).

Unfortunately I don't see a good solution. The only build systems that get this right are those like Ninja/Tup which don't have an EDSL and basically restrict you to direct lines.

ndmitchell commented 6 years ago

I think some kind of query mode "why did X recompile last time?" and "why did X not recompile?" in the profile report would make this easy.