search

jfgailleur / fabricate

Automatically exported from code.google.com/p/fabricate
0 stars 0 forks source link

Improve build speed by calling hasher() less often #17

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
As per Michael Haggerty's tests, on big projects fabricate is slower than
it needs to be (especially for do-nothing builds), in part because it MD5s
the same dependencies over and over for different commands. Think about how
to speed this up (without breaking anything).

Also investigate why strace_runner + mtime_hasher REbuilt both times for
Michael.

See Michael's fabricate-users message:
http://groups.google.com/group/fabricate-users/browse_thread/thread/64670b732d07
1b75

Original issue reported on code.google.com by benh...@gmail.com on 7 Aug 2009 at 2:06

GoogleCodeExporter commented 9 years ago 
I came across this problem as well, it was so annoying I have written my own 
fix for it. 

My build was taking 7.6 seconds to complete a do-nothing build. My fix for this 
problem reduces the do-nothing build time down to 1.5 seconds. A 5 times speed 
increase.

The main cause of the issue was the re-hashing of large (50 MB) pre-compiled 
header files many times through out the build. 

To work around this I introduced a "Hash Cache" dictionary to store all 
previously calculated hashes. The Hash Cache is only updated with new hashes or 
hashes of build outputs.

The final icing on the cake is to ensure all file names of dependencies or 
outputs are fully normalised so that they always match to same entry in the 
Hash Cache.

I have code for this change that I can supply if you want to add this 
improvement to fabricate.

Simon.

Original comment by simon.al...@gmail.com on 11 May 2011 at 10:55

GoogleCodeExporter commented 9 years ago 
Hi Simon -- great, thanks for that! Yeah, if you could send through a diff 
(attach it to this bug), that'd be great.

Original comment by benh...@gmail.com on 11 May 2011 at 1:14

GoogleCodeExporter commented 9 years ago 
Attached is the current diff for my modifications.

Works very well with my current build system. Will let you know if I find any 
problems with the changes.

Simon.

Original comment by simon.al...@gmail.com on 12 May 2011 at 8:29

Attachments:

GoogleCodeExporter commented 9 years ago 
Applied Simon's patch in r118 (thanks!). You can see the speed difference here: 
http://code.google.com/p/fabricate/source/diff?spec=svn119&r=119&format=side&pat
h=/wiki/Benchmarks.wiki

You'll notice that AtimesRunner is so slow it doesn't really do anything for 
that. However, StraceRunner is fast, and it sped it up 2.4x in the benchmark.py 
benchmark.

Original comment by benh...@gmail.com on 14 Jun 2011 at 1:55