Compile faster - only recompile files that have changed

[GoogleCodeExporter]

What change would like to see?

Avoid recompiling when the sources have not changed.  Just reuse the compiled 
.o files from previous compilation.  Make, Ant and other build systems do this. 
 Arduino can too!

Why?

Faster Upload  :-)

Would this cause any incompatibilities with previous versions?  If so, how
can these be mitigated?

Done properly, it should be fully compatible.  I originally submitted this as 
part of a patch on issue 393.  At the time, it was decided to only reorder the 
compile process so errors in the sketch would report faster.  A separate issue 
was suggested, and here it is!

I have recently improved this code and I will be releasing it as part of 
Teensyduino, to solicit feedback and bug reports.  This issue is meant as a 
placeholder for users to comment on how much (or how little) improvement this 
makes in Upload times, and to report any bugs or compatibility problems.

When/if there's enough confidence in the value and reliability to accept this 
feature into Arduino, I'll prepare a patch.

Original issue reported on code.google.com by paul.sto...@gmail.com on 11 Sep 2011 at 3:25

[GoogleCodeExporter]

I am using Arduino 1.0 Beta 4 and installing teensyduino and adding 
"uno.build.dependency=true" in boards.txt has no effect on the compilation 
speed.

Phani J

Original comment by jphani...@gmail.com on 18 Sep 2011 at 6:58

[GoogleCodeExporter]

The speedup is only for the compile done before an upload, and the Arduino's 
cleaning of the build directory when you hit "Save" defeats it as well, but it 
works well when it is invoked.

Original comment by capnbry on 22 Sep 2011 at 3:18

[GoogleCodeExporter]

This video which explains this change... but fails to mention it only works on 
"Upload" but not "Verify".

http://www.youtube.com/watch?v=2dBF1ypQupM

Despite being shown on Hack-a-Day, Adafruit's blog, Dangerous Prototypes, and 
the Arduino Forum - and now over 1000 views accourding to YouTube - it appears 
only a couple people have actually tried this speedup.

Here is the source code, in convenient patch format.  Mostly due to lack of 
response and interest, I'm not planning any further work to contribute this 
Teensy-inspired speedup to the larger Arduino community.  But if anyone does 
want this, here's the patch.

Original comment by paul.sto...@gmail.com on 22 Sep 2011 at 4:20

Attachments:

• compile_speedup.diff

[GoogleCodeExporter]

Thanks for doing this and sending the patch.  Looking it over, it seems very 
straightforward, and probably a good thing to include.  I'd like to hold off 
until after the final 1.0 release, but I'll try to get this in the next version 
after that.  

Original comment by dmel...@gmail.com on 23 Sep 2011 at 3:11

• Changed state: Accepted
• Added labels: Component-IDE, Milestone-Post-1.0

[GoogleCodeExporter]

Here's another user using and grateful for this patch - thanks for making it.  
I come from a makefile environment so having to wait for superfluous compiles 
in this one is a bit frustrating.  

With Arduino 1.0-rc1 and a small sketch (Blink, slightly enhanced from the 
library one) the time to begin the first upload is 4 sec for the complete 
compile vs. 1 sec for subsequent uploads. Not huge, but more than enough for so 
small a sketch! (Intel iMac core 2 duo, 2.16MHz, OS 10.6.8, 3GB; Java for 10.6 
update 5 v1.0).

I'll look forward to the enhancemnt becoming a standard part of the Arduino 
development environment.  

If compatibility is still a concern, how about putting an option in the 
preference dialog, defaulted to "off"?

Original comment by wonkywh...@sent.com on 1 Oct 2011 at 4:24

[GoogleCodeExporter]

I was thinking it might be nice to transition the library and core builds to 
use "make", even if the makefiles are arranged to rebuild all files all the 
time in their default configuration.  Changing the behavior would then become a 
matter for a new makefile, rather than editing the depths of the IDE java.  
This would also allow increased parallelism, and the the ability of a 
user-provided library to provide custom build procedures via their own 
makefiles.  (binary only libraries, for instance.)

Original comment by wes...@gmail.com on 4 Oct 2011 at 10:29

[GoogleCodeExporter]

I would certainly develop a patch to use Make (and perhaps a perl script for 
the preprocessing), if there was genuine interest to accept it into Arduino!  
But you've been following Arduino development for a long time Bill.  You know 
using make has been discussed over and over, with every proposal rejected.  
It's hopeless to even dream of such things!

However, when/if this patch gets used, or maybe even if it never does become 
part of the official Arduino IDE, I may work on a tweak to Compiler.java to run 
multiple instances of gcc simultaneously....

Original comment by paul.sto...@gmail.com on 4 Oct 2011 at 11:04

[GoogleCodeExporter]

As for commenting on compile times. 97% of the compile time for my short 
test/development sketches is spent on the RECOMPILES. 
I turned on vervose in the preferences so I can see what it going on. Of the 
1:15 it takes to complete, 1:10 is recompilation, and of that 85% if for stuff 
my sketch does not use. E.g. Tone.

Surely it is possible to put the time stamp of the source file into the *.O 
file and then check to see that no recomiple is needed, since nothing changed.
Or properly impliment dependencies tracking.

And NOT have the IDE delete the intermediate files on save. (or make it a 
settable preference).

Original comment by RCR...@gmail.com on 9 Oct 2011 at 2:04

[GoogleCodeExporter]

I committed this: 
https://github.com/arduino/Arduino/commit/b0ae6e764d6d1a43eee567dd29869739a6687e
26

I also made a few extra changes:

Removed the per-board preference for this speed-up (if we're going to use it, 
we should use it for everything, although we might want a global preference to 
force recompilation of everything all the time):
https://github.com/arduino/Arduino/commit/1a7fe19348404226c3a28b6fc77fa1ac6a246e
2e

Removing the delete applet folder preference (since it's no longer used):
https://github.com/arduino/Arduino/commit/a29866f7dadc98562e3ba768bf7a0bcd3e20c6
a2

Refactored the compile and upload functions so the speed-up works with verify, 
too:
https://github.com/arduino/Arduino/commit/24a0dfdf000ae22af058f4ea38944075823537
22

Thanks for doing this Paul!

Original comment by dmel...@gmail.com on 17 Dec 2011 at 1:03

• Added labels: Milestone-1.0.1
• Removed labels: Milestone-Post-1.0

[GoogleCodeExporter]

Issue 507 has been merged into this issue.

Original comment by dmel...@gmail.com on 17 Dec 2011 at 1:05

[GoogleCodeExporter]

Original comment by dmel...@gmail.com on 17 Dec 2011 at 1:05

• Changed state: Fixed

[GoogleCodeExporter]

Issue 511 has been merged into this issue.

Original comment by dmel...@gmail.com on 30 Dec 2011 at 10:50

[GoogleCodeExporter]

A minor nuance in windows is that it this optimization appears to ignore 
boards.txt located in the local user directory’s such as:
%USERPROFILE%\My Documents\Arduino\hardware\mega-pro-3.3V\boards.txt

Seem like an obvious oversight as it was created initially on teensy, which 
installs into the %ARDUINO%\arduino-1.0.1\hardware. Where SFE and others are 
instructing to put custom boards into "a 'hardware' directory inside your 
sketches folder".

A quick work-a-round is to simply move the new hardware to the 
%ARDUINO%\arduino-1.0.1\hardware directory.

But definitely worth doing. Especially for those who like to create code bit by 
bit.

Original comment by mpfl...@gmail.com on 7 Jul 2012 at 1:58

[GoogleCodeExporter]

This optimization does not use boards.txt at all.  It doesn't matter if you've 
added stuff in the sketchbook folder or the main boards.txt in the selected 
board/target directory, because nothing in the optimization uses any settings 
from boards.txt.

The original version in Teensyduino did use a line from boards.txt to enable 
this optimization.  That was done because Teensyduino is designed to preserve 
the Arduino IDE's functionality when used with official Arduino boards.  So 
anything extra it adds is either enabled from boards.txt or has some other 
fail-safe mechanism so it won't change behavior for official Arduino boards.

That check for (board).build.dependency=true was removed for the version that 
is now in Arduino 1.0.1.

However, there may be a bug where the dependency checking doesn't work on 
Windows when the pathname contains a space or some other special character.  
Investigating this is on my (horribly backlogged) to-do list.  The 
is_already_compiled() function is designed to force a recompile if anything 
doesn't check out completely right.  There may be a bug in how it parses 
special cases.

If your Arduino 1.0.1 is always recompiling all files, please try moving 
Arduino (at least temporily for the sake of testing) to something like 
C:\arduino-1.0.1 and see if that works?  Even if you never hack on the Arduino 
IDE directly... if someone who uses Windows (I mostly use Linux) could 
investigate this and post definitively what pathnames do and don't work, that 
would help me fix whatever pathname parsing bugs might be happening on Windows.

Original comment by paul.sto...@gmail.com on 9 Jul 2012 at 1:23

[GoogleCodeExporter]

I believe you are correct about the path and what space in the file names.

I just installed "Arduino ERW 1.0.3" using the installer on my XP. And it was 
placed into "C:\Program Files\Arduino\Arduino ERW 1.0.3" and noticed it was 
always a clean compile, taking much longer. Prior, I had been working out of 
"C:\projects\arduino". Native 8.3 compliant. Where these older versions were 
quicker (only a clean compile on first). I moved it to "C:\projects\Arduin3" 
and it behaved as expected, as these filenames are 8.3

I was also able to run it with expected/desired results from the run 
C:\PROGRA~1\Arduino\ARDUIN~1.3\arduino.exe and also from a dos/cmd.exe window 
and using short file names to cd to C:\PROGRA~1\Arduino\ARDUIN~1.3\. (using the 
dir /x command to discover the short names.) 

I found in XP at least the shortcut, as created by ERW's installer on the 
desktop, always reverted its target to long name. Even after I changed it to 
the short name that clearly could work, as expected. Hoping unsuccessfully that 
the shortcut could be amended as a work-a-round. Not sure if this would be the 
same for later windows. Probably.

I just verified that it is white space. 
C:\projects\Arduin12313\arduino.exe works as expected. noting that 
"Arduin12313", shows up as arduin~1.
then changed to "Arduin12313" to "Arduin 3"
C:\projects\Arduin 3\arduino.exe does not. noting that "Arduin 3", also shows 
up as the same arduin~1.
I have also tried various other combination of possible illegal characters such 
as % @ , and . and found they all work. So it appears to be the white space.

So there something about the application not using the short and having 
problems with the long's white space. I have seen this before in other higher 
languages like java. 

Original comment by mpfl...@gmail.com on 23 Dec 2012 at 4:59

[GoogleCodeExporter]

I and a few others on windows have noticed that the temp files are building up, 
understandingly due to this change. As they have raised it with issue 1061 
http://code.google.com/p/arduino/issues/detail?id=1061.

Original comment by mpfl...@gmail.com on 23 Dec 2012 at 5:30

[GoogleCodeExporter]

This should be enabled for me (I'm using it with the teensy 3) but it is still 
recompiling all files each time.  Running arduino 1.0.4 on windows 7 - any 
suggestions?

Original comment by r...@rethinkmedical.com on 18 May 2013 at 8:22

[GoogleCodeExporter]

If you're running on a Teensy board, please report any bugs here:

http://forum.pjrc.com/forums/4-Suggestions-amp-Bug-Reports

Let's not waste the the Arduino Team's valuable time with bug reports on 
hardware they don't support.

Original comment by paul.sto...@gmail.com on 22 May 2013 at 12:05