Auto-detect dependencies between libraries

[GoogleCodeExporter]

What change would like to see?

Enable libraries to include other libraries

Why?

AFAIK at present it's not possible for one library to include another library. 
This makes it difficult 
to have, for example, a SPI device with a library that includes a generic SPI 
communication 
library--the SPI library currently needs to be included in the same folder as 
the device library.

Would this cause any incompatibilities with previous versions?  If so, how
can these be mitigated?

It shouldn't.

(I'm reasonably certain this enhancement has been discussed before but I 
couldn't find an 
existing issue for it.)

Original issue reported on code.google.com by follower@gmail.com on 29 Apr 2010 at 4:16

[GoogleCodeExporter]

Libraries can include other libraries, this just isn't auto-detected.  That is, 
the main sketch must include both 
libraries, otherwise the dependent library will not be linked against (or added 
to the include search path).  But 
yes, we should automatically detect dependencies between libraries.

Original comment by dmel...@gmail.com on 29 Apr 2010 at 5:32

• Changed title: Auto-detect dependencies between libraries
• Added labels: Component-IDE

[GoogleCodeExporter]

Thanks for clarifying the exact issue and terminology.

Is this something that could/would be solved by including the "libraries" 
directory in the avr-gcc include 
search path specified by "-I". (And possibly "-L" although that may be suitable 
for pre-compiled libraries 
only.)

If this approach is taken libraries may need to use "Ethernet/Client.h" style 
includes to work correctly--but 
that may be necessary for includes to be unambiguous anyway.

Is it possible for this functionality possible to be targeted to 1.0--it's 
fairly fundamental for modular library 
development and IMO having to include multiple headers for a single library is 
not very friendly to people 
using the IDE.

Original comment by follower@gmail.com on 7 May 2010 at 2:17

[GoogleCodeExporter]

Adding the main libraries directory to the include path (-I) is an interesting 
approach - I wouldn't mind a 
slightly more verbose syntax for including a library from within another.  
Unfortunately, it doesn't solve the 
problem of linking in the sub-library.  

At the moment, I feel like the best option is to recursively search for library 
dependencies from #include 
statements, in the way that we currently scan the main sketch file for them.  
This seems a bit clunky, though, 
so I'm open to other ideas.

I'll tentatively target it for 1.0, since confusion about how the sketches and 
libraries relate to each other was 
one of the big points of feedback from the survey.

Original comment by dmel...@gmail.com on 7 May 2010 at 3:29

• Added labels: Milestone-1.0

[GoogleCodeExporter]

I'd like to suggest an optional library project file again.
A simple properties file with a few build settings the author like to enforce 
should do .

Possible keys in this file could be 
---------------------------
using=<list of other libraries we depend on>
mcu.includes=<list of mcu's for which this library will compile/run>
or 
mcu.excludes=<list of mcu's for which this library will NOT compile/run> 

arduino.version=<some sort of version-depended entry like needs at least/works 
only on>  
--------------------------- 
Name of this file would be <Lib-Name>.properties

Libraries that don't need this feature don't have to supply this file.

Pros:
Easier to parse than source code (standard java.util.properties object).

The properties file can be added to existing libs without touching the source 
code.

Helps future developement. If a new build feature is needed one could simply 
define a
new key for the file that is to be checked by the IDE. 

Original comment by e.fa...@wayoda.org on 10 May 2010 at 3:49

[GoogleCodeExporter]

Issue 712 has been merged into this issue.

Original comment by dmel...@gmail.com on 15 Nov 2011 at 9:08

[GoogleCodeExporter]

Original comment by dmel...@gmail.com on 16 Dec 2011 at 10:05

• Removed labels: Milestone-1.0

[GoogleCodeExporter]

I can confirm that this still exists in 1.0.1.  I'm working on a website to 
promote engineering and plan on developing numerous sensor specific libraries 
that build on Arduino base libraries (Wire, SPI, etc).  The issue described 
above is a big problem because I will have to explain that in order to use my 
libraries you have to include my library and potentially several others in your 
main sketch. The linker errors will not be helpful to the average user that 
forgets the extra #includes in the main sketch. 

Original comment by samkrist...@gmail.com on 3 Jun 2012 at 6:39

[GoogleCodeExporter]

This issue was discussed on the public Arduino developers mail list on May 
29-30, 2012.  I proposed a solution and asked for input before actually writing 
code, particularly on how a portion of the PdePreprocessor class should be 
modified.

http://arduino.cc/pipermail/developers_arduino.cc/2012-May/006713.html

David decided he wants to keep the current behavior.  This issue should 
probably be closed.

Here his the explanation.

On Wed, May 30, 2012 at 9:27 AM, David A. Mellis <dam@mellis.org> wrote:
> > I do wonder if we're going to detect spurious dependencies between
> > libraries.  Any thoughts on this potential problem (e.g. probability,
> > severity, work-arounds)?
To expand on this: the current system is clunky but explicit.  That
is, the user sees the list of dependencies and explicitly selects (by
#include an appropriate file in their sketch) which ones to link
against.  I'm worried about the possibility for confusion if we start
automatically detecting these inter-library dependencies.  You'll
never be quite sure which ones are getting pulled into your sketch.
(We could print a list of them, of course, but you still wouldn't have
a means of controlling their selection.)

Maybe this is a case where a little bit of extra typing is worth the
trade-off in explicitness and clarity?

Sorry to be changing my mind here but I figure it's better to do so
before you start writing the code.

Original comment by paul.sto...@gmail.com on 3 Jun 2012 at 6:57

[GoogleCodeExporter]

Sorry, I've been meaning to come back to this but it's gotten lost in the 
shuffle.  My previous email (the one you site) wasn't meant as a decision, just 
as trying to think through the issues.  My apologies for this conversation 
being so slow.

Maybe explicit control over which libraries are used only makes sense if it 
actually gives you any control?  For example, if we imagine a situation in 
which there are, say, two SPI libraries (e.g. one hardware and one software) 
and you'd select between them with an #include in the main sketch.  Or, say, 
two servo libraries, one which worked on any pins and another which used 
hardware timers generate pulses to avoid the need to use interrupts.  I 
realize, though, that we don't actually do this currently.  Maybe it wouldn't 
even make sense to do it at all?  It would require weird usage of header files 
(where the sketch would #include, say, SoftwareSPI.h, but the library would 
still #include SPI.h) and so maybe it doesn't make sense?

The other advantage (that I mentioned in the previous email) is that it makes 
it clear (or at least visible) to the user which libraries they're actually 
using in a sketch.  For example, if you install two third-party libraries, one 
of which depends on the other, this might make it clear that you're actually 
using both.  Otherwise I could see someone accidentally uninstalling the 
dependency, since they don't realize they're actually using it.  

Again, I'm sorry I'm not as responsive on these issues as I should be.  They 
seem to me like subtle questions of style, which makes them both difficult to 
evaluate but also (in some ways at least) relatively less important than other 
more obvious problems or missing features.  All of which makes it easy to put 
them off.

Original comment by dmel...@gmail.com on 14 Jul 2012 at 11:47

[GoogleCodeExporter]

Ease of use for novices has traditionally been the overriding criteria in 
Arduino's design.

Indeed requiring #include for all libraries compiled does make the build more 
visible.  But is that really valuable?  In a traditional IDE for programmers, I 
could see making depenencies explicit (most have a project/file browser panel 
with a tab for dependencies), but really, in Arduino?!?

I'm working on a big project right now, so I don't have time for a 
point-by-point response.  It's your decision.  If you decide you want this 
patch, I'll do it in a couple months, after I finish this project.

Original comment by paul.sto...@gmail.com on 15 Jul 2012 at 12:17

[GoogleCodeExporter]

Another idea...  when you use "Sketch > Import library...", the IDE could 
automatically add the #includes for the library you imported and for its 
dependencies.  That would mean your sketch would compile correctly but would 
also mean that you could see which other libraries it used.  What do you think?

Original comment by dmel...@gmail.com on 15 Jul 2012 at 12:27

[GoogleCodeExporter]

That sounds like a good idea too.

That's a separate issue from what the build does for the case where a library 
dependency is needed but the #include is missing in the sketch.  The current 
behavior, a kludge for only Ethernet & SPI and horribly confusing error 
messages for all other cases, is not good.

Original comment by paul.sto...@gmail.com on 15 Jul 2012 at 12:50

[GoogleCodeExporter]

I made the suggestion (I think on the dev list) that we use more of the 
compiler chain's functionality to implement a solution--specifically at link 
time.

One concern raised was that if someone was using a function `foo()` there was 
no way to know if it was defined in multiple libraries due to our general lack 
of namespace use.

One possible solution would be to require that if a header is included that it 
be included with a partial path to the header file. e.g. 'bar/foo.h' Then we 
could parse the preprocessed file (assuming there's no other way to extract the 
information) to find which directories should be added to the libraries path 
for the compiler.

Original comment by follower@gmail.com on 15 Jul 2012 at 2:44

[GoogleCodeExporter]

Duplicate symbols at link time is a completely separate problem.

This issue (#236) manifests at compile time and terminates the build long 
before the linker is ever run.

Original comment by paul.sto...@gmail.com on 15 Jul 2012 at 3:28

[GoogleCodeExporter]

The suggestion I made on the dev list was that we automatically include the 
library directories on the include path as a way of solving this dependency 
issue.

An objection was raised that this could lead to the duplicate symbols issue. 
Thus my suggestion for working around that issue is related to a solution for 
this issue.

Original comment by follower@gmail.com on 15 Jul 2012 at 3:48

[GoogleCodeExporter]

Placing the library directory in the include path wouldn't work (I believe), 
because you also need to link in the .c and .cpp files for the libraries (not 
just provide access to their header files).  Also, it would be a big change (at 
this point) to require an initial path in the #include statements for libraries.

Original comment by dmel...@gmail.com on 15 Jul 2012 at 12:42

[GoogleCodeExporter]

I do not want to make changes that would break library compatibility.  I want 
to sumbit the smallest, simplest patch that solves this problem with the least 
possibility of breaking anything, especially compatibility with all existing 
libraries.

Here is the technical detail for the patch I proposed on May 29, 2012:

> I was imagining just applying the existing code that scans for #includes
> to all .cpp files in all libraries.  The most logical place seems to be at the
> end of step #2 of proprocess() inside Sketch.java, since that's where the
> list of libraries has just been built.  It would just loop through all the 
.cpp
> files in the known libraries, parse for #includes and add to the list, the
> same as is done for the sketch.  It would be iteratively in a loop until a
> pass is made with no additional libraries discovered.
>
> One minor complication is the actual parsing for #includes occurs within
> writePrefix() inside preproc/PdePreprocessor.java.  It's easy to just create
> a new PdePreprocessor object for each library .cpp file, but obviously
> writePrefix() can't be called for the libraries because it writes to the build
> folder.  I'm thinking of 2 possible modifications... #1: split writePrefix() 
into
> 2 functions, one doing the preprocessing work, the other actually writing
> the file; or #2: add a flag to the inputs of writePrefix() which causes it to
> skip the actual writing of files.

I was only hours away from writing this patch on May 30, when David put the 
brakes on this issue.  I am still willing and interested to write and throughly 
test this patch... at least in a couple months when my schedule allows for more 
Arduino contributions.

Currently any real work on issue is stalled, waiting for a clear decision.

Original comment by paul.sto...@gmail.com on 15 Jul 2012 at 5:32

[GoogleCodeExporter]

(a) To handle the linking the intention would be to use some combination of 
'-l' and/or '-L' to link in the object files generated when the library is 
compiled;

(b) If the solution did end up requiring partial paths, it would be possible 
for the approach to be forward-compatible by saying to library authors "if you 
want auto-include of dependent libraries you need to use a partial path". (And, 
this would actually be more robust than the current approach anyway because 
duplicate definitions already cause problems, AFAIK.) Worst case scenario, if 
we don't require partial paths then it's no worse than the current case.

Regardless of the specifics, it "should" be possible to enable auto-include of 
dependencies by making use of the existing compiler/linker functionality. But 
before exerting effort into implementing it we need to know if that's what's 
wanted.

Original comment by follower@gmail.com on 15 Jul 2012 at 5:37

[GoogleCodeExporter]

Just saw your new comment, Paul.

I don't really have an objection to someone else implementing the functionality 
within the Arduino IDE instead but it does seem a bit like reinventing the 
wheel if the compiler chain can already do the hard work for us.

But if the functionality gets implemented either way, I think people using 
Arduino will be better off with fewer lines of boilerplate needing to be 
included in their sketches.

Original comment by follower@gmail.com on 15 Jul 2012 at 5:42

[GoogleCodeExporter]

I agree 100% with you regarding "before exerting effort into implementing it we 
need to know if that's what's wanted".

David and the Arduino Team need to make a decision.  That's simply how open 
source works.  Until the project gatekeepers (who accept or reject patches) 
provide a clear decision, any potential contribution to the project is stalled.

Original comment by paul.sto...@gmail.com on 15 Jul 2012 at 5:42

[GoogleCodeExporter]

Philip, I've read your messages again, actually a couple more times, and to be 
honest I do not fully understand your proposal.  I'm not saying your idea is 
wrong or won't work, but merely that I do not understand how it works, and 
especially how "-l" and "-L" options to the linker solves this particular 
problem... that problem being that the earlier build stages didn't even compile 
code which provides functions called from code it did compile.

It's entirely possible, and in fact quite likely that I'm not seeing how you 
idea works only because I've already planned a very specific solution, right 
down to the exact places where I plan to add specific code.  Had David not 
changed course, I very certainly would have implemented in within the first few 
days of June.  Of all the people who could offer an objective opinion on 
alternate ideas, I'm probably the very last.  I admit that.

Really, as long as it works well, I'm perfectly happy to stand back and let 
someone else take this on.  I only want to see this solved.  If somene else 
does it in a different way, then great.  Less work for me!

My main question is whether you plan to personally develop, test and submit 
this patch?  Are comments a specific proposal of work you intend to do, should 
David and the Arduino Team decide to go with your approach?

Original comment by paul.sto...@gmail.com on 15 Jul 2012 at 6:31

[GoogleCodeExporter]

Thinking about this more, I realize I was being silly.  The requirement that 
the main sketch / file references every library used by any other library is 
beyond what's required in most (all?) development environments / languages.  
For example, in Python, your application doesn't have to import every module 
that used by a module it uses.  I think I was just finding excuses to justify 
the current behavior.

For the record, then, I'm in favor of changing the behavior as suggested in 
this issue.  The approach Paul proposed seems fine to me.  Paul (and others), 
sorry for taking so long to make up my mind about something that's probably 
seemed obvious to you for a long time.  I'd love to get your help with it, when 
and if you have time for it.

Original comment by dmel...@gmail.com on 16 Jul 2012 at 11:26

• Changed state: Accepted

[GoogleCodeExporter]

Thanks for the update David.

Paul, in regard to "that problem being that the earlier build stages didn't 
even compile code which provides functions called from code it did compile"--in 
this case my assumption is that (assuming no additional parsing to find 
included libraries of pre-processed files is done instead) all the libraries 
have been pre-compiled when a new microcontroller is first selected.

"Really, as long as it works well, I'm perfectly happy to stand back and let 
someone else take this on.  I only want to see this solved.  If somene else 
does it in a different way, then great.  Less work for me!" <-- That goes 
double for me. :)

"My main question is whether you plan to personally develop, test and submit 
this patch?" <-- At this stage, no.

If your Java-focussed patch works, then that's good. My only concerns are if 
(a) it's unnecessarily re-inventing the wheel; or, (b) it's pushing into Java 
something that would be better handled by the compiler tool chain.

In this situation a solution beats no solution, I just wanted a tool 
chain-focussed solution to at least be considered as an option. If you're aware 
of the possible trade-off and think the Java-focussed approach is still the 
best one, or at least the one you're prepared to implement then I have no 
problem with you doing so. :)

I look forward to reduced boilerplate when using libraries. :)

Original comment by follower@gmail.com on 16 Jul 2012 at 3:51

[GoogleCodeExporter]

David, thanks.  I'll implement this and submit a patch.  It might be a couple 
months until I can dedicate the time, but it will get done.

Philip, I hear your point about leveraging the tool chain vs implementing 
functionality in the IDE.  That would be really nice for people who want to 
build projects without the IDE, and it might even make the IDE simpler.  As a 
practical matter, it's substantial restructuring of Arduino's entire build 
process.  The IDE is already designed this way.  I'm only writing a small patch 
which builds a better library list using parsing that's already in the IDE.  It 
requires zero changes to the build process.

Original comment by paul.sto...@gmail.com on 16 Jul 2012 at 7:04

[GoogleCodeExporter]

Thanks Paul.  Sorry again for being so slow to make up my mind.  I agree with 
you re: the approach.  Philip, what you're proposing sounds interesting (and 
might make sense for Makefile-based compilation) but I agree with Paul that it 
would be a big change to the build process that doesn't necessarily make sense 
in fixing this issue.  

Original comment by dmel...@gmail.com on 17 Jul 2012 at 2:02

[GoogleCodeExporter]

Any progress on that issue? It seems nothing has changed for 1 year and a half.

Original comment by jfpoilp...@gmail.com on 2 Jan 2014 at 10:02

[GoogleCodeExporter]

Java patch or makefile build process fix? Still nothing new on that topic :(

Original comment by gonzo.ve...@gmail.com on 29 Jun 2014 at 12:25

[GoogleCodeExporter]

This issue has been moved over to GitHub:

https://github.com/arduino/Arduino/issues/236

Original comment by msida...@gmail.com on 11 Feb 2015 at 10:07

[GoogleCodeExporter]

If anyone is still watching this very old issue, a pull request to add library 
dependency detection is now on GitHub and in need of testing & feedback.

https://github.com/arduino/Arduino/pull/2792

Original comment by paul.p...@gmail.com on 19 Mar 2015 at 3:57