Use information gathered by iwyu in fix_includes.py to determine whether headers are project headers

[GoogleCodeExporter]

Projects like boost have a single top-level directory; projects without a 
structure like that may be pulling headers from multiple sub-projects related 
to the overall project.

fix_includes.py allows for a single top-level directory to be specified.  
Quoted headers are [almost?] always assumed to be project headers, but there's 
no facility available to specify more than one.

Changes I would like to see:

* iwyu use -isystem / -I flags to determine whether it's considered a project 
header or not
* have iwyu further distinguish by whether it's a system, 3rd party, or project 
header (eg., if it's specified with -isystem on the command-line and -nostdinc 
is *not* in effect: anything found in default system/compiler search paths is 
system, anything with -isystem is 3rdparty, anything with -I is project)
* alter fix_includes.py's --separate_project_includes to take a list of 
directories
* add an option to iwyu and/or fix_includes.py to automatically prefix project 
headers with a partially qualified path.  This could be used to migrate to a 
more boost-like header directory structure.  For example, supposing 
someHeader.h were found through -I${project_dir}/src/Project/lib:
-#include <someHeader.h>
+#include <src/Project/lib/someHeader.h>

What version of the product are you using? On what operating system?

version: iwyu v 0.3, built against clang/llvm 3.5.1
OS: rhel 6 x86_64

Original issue reported on code.google.com by Phant...@gmail.com on 15 Apr 2015 at 7:28

[GoogleCodeExporter]

To clarify: the main problem I ran into is it currently cannot distinguish 
between project headers and system or 3rd party headers.  I want to keep the 
inclusion order to:

system
3rd party
overall project
sub-project

... but there's currently no facility in these tools to distinguish, unless 
you're using a boost-like hierarchy or they happen to be in the same directory 
as the source being compiled.

Original comment by Phant...@gmail.com on 15 Apr 2015 at 7:35

[GoogleCodeExporter]

I'm not sure I want to introduce any more IWYU logic based on -I/-isystem to 
classify headers. We already have some of those heuristics, and they're a 
constant source of bugs because people don't have the same -I conventions as 
the original IWYU author.

As for migrating to a more qualified include structure, we're planning 
something similar at work, and it can be achieved quite easily without all of 
IWYU's infrastructure; you don't need the use-vs-declaration cross-reference to 
do that.

IIRC, it boils down to:

- For each component/project, find all #include directives in their source files
- Use your build system's include paths for each component to normalize all 
header names to absolute paths
- Cross-reference that with a list of absolute paths for your third-parties
- Rewrite the #include directives to a form you prefer

If you're clever about it, you can do this piece-wise component-at-a-time.

> * alter fix_includes.py's --separate_project_includes to take a list of 
directories

Would this alone solve the problem? I haven't worked much with fix_includes.py, 
so I'm not familiar with the details, but this sounds like something that could 
help it make smarter decisions as to #include ordering, etc.

Original comment by kim.gras...@gmail.com on 25 Apr 2015 at 9:59

[GoogleCodeExporter]

re: directory classification, I can see that.

Maybe the solution would be to have flags for dictating a list of directories 
to be considered project or 3rd party paths.  Adding to that, if a base path 
could be specified and the fully-qualified path of the header it actually found 
were compared against the base path, then identify it as a project header.

The path normalization stuff I can do, though it could get annoying for some of 
the more convoluted things people have done over the years (code shared across 
Linux, Solaris, and vxworks).

>> * alter fix_includes.py's --separate_project_includes
>> to take a list of directories
> Would this alone solve the problem? I haven't worked
> much with fix_includes.py, so I'm not familiar with
> the details, but this sounds like something that could
> help it make smarter decisions as to #include ordering,
> etc.

It would certainly make it possible to distinguish between all project & 
non-project headers.  As I said above (this msg), maybe it'd be better to be 
able to specify a single top-level directory and compare the header actually 
found against the base path ... but that would have to occur in iwyu instead of 
the python script (unless I fully-qualified all inclusions of course).

Original comment by Phant...@gmail.com on 29 Apr 2015 at 7:34