Precompiled-headers support with --prefix_header_includes?

[GoogleCodeExporter]

I am trying to figure out if there is any support for precompiled headers with 
iwyu. The only thing I found is to provide the precompiled header to clang with 
-include and --prefix_header_includes with to iwyu. However the results are not 
what I expected.

For the test I am attaching, the command line reads like this:

F:\tests\build\bin\Release\include-what-you-use.exe 
F:\tests\iwyutest\iwyutest\iwyutest.cpp -w -MD -I"C:\Program Files 
(x86)\Microsoft Visual Studio 10.0\VC\include" -IF:\tests\iwyutest 
-IF:\tests\iwyutest\iwyutest -DWIN32 -DNDEBUG -D_CONSOLE -D_UNICODE -include 
F:\tests\iwyutest\iwyutest\stdafx.h -Xiwyu --prefix_header_includes=remove

And the result is this:

F:/tests/iwyutest/iwyutest/iwyutest.cpp should add these lines:
#include "tchar.h"                      // for _TCHAR, _tmain

F:/tests/iwyutest/iwyutest/iwyutest.cpp should remove these lines:
- #include "stdafx.h"  // lines 4-4

The full include-list for F:/tests/iwyutest/iwyutest/iwyutest.cpp:
#include "tchar.h"                      // for _TCHAR, _tmain

Both suggestions seem incorrect right?

1) #include "tchar.h" should not be necessary because it's already included in 
stdafx.h
2) stdafx.h is needed because this is a precompiled header (although I can 
understand that if there is no real support for this it can complain about it)

Am I doing something wrong here?

Original issue reported on code.google.com by dpun...@gmail.com on 10 Apr 2014 at 2:58

Attachments:

• iwyutest.rar

[GoogleCodeExporter]

Also I must add, "libtest/mytestclass.h" is included in stdafx.h, and this is 
needed by iwyutest.cpp, which is another reason not to remove the include of 
stdafx.h.

Original comment by dpun...@gmail.com on 10 Apr 2014 at 3:01

[GoogleCodeExporter]

Do you normally build this with Visual C++ only, or do you compile with other 
toolchains?

We have a similar setup at work with all translation units explicitly including 
"stdafx.h", and I'm actively working to replace that with a non-intrusive model 
(similar to the one described here: 
http://chadaustin.me/2009/05/unintrusive-precompiled-headers-pch/). My 
realization was: precompiled headers is a technique for build optimization, not 
code organization.

Here's a rhetorical question: How would IWYU know stdafx.h is a precompiled 
header? All you've said is that you want stdafx.h force-included into the 
translation unit (-include). Unfortunately, IWYU does not understand what a 
precompiled header is, so even if you had told it, it wouldn't use that 
information.

So I think IWYU sees stdafx.h as a header that doesn't provide any useful 
symbols, and therefore suggests to remove it. tchar.h, on the other hand, is 
used and should be immediately included.

As for libtest/mytestclass.h, that should also be added to iwyutest.cpp, but I 
think IWYU is confused because stdafx.h is force-included on command-line.

So I think there are multiple interrelated issues here, but I can't quite see 
what's going on

That said, if we can make small changes to enable the default model of 
precompiled headers for MSVC (explicitly included) it would be nice to make 
IWYU play better with Windows apps.

I'll tinker with your project and see what I can figure out...

Original comment by kim.gras...@gmail.com on 10 Apr 2014 at 6:56

[GoogleCodeExporter]

These are my results with my (erroneous) patch from issue 125 applied:

--
D:\temp\iwyutest> include-what-you-use.exe -I. iwyutest\iwyutest.cpp

iwyutest\iwyutest.cpp:14:12: warning: format string is not a string literal 
(potentially insecure) [-Wformat-security]
    printf(buf);
           ^~~

iwyutest/iwyutest.cpp should add these lines:
#include <stdio.h>                      // for printf, sprintf_s
#include <tchar.h>                      // for _TCHAR, _tmain
#include "libtest/mytestclass.h"        // for ZMyTestClass

iwyutest/iwyutest.cpp should remove these lines:
- #include "stdafx.h"  // lines 4-4

The full include-list for iwyutest/iwyutest.cpp:
#include <stdio.h>                      // for printf, sprintf_s
#include <tchar.h>                      // for _TCHAR, _tmain
#include "libtest/mytestclass.h"        // for ZMyTestClass
--

So IWYU suggests removal of stdafx.h because it's not adding any value 
(remember, IWYU doesn't know it's a precompiled header).

I get the same results whether or not I add "-include iwyutest\stdafx.h" to the 
command-line.

But if I also add "-Xiwyu --prefix_header_includes=remove", I can reproduce the 
behavior you're seeing.

I think the reason you're getting different results for tchar.h vs 
mytestclass.h is because the former contains a macro and issue 125 still isn't 
properly resolved.

Original comment by kim.gras...@gmail.com on 10 Apr 2014 at 7:06

[GoogleCodeExporter]

Yes that's what I thought, that there wasn't really support for that. But 
still, to answer how would the tool know about the precompiled header, I 
doubted that it did [understand the whole concept], but at least I thought that 
as far as compilation goes it would have the same result: don't ask to include 
something that is already included through the command line. But I am just 
supposing, I am no expert in compilers really, so excuse me if I'm talking 
nonsense.

To answer your other question, we use other compilers for other platforms but 
we always build from VS (we generate other solutions).

In any case, the thing that confused me the most is that for 2 files that are 
included on the same place (stdafx.h) with the same mechanism (-include) I 
would get different results:

1) include tchar.h
2) no mention to libtest/mytestclass.h

Be aware that all this is tested with the last fix we mentioned on thread 125, 
I just mention it because I don't know if it can affect somehow.

Thanks a lot for all the info, this tool could be great.

Original comment by dpun...@gmail.com on 10 Apr 2014 at 7:08

[GoogleCodeExporter]

Maybe an "-Xiwyu --keep=stdafx.h" switch would be a nice, simple way to tell 
IWYU that includes of "stdafx.h" are not to be touched?

I can imagine there are thousands of Windows apps with #include "stdafx.h" as 
their first line in every translation unit.

Original comment by kim.gras...@gmail.com on 10 Apr 2014 at 7:11

[GoogleCodeExporter]

Thanks for the additional detail.

So are these other compilers set up to also generate/use precompiled headers, 
or is "stdafx.h" treated as just another include? I'm asking out of concern for 
your well-being (:-)), because stdafx.h tends to attract LOTS of includes, and 
if they're not precompiled, they usually add enormously to build times (we had 
this situation for a while).

If you haven't read them, I can recommend the wiki articles on IWYU's 
philosophy:
https://code.google.com/p/include-what-you-use/wiki/WhatIsAUse
https://code.google.com/p/include-what-you-use/wiki/WhyIWYU

The basic idea of IWYU is that any use of a symbol should force a translation 
unit to include the header defining that symbol directly. As far as IWYU is 
concerned, stdafx.h does not provide any symbols, so it's considered useless.

For us sentient beings, we know that stdafx.h brings other benefits (fast 
builds!), but IWYU has no idea. Hence the idea for a --keep switch, even if it 
feels hacky.

Hope that helps explain!

Original comment by kim.gras...@gmail.com on 10 Apr 2014 at 7:21

[GoogleCodeExporter]

My answer is mostly to the first message and is somewhat detached from the 
present discussion.

A little terminology clarification: prefix headers and precompiled headers are 
the same.  These are headers included from the command line.  Header foo.h can 
be stored on a disk in precompiled form foo.h.pch.  But if it's not there, 
Clang will just use foo.h according to Clang User's Manual [1].

Here is my interpretation.  IWYU recommends to remove stdafx.h because
a) nothing from stdafx.h itself is used;
b) it is a prefix header and IWYU is launched with option 
--prefix_header_includes=remove, so prefix headers from source code are removed.

I'm not sure if point b) has any effect, maybe a) trumps it.

Though tchar.h is prefix header too (header prefixness is a transitive 
relation), IWYU recommends to add it because of _TCHAR and _tmain usage.  As we 
now from issue #125 IWYU fails to find correct FileEntry for macros and that's 
why doesn't understand that tchar.h is prefix header.

And IWYU doesn't recommend to add stdio.h, libtest/mytestclass.h because they 
are prefix headers and IWYU is launched with option 
--prefix_header_includes=remove.

[1] http://clang.llvm.org/docs/UsersManual.html#using-a-pch-file

Original comment by vsap...@gmail.com on 10 Apr 2014 at 7:38

[GoogleCodeExporter]

Yes I get all that, it makes a lot of sense. However, it's not feasible to 
maintain two versions (one with precompiled headers and one without). That's 
why to me, just feeding it as normal include for compilation "should" work.

But what you say explains a lot, IWYU tries to enforce proper includes and 
headers, and that's not compatible with the concept of precompiled headers. But 
in practice, there is a lot of people working with this :_)

And yes, we use precompiled headers on all our platforms and compilers.

I appreciate a lot the time you guys take on all this. Thank you!

Original comment by dpun...@gmail.com on 10 Apr 2014 at 7:41

[GoogleCodeExporter]

BTW my last answer was directed to Kim :)

I will read the new one now.

Original comment by dpun...@gmail.com on 10 Apr 2014 at 7:42

[GoogleCodeExporter]

Thank you vsapsai for the clarification.

To me add and remove give the same result. Add is the only one that seems kind 
of coherent (suggests to remove stdafx.h and add everything else), but it's 
still now the behavior I expect.

Original comment by dpun...@gmail.com on 10 Apr 2014 at 7:47

[GoogleCodeExporter]

Sorry, KEEP and REMOVE

Original comment by dpun...@gmail.com on 10 Apr 2014 at 7:47

[GoogleCodeExporter]

@vsapsai:

> A little terminology clarification: prefix headers and precompiled headers 
> are the same.  These are headers included from the command line. 
> Header foo.h can be stored on a disk in precompiled form foo.h.pch. 
> But if it's not there, Clang will just use foo.h [...]

Well, yes and no.

Neither GCC nor MSVC require that you include precompiled headers from the 
command-line. Rather, however foo.h is included, if there's a corresponding 
foo.h.gch (GCC) or foo.pch (MSVC), it will be used instead.

I think GCC users typically use -include to pull in their PCHs, because it's a 
smart thing to do, but it's by no means required.

Clang appears to be different in that PCHs are only allowed on the command-line.

The default MSVC style is to explicitly #include "foo.h" (conventionally called 
stdafx.h), but it also accepts command-line includes (using /FI).

However, -include and /FI are not limited to precompiled headers. You can use 
it to include a global config.h everywhere, for example. Note that the Clang 
manual describes a separate step for generating the PCH:

 $ clang -x c-header test.h -o test.h.pch

We may be using different terminology, but I think of prefix headers as the 
ones put on the command-line (whether they have been precompiled or not) and 
precompiled headers as the ones from which we generate .pch/.gch. They can 
coincide, but are different.

</aside>

> (header prefixness is a transitive relation)

Oh, so explicit in-source includes of headers also specified on the 
command-line are covered by the --prefix_header_includes policy? I didn't 
realize this. I thought it only covered headers included inside the prefix 
header.

Original comment by kim.gras...@gmail.com on 10 Apr 2014 at 8:17

[GoogleCodeExporter]

@dpunset,

I agree, it would be nice if IWYU could respect the "stdafx.h" convention, 
since it's so common.

What do you think about a separate command-line argument for it?

Original comment by kim.gras...@gmail.com on 10 Apr 2014 at 8:44

[GoogleCodeExporter]

A separate command line would allow to choose a specific behavior, and I 
normally try to be as specific always as I can. But in this case, don't you 
think that the fact that we are providing a header with -include should already 
tell the tool: whatever is in there just take it, assume whatever is there is 
already included from it. Basically assuming that every cpp starts including 
that file (which vs enforces), so don't enforce the correctness we talked about 
before.

But if you want to go for something specific I think it's ok too.

Original comment by zzzet...@gmail.com on 10 Apr 2014 at 9:00

[GoogleCodeExporter]

That was me sorry, my iPad did something weird there :(

Original comment by dpun...@gmail.com on 10 Apr 2014 at 9:03

[GoogleCodeExporter]

But you don't use -include or /FI in your normal builds, right, since you have 
stdafx.h explicitly included in source code? Rather, I'm guessing you just have 
/Yc to create a PCH and /Yu to mark stdafx.h as a PCH.

Basically, I think IWYU _should_ ask to remove includes that are already 
satisfied by the command-line, whether they are precompiled headers or not. 
Given the information at hand, the include is redundant (and this would be a 
really nice way for me to clean out all our explicit #include "stdafx.h", for 
example)

An additional arg would be useful to change that default policy, so that a 
specific header can be exempt from IWYU analysis. So instead of what you're 
doing now, you could say:

 $ include-what-you-use <normal compile args> -Xiwyu --keep=stdafx.h

-include would not be necessary.

Since the PCH patterns vary so much (some people use -include, some include 
explicitly, compilers allow different variations, etc), it makes sense to me to 
be explicit. Besides --keep could be used for any include, not just precompiled 
headers.

I'll try to cook up an example patch.

Original comment by kim.gras...@gmail.com on 11 Apr 2014 at 5:02

[GoogleCodeExporter]

Alright! :) Sounds good then.

Original comment by dpun...@gmail.com on 11 Apr 2014 at 1:42

[GoogleCodeExporter]

Till now I've assumed that precompiled headers in MSVC work the same way as in 
Clang.  When I was implementing --prefix_header_includes option I didn't know 
about MSVC peculiarities and that's why --prefix_header_includes doesn't work 
very well for Visual Studio projects.  I'd like to make 
--prefix_header_includes more MSVC-friendly.  Can you please verify that I 
understand MSVC correctly?  I've also described Clang behavior for completeness 
and contrast.

Clang expects precompiled header to be specified in command line.  For each 
-include some_header.h Clang looks for some_header.h.pch and uses .pch file if 
present.  Then in source code you can include any file reachable from prefix 
header and it will incur no performance cost (only if header guards are 
present, I guess).  I don't know if multiple precompiled headers are possible 
and if precompiled header will still work if user -include another header 
before precompiled header.

Here is what MSVC behavior is.  You need to specify precompiled header in 
command line with /Yu[filename] and in .cpp file #include 
"precompiled_header.h" should be the first non-comment line.  In some_class.cpp 
#include "precompiled_header.h" is always before #include "some_class.h".  
Multiple precompiled headers aren't allowed, any non-comment before #include 
"precompiled_header.h" will break precompiled header mechanism.

Original comment by vsap...@gmail.com on 11 Apr 2014 at 5:07

[GoogleCodeExporter]

As far as I know, in clang the precompiled header is specified like this:

-include-pch myprecompiledheader.pch (see 
http://clang.llvm.org/docs/PCHInternals.html)

It seems the same as MSVC right?

Original comment by dpun...@gmail.com on 11 Apr 2014 at 5:21

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Forgive me, there is some difference:

MSVC: 

/Yc"stdafx.h" /Fp"Release\stdafx.h.pch" (to generate the pch)
/Yu"stdafx.h" /Fp"Release\stdafx.h.pch" (to use the pch)

clang: 

-x c++-header"Release/stdafx.h.pch" (to generate the pch)
-include-pch"Release/stdafx.h.pch" (to use the pch)

Original comment by dpun...@gmail.com on 11 Apr 2014 at 6:00

[GoogleCodeExporter]

You've provided internals' documentation intended for Clang developers. User's 
Manual is located at 
http://clang.llvm.org/docs/UsersManual.html#usersmanual-precompiled-headers

And for Clang it is:

-x c++-header "stdafx.h" -o "Release/stdafx.h.pch" (to generate the pch)
-include "Release/stdafx.h" (to use the pch)

Original comment by vsap...@gmail.com on 11 Apr 2014 at 7:28

[GoogleCodeExporter]

Hm, I'm not sure then :_) We use the method I described, but I don't know 
what's the difference.

Original comment by dpun...@gmail.com on 11 Apr 2014 at 7:44

[GoogleCodeExporter]

> Here is what MSVC behavior is.  You need to specify precompiled header
> in command line with /Yu[filename] and in .cpp file
> #include "precompiled_header.h" should be the first non-comment line.

Yes, that's right, and [filename] is "precompiled_header.h", i.e. the source 
file used to geneate the PCH.

But note that you can use /FI (MSVC's spelling of -include) instead of 
including in the source file. The (unfortunate) convention is to use in-source 
includes.

> In some_class.cpp #include "precompiled_header.h" is always before #include 
> "some_class.h".  Multiple precompiled headers aren't allowed, any non-comment
> before #include "precompiled_header.h" will break precompiled header 
mechanism.

Yes.

As a side note, the behavior I've observed is that MSVC discards any content 
before #include "precompiled_header.h" so it won't break the PCH mechanism, but 
rather miscompile the entire translation unit (On several occasions I've had 
#defines at the top of a .cpp file silently removed.)

Original comment by kim.gras...@gmail.com on 12 Apr 2014 at 8:04

[GoogleCodeExporter]

As a follow-up to the previous comment, I'd like to stress, again, the 
difference between prefix headers and precompiled headers.

It makes sense to me to treat them as two separate concepts, because they can 
be used independently, but can also be combined.

Prefix headers
==============
Any header file included from the command-line, using -include on GCC/Clang or 
/FI on MSVC. (I realize that this differs from your definition, but bear with 
me.)

This mechanism is useful to force global header files (e.g. a set of 
configuration #defines) into all translation units without having to explicitly 
#include them in source code.

All compilers allow multiple -include or /FI switches, and they are added in 
the specified order.

It _can_ be used to pull in a precompiled header, but it can also be used to 
pull in regular old headers.

Precompiled headers
===================
Precompiled headers (PCH) are a build-system optimization that lets us collect 
a number of expensive #includes in a single header file (say, precompiled.h), 
and precompile it to some intermediary form (say, precompiled.pch).

Behavior here differs between compilers:
- GCC and MSVC: Whenever they process an #include directive (in source or 
command-line) for "foo.h", they first look for "foo.pch" (naming conventions 
vary here), and if it exists, they pull in the precompiled version instead, 
which saves time on parsing and maybe I/O.
- Clang: Presumably as an optimization, they only perform the PCH check for 
prefix headers (-include), but otherwise the behavior is the same.

Conclusion
==========
So, the combination matrix is a little different between compilers:
- Clang cannot have PCH without prefix headers, but can have prefix headers 
that aren't PCHs
- MSVC and GCC can combine PCH and prefix headers freely

Since a prefix header does not strictly imply PCH on any compiler, I think it's 
unfortunate to commingle the concepts. The current --prefix_header_includes 
switch works great for prefix headers on all compilers (provided we translate 
MSVC's /FI syntax to -include). It only pulls the command-line includes into 
the IWYU analysis, which can help to either remove headers that are already 
pulled in by prefix headers or to make a codebase independent of prefix headers 
by adding them as in-source #includes where necessary. That's a great feature!

PCH really has nothing to do with the #include graph. In a well-implemented PCH 
solution every translation unit should build correctly with or without PCH.

Unfortunately, the conventional MSVC PCH setup is not well-implemented :-). 
Since every source file explicitly has #include "stdafx.h", there's a tendency 
for folks to rely on its presence, and #includes naturally gravitate into 
"stdafx.h", which is in direct conflict with IWYU's philosophy of including 
exactly what you use.

Now, to the real conclusion: Since IWYU does not care about all this, the PCH 
is just another header, and IWYU will do the right thing (even without 
--prefix_header_includes) and copy #includes from the PCH into the source files 
where they're used. The net result of this is that the #include of the PCH 
itself is pointless, they never define any symbols, and IWYU will suggest its 
removal.

But by removing the #include "stdafx.h" we interfere with the MSVC build 
system, which has /Yu "stdafx.h", and the project will fail to compile since 
we're no longer including "stdafx.h".

We could try and scan the command-line args for "/Yu", but that would only be 
valid for MSVC, not GCC.

This is why I think it would be fair to give IWYU a hint that the PCH header 
should be left alone. And I can imagine it would be useful to name any header, 
PCH or not, that IWYU shouldn't touch, which is why I think --keep=<include 
name> is nice.

(Sorry about the novel-length response, but I wanted to make sure we were on 
the same page.)

Original comment by kim.gras...@gmail.com on 12 Apr 2014 at 9:06

[GoogleCodeExporter]

Here's a first draft at a patch for --keep. It's essentially the same as // 
IWYU pragma: keep, but from the command-line.

For the original example, here's how you'd use it:

F:\tests\iwyutest\> include-what-you-use.exe iwyutest\iwyutest.cpp -w -MD -I. 
-DWIN32 -DNDEBUG -D_CONSOLE -D_UNICODE -Xiwyu --keep=stdafx.h

A few problems:

- While stdafx.h is now retained, it ends up at the end of the #include list, 
which isn't legal
- IWYU now can't be instructed not to copy includes from stdafx.h to the source 
file. We could work around this by also adding -include stdafx.h and -Xiwyu 
--prefix_header_includes=keep, but then we're back to mixing the two concepts 
again...

I could try and make this more explicit by changing the argument to "--pch" and 
then use that extra semantics to

a) recognize that this header should sort first
b) also take this header into consideration as a prefix header, so 
prefix_header_includes can apply there too

I'm not excited about these suggestions :-/

Any other ideas?

Original comment by kim.gras...@gmail.com on 12 Apr 2014 at 2:02

Attachments:

• issue126-keep.patch

[GoogleCodeExporter]

OK, given the challenges with my patch, I went back and looked at my braindump 
on prefix headers vs. precompiled headers.

The information is still correct, but this time I came to another conclusion 
that I think will solve everything :-)

- Prefix headers and precompiled headers are two different beasts
- Instead of considering them different, I'm inclined to follow your lead and 
let IWYU just call them prefix headers, because...
- ... here's what I realized: the way we want IWYU to handle of them is very 
similar (both of them should obey some add|keep|remove rule)
- IWYU has no way to know that MSVC-style, in-source precompiled headers are 
special
- So... We need to tell it. The switch should not be --keep, but rather 
--prefix_header (name suggestions welcome)
- This means the total set of prefix headers is the union of all "-include" 
switches and all #include directives that are marked as "-Xiwyu 
--prefix_header" on the command-line
- -Xiwyu --prefix_header_includes now applies to both the actual -include 
prefix headers and the in-source headers marked as prefix headers. 
CleanupPrefixHeaderIncludes does its work on all of them => Life is good.

There's still the issue of making sure prefix headers in-source sort before all 
other headers, but that's probably doable.

Does this sound like a viable strategy?

Original comment by kim.gras...@gmail.com on 12 Apr 2014 at 8:07

[GoogleCodeExporter]

After all the explanation it's understandable that they have to be treated 
different (precompiled headers and prefix headers), thank you for all the info.

I'm a bit confused about the command line options though. Can you give an 
example, like the one I provided at the top? How would that be with the new 
format?

Original comment by dpun...@gmail.com on 12 Apr 2014 at 10:38

[GoogleCodeExporter]

I'm experimenting with a patch that allows:

  F:\tests\iwyutest\> include-what-you-use.exe -I. -Xiwyu --explicit_prefix_header=stdafx.h -Xiwyu --prefix_header_includes=keep iwyutest\iwyutest.cpp

So we specify that stdafx.h is included in source code using 
--explicit_prefix_header and then how we want to treat it, using 
--prefix_header_includes.

Original comment by kim.gras...@gmail.com on 13 Apr 2014 at 7:30

[GoogleCodeExporter]

Here's a patch that implements this.

I also added a minimal, wrong, fix to the problem in issue 125 to be able to 
test all forms of --prefix_headers_include.

If we can agree on the general direction of this, I'd be happy to add tests and 
handle the sorting issue.

Original comment by kim.gras...@gmail.com on 13 Apr 2014 at 8:53

Attachments:

• issue126-commingle.patch

[GoogleCodeExporter]

What is the desired behavior? According to the patch

--explicit_prefix=stdafx.h --prefix_header_includes=add: include actually used 
headers, keep stdafx.h though it's unnecessary

--explicit_prefix=stdafx.h --prefix_header_includes=keep: keep all used prefix 
headers, don't add new used headers, remove unused but keep stdafx.h

--explicit_prefix=stdafx.h --prefix_header_includes=remove: remove all prefix 
headers

What is the main goal of the patch? To maintain precompiled headers in Windows 
code bases working regardless of all IWYU recommendations? I am asking because 
I have other ideas how to separate "prefix headers" and "precompiled header 
include should be in source" aspects. I don't claim that my ideas are better, 
just want to discuss them.

Original comment by vsap...@gmail.com on 13 Apr 2014 at 10:22

[GoogleCodeExporter]

Yes, the desired behavior is exactly as you state it.

Yes, I suppose you could state the goal of the patch like that, but it also 
gives us the opportunity to automatically clean up MSVC-style precompiled 
headers by inlining all #includes from the PCH and removing the PCH include. 
That's my favorite aspect.

I'd love to hear more ideas!

Original comment by kim.gras...@gmail.com on 14 Apr 2014 at 4:48

[GoogleCodeExporter]

I applied the patch, but I see no difference with the test.

This is the command line:

F:\tests\build\bin\debug\include-what-you-use.exe 
F:\tests\iwyutest\iwyutest\iwyutest.cpp -w -MD -IF:\tests\iwyutest 
-IF:\tests\iwyutest\iwyutest -DWIN32 -DNDEBUG -D_CONSOLE -D_UNICODE -Xiwyu 
--explicit_prefix=F:\tests\iwyutest\iwyutest\stdafx.h -Xiwyu 
--prefix_header_includes=keep > f:\iwyulog.txt 2>&1

This is the output I get:

F:/tests/iwyutest/iwyutest/iwyutest.cpp should add these lines:
#include <stdio.h>                      // for printf, sprintf_s
#include <tchar.h>                      // for _TCHAR, _tmain
#include "libtest/mytestclass.h"        // for ZMyTestClass

F:/tests/iwyutest/iwyutest/iwyutest.cpp should remove these lines:
- #include "stdafx.h"  // lines 4-4

The full include-list for F:/tests/iwyutest/iwyutest/iwyutest.cpp:
#include <stdio.h>                      // for printf, sprintf_s
#include <tchar.h>                      // for _TCHAR, _tmain
#include "libtest/mytestclass.h"        // for ZMyTestClass
---

Is this patch supposed to be applied only with the original latest clean source 
of IWYU? Or it can be applied on top of all the other modifications we talked 
about to remove the assert and keep files that don't have decl?

Original comment by dpun...@gmail.com on 14 Apr 2014 at 2:08

[GoogleCodeExporter]

--explicit_prefix=F:\tests\iwyutest\iwyutest\stdafx.h
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This should just be:

--explicit_prefix=stdafx.h

It's not a file path, but the include name as it appears in source.

I think it might conflict with the changes for issue #125, I added a quick & 
dirty workaround to be able to test this patch. You can revert the change to 
iwyu_output.cc from this patch if you want to mix with other patches for issue 
#125.

Original comment by kim.gras...@gmail.com on 14 Apr 2014 at 2:14

[GoogleCodeExporter]

With the command line corrected I get this:

F:/tests/iwyutest/iwyutest/iwyutest.cpp should add these lines:
#include <stdio.h>                      // for printf, sprintf_s
#include <tchar.h>                      // for _TCHAR, _tmain
#include "libtest/mytestclass.h"        // for ZMyTestClass

F:/tests/iwyutest/iwyutest/iwyutest.cpp should remove these lines:

The full include-list for F:/tests/iwyutest/iwyutest/iwyutest.cpp:
#include <stdio.h>                      // for printf, sprintf_s
#include <tchar.h>                      // for _TCHAR, _tmain
#include "libtest/mytestclass.h"        // for ZMyTestClass
#include "stdafx.h"
---

It shouldn't be reporting that I need to add the includes that exist already in 
the precompiled header, does it?

Original comment by dpun...@gmail.com on 14 Apr 2014 at 2:21

[GoogleCodeExporter]

Sorry, my mistake. I'd cut out the one thing from the patch that made any 
difference :-). New patch attached.

Original comment by kim.gras...@gmail.com on 14 Apr 2014 at 2:44

Attachments:

• issue126-commingle-2.patch

[GoogleCodeExporter]

The first impression of it is that it worked really well on the test.

Now I'm testing it on a regular source file from one of our projects, and the 
first result is very nice, I get many occurrences of files to remove. However, 
I do get one case of a file that is suggested to be removed and when done so 
the source doesn't compile.

I will try to prepare a basic example with the case with as less files as 
possible and investigate a bit.

I'll keep you posted. Good job!

Original comment by dpun...@gmail.com on 14 Apr 2014 at 3:01

[GoogleCodeExporter]

I offer to keep explicit_prefix even if prefix_header_includes=remove.  To my 
mind, it will make options orthogonal.  Though prefix headers and precompiled 
headers are fairly intertwined concepts, we can separate them the following way.

Prefix and precompiled headers are important for IWYU because they are 
omnipresent.  So --prefix_header_includes actually means 
--includes_from_omnipresent_headers.  And from this point of view, prefix 
headers and precompiled headers are completely the same for IWYU.  Especially 
that Clang requires precompiled header to be prefix headers.  I don't suggest 
to change option name to --includes_from_omnipresent_headers because it's 
unclear and long.

The fact that precompiled header should be specified in the source code is 
completely independent on includes_from_omnipresent_headers policy.  User might 
want to remove headers already provided by omnipresent precompiled header, but 
still needs #include "precompiled_header.h" so that precompiled header actually 
works.

Original comment by vsap...@gmail.com on 14 Apr 2014 at 3:43

[GoogleCodeExporter]

I've just realized my failed test is not related to precompiled headers, so I 
will move it to another thread.

Original comment by dpun...@gmail.com on 14 Apr 2014 at 4:34

[GoogleCodeExporter]

> Prefix and precompiled headers are important for IWYU because they
> are omnipresent.  So --prefix_header_includes actually means
> --includes_from_omnipresent_headers.  And from this point of view,
> prefix headers and precompiled headers are completely the same for IWYU.

Yes, this is the conclusion I came to as well.

> I offer to keep explicit_prefix even if prefix_header_includes=remove.
> To my mind, it will make options orthogonal.

That makes sense. If a user (e.g. me) would want to get rid of explicit 
precompiled header includes, sed is a perfectly capable tool.

However, I don't see how this makes them orthogonal, they depend on each other 
in some sense -- prefix_header_includes will still apply to the contents of the 
header named by explicit_prefix, right?

The way I look at explicit_prefix is it hints to IWYU that something _not_ 
mentioned on the command-line is a prefix header, it just extends the set of 
prefix headers.

So for consistency, I think I'd prefer it if prefix_header_includes never 
removed in-source includes of _any_ prefix headers, which would eliminate the 
special case. But the current behavior is useful enough that I can live with 
the special case :-)

To get this to work, however, you'd have to add the special case on top of my 
latest patch. Normal IWYU policy removes precompiled headers since they don't 
provide any symbols, and that's what my patch fixes.

Also I find --explicit_prefix an appaling switch name, any suggestions for 
better names?

Glad to hear this is converging!

Original comment by kim.gras...@gmail.com on 14 Apr 2014 at 5:54

[GoogleCodeExporter]

> prefix_header_includes will still apply to the contents of the header named 
by explicit_prefix, right?

What if instead of specifying file name in --explicit_prefix we have a boolean 
option --in_source_precompiled_header and treat the first include as special? 
We can mark this special include as prefix header or require it to be included 
from command line, i.e. to be Clang-compatible precompiled header. I guess, it 
is more user friendly not to require -include "precompiled_header.h". Doubt, 
that I've made naming problem with in_source_precompiled_header easier.

Original comment by vsap...@gmail.com on 14 Apr 2014 at 8:09

[GoogleCodeExporter]

This looks like it would be compatible with both GCC and MSVC.

How about --assume_pch?

We would need to:

- Detect the first include in a translation unit, and mark it as a prefix 
header if --assume_pch is set
- Protect it from removal in MaybeProtectInclude
- Not remove it in CleanupPrefixHeaderIncludes
- Split the set of desired headers into desired prefix headers and normal 
desired headers, and emit prefix headers before normal in iwyu_output

The problem with not naming the assumed pch is that the third step feel more 
difficult, we'd have to magically figure out which included header is the PCH, 
and leave it alone. One option might be to leave all prefix headers in, like I 
suggested earlier.

Original comment by kim.gras...@gmail.com on 14 Apr 2014 at 8:44

[GoogleCodeExporter]

I've thought more about a name that emphasises the fact that precompiled header 
should be mentioned in source, something like --in_source_pch or --in_code_pch 
or --[is_]pch_in_code.

We can figure out which included header is the PCH the same way we figure out 
if header is prefix header, i.e. we can obtain IwyuFileInfo from 
IwyuPreprocessorInfo like preprocessor_info->FileInfoFor(file_entry).

Original comment by vsap...@gmail.com on 19 Apr 2014 at 1:46

[GoogleCodeExporter]

"--pch_in_code" sounds good, I think.

As, so we'd have code in IwyuPreprocessorInfo::FileChanged_EnterFile to detect 
the first #include in the main file, mark that IwyuFileInfo as the PCH (e.g. 
file_info->set_pch_in_code()), and then use the same lookup as for prefix 
headers in iwyu_output?

Sounds good, I'll try and put a patch together.

Original comment by kim.gras...@gmail.com on 20 Apr 2014 at 7:47

[GoogleCodeExporter]

Yes, that was the idea, to use FileChanged_EnterFile to detect the first 
#include and to add pch_in_code property to IwyuFileInfo.

Original comment by vsap...@gmail.com on 20 Apr 2014 at 8:53

[GoogleCodeExporter]

Please take a look at the attached patch.

I still haven't added any tests, I thought I'd do that once the dust settles. I 
removed the tchar.h dependency from one of dpunset's examples, to get rid of 
the macro bug in issue 125, and then tried all variations of --pch_in_code and 
--prefix_header_includes. They act as I would expect.

I've wired it up so that IwyuFileInfo has separate flags for is_prefix_header 
and is_pch_in_code, because

- is_pch_in_code is used to sort the PCH header first
- The PCH header itself should not be considered in CleanupPrefixHeaderIncludes

However, all headers included by a PCH header are marked as prefix headers (see 
iwyu_preprocessor.cc L725), so that they are considered the same way in 
CleanupPrefixHeaderIncludes.

Let me know if this approach looks good, and I'll try to wrap it up into a full 
patch with tests.

Original comment by kim.gras...@gmail.com on 21 Apr 2014 at 3:34

Attachments:

• 126-pch-in-code.patch

[GoogleCodeExporter]

Hello!

I am trying to put the latest patches together. I removed my current source 
code of iwyu, I integrated the patch from issue #125 (issue125-r3.patch), and 
now I am trying to integrate this patch (126-pch-in-code.patch).

I am runing into a compilation issue with this line:

> case 'h': pch_in_code = true; break;

since 'h' is already used in the case as this:

> case 'h': PrintHelp(""); exit(0); break;

Is it possible I am missing another patch?

Original comment by dpun...@gmail.com on 22 Apr 2014 at 2:48

[GoogleCodeExporter]

By the way, apart from this compilation error the results seem to be good for 
the tests made on 125 and 126.

Original comment by dpun...@gmail.com on 22 Apr 2014 at 3:14

[GoogleCodeExporter]

Kim, overall the patch looks good to me. There are a few tweaks I'd like to 
make, like pch_in_code explanation in PrintHelp, but it's irrelevant to general 
patch logic.

dpunset, are you using trunk or branch/tag? Because in trunk 

  case 'h': pch_in_code = true; break;
is in CommandlineFlags::ParseArgv and 

  case 'h': PrintHelp(""); exit(0); break;
is in ParseInterceptedCommandlineFlags. See r526 for more details.

And my late response to comment #23 about the difference between -include and 
-include-pch. Personally I think -include is preferred because it's supported 
by GCC too. Except of that I see no reasons how -include is better than 
-include-pch.

Original comment by vsap...@gmail.com on 22 Apr 2014 at 4:17

[GoogleCodeExporter]

I'm using the official version sorry, 
http://include-what-you-use.com/downloads/include-what-you-use-3.4.src.tar.gz

I'm behind a firewall and I can't use svn right now :( I will get it later when 
I get home.

Original comment by dpun...@gmail.com on 22 Apr 2014 at 5:14