iwyu built against MacPorts LLVM-3.3 prelease can't find stdarg.h

[GoogleCodeExporter]

What steps will reproduce the problem?  Give the *exact* arguments passed
to include-what-you-use, and attach the input source file that gives the
problem (minimal test-cases are much appreciated!)
1. Run IWYU against the following file:
➜  iwyu  clear
➜  iwyu  cat ./cstdarg.cpp
#include <cstdarg>

int main(int argc, char* argv[]) {
    return 0;
}.

What is the expected output? What do you see instead?

I'd expect this file to compile without warnings or errors. Potentially, IWYU 
might tell me that cstdarg is not needed. However, what I actually get is a 
compilation error:

➜  iwyu  include-what-you-use ./cstdarg.cpp
In file included from ./cstdarg.cpp:1:
/usr/include/c++/4.2.1/cstdarg:51:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
         ^

/usr/include/c++/4.2.1/cstdarg should add these lines:

/usr/include/c++/4.2.1/cstdarg should remove these lines:
- #include <bits/c++config.h>  // lines 50-50

The full include-list for /usr/include/c++/4.2.1/cstdarg:

---

(cstdarg.cpp has correct #includes/fwd-decls)

What version of the product are you using? On what operating system?
I'm using r472 on OS X 10.8 built against MacPorts LLVM 3.3 prerelease:
➜  iwyu  port list clang-3.3 llvm-3.3
clang-3.3                      @3.3-r180025    lang/llvm-3.3
llvm-3.3                       @3.3-r180025    lang/llvm-3.3

Please provide any additional information below.

Original issue reported on code.google.com by andrew.c.morrow@gmail.com on 29 Jun 2013 at 6:18

[GoogleCodeExporter]

Can you compile cstdarg.cpp with clang-3.3? Looks like you are hitting 
http://clang.llvm.org/docs/FAQ.html#i-get-errors-about-some-headers-being-missin
g-stddef-h-stdarg-h

With current LLVM/Clang and IWYU I see the following output

/usr/include/c++/4.2.1/cstdarg should add these lines:
#include <cstddef>                      // for _GLIBCXX_BEGIN_NAMESPACE, etc

/usr/include/c++/4.2.1/cstdarg should remove these lines:
- #include <bits/c++config.h>  // lines 50-50

The full include-list for /usr/include/c++/4.2.1/cstdarg:
#include <stdarg.h>                     // for va_end, va_list
#include <cstddef>                      // for _GLIBCXX_BEGIN_NAMESPACE, etc
---

(cstdarg.cpp has correct #includes/fwd-decls)

Original comment by vsap...@gmail.com on 30 Jun 2013 at 6:31

[GoogleCodeExporter]

Volodymyr, I think IWYU is confused because your source file is named 
"cstdarg.cpp"; it considers "cstdarg" its associated header and tries to 
suggest improvements for it.

I think the built-in include search is different on MSVC/Windows, so I can't 
reproduce here.

Original comment by kim.gras...@gmail.com on 30 Jun 2013 at 7:05

[GoogleCodeExporter]

Since issue-99 had turned out to be resolved by rebuilding against the 
clang-3.3 stable, I tried this one again. It still has problems. I also renamed 
the file from cstdarg.cpp to blah.cpp to avoid the issue mentioned above. I 
still find that it errors out. However, I found that if I explicitly include 
the include path to the llvm-3.3 directory containing its stdarg.h, then all is 
well:

➜  cat ./blah.cpp
#include <cstdarg>

int main(int argc, char* argv[]) {
    va_list list;
    return 0;
}
➜  ./include-what-you-use ./blah.cpp
In file included from ./blah.cpp:1:
/usr/include/c++/4.2.1/cstdarg:51:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
         ^

blah.cpp should add these lines:

blah.cpp should remove these lines:
- #include <cstdarg>  // lines 1-1

The full include-list for blah.cpp:
---

Here it is with the additional include path:

➜  ./include-what-you-use ./blah.cpp 
-I/opt/local/libexec/llvm-3.3/lib/clang/3.3/include/

(blah.cpp has correct #includes/fwd-decls)

So this suggests to me that when building IWYU it isn't incorporating 'builtin' 
LLVM/clang header paths somehow.

Original comment by andrew.c.morrow@gmail.com on 1 Jul 2013 at 2:06

[GoogleCodeExporter]

Per the link posted earlier, Clang searches for its builtin headers relative to 
the binary. I think the same rule applies for IWYU.

This suggests you should move the include-what-you-use executable to the same 
directory as clang, or copy the builtin includes and libs from 
/opt/local/libexec/llvm-33/lib/clang/3.3 to just next to include-what-you-use.

I still can't repro here, neither on Windows nor Ubuntu, but I'm not sure what 
headers it finds and how. This is a bit of a problem with Clang, I think, that 
its header search algorithm depends on what platform it was built on.

Note that if you build IWYU, Clang and LLVM together, this never becomes a 
problem, because include-what-you-use will end up next to clang in the install 
tree.

I'd love to hear if you can get it to work by copying include and lib from 
clang/3.3, and then we can think about a more transparent solution.

Original comment by kim.gras...@gmail.com on 1 Jul 2013 at 7:02

[GoogleCodeExporter]

This applies to LLVM 3.5 in MacPorts as well. Doing a regular build with 
clang++-mp-3.5 shows no problem, so it's iwyu related.

Original comment by dsvens...@gmail.com on 12 Feb 2014 at 12:19

[GoogleCodeExporter]

For what it's worth, I was running into the same problem with 3.4 and when I 
copied include-what-you-use to the same directory as clang, the problem went 
away. So it appears that the issue is caused by the headers that are shipped 
with clang not being resolved correctly when include-what-you-use is located in 
a different directory.

Original comment by davejohansen on 24 Apr 2014 at 8:27

[GoogleCodeExporter]

I ran this by the Clang development list and it seems we need to distribute the 
builtin headers with include-what-you-use.

Discussion here:
http://clang-developers.42468.n3.nabble.com/Standalone-tools-td4039043.html

And this is actually mentioned in passing here:
http://clang.llvm.org/docs/LibTooling.html#builtin-includes

I'll try and update our packaging script to include Clang's builtin headers.

Original comment by kim.gras...@gmail.com on 28 Apr 2014 at 6:02

[GoogleCodeExporter]

Do you want to add Clang's builtin headers to the distributed archive or to 
bake headers into IWYU binary?

Original comment by vsap...@gmail.com on 29 Apr 2014 at 1:01

[GoogleCodeExporter]

Since we have more than one build system, I thought I'd add the builtin headers 
to the archive as a first step, but ideally I'd like to bake the headers in.

Original comment by kim.gras...@gmail.com on 29 Apr 2014 at 4:03

[GoogleCodeExporter]

OK, so I've experimented a little bit in my local Windows working copy.

It seems that building a directory tree like this works reasonably well:

include-what-you-use/
  LICENSE.TXT
  README

  bin/
    include-what-you-use.exe
    fix_includes.py

  lib/
    clang/
      3.5.0/
        include/
          <built-in headers>

I can copy this anywhere, add the include-what-you-use/bin root to my path in a 
clean shell, and it works.

Does this look at all FHS-friendly?

@sylvestre on CC: How does this match what you're doing for Debian?

I wonder if we should add the equivalent of a make install step...?

Original comment by kim.gras...@gmail.com on 3 May 2014 at 8:12

[GoogleCodeExporter]

Actually, I am not sure how useful is a temporary fix by adding headers to 
distributed archives.  At least people in this thread haven't complained about 
distributed IWYU binary, they've built IWYU manually.

I was thinking about creating a file containing baked in builtin headers with a 
Python script.  Reasons for that: I am not very good with build systems; there 
will be less differences between different build systems.

Original comment by vsap...@gmail.com on 5 May 2014 at 9:35

[GoogleCodeExporter]

They're building out-of-tree and so run into the same problem -- they can only 
get IWYU to work if they put it in the Clang bin root.

But you're right; solving this in packaging won't help users building 
out-of-tree.

Yeah, a Python script sounds good, we should be able to run it from both CMake 
and Make. The example from Olivier Goffart seemed pretty straightforward and 
should port to Python easily.

I can build a draft implementation and we'll see how far it goes.

Original comment by kim.gras...@gmail.com on 6 May 2014 at 9:01

[GoogleCodeExporter]

I'm working on packaging IWYU for Fedora/RHEL so it can be easily installed 
using the standard "yum install" process on those platforms. In that case, the 
header files are already in the right place because both clang and IWYU are 
installed in the system directories. So if an option is added to copy/package 
the headers as part of the install process, then can an option to disable that 
please be added as well?
Thanks,
Dave

Original comment by davejohansen on 6 May 2014 at 3:09

[GoogleCodeExporter]

Kim, that looks great. Under Debian, I have clang installed here:
/usr/lib/llvm-3.5/lib/clang/3.5.0/include 

Original comment by sylvestre.ledru on 6 May 2014 at 3:40

[GoogleCodeExporter]

I've worked on embedding the headers, but I can't get it to work properly.

After convincing CMake to run the Python script, I now get a compiler error 
from MSVC that the string literals are larger than 64K. Since some of the 
builtin headers are in the 3-500K range, this seems hard to get around.

I'll take a break from this and come back to it, but ideas are welcome!

Original comment by kim.gras...@gmail.com on 11 May 2014 at 7:12

[GoogleCodeExporter]

I just symlink the lib folder from my Clang install to a folder named lib where 
the iwyu binary would expect it and it finds the builtins that way.

Original comment by ja...@citusdata.com on 25 Sep 2014 at 4:01

[GoogleCodeExporter]

Is there any fix or work-around which does not require root? I don't have root 
access to my machine and I would like to use IWYU.

I tried copying the folders from /usr/lib64/clang/3.5.0/* to my_iywu_folder/ 
but without success.

Original comment by ch.schmi...@gmail.com on 4 Dec 2014 at 1:04

[GoogleCodeExporter]

The relative path distance between /usr/bin/clang and /usr/lib/clang/<ver>/ 
needs to be preserved, so if you have the IWYU executable at 
~/my_iwyu_folder/include-what-you-use the builtins need to be at 
~/my_iwyu_folder/../lib/clang/<ver>/.

Also, <ver> needs to match the version IWYU was built for, double-check with 
`include-what-you-use --version` which Clang version it's built against.

Original comment by kim.gras...@gmail.com on 4 Dec 2014 at 7:13

[GoogleCodeExporter]

I built iwyu 0.3 with clang 3.5 on Fedora 21 and this issue seems to be fixed.

Original comment by davejohansen on 4 Feb 2015 at 11:59

[GoogleCodeExporter]

We haven't done anything actively, so chances are you've installed iwyu into 
/usr/bin or built in-tree with Clang or had the builtin headers available in 
the right place some other way.

Original comment by kim.gras...@gmail.com on 7 Feb 2015 at 8:12

[GoogleCodeExporter]

I'm not sure but I believe that this is a change in llvm/clang, because
with IWYU 0.2/clang 3.4 I had to set up the relative paths to get the tests
to pass, but with 0.3/3.5 the tests pass without the relative paths.

Original comment by davejohansen on 20 Feb 2015 at 4:37