Allow build outside of clang source tree

[GoogleCodeExporter]

Hi

I would like to package your work into Debian. However, the fact that 
include-what-you-use needs the clang source tree greatly complexifies the build 
process.
It would be nice if you could update your work to build against clang headers 
and libs.
Here is a list of headers provided by the libclang-dev package in Debian and 
Ubuntu:
http://packages.debian.org/sid/amd64/libclang-dev/filelist

Original issue reported on code.google.com by sylvestre.ledru on 11 Dec 2011 at 4:50

[GoogleCodeExporter]

That would be great to get this into debian!  I'll help how I can.  
Unfortunately, the build system for clang is a bit of a black box for me.

I do have a version of iwyu that is built against llvm-3.0.  Assuming your 
libclang-dev is for llvm-3.0, you should be able to use that:
   http://code.google.com/p/include-what-you-use/downloads/list

I believe that if you unpacked that tarball, it would be a single compile 
command -- possibly a long one -- to build iwyu against the llvm/clang headers. 
 But I don't know if that's exactly what you're looking for.  I can come up 
with the clang command if that would help; or else let me know what would.

Original comment by csilv...@gmail.com on 12 Dec 2011 at 7:34

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

I've attached a basic wscript (waf) which will detect llvm/clang and build iwyu 
outside clang's source tree. Just grab the waf binary [1], put waf and wscript 
into the directory of iwyu and it should work. Requires llvm-config to be in 
the path.

[1] 
http://docs.waf.googlecode.com/git/book_16/single.html#_obtaining_the_waf_file

Original comment by peter.wa...@gmail.com on 4 Mar 2012 at 7:11

Attachments:

• wscript

[GoogleCodeExporter]

I don't know anything about wscript or waf; I've never heard of it before.  I 
think clang is trying to limit the build systems to just make and cmake (and 
maybe just one of those :-), though which one depends on who you ask :-) ), so 
I'll refrain from applying this new file to the tree, but will keep it around 
in this issue report for those who might find it useful.

Original comment by csilv...@gmail.com on 6 Mar 2012 at 11:12

[GoogleCodeExporter]

As the Debian maintainer of clang, I would be interested to stick with cmake or 
autotools...

Original comment by sylvestre.ledru on 10 Mar 2012 at 4:08

[GoogleCodeExporter]

Issue #69 is related to this one.

Original comment by vsap...@gmail.com on 17 Jul 2012 at 8:40

[GoogleCodeExporter]

I took some information from issue #69 to patch the build of 
include-what-you-use. With the attached patch to Subversion revision 371 I am 
able to build it against the Debian libraries using CMake. Of course, the 
changes are only a hack. The paths should never be hard-coded into the 
CMakeLists.txt, but a CMake find module should be used instead.

System: Debian GNU/Linux sid amd64
Build tool: CMake 2.8.9
Compiler: GCC 4.7.1
Libraries: llvm-3.1-dev (3.1-2), libclang-dev (3.1-8)

Original comment by googlec...@eikel.org on 28 Aug 2012 at 3:44

Attachments:

• build-fix-Debian.patch

[GoogleCodeExporter]

Here is a patch which allows IWYU to be built exactly how it is now, but 
optionally adds support for out of tree builds. To build out of tree, just 
compile llvm & clang, then set the LLVM_LIB_PATH LLVM_INCLUDE_PATH 
CLANG_LIB_PATH and CLANG_INCLUDE_PATH variables when running cmake. For 
example, on my machine I did:

cmake -DLLVM_LIB_PATH=/Users/cberner/llvm_build/lib/ 
-DCLANG_LIB_PATH=/Users/cberner/llvm_build/tools/clang/lib/ 
-DCLANG_INCLUDE_PATH="/Users/cberner/llvm/tools/clang/include/;/Users/cberner/ll
vm_build/tools/clang/include" 
-DLLVM_INCLUDE_PATH="/Users/cberner/llvm_build/include;/Users/cberner/llvm/inclu
de" ../include-what-you-use-read-only/

Original comment by christop...@gmail.com on 21 Dec 2012 at 8:36

Attachments:

• cmake.patch

[GoogleCodeExporter]

Nice direction!

It seems that moving away from add_clang_executable() requires that we specify 
all link dependencies. At first I thought that was a shame, but after thinking 
about it, I like that it keeps the out-of-tree and in-tree builds in sync -- if 
we fix a problem with one, the other works too.

A few comments:

+  pthread
+  dl

This will break MSVC, there are no pthread or dl libraries there. I don't know 
if there are other libraries required in the MSVC case, but I don't think so -- 
threading and dynamic linking are all part of the Windows base libraries.

+IF (DEFINED LLVM_INCLUDE_PATH)
+  include_directories("${LLVM_INCLUDE_PATH}")
+ENDIF(DEFINED LLVM_INCLUDE_PATH)

All other if statements are lower-case, please downcase. If HEAD of this file 
diverges from Clang/LLVM at large in this respect, maybe we should first make 
it case consistent, and then apply your changes.

+IF (DEFINED CLANG_INCLUDE_PATH)
+  include_directories("${CLANG_INCLUDE_PATH}")
+ENDIF(DEFINED CLANG_INCLUDE_PATH)

Should be lower-case.

+add_definitions(
+  -D__STDC_LIMIT_MACROS
+  -D__STDC_CONSTANT_MACROS
+  -fno-rtti

Again, not sure how this plays with MSVC. The defines are probably fine, but I 
don't know what problem they're solving, and if that problem needs another 
solution for MSVC...?

Thanks,
- Kim

Original comment by kim.gras...@gmail.com on 22 Dec 2012 at 9:58

[GoogleCodeExporter]

Ok, I addressed the formatting issues and moved pthread and dl to be 
conditional.

Those defines are needed for llvm to compile, although I'm honestly not sure 
why. Without them you get this error though: #error "Must #define 
__STDC_LIMIT_MACROS before #including Support/DataTypes.h"

Let me know if there's anything else you'd like to see addressed.

Original comment by christop...@gmail.com on 27 Dec 2012 at 9:07

Attachments:

• cmake.patch

[GoogleCodeExporter]

Thanks, Christopher. Can you make similar changes for Makefile?

Original comment by vsap...@gmail.com on 30 Dec 2012 at 9:56

[GoogleCodeExporter]

Thanks for the updated patch!

+  -L"${LLVM_LIB_PATH}"
+  -L"${CLANG_LIB_PATH}"

-L doesn't port, either. Can you use CMake's link_directories like you did with 
the include paths?

+add_definitions(
+  -D__STDC_LIMIT_MACROS
+  -D__STDC_CONSTANT_MACROS

These two shouldn't mean any harm...

+  -fno-rtti
+)

... but this is not recognized by MSVC. Also, I've read somewhere that generic 
compiler switches should not be added with add_definitions, but rather appended 
to CXX_FLAGS.

MSVC's spelling of -fno-rtti is /GR- so you should be able to do;

 if( MSVC )
   SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /GR-")
 else( MSVC )
   SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-rtti")
 endif( MSVC )

I'd be happy to test for MSVC with these changes applied.

Original comment by kim.gras...@gmail.com on 1 Jan 2013 at 1:37

[GoogleCodeExporter]

Ok, here's take #3. Let me know if this works for you with MSVC :)

Original comment by christop...@gmail.com on 2 Jan 2013 at 10:29

Attachments:

• cmake.patch

[GoogleCodeExporter]

I've tested this with MSVC and it generates correct projects.

Everything builds and the tests run as expected (i.e. same number of failures 
as before the patch :-))

The only difference I noticed was that the include-what-you-use project ended 
up in a different place in the VS solution tree -- add_clang_executable added 
IWYU to the "Clang executables" folder.

Attached is a modified patch to include this folder setting. With that, looks 
good to me, thanks!

Original comment by kim.gras...@gmail.com on 3 Jan 2013 at 9:29

Attachments:

• cmake-folder.patch

[GoogleCodeExporter]

The Makefile changes would be nice to get in there as well, but I don't know my 
way around Make enough to know where to start.

Unless I'm missing something, it should be as simple as modifying INCLUDE and 
LIB to include the out-of-tree headers and libraries, similar to how 
LLVM_*_PATH and CLANG_*_PATH are passed to CMake.

Maybe documenting that fact would be good enough?

Original comment by kim.gras...@gmail.com on 3 Jan 2013 at 9:44

[GoogleCodeExporter]

Awesome! I took a look at the make file, but I'm not too familiar with make 
syntax either and it wasn't obvious how to get out of tree builds working with 
it. I did try the CLANG_LEVEL variable, but that didn't work for me.

Original comment by christop...@gmail.com on 3 Jan 2013 at 7:37

[GoogleCodeExporter]

As far as I can tell, INCLUDE and LIB are MSVC concepts, for some reason I 
thought it was a universal convention.

My current theory is that adding include paths to CXXFLAGS and library paths to 
LDFLAGS should work. It won't work with a CMake-generated Makefile (see e.g. 
issue 87), but it might work with an autoconf-generated system.

Further, I'm sure we could make our current Makefile accept environment 
vars/commandline args for LLVM_*_PATH and CLANG_*_PATH and transform them into 
CXXFLAGS and LDFLAGS, but I'm not sure which is more idiomatic.

Original comment by kim.gras...@gmail.com on 3 Jan 2013 at 10:36

[GoogleCodeExporter]

I looked into parameterising the Makefile, and it wasn't as obvious. I don't 
know my way around Make very well, but I think what needs to be done is 
dual-mode Makefile that can either

- take no arguments and build in-tree in its current form
- take include/lib arguments and use the command-line from issue 69 with the 
respective -I and -L flags added (or a more Make-friendly one using CXX, etc).

If someone has the Make chops to put in the big if-statement, well, patches are 
welcome :-)

I don't know what an installed LLVM/Clang tree looks like, but hopefully we 
could avoid having to specify both LLVM and Clang include and library paths.

Original comment by kim.gras...@gmail.com on 9 Jan 2013 at 9:26

[GoogleCodeExporter]

OK, let's not update Makefile build now, we'll do that later.

I've noticed that in Clang binaries for Ubuntu-12.04/x86 libclangSema depends 
on libLLVMMCParser and in CMakeLists.txt you don't link against LLVMMCParser. I 
need to recheck how build behaves with applied patch on Mac OS X.

Original comment by vsap...@gmail.com on 14 Jan 2013 at 9:31

[GoogleCodeExporter]

I don't understand why, but generating a Visual Studio solution from the 
CMakeLists.txt in the path produces a result that links against LLVMMCParser 
among other libraries not mentioned:

--
kernel32.lib
user32.lib
gdi32.lib
winspool.lib
shell32.lib
ole32.lib
oleaut32.lib
uuid.lib
comdlg32.lib
advapi32.lib
..\..\..\..\lib\Debug\clangFrontend.lib
..\..\..\..\lib\Debug\clangSerialization.lib
..\..\..\..\lib\Debug\clangDriver.lib
..\..\..\..\lib\Debug\clangParse.lib
..\..\..\..\lib\Debug\clangSema.lib
..\..\..\..\lib\Debug\clangAnalysis.lib
..\..\..\..\lib\Debug\clangAST.lib
..\..\..\..\lib\Debug\clangLex.lib
..\..\..\..\lib\Debug\clangBasic.lib
..\..\..\..\lib\Debug\clangEdit.lib
..\..\..\..\lib\Debug\LLVMipo.lib
..\..\..\..\lib\Debug\LLVMScalarOpts.lib
..\..\..\..\lib\Debug\LLVMInstCombine.lib
..\..\..\..\lib\Debug\LLVMTransformUtils.lib
..\..\..\..\lib\Debug\LLVMipa.lib
..\..\..\..\lib\Debug\LLVMAnalysis.lib
..\..\..\..\lib\Debug\LLVMTarget.lib
..\..\..\..\lib\Debug\LLVMMC.lib
..\..\..\..\lib\Debug\LLVMCore.lib
..\..\..\..\lib\Debug\LLVMSupport.lib
shlwapi.lib
..\..\..\..\lib\Debug\LLVMX86CodeGen.lib
..\..\..\..\lib\Debug\LLVMX86AsmParser.lib
..\..\..\..\lib\Debug\LLVMX86Disassembler.lib
..\..\..\..\lib\Debug\LLVMAsmParser.lib
..\..\..\..\lib\Debug\LLVMAsmPrinter.lib
..\..\..\..\lib\Debug\LLVMSelectionDAG.lib
..\..\..\..\lib\Debug\LLVMX86Desc.lib
..\..\..\..\lib\Debug\LLVMMCParser.lib
..\..\..\..\lib\Debug\LLVMCodeGen.lib
..\..\..\..\lib\Debug\LLVMX86AsmPrinter.lib
..\..\..\..\lib\Debug\LLVMX86Info.lib
..\..\..\..\lib\Debug\LLVMX86Utils.lib
..\..\..\..\lib\Debug\LLVMVectorize.lib
..\..\..\..\lib\Debug\LLVMObject.lib
--

It looks like four segments;

1) Windows libraries that most apps use, not mentioned in CMakeLists.txt
2) Clang and LLVM libraries explicitly listed in CMakeLists.txt
3) shlwapi.lib, special case for Win32 from CMakeLists.txt
4) LLVM libraries not mentioned in CMakeLists.txt

I should get a book on CMake to understand how this works.

Original comment by kim.gras...@gmail.com on 14 Jan 2013 at 9:34

[GoogleCodeExporter]

I've tried to use updated CMakeLists.txt and encountered the following problems:
  - when build in-tree on Mac OS X following warnings are shown: "__STDC_LIMIT_MACROS" redefined, "__STDC_CONSTANT_MACROS" redefined
  - when build with prebuilt binaries on Mac OS X, linking fails without LLVMMCParser. After adding LLVMMCParser build succeeds, but a bunch of warnings are shown. Warnings are like

  ld: warning: direct access in clang::ASTUnit::mapLocationFromPreamble(clang::SourceLocation)   to global weak symbol llvm::IntrusiveRefCntPtr<clang::SourceManager>::getPtr() const   means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

So I don't think it's IWYU problem.

  - when build with prebuilt binaries on Ubuntu, linking fails without LLVMMCParser, tests fail.

Also I have a question, why do we need both LLVM and Clang INCLUDE_PATH, 
LIB_PATH?

Original comment by vsap...@gmail.com on 20 Jan 2013 at 5:30

[GoogleCodeExporter]

It sounds like LLVMMCParser should always be among the target_link_libraries. 
Volodymyr, deos it work if you change that locally?

I can also see that the projects I generate in-tree have __STDC_LIMIT_MACROS 
and __STDC_CONSTANT_MACROS added twice, so that should probably be conditioned 
on CLANG_*_PATH or LLVM_*_PATH.

When you say "tests fail", do you mean run_iwyu_tests.py fails? Does it only 
fail if you build IWYU with prebuilt binaries or is it consistent across 
configurations?

As for whether both are necessary, that's a good question... On my Windows 
machine, all headers and libraries are in the same tree.

Christopher -- which platform did you test your original patch on?

Original comment by kim.gras...@gmail.com on 2 Feb 2013 at 3:09

[GoogleCodeExporter]

I tested it on Linux and OSX

Original comment by christop...@gmail.com on 3 Feb 2013 at 1:48

[GoogleCodeExporter]

With which compiler/linker versions?

Original comment by kim.gras...@gmail.com on 3 Feb 2013 at 9:03

[GoogleCodeExporter]

When I add LLVMMCParser, build succeeds.

run_iwyu_tests.py fails only on Linux (actually it fails on Mac OS X too, but 
it is covered by issue #80). On Linux I've tried to build IWYU only with 
prebuilt binaries. On Mac OS X all build methods work with added LLVMMCParser.

Original comment by vsap...@gmail.com on 3 Feb 2013 at 10:21

[GoogleCodeExporter]

Updated patch:

- Adds LLVMMCParser to the list of target_link_libraries
- New macro idempotent_add_definitions, which only adds definitions if they do 
not already exist
- Use idempotent_add_definitions for _D__STDC_LIMIT_MACROS and 
__STDC_CONSTANT_MACROS
- Remove names from else() and end*() blocks

This appears to make no difference on my Windows/MSVC setup (except I can see 
by inspection that the __STDC_* macros are only added once), could you guys 
check it on your respective platforms?

I did some small amount of research, and it seems like separate Clang and LLVM 
paths are useful: the Debian packages in the original issue report has them in 
different roots.

Unless we can tie the test failure to this change, I think we should go ahead 
with it, so we have at least one documentable way of building against 
Clang/LLVM prebuilts.

Original comment by kim.gras...@gmail.com on 4 Feb 2013 at 9:07

Attachments:

• cmake-prebuilt.patch

[GoogleCodeExporter]

The last patch doesn't add __STDC_LIMIT_MACROS and __STDC_CONSTANT_MACROS when 
I built with prebuilt binaries. And build fails with
In file included from 
llvm-prebuilt/clang+llvm-3.2-x86_64-apple-darwin11/include/llvm/Support/type_tra
its.h:20,
                 from llvm-prebuilt/clang+llvm-3.2-x86_64-apple-darwin11/include/llvm/Support/Casting.h:18,
                 from llvm-prebuilt/include-what-you-use/iwyu_ast_util.h:20,
                 from llvm-prebuilt/include-what-you-use/iwyu.cc:109:
llvm-prebuilt/clang+llvm-3.2-x86_64-apple-darwin11/include/llvm/Support/DataType
s.h:49:3: error: #error "Must #define __STDC_LIMIT_MACROS before #including 
Support/DataTypes.h"
llvm-prebuilt/clang+llvm-3.2-x86_64-apple-darwin11/include/llvm/Support/DataType
s.h:53:3: error: #error "Must #define __STDC_CONSTANT_MACROS before " 
"#including Support/DataTypes.h"

Original comment by vsap...@gmail.com on 10 Feb 2013 at 7:04

[GoogleCodeExporter]

OK, this issue has been lurking at the bottom of my inbox for a long while.

Attached is my latest attempt at a patch to solve this (hopefully in time for 
the Clang 3.3 release):

- Only one input argument -DLLVM_PATH expected
- Chooses between in-tree and out-of-tree based on the same rule used by 
Clang's CMakeLists -- if the root CMake dir is the current source dir, it's 
out-of-tree.
- Adjusts MSVC warning levels for out-of-tree builds (this is done by LLVM's 
cmake config otherwise)
- Prints a message to signify if it's using in-tree/out-of-tree configuration

Now two ways to test:
- In-tree: as before, as detailed in the How to Build instructions
- Out-of-tree (on a Unix platform):
  - Install the llvm-*-dev and libclang-*-dev packages
  - $ mkdir iwyu-trunk && cd iwyu-trunk
  - $ svn co <iwyu svn> iwyu
  - $ mkdir build && cd build
  - $ cmake "Unix Makefiles" -DLLVM_PATH=/usr/lib/llvm-3.3 ../iwyu
  - $ make

I've tested the following configurations:
- MSVC in-tree
- MSVC out-of-tree
- Ubuntu out-of-tree

If someone could test in-tree on a Unix platform and both on a Mac I would be 
delighted.

Original comment by kim.gras...@gmail.com on 26 May 2013 at 7:25

Attachments:

• cmake-out-of-tree.patch

[GoogleCodeExporter]

On Mac OS X in-tree build fails at linking stage with

Linking CXX executable ../../../../bin/include-what-you-use
Undefined symbols for architecture x86_64:
  "typeinfo for clang::ASTConsumer", referenced from:
      typeinfo for include_what_you_use::IwyuAstConsumer in iwyu.cc.o
  "typeinfo for clang::PPCallbacks", referenced from:
      typeinfo for include_what_you_use::IwyuPreprocessorInfo in iwyu_preprocessor.cc.o
  "typeinfo for clang::CommentHandler", referenced from:
      typeinfo for include_what_you_use::IwyuPreprocessorInfo in iwyu_preprocessor.cc.o
  "typeinfo for clang::ASTFrontendAction", referenced from:
      typeinfo for include_what_you_use::IwyuAction in iwyu.cc.o
  "typeinfo for clang::NamedDecl", referenced from:
      typeinfo for include_what_you_use::internal::FakeNamedDecl in iwyu_output.cc.o
ld: symbol(s) not found for architecture x86_64

Will investigate what's wrong.

Original comment by vsap...@gmail.com on 2 Jun 2013 at 11:03

[GoogleCodeExporter]

Thanks! I've finally been able to set up a Linux+CMake environment, and I've 
been able to reproduce that error.

I suspect link order, now I just need to figure out how to convince CMake to 
make said link order what I want :-)

Original comment by kim.gras...@gmail.com on 4 Jun 2013 at 8:24

[GoogleCodeExporter]

Looks like -fno-rtti should be specified both in out-of-tree and in-tree 
configurations. I think it was specified in add_clang_executable, but with 
add_executable we need to specify it explicitly.

Original comment by vsap...@gmail.com on 6 Jun 2013 at 9:01

[GoogleCodeExporter]

Thank you for that! Indeed, RTTI should always be disabled, except for MSVC 
builds.

Updated patch attached:
- Disable RTTI for in-tree as well as out-of-tree, but never for MSVC
- Only set solution folder for MSVC

Let me know if how this looks, thanks!

Original comment by kim.gras...@gmail.com on 6 Jun 2013 at 8:04

Attachments:

• cmake-out-of-tree.patch

[GoogleCodeExporter]

With the last patch on Mac OS X both in-tree and out-of-tree builds succeed, 
tests run as expected. Though there are still linker warnings like those I've 
mentioned in post 20. Overall, patch looks good to me.

Original comment by vsap...@gmail.com on 8 Jun 2013 at 11:11

[GoogleCodeExporter]

The warnings are related to -fvisibility-hidden. Unfortunately, there's quite a 
lot of CMake infrastructure necessary to detect whether the target C++ compiler 
supports -fvisibility-hidden, and I couldn't come up with a consistent way to 
add it to the CXX_FLAGS without adding lots of boilerplate CMakee to IWYU's 
CMakeLists.txt.

Out-of-tree works on Ubuntu with GCC 4.6.3 without any visibility adjustments, 
so I'm inclined to keep it as-is.

I'll commit this tomorrow, thanks for the help!

Original comment by kim.gras...@gmail.com on 8 Jun 2013 at 9:58

[GoogleCodeExporter]

Committed r466.

Original comment by kim.gras...@gmail.com on 9 Jun 2013 at 6:54

[GoogleCodeExporter]

It works now perfectly. I have been able to make a Debian package for iwyu.

thanks

Original comment by sylvestr...@scilab-enterprises.com on 29 Aug 2013 at 5:24

[GoogleCodeExporter]

This bug can be closed!

Original comment by sylvestr...@scilab-enterprises.com on 29 Aug 2013 at 5:25

[GoogleCodeExporter]

Hooray!

Original comment by kim.gras...@gmail.com on 29 Aug 2013 at 6:22

• Changed state: Fixed