Embed libclang as a submodule

sethfowler commented 11 years ago

Every project I've seen that uses libclang ends up embedding it as a submodule. Using the system libclang makes life much harder for end users unless you want to avoid using any feature introduced by newer versions of libclang. Even then an embedded version of libclang is preferable since IIRC, on OS X the system libclang is stripped and you can't link against it.

sethfowler commented 11 years ago

There is actually another reason to do this, which is that I don't think we'll get documentation on Hackage unless we build cleanly on their servers, and it's doubtful they'll have LLVM 3.3 straight away.

ghorn commented 11 years ago

Are you proposing to build llvm and libclang within the cabal project?

sethfowler commented 11 years ago

Yup.

ghorn commented 11 years ago

Using LibClang and llvm haskell bindings, it's always been a pain for me to deal with the underlying libraries. If LibClang handles it for us there are many benifits:

distribute matching versions of llvm and LibClang for API compatibility
ensure LibClang works in ghci for everyone by using the shared library.
users don't have to build llvm if their system doesn't have it (llvm puts tons of stuff in /usr/local/)

I have 3 concerns:

Build times will be significantly slower (this is totally fine with me though)
I would want to make sure that this would not have any conflicts with system or user-installed llvm/clang, especially when using the haskell llvm bindings
Can you do this in a 100% portable way? Otherwise there should be a configure option to use a user-specified library

This would be huge, though

sethfowler commented 11 years ago

Those are some great points, particularly the one about ghci; I'll be paying attention to getting ghci working as I investigate this.

Regarding your concerns:

I'm hoping to avoid making build times too slow. We really only need to build libclang, not all of llvm, and since libclang only involves the clang frontend and doesn't do any code generation, it may be possible to build it independently of the outer llvm project. I still need to look into this, though; this really depends on exactly how the llvm devs have structured things.
There definitely won't be any conflict with the system libclang.
This won't be 100% portable - in particular, I do not plan to support Windows. AFAIK that's the same situation we have now, though. Regardless, I agree that we need an option to use a user-specified copy of libclang.

sethfowler commented 11 years ago

So I've done this, although things are still a bit rough. This is happening on the new branch llvm3.4 (since we're leaving behind what's available in LLVM 3.3). I've only tested on OS X so far, and there everything is working fine so far. I'm able to use LibClang in ghci!

I'm not closing this yet because I want to make sure things work alright on Linux (or for that matter on any computer other than mine), and there is some cleanup yet to be done.

sethfowler commented 11 years ago

@ghorn and @chetant - if you have a chance to test this branch out, please let me know how things go. (Remember to run git submodule update --init before trying to build.)

ghorn commented 11 years ago

Wow, great job. I'm really excited for this feature.

unfortunately:

ghorn@deb-kerman:~/hslibs/LibClang (llvm3.4)$ cabal build
Building llvm and clang...
llvm[0]: Constructing LLVMBuild project information.
llvm[0]: Installing include files
llvm[0]: Making install directory /home/ghorn/hslibs/LibClang/build/out/include/./llvm-c
llvm[0]: Making install directory /home/ghorn/hslibs/LibClang/build/out/include/./llvm-c/Transforms

........... a gazillion lines of llvm building ...............

llvm[4]: Linking Release+Asserts Shared Library libclang.so
llvm[4]: Building Release+Asserts Archive Library libclang.a
llvm[4]: Installing Release+Asserts Archive Library /home/ghorn/hslibs/LibClang/build/out/lib/libclang.a
/usr/bin/ld: /home/ghorn/hslibs/LibClang/build/Release+Asserts/lib/libclangFrontend.a(ASTUnit.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/home/ghorn/hslibs/LibClang/build/Release+Asserts/lib/libclangFrontend.a: could not read symbols: Bad value
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [/home/ghorn/hslibs/LibClang/build/Release+Asserts/lib/libclang.so] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: Leaving directory `/home/ghorn/hslibs/LibClang/build/tools/clang/tools/libclang'
make[3]: *** [install] Error 1
make[3]: Leaving directory `/home/ghorn/hslibs/LibClang/build/tools/clang/tools'
make[2]: *** [install] Error 1
make[2]: Leaving directory `/home/ghorn/hslibs/LibClang/build/tools/clang'
make[1]: *** [clang/.makeinstall] Error 2
make[1]: Leaving directory `/home/ghorn/hslibs/LibClang/build/tools'
make: *** [install] Error 1
Ensuring libclang is available as libclang_static...
Building LibClang-0.2.0...
Preprocessing library LibClang-0.2.0...
Warning: unable to find Foreign.GreenCard along the import path: 
  using built-in defaults instead.
[ 1 of 16] Compiling Clang.Internal.ClangApp ( src/Clang/Internal/ClangApp.hs, dist/build/Clang/Internal/ClangApp.o )
[ 2 of 16] Compiling Clang.Internal.FFI ( dist/build/Clang/Internal/FFI.hs, dist/build/Clang/Internal/FFI.o )
[ 3 of 16] Compiling Clang.TranslationUnit ( src/Clang/TranslationUnit.hs, dist/build/Clang/TranslationUnit.o )
[ 4 of 16] Compiling Clang.Traversal  ( src/Clang/Traversal.hs, dist/build/Clang/Traversal.o )
[ 5 of 16] Compiling Clang.Monad      ( src/Clang/Monad.hs, dist/build/Clang/Monad.o )
[ 6 of 16] Compiling Clang.String     ( src/Clang/String.hs, dist/build/Clang/String.o )
[ 7 of 16] Compiling Clang.Completion ( src/Clang/Completion.hs, dist/build/Clang/Completion.o )
[ 8 of 16] Compiling Clang.Diagnostic ( src/Clang/Diagnostic.hs, dist/build/Clang/Diagnostic.o )
[ 9 of 16] Compiling Clang.Source     ( src/Clang/Source.hs, dist/build/Clang/Source.o )
[10 of 16] Compiling Clang.Token      ( src/Clang/Token.hs, dist/build/Clang/Token.o )
[11 of 16] Compiling Clang.Type       ( src/Clang/Type.hs, dist/build/Clang/Type.o )
[12 of 16] Compiling Clang.File       ( src/Clang/File.hs, dist/build/Clang/File.o )
[13 of 16] Compiling Clang.Debug      ( src/Clang/Debug.hs, dist/build/Clang/Debug.o )
[14 of 16] Compiling Clang.Cursor     ( src/Clang/Cursor.hs, dist/build/Clang/Cursor.o )
[15 of 16] Compiling Clang.CrossReference ( src/Clang/CrossReference.hs, dist/build/Clang/CrossReference.o )
[16 of 16] Compiling Clang            ( src/Clang.hs, dist/build/Clang.o )
[ 1 of 16] Compiling Clang.Internal.ClangApp ( src/Clang/Internal/ClangApp.hs, dist/build/Clang/Internal/ClangApp.p_o )
[ 2 of 16] Compiling Clang.Internal.FFI ( dist/build/Clang/Internal/FFI.hs, dist/build/Clang/Internal/FFI.p_o )
[ 3 of 16] Compiling Clang.TranslationUnit ( src/Clang/TranslationUnit.hs, dist/build/Clang/TranslationUnit.p_o )
[ 4 of 16] Compiling Clang.Traversal  ( src/Clang/Traversal.hs, dist/build/Clang/Traversal.p_o )
[ 5 of 16] Compiling Clang.Monad      ( src/Clang/Monad.hs, dist/build/Clang/Monad.p_o )
[ 6 of 16] Compiling Clang.String     ( src/Clang/String.hs, dist/build/Clang/String.p_o )
[ 7 of 16] Compiling Clang.Completion ( src/Clang/Completion.hs, dist/build/Clang/Completion.p_o )
[ 8 of 16] Compiling Clang.Diagnostic ( src/Clang/Diagnostic.hs, dist/build/Clang/Diagnostic.p_o )
[ 9 of 16] Compiling Clang.Source     ( src/Clang/Source.hs, dist/build/Clang/Source.p_o )
[10 of 16] Compiling Clang.Token      ( src/Clang/Token.hs, dist/build/Clang/Token.p_o )
[11 of 16] Compiling Clang.Type       ( src/Clang/Type.hs, dist/build/Clang/Type.p_o )
[12 of 16] Compiling Clang.File       ( src/Clang/File.hs, dist/build/Clang/File.p_o )
[13 of 16] Compiling Clang.Debug      ( src/Clang/Debug.hs, dist/build/Clang/Debug.p_o )
[14 of 16] Compiling Clang.Cursor     ( src/Clang/Cursor.hs, dist/build/Clang/Cursor.p_o )
[15 of 16] Compiling Clang.CrossReference ( src/Clang/CrossReference.hs, dist/build/Clang/CrossReference.p_o )
[16 of 16] Compiling Clang            ( src/Clang.hs, dist/build/Clang.p_o )

src/Clang/Internal/FFI_stub_ffi.c:3:27:
     fatal error: clang-c/Index.h: No such file or directory
compilation terminated.

sethfowler commented 11 years ago

Thanks for trying the branch out!

Looks like this is the problem:

/usr/bin/ld: /home/ghorn/hslibs/LibClang/build/Release+Asserts/lib/libclangFrontend.a(ASTUnit.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC

Oddly, it seems like it's trying to build a shared library even though I disabled that in the configure flags. I'll look into this. It's probably not a hard thing to fix.

sethfowler commented 11 years ago

Now with Travis CI set up (thanks Greg and Chetan!) it's much easier to test things. And it immediately became obvious that I left some hard-coded paths and some unneeded code in Setup.hs. I've pushed some fixes.

sethfowler commented 11 years ago

And we're green on Travis CI! Nice!

Though there are lots of ways in which the build process can be further improved, I think we should handle them in other issues. I consider the initial embedding of libclang a success! Closing.

chetant / LibClang

Embed libclang as a submodule #25