search

JuliaInterop / Cxx.jl

The Julia C++ Interface
Other
757 stars 108 forks source link

Can't build on OSX #4

Closed jtravs closed 9 years ago

jtravs commented 10 years ago

I tried to build following your mailing list instructions. However I get the following error during compilation of Julia itself:

In file included from /Users/john/Documents/code/downloaded/julia/deps/llvm-svn/tools/lldb/source/../include/lldb/Core/FormatNavigator.h:29:
/Users/john/Documents/code/downloaded/julia/deps/llvm-svn/tools/lldb/source/../include/lldb/Symbol/ClangASTContext.h:21:10: fatal error: 
      'llvm/ADT/OwningPtr.h' file not found
#include "llvm/ADT/OwningPtr.h"
         ^
1 error generated.

My Make.user:

USE_SYSTEM_LIBM=1
BUILD_LLVM_CLANG=1
BUILD_LLDB=1

My deps/Versions.make

USE_SYSTEM_LIBM=1
BUILD_LLVM_CLANG=1
BUILD_LLDB=1
...

I also merged the branches and applied the patch as stated on the mailing list.

In issue #3 a similar problem arose on Ubuntu, but I'm on OSX. Any ideas how I can solve this? How do I try "the master branch of lldb instead of release_33"

Apologies for the stupid questions. I'm unreasonably excited about CXX.jl! Thanks!

ihnorton commented 10 years ago

How do I try "the master branch of lldb instead of release_33"

in /path/to/julia/:

echo "LLDB_VER = master" >> Make.user && make -C deps configure-llvm

or go to deps/llvm-svn/tools/lldb and git checkout master. Then run make install in llvm-svn/build_Release+Asserts.

acroy commented 10 years ago

I am on OSX 10.8.5 and got past the "llvm/ADT/OwningPtr.h" error, but now I am stuck with

llvm[8]: Compiling Host.cpp for Release+Asserts build
/Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/tools/lldb/source/Host/common/Host.cpp:371:68: error: use of undeclared identifier 'CPU_SUBTYPE_X86_64_H'
                if (cpusubtype == CPU_SUBTYPE_486 || cpusubtype == CPU_SUBTYPE_X86_64_H)
                                                                   ^
1 error generated.

To get there at all I had to use

USECLANG=1
USE_LIBCPP=1

(not sure about USECLANG though) and I had to change "-mmacosx-version-min=10.6" to "-mmacosx-version-min=10.7" in deps/llvm-svn/build_Release+Asserts/Makefile.config (I am sure there is a better way to do this). I hope this makes some sense.

Any ideas how to proceed are greatly appreciated!

jtravs commented 10 years ago

Thanks @ihnorton that worked. I now also have the same issue as @acroy.

ihnorton commented 10 years ago

Depending on when you pulled the first time, something may have changed upstream in llvm. You should fetch and rebase to origin/master in llvm-svn and llvm-svn/tools/clang too, and then reconfigure and compile llvm. I don't think it should be necessary to edit any generated makefiles.

acroy commented 10 years ago

I agree that one shouldn't edit the makefiles, but I couldn't figure out how to change "10.6" to "10.7" properly. Without that I ran into the problem described here.

Anyways, I will try to fetch and rebase ...

ihnorton commented 10 years ago

This may go without saying or not be applicable, but just in case: "transitioning" an existing build from 3.3 to llvm-svn is a bad idea. It will be much better to do a separate clean build with the Make.user settings specified by Keno in place at the start of the build.

ihnorton commented 10 years ago

Hmm, sorry, I take that back. @Keno did not specify building Julia against LLVM master in the original email. (however, I know Keno does most of his work on llvm master so this may be more fully tested with that configuration).

Keno commented 10 years ago

@acroy I've seen that issue, I think it's a bug in lldb.

Try this:

diff --git a/source/Host/common/Host.cpp b/source/Host/common/Host.cpp
index 4aa362e..1519ff5 100644
--- a/source/Host/common/Host.cpp
+++ b/source/Host/common/Host.cpp
@@ -313,6 +313,8 @@ Host::SystemLog (SystemLogType type, const char *format, ...
     va_end (args);
 }

+#define CPU_SUBTYPE_X86_64_H 8
+
 const ArchSpec &
 Host::GetArchitecture (SystemDefaultArchitecture arch_kind)
 {

in the lldb sources

acroy commented 10 years ago

Thanks @Keno and @ihnorton. I did a completely fresh start (just to be sure) and Keno's patch got me past the "CPU_SUBTYPE_X86_64_H" issue. Now I am getting

In file included from /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/tools/lldb/tools/debugserver/source/DNB.cpp:14:
In file included from /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/tools/lldb/tools/debugserver/source/DNB.h:17:
In file included from /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/tools/lldb/tools/debugserver/source/MacOSX/Genealogy.h:20:
In file included from /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/tools/lldb/tools/debugserver/source/MacOSX/MachThreadList.h:17:
In file included from /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/tools/lldb/tools/debugserver/source/MacOSX/MachThread.h:27:
/Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/tools/lldb/tools/debugserver/source/MacOSX/MachException.h:20:10: fatal error: 'DNBConfig.h' file not found
#include "DNBConfig.h"
         ^
1 error generated.
rm: /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/build_Release+Asserts/tools/lldb/tools/debugserver/source/Release+Asserts/DNB.d.tmp: No such file or directory
Keno commented 10 years ago

@acroy I've seen that one as well at times. I think there's a problem with make dependencies not being set up correctly. IIRC, the solution is to manually:

cd /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/build_Release+Asserts/tools/lldb/tools/debugserver/
make -C scripts
acroy commented 10 years ago

@Keno: You saved me once again! It goes almost through. I end up with

In file included from codegen.cpp:742:
./debuginfo.cpp:283:63: error: no matching function for call to 'createObjectFile'
            ErrorOr<llvm::object::ObjectFile*> origerrorobj = llvm::object::ObjectFile::createObjectFile(
                                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/acr/Development/Julia/julia-cxx/usr/include/llvm/Object/ObjectFile.h:346:32: note: candidate function not viable: requires single argument 'ObjectPath',
      but 3 arguments were provided
  static ErrorOr<ObjectFile *> createObjectFile(StringRef ObjectPath);
                               ^
/Users/acr/Development/Julia/julia-cxx/usr/include/llvm/Object/ObjectFile.h:348:3: note: candidate function not viable: requires 2 arguments, but 3 were provided
  createObjectFile(std::unique_ptr<MemoryBuffer> &Object,
  ^
/Users/acr/Development/Julia/julia-cxx/usr/include/llvm/Object/ObjectFile.h:351:3: note: candidate function not viable: requires single argument 'Object', but 3
      arguments were provided
  createObjectFile(std::unique_ptr<MemoryBuffer> &Object) {
  ^
Keno commented 10 years ago 
diff --git a/src/debuginfo.cpp b/src/debuginfo.cpp
index c79ba97..1886e59 100644
--- a/src/debuginfo.cpp
+++ b/src/debuginfo.cpp
@@ -279,17 +279,18 @@ void jl_getDylibFunctionInfo(const char **name, int *line, const char **filename
             // First find the uuid of the object file (we'll use this to make sure we find the
             // correct debug symbol file).
             uint8_t uuid[16], uuid2[16];
+
+            MemoryBuffer *membuf = MemoryBuffer::getMemBuffer(
+                StringRef((const char *)fbase, (size_t)(((uint64_t)-1)-fbase)),"",false);
+
 #ifdef LLVM35
+            std::unique_ptr<MemoryBuffer> buf(membuf);
             ErrorOr<llvm::object::ObjectFile*> origerrorobj = llvm::object::ObjectFile::createObjectFile(
+                buf, sys::fs::file_magic::unknown);
 #else
             llvm::object::ObjectFile *origerrorobj = llvm::object::ObjectFile::createObjectFile(
+                membuf);
 #endif
-                    MemoryBuffer::getMemBuffer(
-                    StringRef((const char *)fbase, (size_t)(((uint64_t)-1)-fbase)),"",false)
-#ifdef LLVM35
-                    ,false, sys::fs::file_magic::unknown
-#endif
-            );
             if (!origerrorobj) {
                 objfileentry_t entry = {obj,context,slide};
                 objfilemap[fbase] = entry;

I'll commit this on master soon

acroy commented 10 years ago

Thanks again! Now Julia seems to build fine, but building CXX.jl fails (see next comment).

In case someone else encounters the same problems, this is what I did to get to this point:

-static LLVMContext &jl_LLVMContext = getGlobalContext(); 
+DLLEXPORT LLVMContext &jl_LLVMContext = getGlobalContext(); 
 static IRBuilder<> builder(getGlobalContext()); 
 static bool nested_compile=false; 
-static ExecutionEngine *jl_ExecutionEngine; 
+DLLEXPORT ExecutionEngine *jl_ExecutionEngine; 
LLVM_VER = svn
LLDB_VER = master

BUILD_LLVM_CLANG=1 
BUILD_LLDB=1
LLVM_ASSERTIONS=1

USECLANG=1
USE_LIBCPP=1
cd deps/llvm-svn/build_Release+Asserts/tools/lldb/tools/debugserver/
make -C scripts

This got a bit longer than anticipated, but maybe it is helpful for someone.

acroy commented 10 years ago

Ok, back to the building error. After removing -lclangRewriteCore I get

Tuning for julia installation at: /Users/acr/Development/Julia/julia-cxx/usr/bin
    CC /Users/acr/Development/Julia/Cxx.jl/deps/build/bootstrap.o
    LINK /Users/acr/Development/Julia/Cxx.jl/deps/usr/lib/libcxxffi.dylib
Undefined symbols for architecture x86_64:
  "_PyArg_Parse", referenced from:
      lldb_private::ScriptInterpreterPython::ExecuteOneLineWithReturn(char const*, lldb_private::ScriptInterpreter::ScriptReturnType, void*, lldb_private::ScriptInterpreter::ExecuteScriptOptions const&) in liblldbInterpreter.a(ScriptInterpreterPython.o)
  "_PyArg_ParseTuple", referenced from:
      __wrap_new_SBAddress in liblldbInterpreter.a(LLDBWrapPython.o)
      __wrap_delete_SBAddress in liblldbInterpreter.a(LLDBWrapPython.o)
      __wrap_SBAddress_IsValid in liblldbInterpreter.a(LLDBWrapPython.o)
      __wrap_SBAddress_Clear in liblldbInterpreter.a(LLDBWrapPython.o)
      __wrap_SBAddress_GetFileAddress in liblldbInterpreter.a(LLDBWrapPython.o)
      __wrap_SBAddress_GetLoadAddress in liblldbInterpreter.a(LLDBWrapPython.o)
      __wrap_SBAddress_SetLoadAddress in liblldbInterpreter.a(LLDBWrapPython.o)
      ...
[/snip/]
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [usr/lib/libcxxffi.dylib] Error 1

Maybe there is some mismatch between the build flags used in Make.inc and BuildBootstrap.Makefile?

Keno commented 10 years ago 
diff --git a/deps/Makefile b/deps/Makefile
index 49f1661..c0b7fb4 100644
--- a/deps/Makefile
+++ b/deps/Makefile
@@ -211,8 +211,8 @@ endif
 endif

 LLVM_TARGET_FLAGS= --enable-targets=host
-LLVM_FLAGS += --disable-profiling --enable-shared --enable-static $(LLVM_TARGET_FLAGS) --disable-bindings --disable-docs
-LLVM_MFLAGS =
+LLVM_FLAGS += --disable-profiling --enable-shared --enable-static $(LLVM_TARGET_FLAGS) --disable-bindings --disable-docs CXXFLAGS="-DLLDB_DISABLE_PYTHON"
+LLVM_MFLAGS = CXXFLAGS="-DLLDB_DISABLE_PYTHON"
 ifeq ($(LLVM_ASSERTIONS), 1)
 LLVM_FLAGS += --enable-assertions
 ifeq ($(OS), WINNT)

LLDB seems to be built with python. Try the above.

Keno commented 10 years ago

On master, I also added a LLDB_DISABLE_PYTHON flag which you can put in your Make.user.

acroy commented 10 years ago

Unfortunately, the patch doesn't help, but maybe I did it wrong. Rebuilding llvm gives me:

[/snip/]
LLDBWrapPython.cpp:150:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^
1 error generated.
rm: /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/build_Release+Asserts/tools/lldb/source/Interpreter/Release+Asserts/LLDBWrapPython.d.tmp: No such file or directory
make[7]: *** [/Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/build_Release+Asserts/tools/lldb/source/Interpreter/Release+Asserts/LLDBWrapPython.o] Error 1
make[6]: *** [all] Error 1
make[5]: *** [all] Error 1
make[4]: *** [all] Error 1
make[3]: *** [all] Error 1
make[2]: *** [llvm-svn/build_Release+Asserts/Release+Asserts/lib/libLLVMJIT.a] Error 2
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2
Keno commented 10 years ago

That's odd. That file shouldn't really exist with those flags. How did you rebuild llvm? I would recommend throwing away the entire llvm-svn/build_Release+Asserts directory in case you have stale state. Thanks for being the guinea pig and trying this out BTW.

acroy commented 10 years ago

No problem. Like everyone else I am quite excited to try this out :-)

Regarding the rebuild: I just did make -C deps clean-llvm; make. Maybe this was not sufficient. I will get rid of the directory you mentioned and start the build again. Might take a while before I can report the result ...

Keno commented 10 years ago

The patch to Host.cpp has been committed upstream.

acroy commented 10 years ago

Nope, doesn't work. I started from scratch and as you say the patches to Host.cpp and debuginfo.cpp are already included. I set LLDB_DISABLE_PYTHON=1, but get the same error regarding "Python.h".

acroy commented 10 years ago

Ok, I got it to build. I suspect that tools/lldb/source/Interpreter/Makefile doesn't respect the flag LLDB_DISABLE_PYTHON (I removed a few lines and now it doesn't complain about python). Anyways, I had to replace -lclangRewriteCore by -lclangRewrite in your makefile, but now CXX.jl also builds:

julia> Pkg.build("CXX")
INFO: Building CXX
Tuning for julia installation at: /Users/acr/Development/Julia/julia-cxx/usr/bin
    CC /Users/acr/.julia/v0.3/Cxx/deps/build/bootstrap.o
    LINK /Users/acr/.julia/v0.3/Cxx/deps/usr/lib/libcxxffi.dylib
ld: warning: direct access in clang::CodeGen::CodeGenFunction::EmitObjCMessageExpr(clang::ObjCMessageExpr const*, clang::CodeGen::ReturnValueSlot) to global weak symbol clang::ValueDecl::getType() const means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
Not building debug library because corresponding julia DEBUG library does not exist.
To build, simply run the build again once the the library has been built.

Running a test doesn't work

julia> include("/Users/acr/.julia/v0.3/Cxx/test/llvmtest.jl")
ERROR: error compiling cxxsizeof: ccall: could not find function julia_type_to_llvm
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
while loading /Users/acr/.julia/v0.3/Cxx/test/llvmtest.jl, in expression starting on line 44

Any ideas?

ihnorton commented 10 years ago

@acroy see https://github.com/Keno/Cxx.jl/pull/7#issue-38891596 for a (temporary) solution.

acroy commented 10 years ago

@ihnorton : Thanks for the pointer. Where exactly did you add cjulia_type_to_llvm?

jakebolewski commented 10 years ago

cgutil.cpp

ihnorton commented 10 years ago

Right, just below the def'n of julia_type_to_llvm On Jul 28, 2014 12:33 PM, "Jake Bolewski" notifications@github.com wrote:

cgutil.cpp https://github.com/jakebolewski/julia/blob/master/src/cgutils.cpp#L397

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/4#issuecomment-50362714.

acroy commented 10 years ago

Sorry for my ignorance, but what then? make clean; make and running the test again doesn't work.

ihnorton commented 10 years ago

You need to change the corresponding call in Cxx.jl On Jul 28, 2014 1:24 PM, "acroy" notifications@github.com wrote:

Sorry for my ignorance, but what then? make clean; make and running the test again doesn't work.

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/4#issuecomment-50369442.

acroy commented 10 years ago

I tried that as well :-) But

julia> ccall(:cjulia_type_to_llvm,Ptr{Void},(Any,),1)
ERROR: ccall: could not find function cjulia_type_to_llvm

Actually, it seems that cgutils.cpp isn't compiled at all ...

Keno commented 10 years ago

It's #include'd in codegen.cpp.

On Mon, Jul 28, 2014 at 1:58 PM, acroy notifications@github.com wrote:

I tried that as well :-) But

julia> ccall(:cjulia_type_to_llvm,Ptr{Void},(Any,),1) ERROR: ccall: could not find function cjulia_type_to_llvm

Actually, it seems that cgutils.cpp isn't compiled at all ...

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/4#issuecomment-50374005.

ihnorton commented 10 years ago

Try adding cjulia_* to julia.expmap On Jul 28, 2014 2:00 PM, "Keno Fischer" notifications@github.com wrote:

It's #include'd in codegen.cpp.

On Mon, Jul 28, 2014 at 1:58 PM, acroy notifications@github.com wrote:

I tried that as well :-) But

julia> ccall(:cjulia_type_to_llvm,Ptr{Void},(Any,),1) ERROR: ccall: could not find function cjulia_type_to_llvm

Actually, it seems that cgutils.cpp isn't compiled at all ...

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/4#issuecomment-50374005.

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/4#issuecomment-50374345.

acroy commented 10 years ago

No, doesn't help. In which lib does it end up?

Keno commented 10 years ago

should be in libjulia.

acroy commented 10 years ago

Mhmm, it is in libjulia but ccall refuses to find it.

Keno commented 10 years ago

Try the following diff:

index d9ed432..8f0a81b 100644
--- a/src/cgutils.cpp
+++ b/src/cgutils.cpp
@@ -394,7 +394,8 @@ static Type *julia_struct_to_llvm(jl_value_t *jt);

 static bool jltupleisbits(jl_value_t *jt, bool allow_unsized = true);

-static Type *julia_type_to_llvm(jl_value_t *jt)
+extern "C" {
+DLLEXPORT Type *julia_type_to_llvm(jl_value_t *jt)
 {
     if (jt == (jl_value_t*)jl_bool_type) return T_int1;
     if (jt == (jl_value_t*)jl_bottom_type) return T_void;
@@ -478,6 +479,7 @@ static Type *julia_type_to_llvm(jl_value_t *jt)
     }
     return jl_pvalue_llvmt;
 }
+}

 static Type *julia_struct_to_llvm(jl_value_t *jt)
 {

(no need for cjulia then, just Cxx.jl as is). This definitely worked for me on OS X.

acroy commented 10 years ago

Bingo! Thanks a ton. Now the function is found, but your test kicks me out:

julia> include("/Users/acr/.julia/v0.3/Cxx/test/llvmtest.jl")
Assertion failed: (Val && "isa<> used on a null pointer"), function doit, file /Users/acr/Development/Julia/julia-cxx/deps/llvm-svn/include/llvm/Support/Casting.h, line 95.

signal (6): Abort trap: 6
??? at ???:259608915
??? at ???:259600885
??? at ???:-1761896182
Abort trap: 6
Keno commented 10 years ago

Can you run it under lldb and get me a backtrace.

acroy commented 10 years ago

I hope I did it right. The backtrace is in this gist.

Keno commented 10 years ago

Unfortunately that backtrace doesn't make alll that much sense. You might have to rebuild llvm in debug mode to get a better backtrace (LLVM_DEBUG=1 in Make.user). Before you do that though, try just running the test from the command line.

acroy commented 10 years ago

Same error when I run it from the command line. I can try LLVM_DEBUG=1 but it will probably take while before I can look into it. Is there something else I could test?

Keno commented 10 years ago

Not that I can think of, sorry. This is usually the point where I give up and build the debug version ;).

acroy commented 10 years ago

Well, I always wanted to build llvm in debug mode :-) I will let you know once I got the chance to look into it. Thanks again.

Keno commented 10 years ago

The good news is I can now reproduce this failure so, I should have a fix soon.

Keno commented 10 years ago

Segfault should be fixed in http://reviews.llvm.org/rL215686. There's one more issue I'll need to work around and I have a build fix pending, so maybe don't try just yet, but soon.

Edit: GitHub link: https://github.com/llvm-mirror/clang/commit/8444ae7cfeaefae031f8fedf0d1435ca3b14d90b

acroy commented 10 years ago

Exciting! I am looking forward to try this.

maxruby commented 9 years ago

I really would like to try CXX on MacOSX to wrap some C++ libraries and code. I tried to install Cxx.jl on Julia v0.3, OSX 10.9 but I get a fatal error:

This version of julia does not have staged functions.

This is what I did:

Added to ~/.julia/Make.user

LLVM_VER=svn
LLVM_ASSERTIONS=1
#LLVM_DEBUG=1
BUILD_LLVM_CLANG=1
BUILD_LLDB=1
LLDB_VER = master
USE_LLVM_SHLIB=1
LLDB_DISABLE_PYTHON=1

In the REPL I typed,

Pkg.clone("https://github.com/Keno/Cxx.jl")  
Pkg.build("CXX")

I looked for the staged functions, but I am not sure where/how I should install them.
How to proceed? Is CXX working well enough that it is worth trying now?

StefanKarpinski commented 9 years ago

You need a development version of Julia – v0.3 doesn't have staged functions.

maxruby commented 9 years ago

Thanks! I will try my currently dormant fork of v0.4 . . .

jtravs commented 9 years ago

Still can't build this. On julia commit 7dd805feb0e18d08a8159d4b49415af685cf62dc

With my Make.user:

USE_SYSTEM_LIBM=1
LLDB_VER=master
LLVM_VER=svn
LLVM_ASSERTIONS=1
#LLVM_DEBUG=1
BUILD_LLVM_CLANG=1
BUILD_LLDB=1
USE_LLVM_SHLIB=1
LLDB_DISABLE_PYTHON=1

I get this error:

rl: (22) The requested URL returned error: 404 Not Found
make[2]: *** [lldb-svn.src.tar.gz] Error 22
Keno commented 9 years ago

@jtravs That error should be fixed by https://github.com/JuliaLang/julia/pull/9263