search

JuliaInterop / Cxx.jl

The Julia C++ Interface
Other
757 stars 108 forks source link

Using CXX #21

Closed wbhart closed 9 years ago

wbhart commented 9 years ago

Sorry to open issues in order to ask questions, but I'm not sure where I should be directing questions.

In the examples I can find for using CXX people seem to be doing 

using CXX

However, when I do this it doesn't seem to find a module called CXX.

I notice when the build was done it did not build a debug build because there was not a corresponding Julia debug build. Is this what I'm missing?

Keno commented 9 years ago

It's not really modulified yet, so even though you write using, it doesn't load a module. The functionality will still be available. You can have a look at the tests.

wbhart commented 9 years ago

Hmm, at the moment the tests don't run for me. Here is what I get when typing using CXX

julia> using CXX ERROR: CXX not found in require at ./loading.jl:49 in require3953 at /home/wbhart/julia/usr/bin/../lib/julia/sys.so

Keno commented 9 years ago

Probably case sensitivity. Try using Cxx.

wbhart commented 9 years ago

Same problem sorry. Also tried cxx.

Keno commented 9 years ago

I see. When I do that, I get:

julia> using Cxx
Warning: requiring "Cxx" did not define a corresponding module.

Can you use other packages as usual? Try restarting your julia session if you haven't.

wbhart commented 9 years ago

No, restarting Julia didn't fix it.

I think it only does something if I am currently in the directory containing Cxx.jl.

But now I get the following message:

julia> using Cxx ERROR: error compiling init: could not load module libcxxffi: libcxxffi: cannot open shared object file: No such file or directory in include at ./boot.jl:242 in include_from_node1 at ./loading.jl:128 in reload_path at ./loading.jl:152 in _require at ./loading.jl:67 in require at ./loading.jl:52 in require3953 at /home/wbhart/julia/usr/bin/../lib/julia/sys.so while loading /home/wbhart/.julia/v0.4/CXX/src/Cxx.jl, in expression starting on line 37

Are there some dependencies I'm missing?

Bill.

Keno commented 9 years ago

Try this patch: https://github.com/JuliaLang/julia/pull/7940

wbhart commented 9 years ago

Same problem I'm afraid. I even did make clean and rebuilt Julia, but it doesn't seem to fix that issue.

wbhart commented 9 years ago

I checked what happens at that point and error is -1 but the string does contain "No such file", so those lines are not run.

Keno commented 9 years ago

Can you redo Pkg.build("Cxx"). Additionally perhaps lld on the libcxxffi shared library.

wbhart commented 9 years ago

I tried Pkg.build("Cxx"). Same message.

julia> Pkg.build("CXX")
INFO: Building CXX
Tuning for julia installation at: /home/wbhart/julia/usr/bin
Not building debug library because corresponding julia DEBUG library does not exist.
To build, simply run the build again once the library at
/home/wbhart/julia/usr/bin/../../usr/lib/libjulia-debug.so
has been built.

julia> using Cxx
ERROR: error compiling init: could not load module libcxxffi: libcxxffi: cannot open shared object file: No such file or directory
 in include at ./boot.jl:242
 in include_from_node1 at ./loading.jl:128
 in reload_path at ./loading.jl:152
 in _require at ./loading.jl:67
 in require at ./loading.jl:52
 in require3947 at /home/wbhart/julia/usr/bin/../lib/julia/sys.so
while loading /home/wbhart/julia/Cxx.jl, in expression starting on line 37

Here is the output of ldd

wbhart@hilbert:~/julia$ ldd usr/lib/libcxxffi.so
        linux-vdso.so.1 =>  (0x00007fff331ff000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fad6a09f000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fad69e97000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fad69c92000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fad6998e000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fad69778000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fad693b8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fad6caf4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fad6919b000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fad68e9f000)
Keno commented 9 years ago

Ah, that shows that you're not linking LLVM as a shared library. You need to have USE_LLVM_SHLIB in your Make.user, then rebuild julia, then delete the build artifacts in ~/.julia/Cxx/deps/

wbhart commented 9 years ago

I already have USE_LLVM_SHLIB=1 in my Make.user.

Keno commented 9 years ago

Hmm, for some reason that did not get picked up by libcxxffi.so

wbhart commented 9 years ago

Make.user should just be at the very top level of the Julia source tree, i.e. not in src/ ?

Keno commented 9 years ago

Yes. Can you try ldd on libjulia and see if that at least picked that up?

wbhart commented 9 years ago 
wbhart@hilbert:~/julia$ ldd usr/lib/libjulia.so
        linux-vdso.so.1 =>  (0x00007fffab5b7000)
        libLLVM-3.6svn.so => /home/wbhart/julia/usr/lib/libLLVM-3.6svn.so (0x00007fb901aee000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb9018db000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fb9016d2000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb9014b5000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb9011b1000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb900f9a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb900bdb000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb9009c4000)
        libedit.so.2 => /usr/lib/x86_64-linux-gnu/libedit.so.2 (0x00007fb90079c000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fb900575000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb900279000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb9039e9000)
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007fb90006c000)
Keno commented 9 years ago

Hmm, I don't know. You could try moving -ljulia after LDFLAGS in BuildBootstrap.Makefile. That might make a difference depending on your linker.

wbhart commented 9 years ago

I didn't realise libcxxffi.so is built by CXX. I thought it was part of LLVM.

So I deleted the usr and build directories in CXX and rebuilt. I also ran from the actually CXX/src directory, and this changes the error message:

WARNING: while loading libcxxffi: /home/wbhart/.julia/v0.4/CXX/src/../deps/usr/lib/libcxxffi.so: undefined symbol: _ZN12lldb_private15HostThreadPosix4JoinEPPv

followed by the usual message about not being able to find libcxxffi.so.

I think part of the problem is that it was building against the system Julia, which is an old julia-0.3, not against the julia instance I'm running. After changing JULIA_HOME, deleting usr and build and rebuilding CXX it now says "Tuning for julia installation at: /home/wbhart/julia/usr/share", which is the correct julia (though I don't know which exact directory JULIA_HOME is supposed to be set to).

I tried moving the -ljulia after LDFLAGS and adding -lpthread. Neither of these things changed anything.

Keno commented 9 years ago

It might actually just be that it's missing some LLDB library. I don't have access to our linux system at the moment, so I can't check, but once that's back up, I'll take a look.

wbhart commented 9 years ago

There's still no mention of LLVM in the ldd output anyway:

wbhart@hilbert:~/.julia/v0.4/CXX/deps/usr/lib$ ldd libcxxffi.so
        linux-vdso.so.1 =>  (0x00007fffe5f7b000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fd283f00000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd283cf8000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd283af3000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd2837ef000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd2835d9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd283219000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd286955000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd282ffc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd282d00000)
Keno commented 9 years ago

What julia prompt are you running Pkg.build from? It should automatically pick up the Make.user file of whatever julia instance is currently running.

wbhart commented 9 years ago

I'm starting in the directory ~/.julia/v0.4/CXX/src and I'm running ~/julia/julia instead of running the system julia. JULIA_HOME is currently set to /home/wbhart/julia/usr/lib.

wbhart commented 9 years ago

Actually what I said above must be incorrect. Either I accidentally ran the system julia once or I just misread what it said HOME was. When I unset JULIA_HOME and check what JULIA_HOME is from within julia, you are right, it gets the right one. And CXX seems to use that. So that wasn't a part of the problem.

wbhart commented 9 years ago

I took the PRINT_LINK out so that the full g++ invocation can be seen. I also removed the ifeq around the USE_LLVM_SHLIB test so that it unconditionally does that. Here is the full g++ invocation. It appears to be doing the right thing to build against LLVM-3.6svn.so, which exists in the location it is looking. So I don't understand why ldd doesn't show it linked against that.

g++ -shared -fPIC -L/home/wbhart/julia/usr/bin/../../usr/lib -L/home/wbhart/julia/usr/bin/../../usr/lib -lLLVM-3.6.0svn -ljulia -lpthread -o usr/lib/libcxxffi.so -Wl,--whole-archive -lclangFrontendTool -lclangBasic -lclangLex -lclangDriver -lclangFrontend -lclangParse -lclangAST -lclangASTMatchers -lclangSema -lclangAnalysis -lclangEdit -lclangRewriteFrontend -lclangRewrite -lclangSerialization -lclangStaticAnalyzerCheckers -lclangStaticAnalyzerCore -lclangStaticAnalyzerFrontend -lclangTooling -lclangCodeGen -lclangARCMigrate -llldbAPI -llldbBreakpoint -llldbCommands -llldbCore -llldbDataFormatters -llldbExpression -llldbHostCommon -llldbInitAndLog -llldbInterpreter -llldbPluginABISysV_x86_64 -llldbPluginDisassemblerLLVM -llldbPluginDynamicLoaderPOSIX -llldbPluginDynamicLoaderStatic -llldbPluginEmulateInstructionARM -llldbPluginEmulateInstructionARM64 -llldbPluginJITLoaderGDB -llldbPluginLanguageRuntimeCPlusPlusItaniumABI -llldbPluginObjectFileELF -llldbPluginObjectFileJIT -llldbPluginObjectContainerBSDArchive -llldbPluginObjectFilePECOFF -llldbPluginOperatingSystemPython -llldbPluginPlatformFreeBSD -llldbPluginPlatformGDBServer -llldbPluginPlatformLinux -llldbPluginPlatformPOSIX -llldbPluginPlatformWindows -llldbPluginPlatformKalimba -llldbPluginPlatformMacOSX -llldbPluginLanguageRuntimeObjCAppleObjCRuntime -llldbPluginProcessElfCore -llldbPluginProcessGDBRemote -llldbPluginMemoryHistoryASan -llldbPluginSymbolFileDWARF -llldbPluginSymbolFileSymtab -llldbPluginSymbolVendorELF -llldbSymbol -llldbUtility -llldbPluginUnwindAssemblyInstEmulation -llldbPluginUnwindAssemblyx86 -llldbPluginUtility -llldbTarget -llldbPluginInstrumentationRuntimeAddressSanitizer -lz -ltinfo -lrt -ldl -lm -llldbPluginABIMacOSX_arm -llldbPluginABIMacOSX_arm64 -llldbPluginABIMacOSX_i386 -llldbHostLinux -llldbPluginProcessLinux -llldbPluginProcessPOSIX -Wl,--no-whole-archive build/bootstrap.o

wbhart commented 9 years ago

It seems -llldb was missing. It works now.

wbhart commented 9 years ago

Oh I also added -llldbHostPosix and -llldbHostLinux (the latter was already there on linux I think but I added it unconditionally for good measure since a grep for the missing symbol turned up all three of those libraries, so I thought it couldn't hurt to add them all).

rhl- commented 9 years ago

Hi, I have a fresh install of the nightly Julia, can you summarize what you did to get Cxx.jl working?

Keno commented 9 years ago

What OS?

rhl- commented 9 years ago

OS/X I am also doing a compile from source right now...

rhl- commented 9 years ago

and I can do all of this on a linux machine if need be...

Keno commented 9 years ago

OS X is just fine (linux is too, my machine is just down, so I can't test at the moment).

Step 1: Place the following into a Make.user file inside your julia source tree:

LLVM_ASSERTIONS=1
LLVM_VER=svn
BUILD_LLVM_CLANG=1
BUILD_LLDB=1
LLDB_VER=master
USE_LLVM_SHLIB=1
LLDB_DISABLE_PYTHON=1
#LLVM_DEBUG=1

Step 2: (Re)build julia. You may have to do make cleanall first Step 3: From within julia (it is important that this is the julia version that has the above Make.user) run Pkg.build("Cxx"). Step 4: Done! using Cxx should now work.

Let me know if you run into any trouble. Also let me know if you use it to do something cool :).

rhl- commented 9 years ago

Thanks. My goal is to use julia as a repl for my computational topology library ctl.appliedtopology.org On Nov 3, 2014 3:51 PM, "Keno Fischer" notifications@github.com wrote:

OS X is just fine (linux is too, my machine is just down, so I can't test at the moment).

Step 1: Place the following into a Make.user file inside your julia source tree:

LLVM_ASSERTIONS=1 LLVM_VER=svn BUILD_LLVM_CLANG=1 BUILD_LLDB=1 LLDB_VER=master USE_LLVM_SHLIB=1 LLDB_DISABLE_PYTHON=1

LLVM_DEBUG=1

Step 2: (Re)build julia. You may have to do make cleanall first Step 3: From within julia (it is important that this is the julia version that has the above Make.user) run Pkg.build("Cxx"). Step 4: Done! using Cxx should now work.

Let me know if you run into any trouble. Also let me know if you use it to do something cool :).

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/21#issuecomment-61571172.

Keno commented 9 years ago

Cool! Let me know if you need any help.

rhl- commented 9 years ago

Any kind of documentation would be great, this is my first foray into Julia On Nov 3, 2014 3:55 PM, "Keno Fischer" notifications@github.com wrote:

Cool! Let me know if you need any help.

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/21#issuecomment-61571576.

Keno commented 9 years ago

Yeah, I need to do that (most of this is still experimental, but it works well for me). Quick overview of the Cxx.jl julia side features.

# First load the package
using Cxx

# Call a C++ function or access a C++ variable
@cxx foo::bar()
@cxx myclass->method()
@cxx myclass->variable

# Write raw C++ code at global scope
cxx"""
class MyClass {...}

// You can embed julia code by splicing expressions
int foo() {
$:(println("foo"));
// Or values by splicing them
return $(int32(1.0));
}
"""

# Write raw C++ code at function scope
foo() = icxx"long i = 0; i++; i;" # Last expression gets an implicit return

For more julia-specific references, there's of course docs.julialang.org and a bunch of other references at julialang.org.

rhl- commented 9 years ago

I see. What about namespaces and template types? My library uses them both, the latter quite heavily. Do you have auto template expansion? Is that planned?

Best, -rhl

On Mon, Nov 3, 2014 at 4:06 PM, Keno Fischer notifications@github.com wrote:

Yeah, I need to do that (most of this is still experimental, but it works well for me). Quick overview of the Cxx.jl julia side features.

First load the package

using Cxx

Call a C++ function or access a C++ variable

@cxx foo::bar() @cxx myclass->method() @cxx myclass->variable

Write raw C++ code at global scope

cxx""" class MyClass {...}

// You can embed julia code by splicing expressions int foo() { $:(println("foo")); // Or values by splicing them return $(int32(1.0)); } """

Write raw C++ code at function scope

foo() = icxx"long i = 0; i++; i;" # Last expression gets an implicit return

For more julia-specific references, there's of course docs.julialang.org and a bunch of other references at julialang.org.

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/21#issuecomment-61572667.

Keno commented 9 years ago

Yes, that's no problem. In julia land templates are expressed as eg @cxx fooclass{pcpp"foo::bar"}::staticmethod, though for complicated things I would recommend, writing the C++ call using icxx directly (and splicing in julia values using $).

rhl- commented 9 years ago

Wow, that is great! Would it be difficult to write something like julia_foo = cxx( " abazaba::Foo< Bar< Baz< int > > > foo( ..parameters..)") which takes a valid C++ type instantiation /function call and returns the result of @cxx .... ? If I had that, it seems like it would be painless to use C++ with julia.

Also, once all this compiling is done, is it cached somewhere? like in a .julia/cxx/cache.so ?

Best, -rhl

On Mon, Nov 3, 2014 at 6:43 PM, Keno Fischer notifications@github.com wrote:

Yes, that's no problem. In julia land templates are expressed as eg @cxx fooclass{pcpp"foo::bar"}::staticmethod, though for complicated things I would recommend, writing the C++ call using icxx directly (and splicing in julia values using $).

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/21#issuecomment-61585752.

rhl- commented 9 years ago

more to the point, suppose you have a main() function in some to_become_a_binary.cpp could that easily be eaten and turned into a C++/Julia script?

Best, -rhl

On Mon, Nov 3, 2014 at 6:50 PM, Ryan Lewis me@ryanlewis.net wrote:

Wow, that is great! Would it be difficult to write something like julia_foo = cxx( " abazaba::Foo< Bar< Baz< int > > > foo( ..parameters..)") which takes a valid C++ type instantiation /function call and returns the result of @cxx .... ? If I had that, it seems like it would be painless to use C++ with julia.

Also, once all this compiling is done, is it cached somewhere? like in a .julia/cxx/cache.so ?

Best, -rhl

On Mon, Nov 3, 2014 at 6:43 PM, Keno Fischer notifications@github.com wrote:

Yes, that's no problem. In julia land templates are expressed as eg @cxx fooclass{pcpp"foo::bar"}::staticmethod, though for complicated things I would recommend, writing the C++ call using icxx directly (and splicing in julia values using $).

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/21#issuecomment-61585752.

Keno commented 9 years ago

I haven't worked on caching yet, but it's on the todo list. What you're describing above is what icxx"abazaba::Foo< Bar< Baz< int > > > foo(...)" does.

zenna commented 9 years ago

I did a fresh compile from master using the instructions above but I am seeing the same problem as @wbhart did. When I call Pkg.build("Cxx") I get the same debug message. And when I try using Cxx I get ERROR: error compiling init: could not load module libcxxffi: libcxxffi: cannot open shared object file: No such file or directory.

I don't really follow on what was done to resolve this, any tips?

Keno commented 9 years ago

Few things to try:

  1. Verify that the library exists in ~/.julia/Cxx/deps/usr/lib
  2. Use ldd to verify that it links against the LLVM shared library.
wbhart commented 9 years ago

If I recall correctly, I did the following:

On 9 November 2014 00:48, Keno Fischer notifications@github.com wrote:

Few things to try:

  1. Verify that the library exists in ~/.julia/Cxx/deps/usr/lib
  2. Use ldd to verify that it links against the LLVM shared library.

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/21#issuecomment-62283809.

rhl- commented 9 years ago

@Keno @wbhart it seems like that line is necessary and causing an error.

Also, in my git clone, there is no Make.user, just a Make.inc, and in the line #LLVM_DEBUG=1 from early gives me an error.

I must write LLVM_DEBUG=0

Keno commented 9 years ago

You need to create Make.user. It's not in the git checkout because it's specific to an individual user.

rhl- commented 9 years ago

oh!

Best, -rhl

On Wed, Nov 12, 2014 at 12:03 PM, Keno Fischer notifications@github.com wrote:

You need to create Make.user. It's not in the git checkout because it's specific to an individual user.

— Reply to this email directly or view it on GitHub https://github.com/Keno/Cxx.jl/issues/21#issuecomment-62783852.

Keno commented 9 years ago

I'm going to close this issue. Please open new issues for any specific new failures.