Cxx as standard Julia package??

[wbhart]

Sorry to use a GitHub issue to ask questions, but I couldn't think of a better way to ask the following.

My colleague Oleksandr Motsak is about a week away from completing his initial prototype Julia/Cxx interface to a large ~600,000 line C++ library for computer algebra, called Singular [1]. This relies heavily on Cxx for C++ interface to Julia and our Nemo.jl computer algebra package for Julia [2].

This project is a prototype for a much larger project for which we are currently applying for an enormous grant. It will bring together millions of lines of Asm/C/C++/Perl code, plus library code in various custom languages, all devoted to various aspects of computer algebra. I can't say too much about it, except that it brings together many experts and is a long term endeavour with many aspects.

Julia is being used as part of our strategy for efficient interfacing and to provide highly efficient generics.

As such, we are trying to get an overview of possible future development of Cxx. Essentially we are ready right now to merge Oleksandr's work into mainstream Nemo so we can begin refining it, extending it and cleaning it up. But we can't really do that quite yet due to the experimental nature of Cxx.

As I understand it, Cxx relies on LLVM-3.7, which is now a standard part of Julia-0.5-dev. But it also relies on libclang and lldb plus a few other dependencies? So I guess my question is, what has been decided with regard to the future of Cxx? We can't really begin to use it very seriously unless it is as easy as loading some standard packages for people to begin using it. We can't really ask all our users to build a special version of Julia, etc. This is especially true for online notebooks such as cloud.sagemath.com and JuliaBox which both support Julia.

Would it be feasible at all to make a standard libclang package for Julia for example? Is Cxx likely to become a standard Julia package any time soon?

Also, what in your opinion is still missing from Cxx? Are there any plans for closer integration into Julia itself?

Are there still upstream LLVM patches that are being merged that Cxx will rely on, so that Julia will need to keep updating LLVM in order to keep maintaining Cxx?

Do you have any kind of notional timeline for any of the Cxx plans?

Thanks in advance for any answers you can give. I'm also happy to discuss off list if you prefer.

[1] https://github.com/malex984/Nemo.jl/tree/singular_interpreter/src/singular [2] https://github.com/Nemocas/nemo.jl

[Keno]

Hi there,

But it also relies on libclang and lldb plus a few other dependencies?

Yes, it relies on Clang (needs to be a custom source build because it uses private APIs, vanilla libclang is unfortunately not enough). The LLDB dependency was added for Gallium, but it looks like we may be dropping LLDB from Gallium entirely, which would mean dropping LLDB here.

Would it be feasible at all to make a standard libclang package for Julia for example? Is Cxx likely to become a standard Julia package any time soon?

Yes, I'm planning to start building Cxx binaries as soon as 0.5 stabilizes a bit. That should cover at least Mac/Windows. The Linux situation may be a bit more tricky. I can build binaries against the official julialang binaries, but there are likely distribution specific requirements that I may not be able to meet.

Also, what in your opinion is still missing from Cxx? Are there any plans for closer integration into Julia itself?

There is a couple of things I'd like to add, such as an easier way to handle some of the common wrapping tasks, but other than that I consider Cxx to be essentially feature complete. The main thing I'm hoping for in the future is performance improvements, integration with precompilation and better integration for debugging, etc.

Are there still upstream LLVM patches that are being merged that Cxx will rely on, so that Julia will need to keep updating LLVM in order to keep maintaining Cxx?

No, Cxx works on LLVM 3.7.1 onwards. There may be additional LLVM patches required in the future, but I'll try to keep Cxx working on the LLVM version utilized by the last major julia release (beginning with 0.5 obviously).

Do you have any kind of notional timeline for any of the Cxx plans?

Binaries by the 0.5 release. Any other sorts of improvements are mostly on hold while I focus on other things (particularly Gallium). I do try to get around to fixing bugs though.

[wbhart]

As always, thanks for all the really amazing work you are doing, and thanks for the answers to my questions. This will help us a lot with planning and answering the all important questions of referees when the time comes, and of our colleagues in the mean time.

[SirFizX]

Great work! It will be awesome to see Cxx in Juliabox!

[abbradar]

Hi,

I was trying to set up Cxx.jl some time ago. One thing that in my opinion could be improved is that if there's no real requirement of the whole Julia tree it would be nice to support building just libclang (for private headers) and Cxx.jl itself. That is -- be able to use system installation of both Julia and LLVM. If you already plan to move in this direction after 0.5 release sorry for the noise. Some Linux distributions for example already provide git snapshots of Julia, so I has expected the package to just use it.

That said, thank you very much for your work!

[Keno]

This is done.

[ihnorton]

I'm not quite sure what to expect here, but tried the following with latest OSX nightly, clean package directory:

julia> Pkg.add("Cxx")
INFO: Initializing package repository /Users/inorton/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
INFO: Cloning cache of Cxx from https://github.com/Keno/Cxx.jl.git
INFO: Installing Cxx v0.0.1
INFO: Building Cxx
writing path.jl file
Tuning for julia installation at: /opt/sw/Julia-0.5-dev-5f50872dd4.app/Contents/Resources/julia/bin
 c++ -D_GLIBCXX_USE_CXX11_ABI=0 -fno-rtti -DLIBRARY_EXPORTS -fPIC -O0 -g -std=c++11   -I/opt/sw/Julia-0.5-dev-5f50872dd4.app/Contents/Resources/julia/bin/../../src/support -I/opt/sw/Julia-0.5-dev-5f50872dd4.app/Contents/Resources/julia/bin/../include -Isrc/clang-3.7.1/lib -Ibuild/clang-3.7.1/include -Isrc/clang-3.7.1/include -Isrc/llvm-3.7.1/include -Ibuild/llvm-3.7.1/include  -DLLVM_NDEBUG -c ../src/bootstrap.cpp -o build/bootstrap.o
../src/bootstrap.cpp:19:10: fatal error: 'llvm/ADT/DenseMapInfo.h' file not found
#include "llvm/ADT/DenseMapInfo.h"
         ^
1 error generated.
make: *** [build/bootstrap.o] Error 1
========================================================[ ERROR: Cxx ]========================================================

LoadError: failed process: Process(`make -j4 -f BuildBootstrap.Makefile BASE_JULIA_HOME=/opt/sw/Julia-0.5-dev-5f50872dd4.app/Contents/Resources/julia/bin`, ProcessExited(2)) [2]
while loading /Users/inorton/.julia/v0.5/Cxx/deps/build.jl, in expression starting on line 31

==============================================================================================================================

=======================================================[ BUILD ERRORS ]=======================================================

WARNING: Cxx had build errors.

 - packages with build errors remain installed in /Users/inorton/.julia/v0.5
 - build the package(s) and all dependencies with `Pkg.build("Cxx")`
 - build a single package by running its `deps/build.jl` script

==============================================================================================================================
INFO: Package database updated

[Keno]

File a separate issue?

[blegat]

This may be a stupid question but since you say that the LLVM patches are now upstream, why does the build.jl script needs to build julia ? What is the issue with the Julia version already installed ?

[Keno]

Julia doesn't ship LLVM headers, so if you have a binary distribution of julia, we need to rebuild LLVM. We always need to build Clang.