calling ROOT via Cxx.jl

[jpata]

I was able to get ROOT.jl (on https://github.com/jpata/ROOT.jl/tree/cxx) with ROOT 6 to work directly through Cxx.jl under Mac OS X without re-compiling the julia binary to load-and-hide ROOT before the julia LLVM is called.

This seems to be possible due to LLVM >=3.6 using versioned symbols as described in https://bugs.freedesktop.org/show_bug.cgi?id=93103. However, ROOT still uses an old LLVM with un-versioned symbols.

julia> versioninfo()
Julia Version 0.5.0-dev+5503
Commit 93ee8dd* (2016-07-19 06:48 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin15.4.0)
  CPU: Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, haswell)

$ root-config --version
6.06/02

Enabling RTTI in Cxx.jl: https://github.com/jpata/Cxx.jl/pull/2, https://github.com/Keno/Cxx.jl/issues/104 (on top of https://github.com/Keno/Cxx.jl/commit/f6bc213b335e3c87a6dd2a08a0846813a259abfe)

ROOT prints the usual error but runs successfully:

$ ~/Documents/julia/julia ~/Documents/.julia/v0.5/ROOT/test/rootcxx.jl
Error in <UnknownClass::InitInterpreter()>: LLVM SYMBOLS ARE EXPOSED TO CLING! This will cause problems; please hide them or dlopen() them after the call to TROOT::InitInterpreter()!

Trying the same under linux (via https://github.com/jpata/docker-julia-root-cc7/blob/master/Dockerfile) fails with a segfault

(gdb) bt
#0  0x00007ffff6d145f7 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d15ce8 in abort () from /lib64/libc.so.6
#2  0x00007ffff6d54327 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff6d5a194 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffddd0081e3 in clang::LangOptions::~LangOptions() () from /root/.julia/v0.5/Cxx/src/../deps/usr/lib/libcxxffi-debug.so
#5  0x00007ffdde068152 in clang::PrintingPolicy::~PrintingPolicy (
    this=0x7ffddaa809a0 <TCling::InspectMembers(TMemberInspector&, void const*, TClass const*, bool)::printPol>, __in_chrg=<optimized out>)
    at /opt/julia/usr/bin/../../usr/include/clang/AST/PrettyPrinter.h:35
#6  0x00007ffff6d17e69 in __run_exit_handlers () from /lib64/libc.so.6
#7  0x00007ffff6d17eb5 in exit () from /lib64/libc.so.6
#8  0x00007ffff6d00b1c in __libc_start_main () from /lib64/libc.so.6
#9  0x00000000004012e9 in _start ()

Here's the docker image to reproduce that

jpata/julia               v0.5_root_cxx       f7e4de6be62a        About an hour ago   5.989 GB

ROOT looks for the following LLVM symbol to detect a clash: which is named a bit differently under OSX

$ nm -a /Users/joosep/Documents/julia/usr/lib/libLLVM-3.7.dylib | grep Pretty
0000000000cb8c10 T _LLVMEnablePrettyStackTrace

and linux

$ nm -a /opt/julia/usr/lib/libLLVM-3.7.so | grep Pretty
00000000004a6b40 T LLVMEnablePrettyStackTrace

The long-term fix would renaming the LLVM symbols when building it for julia: https://github.com/JuliaLang/julia/issues/12644

@oschulz @jpivarski you may be interested in this

[oschulz]

@jpata: Very nice! My Cxx.jl-based ROOT bindings (have a look: https://github.com/mppmu/ROOTFramework.jl/blob/master/examples/histgen.jl) were limited to ROOT-5 until now, because of the LLVM issue. I'll try with ROOT-6 and your patches.

[oschulz]

@jpata: I tried using ROOT6 via jpata/Cxx.jl#2 (with my ROOTFramework.jl and your example from your DIANA talk), but as soon as the first ROOT class is called, julia exits with this error:

Error in <UnknownClass::InitInterpreter()>: LLVM SYMBOLS ARE EXPOSED TO CLING! This will cause problems; please hide them or dlopen() them after the call to TROOT::InitInterpreter()!
: CommandLine Error: Option 'inline-threshold' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

Any ideas on what's causing the Option 'inline-threshold' registered more than once (resp. why it doesn't happen with your setup)?

[jpata]

@oschulz I don't really know. I also can't get ROOT6 via Cxx to work under linux, giving me the impression that there is still some fundamental conflict between the two LLVM-s.

[jpata]

I've made some progress.

@oschulz Try the cxx branch now. It will still compile a ROOT-enable julia binary, but it will live in the julia source tree in usr/bin/rjulia and behave in general exactly like the main binary. It seems to work well under OSX and Linux. In the end at CERN we'll probably make a custom compilation that is Cxx and ROOT enabled.

[oschulz]

Thanks, @jpata, I'll try!

I know that Keno also was trying some things to get ROOT-6 compatibility in CxxROOT.jl, but I don't know what the status is.

[oschulz]

Hm, on further reflection, I'm not so sure it's a good idea for a package to install something to the julia source tree. For example, we use a share julia installation for several people in our group - so that would mean that only the admin can install ROOT.jl - and things like ROOT.jl will warrant frequent updates. Or maybe the rjulia could live in a package of it's own (and wouldn't need to be updated often), separate from Julia wrappers around ROOT classes?

[oschulz]

Just an idea: rjulia could be a small separate C++ project, to be built independenty and linked against an existing Julia and ROOT installation - but installed separately. That way, one could also use one Julia installation with several ROOT versions (or the other way around). Or does rjulia need to live next to the julia binary for some reason? If not, it should be enough to install it somewhere in PATH.

Of course I still hope that Keno and the ROOT team will find a way to make Julia and ROOT play nicely together, eventually. :-)

[oschulz]

@jpata: I was able to apply and build your patch. However, I had to change

[...] `root-config --cflags --libs --ldflags` $^ -o $@ [...]

to

[...] $^ -o $@ `root-config --cflags --libs --ldflags` [...]

in the Makefile to avoid linker errors. (I'm on Ubuntu. Depending on the linker configuration, linking order can matter nowadays, https://wiki.debian.org/ToolChain/DSOLinking).

[jpata]

Thanks for trying.

1) CxxROOT.jl is a great idea, I first heard about it in December '15 when Keno wrote the code. It aims to replace TCling in ROOT with a Cxx.jl-based interpreter. It needs significant work that is beyond my knowledge. Depending on how modular ROOT is in that part, it may not offer full compatibility and may need significant changes to ROOT. In any case, I don't know how much time @Keno has to work on it in the near future.

2) Indeed, one would have to think about the packaging. I used to compile the binary in the Pkg directory, but it's somewhat easier (sysimg, JULIA_HOME, access to julia build env etc) to do it directly in the source tree. It wouldn't be a user-installed package on a shared system. One can think to still compile it in the Pkg directory (and later add it to PATH) by propagating -H -J flags, but I'm not sure it's better.

3) In my view, it's not necessary to re-create all the ROOT class structure and hierarchy on the julia side, like is done in PyROOT. I would have ROOT.jl be just a very thin layer that allows you to call ROOT via @cxx. We can think about offering julia-esque wrappers around some ROOT structures akin to rootpy, but re-coding all the structure of ROOT in julia seems hopeless (unless you do it automatically), especially as you can never offer full syntactic equivalence to C++.

[jpivarski]

For reference, PyROOT dynamically generates proxies to the corresponding ROOT classes. It's not just automated, but also just-in-time.

Unfortunately, this magic doesn't seem to work in Julia's PyCall. You can @pyimport ROOT, but none of the proxies exist. (In Julia 0.4. And even if it did work, using PyROOT in Julia would not be an efficient way to access ROOT.)

Jim

On Jul 29, 2016 1:48 AM, "Joosep Pata" notifications@github.com wrote:

Thanks for trying.

1) CxxROOT.jl is a great idea, I first head about it in December '15 when Keno wrote the code. It aims to replace TCling in ROOT with a Cxx.jl-based interpreter. It needs significant work that is beyond my knowledge. Depending on how modular ROOT is in that part, it may not offer full compatibility and may need significant changes to ROOT. In any case, I don't know how much time @Keno https://github.com/Keno has to work on it in the near future.

2) Indeed, one would have to think about the packaging. I used to compile the binary in the Pkg directory, but it's somewhat easier (sysimg, JULIA_HOME, access to julia build env etc) to do it directly in the source tree. It wouldn't be a user-installed package on a shared system. One can think to still compile it in the Pkg directory (and later add it to PATH) by propagating -H -J flags, but I'm not sure it's better.

3) In my view, it's not necessary to re-create all the ROOT class structure and hierarchy on the julia side, like is done in PyROOT. I would have ROOT.jl be just a very thin layer that allows you to call ROOT via @cxx. We can think about offering julia-esque wrappers around some ROOT structures akin to rootpy, but re-coding all the structure of ROOT in julia seems hopeless (unless you do it automatically), especially as you can never offer full syntactic equivalence to C++.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jpata/ROOT.jl/issues/4#issuecomment-236107428, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxEHxBsqAtjUe2sm2KHyErqJhnML5-Eks5qaaJKgaJpZM4JQo5Q .

[jpata]

Yep, I've seen the code that does it in PyROOT. PyCall+PyROOT was something I considered a while back, before embarking on ROOT+Clang.jl. There's also a way of directly accessing the ROOT proxies from PyCall using some python trickery, but it's all rather brittle.

To me, reflecting the whole of a complex C++ library in julia or python is bound to be doomed. In particular, the semantics of references, pointers and so on are lost in the PyROOT approach. Until recently, one could not really create any class from the PyROOT side which didn't have explicit "dictionaries" generated, so using functions like func(std::vector<Object>) was impossible.

Given that we have a way of directly calling C++ from julia, I would endeavour to make it as painless as possible. I think providing bespoke julian interfaces on top of TFile, TTree, TH*, ... as much as possible is useful, this is mostly what ROOTFramework.jl does.

If calling some more obscure ROOT functionality from julia is needed, one can still do it using cxx""" //C++ code here """.

The one issue that is killer is still the LLVM incompatibility, I don't have any good ideas about that.

On 29 Jul 2016, at 13:32, Jim Pivarski notifications@github.com wrote:

For reference, PyROOT dynamically generates proxies to the corresponding ROOT classes. It's not just automated, but also just-in-time.

Unfortunately, this magic doesn't seem to work in Julia's PyCall. You can @pyimport ROOT, but none of the proxies exist. (In Julia 0.4. And even if it did work, using PyROOT in Julia would not be an efficient way to access ROOT.)

Jim

On Jul 29, 2016 1:48 AM, "Joosep Pata" notifications@github.com wrote:

Thanks for trying.

1) CxxROOT.jl is a great idea, I first head about it in December '15 when Keno wrote the code. It aims to replace TCling in ROOT with a Cxx.jl-based interpreter. It needs significant work that is beyond my knowledge. Depending on how modular ROOT is in that part, it may not offer full compatibility and may need significant changes to ROOT. In any case, I don't know how much time @Keno https://github.com/Keno has to work on it in the near future.

2) Indeed, one would have to think about the packaging. I used to compile the binary in the Pkg directory, but it's somewhat easier (sysimg, JULIA_HOME, access to julia build env etc) to do it directly in the source tree. It wouldn't be a user-installed package on a shared system. One can think to still compile it in the Pkg directory (and later add it to PATH) by propagating -H -J flags, but I'm not sure it's better.

3) In my view, it's not necessary to re-create all the ROOT class structure and hierarchy on the julia side, like is done in PyROOT. I would have ROOT.jl be just a very thin layer that allows you to call ROOT via @cxx. We can think about offering julia-esque wrappers around some ROOT structures akin to rootpy, but re-coding all the structure of ROOT in julia seems hopeless (unless you do it automatically), especially as you can never offer full syntactic equivalence to C++.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jpata/ROOT.jl/issues/4#issuecomment-236107428, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxEHxBsqAtjUe2sm2KHyErqJhnML5-Eks5qaaJKgaJpZM4JQo5Q . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[oschulz]

@jpata: In my view, it's not necessary to re-create all the ROOT class structure and hierarchy on the julia side, like is done in PyROOT. I would have ROOT.jl be just a very thin layer that allows you to call ROOT via @cxx

To me, reflecting the whole of a complex C++ library in julia or python is bound to be doomed. [...] I think providing bespoke julian interfaces on top of TFile, TTree, TH*, ... as much as possible is useful, this is mostly what ROOTFramework.jl does.

I fully agree, and I like your idea of having one slim package to provide the necessary low-level compatibility support (ROOT.jl), and a separate package (like my current ROOTFramework) to provide Julianic convenience wrapper around frequently used ROOT functionalities (and maybe a Plots.jl backend, one day?). Certainly wrapping all of ROOT is not feasible - nor necessary with Cxx.jl. And the parts that we do want provide Julia wrapper for (like THxx, TFile, TTree) have to be carefully crafted manually, not automatically, to be ideomatic Julia.

Until now I thought we would merge ROOT.jl and ROOTFramework.jl at some point - but I think I prefer your layered approach, Joosep. In that case, I want to find a better name for ROOTFramework.jl though, it was just chosen to not conflict with ROOT.jl, and always considered temporary. :-) I'm open for proposals, anyone got a good idea?

[jpata]

I wouldn't still outrule merging the two packages at some point, I think for users it might be clearer if there is only one ROOT package, providing the base compat layer through Cxx and some limited "nice-to-have" julian wrappers.

The need to recompile the binary etc might go away and then we'd just have a single ROOT package.

IMHO first we've to figure out a strategy of what we wrap and how. rootpy has gone the way of basically manually pythonizing File, Tree, Chain, Hist etc. Their interface quite battle-tested, I would strongly favour taking example from them, e.g.

type Tree
    baseobj::CppValue{:TTree}
   ...
end
function Tree(name=nothing, title=nothing, model=nothing)
#like https://github.com/rootpy/rootpy/blob/master/rootpy/tree/tree.py#L886
end

In ROOT.jl I attempted to autogenerate the hierarchies using Clang.jl, but it will never map properly I think.

[oschulz]

The need to recompile the binary etc might go away and then we'd just have a single ROOT package.

Sure, once the LLVM problem has gone away, there will hopefully be no need for two packages. As long as we need to compile stuff to make Julia and ROOT compatible, though, it's probably better to keep that separated from the easy-to-update convenience wrappers.

rootpy has gone the way of basically manually pythonizing File, Tree, Chain, Hist etc. [...] I would strongly favour taking example from them

Well, yes and no. Julia is not Python, and not object oriented. So IMHO we shouldn't follow Python conventions where there is already a different Julia way to to a certain kind of task. We should also take full advantage of multiple dispatch, etc. in the Julia wrappers.

Also, we should strive free the Julia users from some of the historic awkwardness of ROOT (the the ROOT developers want to get away from long-term, too). Most importantly, I'd like to shield them from ROOT's way of having a global "current" file and putting some things into it automatically (TTree creation is a good example). For example, we could use getindex to access TDirectory (and therefore TFile) contents, like mytfile[TTree, "treename] or so.

Good Interface- / API-design is never independent from the language and it's conventions, and in several ways, Julia and Python differ very much. So instead of pythonizing ROOT, I'd like to julianize it. :-)

Also, it may be better not to wrap the CppValues, like in

type Tree
    baseobj::CppValue{:TTree}
   ...
end

but use them directly. It will make it easier to access non-wrapped functionality via @cxx in a natural fashion.

[Keno]

The one issue that is killer is still the LLVM incompatibility, I don't have any good ideas about that.

We're working on it. ROOT is carrying a number of patches that are not acceptable to upstream, so we're trying to work with them to figure out how to clean that up.

As for API design in general, I've strongly advocated not to just do some sort of automatic translation, but let C++ be C++ and have the wrapper packages expose more julian APIs on top of what C++ provides (i.e. integration with julia side abstractions, iterators, trees, arrays, etc.)

[oschulz]

the LLVM incompatibility We're working on it.

@Keno: Thanks!! I can imagine that it's not a trivial problem at all.

API design in general, I've strongly advocated not to just do some sort of automatic translation, but let C++ be C++ and have the wrapper packages expose more julian APIs on top of what C++ provides

Yes, that's what I've tried to do with ROOTFramework, so far. It's still very much a work in progress (and I probably need to revise some concepts). But my goal is to make it as julian as possible for the wrapped functionality, and to allow users to easily fall back to @cxx for the non-wrapped stuff. As Joosep said, ROOT is just to big to wrap it all. ;-)

[jpata]

@oschulz

• getindex is used in HDF5.jl, perhaps we can think if TFile, TDirectory etc map onto that.
• For TTree, TChain, I've investigated an AbstractDataFrame interface in ROOTDataFrames.jl

[oschulz]

getindex is used in HDF5.jl, perhaps we can think if TFile, TDirectory etc map onto that.

I didn't look into HDF5 in detail yet, but that's certainly a good reason to emulate it. One thing that held me back so far is that ROOT objects already have a name (not a design concept I like, but ...) and are usually written to a file under that name. So mytfile["myhist"] = h would not write h under it's current name. Though maybe that's not such a bad thing.

For TTree, TChain, I've investigated an AbstractDataFrame interface in ROOTDataFrames.jl

Yes, I saw that. I think something like that should be an add-on package (like it is now). ROOTPlots.j (Plot.jl integration) would be another candidate. I'm considering to directly include things like THxx<->StatsBase.Histogram, though, because histograms are so central to ROOT and one can hardly avoid having StatsBase installed anyhow.

[jpata]

On the topic of histograms and StatsBase:

https://github.com/jpata/ROOTHistograms.jl https://github.com/jpata/Histograms.jl

[jpivarski]

If this e-mail thread is a brainstorming session, here's an idea:

• Don't dynamically link into ROOT from Julia, which has problems with incompatible dependencies (LLVM). Instead, run it as a separate process.

Julia has an equivalent of Python's "multiprocessing," but deeply built into the language. When you @spawn an expression, it must serialize all the objects and send them through a pipe, and it must have some protocol for worker Julias to talk to the master Julia.

Here's the idea: implement that worker protocol in C++ in ROOT. When ROOT gets serialized Julia objects (a derivative of HDF5), it should unpack them as C++ objects, possibly using Cling to make the types and to compile and execute the desired expression. For user familiarity, it should use the same Julia data <--> C++ data mapping as Cxx.jl.

The @spawnroot macro would work a little differently than @spawn, in that the expression it gets has to be turned into a C++ AST or string for ROOT to evaluate. Since Julia macros already act on ASTs, there's a lot of flexibility here. For instance, Julia ASTs can look like objects and methods in dot-notation, such as '@spawnroot tfile.ls()'. If you like, you can retain C++'s distinctions between '.' and '->' by faking a pointer: transform '@spawnroot tfile.pointer.ls()' into 'tfile->ls()'. (Or use some shorter word like 'to'.)

This separate-process interface wouldn't be as fast as direct linkage, but it would be more flexible. Objects would not live in the same memory space, which can help to avoid death due to segmentation fault. In fact, if a ROOT process dies, Julia can spawn another one with less disruption to the user. Remote ROOT processes can be accessed on clusters with ROOT installed and no Julia installed (typical of physics installations, right now at least). Although each invocation of @spawnroot has a lot of overhead and shouldn't be used in a tight loop, it can contain a tight loop that runs as fast as C++ (because after conversion it is just C++). This is still faster than PyROOT because PyROOT does transformations every time it sees a dot, every time it steps through a loop.

In some systems (Spark and other tools from the Hadoop ecosystem come to mind), this separation of work into independent processes is not seen as a concession or a hack, but as a way of isolating and recovering from failure (because operating systems are more robust than most process runtimes).

And then Julianic interfaces can be built on top of this, like rootpy on PyROOT. (Is "Julianic" really the word? Not "Julian" or "Jubilant?")

-- Jim

On Fri, Jul 29, 2016 at 9:37 AM, Joosep Pata notifications@github.com wrote:

I wouldn't still outrule merging the two packages at some point, I think for users it might be clearer if there is only one ROOT package, providing the base compat layer through Cxx and some limited "nice-to-have" julian wrappers.

The need to recompile the binary etc might go away and then we'd just have a single ROOT package.

IMHO first we've to figure out a strategy of what we wrap and how. rootpy https://github.com/rootpy/rootpy/tree/master/rootpy has gone the way of basically manually pythonizing File, Tree, Chain, Hist etc. Their interface quite battle-tested, I would strongly favour taking example from them, e.g.

type Tree baseobj::CppValue{:TTree} ... end function Tree(name=nothing, title=nothing, model=nothing)

like https://github.com/rootpy/rootpy/blob/master/rootpy/tree/tree.py#L886

end

In ROOT.jl I attempted to autogenerate the hierarchies using Clang.jl, but it will never map properly I think.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jpata/ROOT.jl/issues/4#issuecomment-236197606, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxEHwcw6INORNqarqTZ0oO0I-uI5Bxwks5qahAMgaJpZM4JQo5Q .

[jpata]

@jpivarski this is certainly an idea I toyed with in desperation in early 2015, before I got the dynamic linking to work (what we do is dlopen julia in an exe dynamically linked with ROOT). For the record, using this kind of linking is fully functional, one just needs to recompile the julia REPL.

I thought about using Twisted, named pipes etc. I played around with it a bit but never got it to work well. You are probably much better prepared for this kind of an experiment than I was, though.

It seems kind of a nightmare though to have the two live in separate memory spaces so you can't exchange function pointers, arrays without overhead. TTree-backed columnar analysis like the following would be much clunkier.

values = tree[cuts, "branch"] #possibly an expensive operation, generating a large array in memory
println(mean(values)) #julia function on ROOT-derived data

I do like the idea of julia not dying in case of ROOT segfaulting with this encapsulation. However, from the practical distributed analysis world, it can be problematic forking in your jobs if you only want to be charged for the resources of 1 CPU.

It would be an additional layer of complexity. My experience with ROOT.jl (and CMSSW.jl) in a practical physics analysis was overall quite good, I had near native performance and good stability.

[oschulz]

This separate-process interface wouldn't be as fast as direct linkage, but it would be more flexible.

@jpivarski: I have thought about this as well - and in fact, used to in the past to write ROOT files from Scala - but I don't think it would be a good approach, in this case. The latency inherent in this model, for every function call, means that you need to use a programming API that supports queued/pipelined requests (e.g. promise/future) to get decent performance. And you need to users to understand how to properly use it. Our specific community is not used to this kind of programming model at all ... ;-)

[jpata]

Our specific community is not used to this kind of programming model at all ... ;-)

@jpivarski I think this is a crucial point, also about the question of any pushback inside HEP. It's good to change a few things for the better, but change too many things at once and nobody will use it. I'm not saying new is bad (I like Spark and @spawn), but we have to be careful to not be too far removed from how people are actually using ROOT: kind of a kitchen-sink Pandas/numpy/scipy equivalent.

[jpata]

@oschulz thinking about it more: we have a chance to get the API right, but it needs serious thought and also discussion with people who are not us. shall we make a repo like JuliaHEP/API and have some markdown documents there? We can start to flesh out our ideas, then ask for feedback from other people who have undertaken similar efforts.

[jpivarski]

On Fri, Jul 29, 2016 at 1:07 PM, Joosep Pata notifications@github.com wrote:

@oschulz https://github.com/oschulz thinking about it more: we have a chance to get the API right, but it needs serious thought and also discussion with people who are not us. shall we make a repo like JuliaHEP/API and have some markdown documents there? We can start to flesh out our ideas, then ask for feedback from other people who have undertaken similar efforts.

I like that idea a lot. I'll contribute, but I'll also find people who have physics deadlines to post their opinions as well. We might want to categorize suggestions from computing-focused people (who know the technical issues) and physics-focused people (who will be working with it, day in, day out) differently. I know; some are both.

-- JIm

[jpata]

thanks @jpivarski. I certainly have some physics deadlines :D I should open an issue to myself.

[jpata]

done https://github.com/JuliaHEP/API/blob/master/ROOT.md

[oschulz]

@jpata: thinking about it more: we have a chance to get the API right, but it needs serious thought and also discussion with people who are not us.

I fully agree. I'd like to get inputs from lots of people in the related fields - and on some issues, I hope we can also get advice from a Julia expert like Keno.

shall we make a repo like JuliaHEP/API and have some markdown documents there? We can start to flesh out our ideas,

I would actually prefer not to start with an abstract API spec, but with actual code: Prototyping with code, in my experience, often gives one a clearer idea of what work and what doesn't, than an abstract spec. Also, we already have some (I hope not too horrible) ROOT wrappers in ROOTFramework.jl, so I propose to use that as a starting point. We're already using that code in our (small) group here, and I need something that runs. However, we don't have a lot of code based on it yet, so I don't consider anything that's currently in ROOTFramework as sacred. For discussion and API proposals, I propose to use GitHub issues instead of a repo with Markdown docs - the issue system works well for such things, in my experience.

I mainly need TTree's and Histograms/Graphs at the moment, so these areas have a high priority for me, personally. I'm also currently in the lucky situation that I can officially devote some of my time to this. :-)

[oschulz]

Sorry, I just saw that you created the ROOT.md already. I'd really like to convince you that issues will work better for this than documents, though. The Julia people use the GitHub issue system very effectively, for example, also for semi-permanent content like milestone checklists, etc. It makes it easier to discuss.

[oschulz]

Like I said, I'm more than open to rename ROOTFramework to something else. Uh, and I'd also like a few days to clean it up a little, before we start in earnest. :-)

[jpata]

I'd say issues for discussion, .md for a status overview that we can easily show. I'm hesitant to write API code without a clear plan, having written already some useless APIs in the past. You're welcome to go ahead though. I'm not currently using julia for active work so my timeline is a bit different. Prototypes are of course required.

[oschulz]

@jpata: For more static content - like developing a user guide - why don't we just use a GitHub wiki (e.g. enable the wiki functionality for ROOTFramework.jl) instead a separate repo with markdown docs? I think a wiki would be a better fit. Then in the end, some of it can end up in the README and docs.

[oschulz]

I'm hesitant to write API code without a clear plan, having written already some useless APIs in the past

I actually did quite a bit of thinking before I put code into ROOTFramework. :-) I'm not saying it's perfect - probably far from it - but the was some planning. :-)

Prototypes are of course required.

Very much so. Since we're not writing a pure Julia API here, the fact that we're wrapping C++ will drive (and has for me, so far) quite a few design decisions. That why I think it important to try things out (and even use them) while they are developed.

You're welcome to go ahead though. I'm not currently using julia for active work so my timeline is a bit different.

Ok, thanks. I'll go ahead and flesh ROOTFramework out a bit more, then, in the areas we need immediately. I'll try to quickly get the master to a state anyone can try out very (say two weeks) - if you can find some time (I know your're busy with your thesis), I'd be happy if you gave it a go. Please feel free to suggest (even drastic) changes, I really want to avoid tailoring this too much to our needs.

Uh, would you be Ok with removing the "API" repo in the mean time? As not a lot may happen on it, it'll be confusing to people who look into JuliaHEP.

[jpata]

I'd keep the API it for the moment (if you think it's confusing I'll move it under my own name). But however smart any one of us is and how much thinking and designing we do, we are not smarter than the hivemind of ROOT users, julia users, rootpy developers etc.

If you develop something now, in my experience, it will be harder to change in the future. For example, all ROOTDataFrames, ROOTHistograms etc are at the moment useless.

Strong vote for a design document.

[jpivarski]

Okay, another weird suggestion: as part of my DIANA-HEP work, I'm going to try to solicit opinions about analysis script APIs (generally) in a focus group a few weeks from now. If that works well, future focus groups could be about the Julia-ROOT interface. We could prepare several mock-up APIs, convince ourselves that they'll work, and then present them to a group of physicists with laptops to try them all out and say what they like/don't like about them. That's pretty close to how focus groups work in industry, replacing "data analysis API" for "brand of soap."

In that case, our job would be to come up with a basis set of trial APIs--- they don't have to have working machinery behind them, just enough for participants to get the "feel" of them. It should be a basis set--- covering the space of possibilities with as few vectors as possible to avoid overwhelming the participants with nearly identical choices. For our own development, the design documents can be private, and I find Google Docs to be a very responsive way to collaboratively edit.

What do you think?

[jpata]

Focus group sounds really good. Given that I've been doing HEP analysis for some years, I have also some feeling for what works and what doesn't, but I'd like to take the average over some ensemble. For many questions, we might even get an answer from the root devs.

Trial APIs are also good, though I'm not sure we can have several meaningful ones very quickly. Maybe one and iterate on that.

Private google docs I'm less a fan of. In case dev discussions are held in private, they will be lost for future collaborators, who will keep offering the same old ideas. I think anybody should be able to trace why some design choice was adopted.

I'm reluctant to turn this into an all-talk and no-action kind of gig, so I propose we think a bit, write some code, and then discuss again in a couple of days.

[oschulz]

I'd keep the API it for the moment (if you think it's confusing I'll move it under my own name).

Would you, at least for the moment? Like I said, I don't want to do ROOTFramework (or whatever it's name will be) by myself, and I'm more than open to any suggestions concerning it's current (very limited) API. However, I think I'll have it in a state that people can actually use for simple things soon, and having API with different/future API concepts right next to it may be confusing.

[jpata]

Done.

I strongly believe that a solution that we code away in the next weeks with thought but without discussion will result in a local optimum, but not a global one.

We must use the experience of other devs out there, otherwise it's unlikely we'll get something much better than we already have.

I'm also closing this issue as it's no longer about the technicalities of using ROOT with Cxx.jl.

[oschulz]

We must use the experience of other devs out there, otherwise it's unlikely we'll get something much better than we already have.

I fully agree!

[jpivarski]

A heretical idea occurred to me about accessing ROOT in Cxx.jl: what about ROOT 5, rather than ROOT 6? It was based on CINT, not LLVM, and many analyses are still using it. I know it's not a good idea to depend on obsolete software, but the compatibility between ROOT and Julia could be advertised as "ROOT 5 or ROOT 6.x.y" where "6.x.y" is the version with the LLVM issues resolved.

[jpata]

ROOT 5 is a no go, an unmaintained and buggy evolutionary dead end. I haven't had a lot of time now to work on this, I think as Julia 0.5 and cxx are released and stable, this project just needs someone elbow grease to flesh out the essentials in Cxx as @oschulz has started.

[jpivarski]

I apologize if this information is available and I just can't find it, but can an appropriately-built version of ROOT and Julia be found anywhere? There was some talk earlier of putting it on CVMFS. Has this happened?

I want to do a tutorial session with some physicists at the LPC (cmslpc-sl6.fnal.gov or lxplus.cern.ch) to convince them that Julia is interesting/useful for their analyses. Without any way to access data in ROOT files, it's a hard sell. So we don't need the frameworks to be fully worked out, such as API differences between ROOT.jl and ROOTFramework.jl: we just need enough compatibility to load data into Julia data structures so that I can show them Julia. If ROOT 5 works, I'd use that. (I'm trying it now and am running into compilation errors.)

If the physics audience gets hooked on the idea of doing analysis in Julia and then find that the Julia-ROOT link is not production ready (either because of the custom build or ROOT 5), that's okay. They'll demand compatibility in a future ROOT release. In fact, the existence of vocal users might be necessary to get this incompatibility resolved.

So let's get these eggs laying chickens! (Or chickens laying eggs, whichever...)

[jpata]

Pere Mato Vila was working on the julia+ROOT cvmfs install, but they've different constraints at CERN for installing software and it's not high on their priority list I suppose. Did you try setting it up based on this repo? It works under OSX and linux when I last tried, there's also a docker image: https://github.com/jpata/docker-julia-root-cc7/blob/master/Dockerfile There is no way to install on a non-source julia. I currently have no time to invest in making a community around this project, but if you have some people interested, that would be great. I'd be happy if you and @oschulz exercised commit rights to this project under the JuliaHEP org.

[jpivarski]

I'm trying to create the community interest, and for that I'll need a minimally working version. If you have a recipe, I'll build an Amazon AMI and give everyone in the tutorial access to log in and try it. Maybe as many instances as there are participants.

This repo says you're changing things around for Cxx.jl, so don't expect it to work. Is there a branch or a revision number that I can Pkg.clone?

[oschulz]

@jpivarski:

A heretical idea occurred to me about accessing ROOT in Cxx.jl: what about ROOT 5

I'm actually using ROOT-5 at the moment, without problems, because of the LLVM conflict between Julia and ROOT-6 (I haven't managed to get @jpata's workaround to run on my system, but I'll try again). Joseep is right, though, of course: ROOT-5 is a dead and, and won't be maintained for much longer (if it's still supported at all).

[oschulz]

I haven't found time in the last weeks to work on the ROOT/Cxx.jl integration, but I hope to make some progress over the next two weeks.

[jpivarski]

I overheard some conversations about a ROOT-LLVM incompatibility due to a symbol clash with LLVM loaded by a graphics library. This is the bug report:

https://sft.its.cern.ch/jira/browse/ROOT-7821

I'm told it was fixed in 6.06/00, but only for CMake builds, not for configure builds. When you encountered the Julia-ROOT incompatibility, were you using this patch?

[oschulz]

@jpivarski: I'm not sure it's the same issue we're running into, but I'll try!

[jpata]

Yes, some newer OpenGL also uses it's own llvm and I've seen this clash as well. ROOT specs simply expect to call TCling::InitInterpreter before referring to any other external LLVM, in which case ROOT can hide itself. This is what I've done in ROOT.jl by recompiling the julia binary.

You can see the dockerfile with the ROOT version here: https://github.com/jpata/docker-julia-root-cc7/blob/master/Dockerfile#L11, so most likely this is already included and didn't fix the fundamental incompatibility.

I've thought about it over the past months and I believe it's not worth it to port ROOT, it's probably better for HEP to go cold turkey.

[oschulz]

@jpata: I believe it's not worth it to port ROOT, it's probably better for HEP to go cold turkey.

I agree that it would be best to do as many things as possible in pure Julia, integrated with the rest of the Julia ecosystem. Unfortunately, it's often still unavoidable to read and write root files, to be able to work with one's community. Also, ROOT plots are often expected or even mandated.

Just curious, what's the current status with ROOT.jl - meaning, rjulia. Does it works reliably on Linux now? If so, it would really be nice to be able to build it separately from Julia, now that Cxx.jl can be build separately (e.g. with official Julia binaries) too.