smaller binary snapshots download (without bundled LLVM)

[ustcscgy]

The binary snapshots listed in https://root.cern.ch/download/cling/ are pretty big, over 200MB, but most of it are LLVM, while cling is static linked against it. After removing bundled LLVM, cling is under 40MB, and after compression, it's 14MB.

Would you consider uploading this cling-only binary snapshots?

BTW, what's the progress of upstreaming LLVM patches? (So that I can build cling against system bundled LLVM instead of downloading binary snapshots.)

[vgvassilev]

Thanks for the report. Yes we are working on this. We have a packaging tool that makes them considerably smaller.

We consider uploading binary releases along with each tag.

Some of the patches could be upstreamed but some of them cannot. I have this on my todo list but it is not with high priority. I am completely in favor of building cling against bundled LLVM, but it may require some more work and time.

[ustcscgy]

Glad to know that you're working on it, I'll wait for the samller binary packages then.

As for the pacthes that can't be upstreamed, can you tell a bit more about it? As far as I am concered, if unpatched LLVM can't provide functionalities that cling needs, isn't that a bug/missing feature for LLVM?

[vgvassilev]

We have uploaded smaller binaries here.

[ustcscgy]

Great! Two small improvements:

1. Fedora version. Which do not use libtinfo and libcxx.
2. Nightly version, like https://root.cern.ch/download/cling/

[ustcscgy]

I see there's a nightly version now, will there be fedora version too?

[vgvassilev]

Currently we are looking into this. I am not quite sure we can succeed, because we are using travis ci public service, running on ubuntu. I've been thinking to add a build which spans fedora environment with docker to deliver the fedora nightly. I don't think I will have enough time though. You are welcome to help us out.

[ustcscgy]

OK, one reason I want to have Fedora version is because it works and Ubuntu version not work: my current system don't have libtinfo and libc++abi.

Anyways to avoid link cling againt these libraries?

[vgvassilev]

Can you download the latest binary and try? I don't think we depend on libtinfo and libc++ at least for the current nightly builds.

[ustcscgy]

In 2016-Aug-08-cling-Ubuntu-12.04-x86_64-0.3.dev-e21ea6d.tar.bz2 :

ldd ./bin/cling
....
        libtinfo.so.5 => not found
....

libc++/libc++rt/libc++abi is in release 0.2 IIRC, but they are gone in this snapshot I think.

[vgvassilev]

We need tinfo for colored diagnostics.

[ustcscgy]

tinfo is a part of ncurses, can be configured off. What cling actually need is ncurses, I think.

[vgvassilev]

We provide our own terminal library, because we don't want to depend on ncurses. Maybe we could have a static/fat binary build.

[ustcscgy]

cling uses its own terminal library? Let me look into source code to provide more comment.

BTW, cling's terminal behavior is odd under emacs (M-x comint-run), I wonder if it's possibe to disable its terminal capability (and let emacs handle that).

[ustcscgy]

First, grep -R tinfo . only returns one match in ./www/index.html, and I think that needs to be fixed.

I think the libtinfo dependency may be come from LLVM, just a guess.

[vgvassilev]

Yes, it comes from llvm. It is used by clang to produce colored diagnostics.

[ustcscgy]

tinfo is old ncurses, I think?

I see that lib/UserInterface/textinput/doc/textinput.txt says the reason to not use ncurses is license and simplicity.

Well, LLVM/Clang has to use ncurses/libtinfo, or BSD licensed libedit.

So, question 1: is it possible to static link ncurses/libtinfo or libedit againt LLVM/Clang? (Is that what happened in Fedora build?)

Question 2: is textinput written from scratch or borrowed from some open source project?

[vgvassilev]

Q1: Could you find out. The LLVM cmake config flag for this is LLVM_ENABLE_TERMINFO. I think it should be possible with minimal changes to our build system to do it. Q2: It was written from scratch by @karies.

[ustcscgy]

I skim through gentoo's llvm build script, I think it's impossible to static link LLVM against ncurses.

However, one can disable ncurses support by -DLLVM_ENABLE_TERMINFO=NO, does that impact anything else? I'd like to disable that, otherwise it would be a mess to distribute binary packages: because there are ncurses5 and ncurses6 nowadays...

[vgvassilev]

Yes you could disable it, but you won't see colored output on error. I don't think we can afford that in mainline.

[ustcscgy]

Colored output... Currently I have a Fedora snapshot from https://root.cern.ch/download/cling, it doesn't have colored error output and doesn't link to tinfo or ncurses.

Do you aware of that?

[vgvassilev]

No. I think this is a bug. Does clang have colored diagnostics on fedora? Eg. echo "error" | clang++ -xc++ - returns non color error message?

[ustcscgy]

I'm not using fedora, I'm using gentoo. And yes, that line have colored error output on my system, no colored error output from bundled clang.

[vgvassilev]

So we need to figure out what's wrong in our version...

[ustcscgy]

May I ask which script generate binary snapshot on github and https://root.cern.ch/download/cling?

[vgvassilev]

On github I use cpt. For the others it is an internal custom post build procedure.

[ustcscgy]

More info: I have cling_2016-07-20_fedora22 on my system, and running ldd on every bin and so from that tarball, none links to tinfo or ncurses.

And echo "error" | ./clang++ -xc++ - has no colored output.

I'm downloading cling_2016-08-09_fedora24.tar.bz2 .

[vgvassilev]

Look at https://root.cern.ch/gitweb/?p=rootspi.git;a=tree;f=jenkins;h=fee15867baf9244d6ed003110c5fa3c133584557;hb=HEAD. cling-* are the builders there. I don't understand why we don't build with colors. Maybe the machine doesn't have tinfo installed.

[ustcscgy]

Same error for cling_2016-08-09_fedora24, I guess there is a bug in fedora building script.

But I have to say, being never have a cling with colored error output, I'm OK with that ...

[ustcscgy]

I see the jenkins build scripts, nothing related with tinfo/ncurses. I'm not familar with jenkins CI, so maybe the build machine don't have ncurses/tinfo installed.

And one correction: libedit is supposed to be a replacement for readline, not ncurses.

So, do you think package libtinfo.so.5 inside the package is OK?

Also, seems Ubuntu version hard code g++4.9 inside binary, Fedora version doesn't have this problem:

ERROR in cling::CIFactory::createCI(): cannot extract standard library include paths!
Invoking:
    echo | LC_ALL=C g++-4.9  -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -fno-strict-aliasing -O3 -DNDEBUG -xc++ -E -v - 2>&1 >/dev/null | awk '/^#include </,/^End of search/{if (!/^#include </ && !/^End of search/){ print }}' | GREP_OPTIONS= grep -E "(c|g)\+\+"
results in
with exit code 256
input_line_1:1:10: fatal error: 'new' file not found
#include <new>
         ^

causes segfault on any input.

[ustcscgy]

One more PS: I think cling's textinput library is more like a readline replacement than a ncurses like terminal library.

So the root problem here is clang uses ncurses to display colored error output, cling don't use ncurses/tinfo.

[SylvainCorlay]

Would it be conceivable to not require any cling patches for llvm and use stock llvm at some point in the future?

[vgvassilev]

Short term no (we lack manpower) long term: yes once we get rid of the 100 patches we have on top of llvm and clang.

[ustcscgy]

@vgvassilev , why not ask the llvm/clang community for help? There's a post by Axel on cfe-dev in 2011, now 5 years later, cling has made huge improvements, maybe more people will find cling useful or interesting, and willing to help to upstream these patches.

[SylvainCorlay]

I was wondering what it would take to create a conda package for cling on conda-forge. We could have a variant of the llvm package which is marked as conflicting with the regular one...

[vgvassilev]

@ustcscgy: I don't think it's llvm/clang job to prepare our patches for a review. I have this in my todo list (as I said before) but not with very high priority. I am hoping for some help by the cling community if that is too big of a trouble.

@SylvainCorlay: I am not sure how exactly conda-forge works but it might be easier to squash all llvm/clang related patches which you would apply (temporarily) to llvm before the cling build. Note that we are very sensitive to revision numbers as the patch may not apply anymore to later versions.

[vgvassilev]

@ustcscgy: on the original issue, we have a PR #108 which is adds a fedora based cpt build. Once we have this integrated we could provide smaller nightlies. If you are still interested I wouldn't mind some help, as I don't think I can get to it soon.

[SylvainCorlay]

@SylvainCorlay: I am not sure how exactly conda-forge works but it might be easier to squash all llvm/clang related patches which you would apply (temporarily) to llvm before the cling build. Note that we are very sensitive to revision numbers as the patch may not apply anymore to later versions.

We could have an llvm-cling package which reuses the regular llvm recipe and applies your patches. conda builds are reproducible in that they target a specific hash for the source.

Also we would need to create a clang package too.

[SylvainCorlay]

The python recipe applies a few patches on the python source code:

https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml

[vgvassilev]

@SylvainCorlay: for packaging you may want to look at cpt. And our 'nightlies' here.

[SylvainCorlay]

@vgvassilev I was playing around with conda, with the latest ubuntu nighly builds.

I put together a recipe for ubuntu conda install -c SylvainCorlay cling, which works well and installs the kernelspec.

On docker though, there is a link issue with cling::printValue(int const*).

[vgvassilev]

@SylvainCorlay cool! Could you open an ticket about the linker issue?

[SylvainCorlay]

It seems to be that issue, but with gcc 4.9 instead of 5.1. I will just need to use another build of cling that is based on gcc4 (like the ubuntu 14 build).

Remark: it would be really nice if it was possible to have a single wheel for all linux64 architectures.

[SylvainCorlay]

@vgvassilev

I put together a conda recipe with the current binary and published a binder so that anyone can play with it online. You can check it out here:

I would love to make the conda packaging so that we could industrialize this sort of thing.

[vgvassilev]

@SylvainCorlay cool. Maybe if it stabilizes (having at least a domain name) you could open a pull request and update cling's readme pointing to it.

Please open a new ticket if you need some more help about conda packaging.

[SylvainCorlay]

@vgvassilev this binder is meant to be a transient thing. What I wanted to emphasize, is that putting a clean conda package together would have a lot of benefits. A lot would come for free then.

• users of anaconda / miniconda can install cling with

  conda install cling

  and run it in the notebook, or jupyterlab.

• we can create binders like the above example, for people who want to share runable snippets in a git repository to show off they library.

If you want something like this for the cling website itself, the best approach would probably to use a tmpnb-style deployment and host it on a subdomain like try.root.cern.ch just like we have try.jupyter.org @rgbkrk.

I am opening a separate issue with the different pitfalls I encountered which I think should be solved to have cling be part of an official conda channel like conda-forge.

[SylvainCorlay]

@vgvassilev @karies I wanted to ping on the release of xtensor.

The project website has a Try it Now button powered by Cling and Jupyter.

[vgvassilev]

Cool! Why does the link resolve to an ip and not a domain? Once that's fixed we can add this to the README page of jupiter/cling.

[SylvainCorlay]

@vgvassilev it is because it is a redirection to the vm that is specifically spawned for you.