Use Docker on Travis

[staticfloat]

This closes https://github.com/JuliaInterop/libcxxwrap-julia/issues/1

The issue is that GCC, when moving to version 5, changed the default C++ ABI to be compatible with the then-new C++ version c++11. For "backwards compatibility" reasons they decided not to bump the SOVERSION of libstdc++ and kept it at libstdc++.so.6; but because the changes were not truly backwards compatible, users must not mix code compiled with the new cxx11 ABI with code compiled with the old ABI.

Julia used to be compiled with the old ABI, however the compiler landscape eventually moved forward enough that we determined it was worthwhile to drop that restriction. Starting from Julia 0.6.1, the releases were built with the new cxx11 ABI, which should be the default for any version of GCC >= 5.2. When I saw the issues in this repository, I thought to myself that it must be as simple as upgrading the gcc you're using to 5.4 and changing some compiler flag to say -D _GLIBCXX_USE_CXX11_ABI=1, however that didn't work. It took me a little while to realize that, since the default compilers on Ubuntu precise (and trusty) were both < 5.0, the newer compilers I was downloading from PPAs all had the new cxx11 abi completely disabled. Even if you explicitly ask for it, the compiler just ignores it and continues to compile against the old abi. (This is set by the --disable-libstdcxx-dual-abi configure argument passed to gcc at build-time. You can check for it by running g++ -v and looking at the configure line)

This means that, unfortunately, no g++ compilers you install through any PPA that I searched through will properly link against the official julia binaries if you're running on Ubuntu Precise or Trusty. There is a workaround, which is to tell the compiler to link against the libstdc++.so.6 that comes with Julia (e.g. just pass that file in as another input file to your failing g++ command) and that works, but I think that mostly works because you are not actually compiling any new C++ source code that makes use of the cxx11 abi here, you're just relinking LLVM code that has already been compiled and uses it, and g++ is complaining that it's default internal libstdc++ doesn't have those methods available.

The better solution, in my opinion, is to use a slightly newer version of Ubuntu as the build environment. This PR adds a .ci folder that has a Dockerfile within it and a build.sh script. The Dockerfile is used to download and install g++ and cmake within a docker image, which is then used to build the libcxxwrap library. To prove that the built library is good and usable anywhere, the tests are run outside of the docker environment.

If you have any questions about this, please do not hesitate to ask. I spent a good while researching why exactly this was breaking on Ubuntu when it worked fine on my local (non-Ubuntu) machines, so I've likely got most of the answers cached in my mind already. :)

[tkelman]

This won't help for users on LTS distros with old-abi compilers, they will all still have this issue. This doesn't fix anything just ignores the problem for CI.

[staticfloat]

This doesn't fix anything just ignores the problem for CI.

This isn't a problem to fix; it's the natural consequence of moving beyond an ABI incompatibility. There are two solutions for users that want to install this package:

• Compile locally, using a non-crippled GCC version. Ubuntu versions 15.10+ will work out-of-the-box. Older versions such as 14.04 and 12.04 will need a different version of GCC than is available through PPAs. If the user must build locally, I would suggest using a docker setup similar to what is showcased here; it will result in binaries sitting in the same locations as a typical build and will function outside of the docker environment without issue.

• Download precompiled binaries. This package is pretty easy to compile ahead of time, and as is clear from this issue, local compiler mismatches can cause plenty of trouble. I've setup a BinaryBuilder repository for Cxxwrap to showcase how can be done, with a release that can be downloaded and unpacked via BinaryProvider. The only issues with that release are that it appears the cmake build script is not passing in any -march flags, you should pass in -march=x86-64 to limit the CPU instructions being generated to be maximally compatible across all client systems. This can be done by adding to CMAKE_C_FLAGS and CMAKE_CXX_FLAGS.

[barche]

Thanks for this very extensive work. I was already planning to look at BinaryBuilder as part of the 0.7 transition, so that certainly looks like the long-term solution.

[tkelman]

That would result in this package no longer working for any users on LTS distros. As seen with BinaryBuilder on any C++ libraries (or Fortran for slightly different reasons), compatibility was lost with source builds of Julia and anything else built with system compilers on those distros. Particularly for this package, a "crippled," really just old, version of gcc is going to be the only option for many users. Centos and redhat aren't going anywhere and use old-abi compilers. The new ABI isn't mandatory to use on new distros, new versions of gcc are going to continue supporting the old ABI as a non-default option for some time.

[barche]

Aha, I was under the impression that the objective of BinaryBuilder was to build binaries that are intended for use together with the official Julia binaries. It seems reasonable to provide cxxwrap binaries built that way, i.e. built with the same compiler and libraries as the official Julia binaries. In other cases, cxxwrap will have to be built from source, but that should be easy for someone who already built Julia from source.

[staticfloat]

That would result in this package no longer working for any users on LTS distros.

That is untrue. As shown within this PR, newer compilers that use the new C++ ABI can be used to compile code that runs just fine on older distros. The compilers within a docker image as used here, or from within the BinaryBuilder environment will both generate binaries that work just fine on whatever CentOS or RHEL distro you wish to use.

Aha, I was under the impression that the objective of BinaryBuilder was to build binaries that are intended for use together with the official Julia binaries

That is correct.

In other cases, cxxwrap will have to be built from source, but that should be easy for someone who already built Julia from source.

If the user already built Julia from source and the user wants to build libcxxwrap from source, then the issues we've been dealing with in this PR shouldn't matter, as their from-source Julia should be compiled with the same compilers they're using to build libcxxwrap. If the user doesn't want to build libcxxwrap from source, then yes, things become very complicated.

[staticfloat]

Thanks for this very extensive work.

You are welcome! :)

[tkelman]

The point of this package is to link to users' c++ code, which will be compiled by their system compilers. Using the new ABI means users have to build all C++ code via BinaryBuilder, which is not always an option

[barche]

Using the new ABI means users have to build all C++ code via BinaryBuilder, which is not always an option

True, but the ultimate objective is to build packages that use a C++ library, so these packages themselves should be encouraged to use BinaryBuilder to compile both the wrapped library (say, Qt in the case of QML.jl) and the custom wrapper code based on libcxxwrap. That way, everything is built using the same compiler. When developing such a package, it's up to the developer to ensure the compilers are compatible. Personally, I use from-source Julia builds on Linux and the official binaries on macOS and Windows. On our CentOS cluster, I use a GCC 7.2 built from source, since the compiler there is so ancient it's useless anyway.

[tkelman]

Packages built that way won't be compatible with source builds of Julia (which includes distro packages) using system compilers on old-ABI distros though. Do you expect everyone using CentOS to first build their own GCC before doing anything?

[SylvainCorlay]

Thanks @staticfloat for the detailed explanation. We have been seeing that issue with Xtensor.jl since the release of Julia 0.6.1 (btw, I responded to the release announcement with a question about the switch to the new ABI, but few people seem to build native extensions like we do and noone replied.)

It took me a little while to realize that, since the default compilers on Ubuntu precise (and trusty) were both < 5.0, the newer compilers I was downloading from PPAs all had the new cxx11 abi completely disabled. Even if you explicitly ask for it, the compiler just ignores it and continues to compile against the old abi. (This is set by the --disable-libstdcxx-dual-abi configure argument passed to gcc at build-time. You can check for it by running g++ -v and looking at the configure line).

Now, it all makes sense. This looks like a bad decision for the distributions to use that option though.

Quick note: it could be interesting to use a conda package for gcc which will always use the new ABI regardless of the linux distribution.

[SylvainCorlay]

More generally, I would find it very interesting if there was a conda build of Julia (but also for other package managers), especially for the case of native extensions. There has been some work done for conda-forge but only the linux build got through.

The main benefit of using conda would be that conda packages for Julia extensions could stop vendoring the binary dependencies and merely depend on the corresponding packages. For those built with CxxWrap, the split of libcxxwrap-julia as a pure C++ package really goes in the right direction. I already have made a conda package for it...

Also, vendoring things in a Julia package places the assets in non-standard locations, making it harder to depend on it. At the moment, the build of a package making use of CxxWrap-vendored libcxxwrap-julia looks like

JlCxx_DIR=$(julia -E "Pkg.dir(\"CxxWrap\", \"deps\", \"usr\", \"lib\", \"cmake\", \"JlCxx\")")
JlCxx_DIR=${JlCxx_DIR//\"/}
cmake -D JlCxx_DIR=$JlCxx_DIR -D CMAKE_INSTALL_PREFIX=/path/to/prefix SOURCE_DIR

while it can simply be

cmake -D CMAKE_INSTALL_PREFIX=/path/to/prefix SOURCE_DIR

if libcxxwrap-julia is install in the prefix as a dependency.

[barche]

Do you expect everyone using CentOS to first build their own GCC before doing anything?

That would be fairly common on e.g. a cluster, where one could also use a package manager like nix or easybuild that also ships its own compiler.

Anyway, if for this package right now we have to choose between being binary compatible with something, I think it's best to pick the official binaries. Maybe the ABI flag on the Julia binaries could be flipped again, but that should probably be discussed in a Julia issue. It is unclear to me what the consequences are of sticking with the old ABI.

@staticfloat I took a look at the CxxwrapBuilder repo you made, it seems to me that the code for building the binaries could be integrated right into this repository?

[SylvainCorlay]

As I am updating Xtensor.jl to the latest CxxWrap.jl (https://github.com/QuantStack/Xtensor.jl/pull/82), I wonder if it would not make sense to have a general installation prefix for binary things under .julia for extensions.

That would be a much more sensible thing than having to specify all the paths to find

Pkg.dir("CxxWrap", "deps", "usr", "lib", "cmake", "JlCxx")
Pkg.dir("Xtensor", "deps", "usr", "lib", "cmake", "xtl")
Pkg.dir("Xtensor", "deps", "usr", "lib", "cmake", "xtensor")
Pkg.dir("Xtensor", "deps", "usr", "lib", "cmake", "xtensor-julia")

[barche]

I agree having a single prefix would be simpler, but I'm not sure how to choose it and afterwards how to manage the files that get placed there. I have some broader questions as well, regarding user-control over build.jl behavior, to set things such as "always build from source", "prefer Conda", "prefer system packages", ... Currently I use environment variables, but I'm not sure that is the best way. Maybe it's best to start a discussion on Discourse about this?

[SylvainCorlay]

I agree having a single prefix would be simpler, but I'm not sure how to choose it and afterwards how to manage the files that get placed there. I have some broader questions as well, regarding user-control over build.jl behavior, to set things such as "always build from source", "prefer Conda", "prefer system packages", ... Currently I use environment variables, but I'm not sure that is the best way. Maybe it's best to start a discussion on Discourse about this?

👍 I would love to get into a discussion with Julia folks interested in these questions.

Julia has a unique occasion to get a nicer story for packaging extensions and preventing the vendoring hell that other languages like R and Python are into. For example RcppArmadillo, RcppEigen vendor armillo and eigen headers respectively, even on debian (which is an infringement of the debian packaging policy...)

[SylvainCorlay]

My take on this is that the scenario should be that of Julia installed in a given prefix, along peer packages, without a special prefix for julia packages having native dependencies. The official distribution channel could rely on any cross-platform package manager. The result would be very amenable to the packaging for linux distributions or homebrew.

So to clarify, the default directory would be the one where julia was installed.

[staticfloat]

Please feel free to start a discourse discussion of this and ping myself and Stefan Karpinski; we've been putting a lot of thought into how this should work with binaries and Pkg3, and there are lots of things to consider.

[barche]

@staticfloat I've been trying to integrate this to use BinaryBuilder directly inside this repository at PR #5 , and it works for x86_64, but trying to extend to other platforms it doesn't work because the build script downloads only Julia-x86_64. Is there an easy way already provided by BinaryBuilder to download the correct Julia tarball based on the target platform considered? For reference, this is the current error: https://travis-ci.org/JuliaInterop/libcxxwrap-julia/builds/392699491

The build before this works because it only targets Linux x86_64.

Also, I assume non-linux platforms are built using cross-compilation? If so, I expect trouble, since we run Julia itself to find out where the library and headers are.

[staticfloat]

Is there an easy way already provided by BinaryBuilder to download the correct Julia tarball based on the target platform considered?

No, there isn't. It's ironic, almost like we want a build.jl for Julia itself. :P Eventually, we'll probably have to make that, just so that you can link against libjulia, but right now the inability to run the julia executable is a bigger problem.

Also, I assume non-linux platforms are built using cross-compilation? If so, I expect trouble, since we run Julia itself to find out where the library and headers are.

Those don't seem like platform-specific things; why not run the x86_64-linux-gnu executable for every platform?

[barche]

Closed with PR #5