Use pkg-config to find library location and headers ?

[stuaxo]

Could xeus-cling use pkg-config to find the a library, it's headers and flags ?

e.g. - for cairo there is:

$ pkg-config --cflags cairo
-I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/uuid -I/usr/include/freetype2 -I/usr/include/libpng16

And it knows the linker flags:

$ pkg-config --libs cairo
-lcairo

[traversaro]

Indeed, discussing with @GiulioRomualdi we also noticed it would be quite convenient to have a single command to import a C/C++ library via its pkg-config files, i.e. to include libxml2 to have instead of:

#pragma cling add_include_path("<prefix>/include")
#pragma cling add_include_path("<prefix>/include/libxml2")
#pragma cling add_library_path("<prefix>/lib")
#pragma cling load("libxml2")

where the specific directories need to be hardcoded (see https://github.com/jupyter-xeus/xeus-cling/issues/87), to just load the necessary info from pkg-config, like:

#pragma cling load_library_from_pkg_config("libxml-2.0")

or, if you want to use CMake imported target, to have:

#pragma cling load_library_from_cmake_config("LibXml2", "LibXml2::LibXml2")

Not immediately, but we could be interested in providing support for such a functionality (even by using a different mechanism rather then #pragma cling that may something quite deep inside cling), so if anyone has some inputs on this it would be great. If I recalled correctly with @wolfv we discussed about something related to this, but I don't remember the details.

[traversaro]

I guess this cling upstream issue could be related: https://github.com/root-project/cling/issues/320 .

[wolfv]

yeah, this would be great!

We have good experience with reproc-cpp and I guess all the information would be only one subprocess call away :)

[traversaro]

I was a bit confused by the fact that #pragma cling add_include_path-style commands were not really usable in a C++ program being preprocessor commands and not proper C++ APIs, but then looking into the code (https://github.com/root-project/cling/blob/v0.8/lib/Interpreter/ClingPragmas.cpp#L224) I noticed that they are just a tiny wrapper over cling C++ API (that can be easily accessible, see https://blog.llvm.org/posts/2020-12-21-interactive-cpp-for-data-science/, so a first proof of concept could be implemented as a simple and independent C++ header only library, something like:

// To find the cling-load-library library. not necessary if it is already in 
// a directory that is already in the include path
#pragma cling add_include_path(<cling_load_library_include_install_prefix>)

// Load library via CMake Config
#include <clingLoadLibrary/LoadLibraryFromCMakeConfig.h>
clingLoadLibrary::LoadLibraryFromCMakeConfig("LibXml2", "LibXml2::LibXml2");

// Load library via pkg-config
#include <clingLoadLibrary/LoadLibraryFromCMakeConfig.h>
clingLoadLibrary::LoadLibraryFromPkgConfig("libxml-2.0");

CMake brainstorming

The clingLoadLibrary::LoadLibraryFromCMakeConfig function inside would just need to invoke cmake via reproc-cpp or similar to configure a dummy CMake project to print and extract the necessary include files and library files for the specified target, and then it can pass them to the cling interpreter by including #include <cling/Interpreter/Interpreter.h> (that provides access to the gCling interpreter singleton) and calling as appropriate:

//  Add include path, see https://github.com/root-project/cling/blob/v0.8/include/cling/Interpreter/Interpreter.h#L421
gCling->AddIncludePath("<include_path>");
// Load shared library
// https://github.com/root-project/cling/blob/v0.8/include/cling/Interpreter/Interpreter.h#L635
gCling->loadLibrary("<library_to_load>");

At the CMake-level a few things needs to be properly accounted for (transitive target dependencies in INTERFACE_LINK_LIBRARIES, properly evaluate CMake generator expressions for example in multiple-config case, raise error if the library type is not shared), but nothing too complex.

pkg-config brainstorming

A pattern similar to CMake of invoking as an external process pkg-config could be used even for .pc files, but perhaps in this case this could be avoided by linking directly to libpkgconf.

Other formats

In the future it would be nice to even add support for more C++ library manifest formats such as cps or libman, see the related discussions: https://gitlab.kitware.com/cmake/cmake/-/issues/20106, https://github.com/conan-io/conan/issues/4878), however until those formats are widely adopted in libraries there is a limited benefit in doing so.

[traversaro]

An alternative solution that I found thanks to https://github.com/PointCloudLibrary/pcl/issues/4838#issuecomment-881540056 is to use a separate generate_cling_3rd_party.py (https://github.com/rapyuta-robotics/jupyter_ros_utils/blob/main/cpp/scripts/generate_cling_3rd_party.py) script to write the #pragma cling directive in a file, that then is loaded in some way from the interpreter (it is not clear to me how, to be honest). See https://github.com/rapyuta-robotics/jupyter_ros_utils/blob/main/cpp/notebooks/pcl-1.10.ipynb for an example of this pattern, kudos to @ardiya .

[ardiya]

@traversaro Thank you for your interest. You can also check out our notebook interactively via binder. here is example interactive notebook for the file you mentioned https://mybinder.org/v2/gh/rapyuta-robotics/jupyter_ros_utils/HEAD?filepath=cpp%2Fnotebooks%2Fpcl-1.10.ipynb

[traversaro]

Ah, I see, you need to first call ! ../scripts/generate_cling_3rd_party.py eigen3 and then #include "load_eigen3.h", now this is clear. Just a (completly OT) curiosity: there is any specific reason why you install some software via conda and some via apt (such as OpenCV or ROS) instead of installing all the software via conda, using conda-forge for packages such as OpenCV (https://github.com/conda-forge/opencv-feedstock) or RoboStack (https://github.com/RoboStack/ros-noetic, https://medium.com/robostack/cross-platform-conda-packages-for-ros-fa1974fd1de3) for ROS related packages?

[ardiya]

@traversaro For C++ stuff, only boost, xeus-cling, and xwidgets are installed by conda. line in Dockerfile: https://github.com/rapyuta-robotics/jupyter_ros_utils/blob/main/Dockerfile#L31

While reason for xeus-cling and xwidgets are straightforward. I was having issue boost installed via apt because some boost header located at /usr/include/boost/* are trying to include <bits/***> header which exists in gcc but doesn't exist in cling. My first attempt was to clone boost from source. It fixes the issue but it was taking too much time to clone and size was pretty big too. Then, someone from the team uses conda to install boost. It fixes the issue and shorten the download time, so we use the boost conda.

But yeah, the rest of the stuff are installed via apt. I might take a look at RoboStack in future, it seems interesting and it also support ROS2