wlav / cppyy

Other
391 stars 40 forks source link

Compatibility with ROOT #105

Closed marc1uk closed 1 year ago

marc1uk commented 1 year ago

I'm trying to add cppyy into a framework that also uses ROOT. As usual, the mesh of dependencies is a pain, so we're using an old version of ROOT: 6.08.06. I can install cppyy just fine, but once I source thisroot.sh for my ROOT installation, cppyy fails when trying to import cppyy.gbl.std vector

Python 3.8.13 (default, Jun 24 2022, 15:27:57) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-13)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cppyy
>>> from cppyy.gbl.std import vector
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cppyy.gbl.std'; 'cppyy.gbl' is not a package

As you mentioned in #77 python is finding the wrong version of cppyy. So I tried disabling the python bindings in ROOT, rebuilding with --Dpyroot=OFF -Dpython=OFF -Dpython3=OFF but I now get a segfault on import cppyy:

[root@44013a4a215f install]# python3
Python 3.8.13 (default, Jun 24 2022, 15:27:57) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-13)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cppyyy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cppyyy'
>>> import cppyy
 Generating stack trace...
 0x00007f5b3683de3a in CppyyLegacy::TSystem::FindHelper(char const*, void*) at /ToolAnalysis/Dependencies/cppyy-backend/cling/
src/core/base/inc/TString.h:244 from /root/.local/lib/python3.8/site-packages/cppyy_backend/lib/libCoreLegacy.so
 0x00007f5b368b49b2 in CppyyLegacy::TUnixSystem::Unlink(char const*) at /ToolAnalysis/Dependencies/cppyy-backend/cling/src/cor
e/unix/src/TUnixSystem.cxx:1368 from /root/.local/lib/python3.8/site-packages/cppyy_backend/lib/libCoreLegacy.so
 0x00007f5b2e1e0988 in TUUID::TUUID() at /ToolAnalysis/ToolDAQ/root-6.08.06/core/base/src/TUUID.cxx:149 (discriminator 1) from
 /ToolAnalysis/ToolDAQ/root-6.08.06/install/lib/libCore.so
 0x00007f5b2e190a09 in TDirectory::TDirectory() at /ToolAnalysis/ToolDAQ/root-6.08.06/core/base/src/TDirectory.cxx:44 (discrim
inator 2) from /ToolAnalysis/ToolDAQ/root-6.08.06/install/lib/libCore.so
 0x00007f5b2e10d7a7 in TROOT::TROOT(char const*, char const*, void (**)()) at /ToolAnalysis/ToolDAQ/root-6.08.06/core/base/src
/TROOT.cxx:618 from /ToolAnalysis/ToolDAQ/root-6.08.06/install/lib/libCore.so
 0x00007f5b2e10ef04 in ROOT::Internal::GetROOT1() at /ToolAnalysis/ToolDAQ/root-6.08.06/core/base/src/TROOT.cxx:356 (discrimin
ator 4) from /ToolAnalysis/ToolDAQ/root-6.08.06/install/lib/libCore.so
 0x00007f5b2e103e30 in <unknown> from /ToolAnalysis/ToolDAQ/root-6.08.06/install/lib/libCore.so
 0x00007f5b39ed405a in <unknown> from /lib64/ld-linux-x86-64.so.2
 0x00007f5b39ed415a in <unknown> from /lib64/ld-linux-x86-64.so.2
 0x00007f5b3861abac in _dl_catch_exception at :? from /lib64/libc.so.6
 0x00007f5b39edb89e in <unknown> from /lib64/ld-linux-x86-64.so.2
 0x00007f5b3861ab54 in _dl_catch_exception at :? from /lib64/libc.so.6
 0x00007f5b39edba81 in <unknown> from /lib64/ld-linux-x86-64.so.2
 0x00007f5b38dfdf8a in <unknown> from /lib64/libdl.so.2
 0x00007f5b3861ab54 in _dl_catch_exception at :? from /lib64/libc.so.6
 0x00007f5b3861ac13 in _dl_catch_error at :? from /lib64/libc.so.6
 0x00007f5b38dfe52e in <unknown> from /lib64/libdl.so.2
 0x00007f5b38dfe02a in dlopen at :? from /lib64/libdl.so.2
 0x00007f5b368177a7 in CppyyLegacy::TROOT::InitInterpreter() at /ToolAnalysis/Dependencies/cppyy-backend/cling/src/core/base/s
rc/TROOT.cxx:1134 from /root/.local/lib/python3.8/site-packages/cppyy_backend/lib/libCoreLegacy.so
 0x00007f5b3681794f in CppyyLegacy::Internal::GetROOT2() at /ToolAnalysis/Dependencies/cppyy-backend/cling/src/core/base/src/T
ROOT.cxx:361 from /root/.local/lib/python3.8/site-packages/cppyy_backend/lib/libCoreLegacy.so
 0x00007f5b31568d88 in <unknown> from /root/.local/lib/python3.8/site-packages/cppyy_backend/lib/libcppyy_backend.so
 0x00007f5b39ed405a in <unknown> from /lib64/ld-linux-x86-64.so.2
 0x00007f5b39ed415a in <unknown> from /lib64/ld-linux-x86-64.so.2
 0x00007f5b3861abac in _dl_catch_exception at :? from /lib64/libc.so.6
 0x00007f5b39edb89e in <unknown> from /lib64/ld-linux-x86-64.so.2
 0x00007f5b3861ab54 in _dl_catch_exception at :? from /lib64/libc.so.6
 0x00007f5b39edba81 in <unknown> from /lib64/ld-linux-x86-64.so.2
 0x00007f5b38dfdf8a in <unknown> from /lib64/libdl.so.2
 0x00007f5b3861ab54 in _dl_catch_exception at :? from /lib64/libc.so.6
 0x00007f5b3861ac13 in _dl_catch_error at :? from /lib64/libc.so.6
 0x00007f5b38dfe52e in <unknown> from /lib64/libdl.so.2
 0x00007f5b38dfe02a in dlopen at :? from /lib64/libdl.so.2
 0x00007f5b376de3bd in <unknown> from /usr/lib64/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so
 0x00007f5b395c1a8a in PyCFunction_Call + 0x11a from /lib64/libpython3.8.so.1.0
 0x00007f5b39575402 in _PyObject_MakeTpCall + 0x442 from /lib64/libpython3.8.so.1.0
 0x00007f5b3962fc67 in _PyEval_EvalFrameDefault + 0x5397 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e8559 in _PyEval_EvalCodeWithName + 0xcd9 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e9292 in _PyFunction_Vectorcall + 0x422 from /lib64/libpython3.8.so.1.0
 0x00007f5b39618076 in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b3957529a in _PyObject_MakeTpCall + 0x2da from /lib64/libpython3.8.so.1.0
 0x00007f5b39630398 in _PyEval_EvalFrameDefault + 0x5ac8 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b3962aeaa in _PyEval_EvalFrameDefault + 0x5da from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b3962fd69 in _PyEval_EvalFrameDefault + 0x5499 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e7c46 in _PyEval_EvalCodeWithName + 0x3c6 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e8e23 in PyEval_EvalCode + 0x23 from /lib64/libpython3.8.so.1.0
 0x00007f5b3969affb in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b395b0630 in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b395b51d0 in PyVectorcall_Call + 0x70 from /lib64/libpython3.8.so.1.0
 0x00007f5b39630c8b in _PyEval_EvalFrameDefault + 0x63bb from /lib64/libpython3.8.so.1.0
 0x00007f5b395e7c46 in _PyEval_EvalCodeWithName + 0x3c6 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e9292 in _PyFunction_Vectorcall + 0x422 from /lib64/libpython3.8.so.1.0
 0x00007f5b3962fd69 in _PyEval_EvalFrameDefault + 0x5499 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b3962b1dd in _PyEval_EvalFrameDefault + 0x90d from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b3962aeaa in _PyEval_EvalFrameDefault + 0x5da from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b3962aeaa in _PyEval_EvalFrameDefault + 0x5da from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b395c1d50 in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b395db2e8 in _PyObject_CallMethodIdObjArgs + 0x168 from /lib64/libpython3.8.so.1.0
 0x00007f5b3957fb62 in PyImport_ImportModuleLevelObject + 0x682 from /lib64/libpython3.8.so.1.0
 0x00007f5b3962df97 in _PyEval_EvalFrameDefault + 0x36c7 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e7c46 in _PyEval_EvalCodeWithName + 0x3c6 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e8e23 in PyEval_EvalCode + 0x23 from /lib64/libpython3.8.so.1.0
 0x00007f5b3969affb in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b395b0630 in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b395b51d0 in PyVectorcall_Call + 0x70 from /lib64/libpython3.8.so.1.0
 0x00007f5b39630c8b in _PyEval_EvalFrameDefault + 0x63bb from /lib64/libpython3.8.so.1.0
 0x00007f5b395e7c46 in _PyEval_EvalCodeWithName + 0x3c6 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e9292 in _PyFunction_Vectorcall + 0x422 from /lib64/libpython3.8.so.1.0
 0x00007f5b3962fd69 in _PyEval_EvalFrameDefault + 0x5499 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b3962b1dd in _PyEval_EvalFrameDefault + 0x90d from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b3962aeaa in _PyEval_EvalFrameDefault + 0x5da from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b3962aeaa in _PyEval_EvalFrameDefault + 0x5da from /lib64/libpython3.8.so.1.0
 0x00007f5b395e902f in _PyFunction_Vectorcall + 0x1bf from /lib64/libpython3.8.so.1.0
 0x00007f5b395c1d50 in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b395db2e8 in _PyObject_CallMethodIdObjArgs + 0x168 from /lib64/libpython3.8.so.1.0
 0x00007f5b3957fb62 in PyImport_ImportModuleLevelObject + 0x682 from /lib64/libpython3.8.so.1.0
 0x00007f5b3962df97 in _PyEval_EvalFrameDefault + 0x36c7 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e7c46 in _PyEval_EvalCodeWithName + 0x3c6 from /lib64/libpython3.8.so.1.0
 0x00007f5b395e8e23 in PyEval_EvalCode + 0x23 from /lib64/libpython3.8.so.1.0
 0x00007f5b39674f0a in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b3969ae52 in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b3954d270 in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b39554035 in PyRun_InteractiveLoopFlags + 0xee from /lib64/libpython3.8.so.1.0
 0x00007f5b395540f9 in PyRun_AnyFileExFlags + 0x3d from /lib64/libpython3.8.so.1.0
 0x00007f5b39554a9c in <unknown> from /lib64/libpython3.8.so.1.0
 0x00007f5b3969cba9 in Py_BytesMain + 0x39 from /lib64/libpython3.8.so.1.0
 0x00007f5b384ebd85 in __libc_start_main + 0xe5 from /lib64/libc.so.6
 0x000055fd29d8078e in _start + 0x2e from python3

I can get around this by putting the path to the corresponding cppyy libraries in LD_LIBRARY_PATH first: export LD_LIBRARY_PATH=/root/.local/lib/python3.8/site-packages/cppyy_backend/lib:/ToolAnalysis/ToolDAQ/root-6.08.06/install/lib after which cppyy works, but now firing up a ROOT shell gives:

[root@44013a4a215f ToolAnalysis]# root -b
Warning in <TInterpreter::ReadRootmapFile>: class  timespec found in libCoreLegacy.so  is already in libCore.so 
Warning in <TInterpreter::ReadRootmapFile>: typedef  timespec_t found in libCoreLegacy.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  is already in libCore.so 
Warning in <TInterpreter::ReadRootmapFile>: var  gSystem found in libCoreLegacy.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  libCore.so  is already in libCore.so 

Granted these are warnings not errors, but I have a sinking feeling that this will cause problems further down the line. It seems I have to choose; ROOT or cppyy. Is it possible for both to play nice? What's the necessary configuration?

wlav commented 1 year ago

Can't mix these: that old ROOT uses Cling based on Clang3; modern cppyy uses Cling based on Clang9. Both use Cling through libCling.so. Clashes will ensue.

marc1uk commented 1 year ago

I see. Is there an old cppyy version that would be compatible?

wlav commented 1 year ago

That release is from March 2017. The first public release of cppyy on PyPI is from July 2017. You probably don't want to go there. You're better off using the old cppyy.py that you had at the beginning (from cppyy.gbl.std import vector doesn't work, but accessing vector = cppyy.gbl.std.vector probably will).

One way or another, both those dates are more than 5 years ago. cppyy has been under non-stop development. The difference in functionality is going to be huge.

marc1uk commented 1 year ago

:pensive: It's great that things keep progressing, but it does make keeping a stable system tough. 2017 may sound like forever ago, but I've been on this experiment since 2015, and at that time ROOT 6 was considered "new". Experiments do typically run over those kinds of timescales, and constantly updating package versions (particularly ones deeply integrated with others like ROOT) is a lot of work. At some point "let's just drop everything and get all the latest stuff" just isn't an option.

Anywho. I guess I'll keep fiddling and see what I can do. Thanks for the quick and definitive reply, it's really helpful. :+1:

wlav commented 1 year ago

But why do you need to have the latest (non-HEP) cppyy in old ROOT? Ie., why do you need to update at all? And if you do need to update, why only update parts of the software and not everything?

Current ROOT has a moderately updated cppyy (2020 or so), and is probably going to suit your needs better.

marc1uk commented 1 year ago

Some newer collaborators want to use machine learning algorithms from python, which of course require all the latest-and-greatest libraries, along with a python bindings that integrate with our existing c++ stuff. I'm not familiar with the python bindings provided by ROOT - I just googled c++/python bindings and found cppyy. If there's a built-in version in ROOT 6.08 that's probably our best option.

wlav commented 1 year ago

Yes! You do not need public cppyy at all in any which way or form. ROOT has PyROOT, which in newer versions is indeed based on cppyy (which itself was originally forked off PyROOT, so the variance in functionality between 2017 PyROOT and 2017 cppyy is going to be minimal; it's mostly cleanup/performance that was done first), but there, too, you won't need public cppyy, as ROOT ships with a builtin copy.