search

accellera-official / PySysC

Public repository for PySysC, (From SC Common Practices Subgroup)
Apache License 2.0
48 stars 6 forks source link

RuntimeError: could not load cppyy_backend library, details: #4

Open amal-khailtash opened 3 years ago

amal-khailtash commented 3 years ago

This is probably not PySysC specific issue but my cppyy installation. I followed the procedure to build PySysC and I get this error when cppyy is imported (when I run any of the python files in tests directory):

Traceback (most recent call last):
  File ".../PySysC/tests/modules.py", line 9, in <module>
    import cppyy
  File ".../PySysC/.venv/lib/python3.9/site-packages/cppyy/__init__.py", line 74, in <module>
    from ._cpython_cppyy import *
  File ".../PySysC/.venv/lib/python3.9/site-packages/cppyy/_cpython_cppyy.py", line 20, in <module>
    c = loader.load_cpp_backend()
  File ".../PySysC/.venv/lib/python3.9/site-packages/cppyy_backend/loader.py", line 85, in load_cpp_backend
    raise RuntimeError("could not load cppyy_backend library, details:\n%s" %
RuntimeError: could not load cppyy_backend library, details:
  libcppyy_backend.so: cannot open shared object file: No such file or directory
  .../.venv/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.so: undefined symbol: _ZN11CppyyLegacy10TClassEdit9ShortTypeEPKci
  libcppyy_backend.cpython-39-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
  .../.venv/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.cpython-39-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

Any ideas?

wlav commented 3 years ago

Is there any libcppyy_backend*.so file in lib/python3.9/site-packages/cppyy_backend/lib?

amal-khailtash commented 3 years ago

Yes!

$ ll .venv/lib/python3.9/site-packages/cppyy_backend/lib/
total 68956
-rwxr-xr-x 1 akhailta asic_100g 62646768 Mar 17 17:23 libCling.so*
-rw-r--r-- 1 akhailta asic_100g     9722 Mar 17 17:23 libCoreLegacy.rootmap
-rwxr-xr-x 1 akhailta asic_100g  3468280 Mar 17 17:23 libCoreLegacy.so*
-rwxr-xr-x 1 akhailta asic_100g   170952 Mar 17 17:23 libcppyy_backend.so*   <<<<<<<
-rw-r--r-- 1 akhailta asic_100g     1386 Mar 17 17:23 libRIOLegacy.rootmap
-rwxr-xr-x 1 akhailta asic_100g  3603224 Mar 17 17:23 libRIOLegacy.so*
-rw-r--r-- 1 akhailta asic_100g      700 Mar 17 17:23 libThreadLegacy.rootmap
-rwxr-xr-x 1 akhailta asic_100g   402952 Mar 17 17:23 libThreadLegacy.so*
amal-khailtash commented 3 years ago

This symbol seems to point to the cppyy library itself:

$ echo _ZN11CppyyLegacy10TClassEdit9ShortTypeEPKci | demangle
CppyyLegacy::TClassEdit::ShortType(char const*, int)
CppyyLegacy::TClassEdit::ShortType(char const*, int)

The symbol is undefined in this library:

$ nm --dynamic --defined-only --extern-only .venv/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.so | grep _ZN11CppyyLegacy10TClassEdit9ShortTypeEPKci
                 U _ZN11CppyyLegacy10TClassEdit9ShortTypeEPKci

As it is confirmed here:

$ nm --dynamic --defined-only --extern-only .venv/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.so | grep _ZN11CppyyLegacy10TClassEdit9ShortTypeEPKci

Not sure which library this symbol should be defined.

wlav commented 3 years ago

CppyyLegacy::TClassEdit::ShortType(char const*, int) is defined in libCoreLegacy.so, but the mangled symbol should be _ZN11CppyyLegacy10TClassEdit9ShortTypeB5cxx11EPKci, not _ZN11CppyyLegacy10TClassEdit9ShortTypeEPKci. (I.e. with abi:cxx11.)

Was the gcc version with which libcppyy_backend.so was build older than 5? (You can use cppyy with gcc4.8 or later, but you'd have to install with the --no-binary=cppyy-cling option given to pip. That'll take a long time to build, though.)

amal-khailtash commented 3 years ago

Yes, I was building with my system default version 4.8.5. I switched to gcc/8.3.0 and I can get passed that error. According to README.md, I am also supposed to build PySysC-SC, which I got from: https://git.minres.com/SystemC/PySysC-SC, bu I am having trouble with cmake -DBUILD_SHARED_LIBS=ON .. step and I get:

cmake -DBUILD_SHARED_LIBS=ON ..
CMake Error at CMakeLists.txt:11 (include):
  include could not find load file:

    Conan

CMake Error at CMakeLists.txt:12 (include):
  include could not find load file:

    BuildType

CMake Error at CMakeLists.txt:13 (include):
  include could not find load file:

    clang-format

CMake Error at CMakeLists.txt:38 (setup_conan):
  Unknown CMake command "setup_conan".

-- Configuring incomplete, errors occurred!
See also "/home/akhailta/work/EDA/Accellera/PySysC-SC/build/CMakeFiles/CMakeOutput.log".

Is there a simple test I can do to see if the PySysC build is successful?

wlav commented 3 years ago

No idea. :) I've never used PySysC-SC ... (I'm author of cppyy, which is why its mention in a bug report drew me here.)

My assumption is that these packages need to either come from the system packager or from pip. Anyway, if I follow the link you provide above and read the README.md, it states:

pip install conan

and a quick search on pypi.org shows that there is also a package called clang-format. As for BuildType, I don't know ...

amal-khailtash commented 3 years ago

I am not sure if this is ready for prime-time use. It seems instructions are lacking or not complete. As part of the setup.py, it already installs conan and I am not sure what it is used for. Thanks for looking into this though.

    install_requires=[
        'cppyy',
        'conan'
        ],
amal-khailtash commented 3 years ago

Almost done compiling, but I still get this when running the example:

$ python router_example.py 
Traceback (most recent call last):
...
    raise RuntimeError("could not load cppyy_backend library, details:\n%s" %
RuntimeError: could not load cppyy_backend library, details:
  .venv/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.so: undefined symbol: _ZN11CppyyLegacy10TClassEdit9ShortTypeEPKci
  libcppyy_backend.so: cannot open shared object file: No such file or directory
  libcppyy_backend.cpython-39-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
  .venv/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.cpython-39-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

These are installed:

$ pip list | grep cppyy
cppyy              2.1.0
cppyy-backend      1.14.6
cppyy-cling        6.25.1

Not sure what is missing for the cppyy-backend. The shared libraries seem to be in order:

$ LD_LIBRARY_PATH=.venv/lib/python3.9/site-packages/cppyy_backend/lib ldd .venv/lib/python3.9/site-packages/cppyy_backend/lib/*.so

.venv/lib/python3.9/site-packages/cppyy_backend/lib/libCling.so:
    linux-vdso.so.1 =>  (0x00007ffdb9b65000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f6a8e3b8000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f6a8e1b4000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6a8df98000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f6a8dc90000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f6a8d98e000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6a8d778000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f6a8d3aa000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6a9276b000)

.venv/lib/python3.9/site-packages/cppyy_backend/lib/libCoreLegacy.so:
    linux-vdso.so.1 =>  (0x00007ffd7bbe5000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fa941674000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa941458000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fa941150000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fa940e4e000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa940c38000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fa94086a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa941b0b000)

.venv/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.so:
    linux-vdso.so.1 =>  (0x00007ffe2daf8000)
    libstdc++.so.6 => python/3.9.6/lib/libstdc++.so.6 (0x00007f9db5675000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f9db52e3000)
    libgcc_s.so.1 => python/3.9.6/lib/libgcc_s.so.1 (0x00007f9db565f000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9db50c7000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f9db4cf9000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9db5625000)

.venv/lib/python3.9/site-packages/cppyy_backend/lib/libRIOLegacy.so:
    linux-vdso.so.1 =>  (0x00007ffe48d5d000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fee6fec5000)
    libThreadLegacy.so => .venv/lib/python3.9/site-packages/cppyy_backend/lib/libThreadLegacy.so (0x00007fee7055c000)
    libCoreLegacy.so => .venv/lib/python3.9/site-packages/cppyy_backend/lib/libCoreLegacy.so (0x00007fee6fc32000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fee6fa16000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fee6f70e000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fee6f40c000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fee6f1f6000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fee6ee28000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fee703c4000)

.venv/lib/python3.9/site-packages/cppyy_backend/lib/libThreadLegacy.so:
    linux-vdso.so.1 =>  (0x00007ffd284f7000)
    libCoreLegacy.so => .venv/lib/python3.9/site-packages/cppyy_backend/lib/libCoreLegacy.so (0x00007efd7f9e2000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efd7f7c6000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007efd7f4be000)
    libm.so.6 => /lib64/libm.so.6 (0x00007efd7f1bc000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007efd7efa6000)
    libc.so.6 => /lib64/libc.so.6 (0x00007efd7ebd8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007efd7fc75000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007efd7e9d4000)
amal-khailtash commented 3 years ago

Using gcc above 5.0 (8.3.0) and doing the following solved this issue:

$ python -m pip install CPyCppyy cppyy-backend --no-cache-dir --upgrade --force-reinstall

But now to a different problem:

$ python router_example.py 
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): center.conan.io:443
DEBUG:urllib3.connectionpool:https://center.conan.io:443 "GET /v1/ping HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:https://center.conan.io:443 "GET /v1/conans/SystemC/2.3.3/minres/stable/download_urls HTTP/1.1" 404 74
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): artifactory.minres.com:443
DEBUG:urllib3.connectionpool:https://artifactory.minres.com:443 "GET /artifactory/api/conan/oss/v1/ping HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:https://artifactory.minres.com:443 "GET /artifactory/api/conan/oss/v1/conans/SystemC/2.3.3/minres/stable/download_urls HTTP/1.1" 404 74
...
...
    raise NotFoundException(msg)
conans.errors.NotFoundException: Unable to find 'SystemC/2.3.3@minres/stable' in remotes
amal-khailtash commented 3 years ago

I think this is related to the conanfile.txt or artifactory. I am not sure if the artifacts exist in the artifactory.

Can I not point to a previously installed systemc location?

eyck commented 2 years ago

Actually the master branch of https://git.minres.com/SystemC/PySysC-SC/ is out-date with the retirement of the free bintray service. You should try https://git.minres.com/SystemC/PySysC-SC/src/branch/develop. The develop branch with the respective changes has been merged to master.

This also honors now the SYSTEMC_HOME setting