search

wlav / cppyy

Other
400 stars 41 forks source link

cppyy fails to load on manylinux2014 docker image #70

Closed N-Coder closed 2 years ago

N-Coder commented 2 years ago

I tried to install cppyy within the official manylinux docker image, but unfortunately the library fails to load afterwards:

$ docker run -ti --rm quay.io/pypa/manylinux2014_x86_64:2022-06-13-c365205 /bin/bash -c '/opt/python/cp310-cp310/bin/pip install cppyy; /opt/python/cp310-cp310/bin/python -c "import cppyy; cppyy.gbl.gSystem.ListLibraries()"'
Collecting cppyy
  Downloading cppyy-2.4.0.tar.gz (24 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cppyy-cling==6.27.0
  Using cached cppyy_cling-6.27.0-py2.py3-none-manylinux2014_x86_64.whl (32.4 MB)
Collecting cppyy-backend==1.14.9
  Using cached cppyy_backend-1.14.9-py2.py3-none-linux_x86_64.whl
Collecting CPyCppyy==1.12.11
  Downloading CPyCppyy-1.12.11.tar.gz (203 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 203.9/203.9 kB 9.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: cppyy, CPyCppyy
  Building wheel for cppyy (pyproject.toml) ... done
  Created wheel for cppyy: filename=cppyy-2.4.0-py3-none-any.whl size=14634386 sha256=839b95db18a36315d767a5e988fef740cae4215252b1d2e00e6278120b025b51
  Stored in directory: /root/.cache/pip/wheels/53/49/3e/00ba114678187aa83f5e55d20dcdd9814f43d13190c5ba74ec
  Building wheel for CPyCppyy (pyproject.toml) ... done
  Created wheel for CPyCppyy: filename=CPyCppyy-1.12.11-cp310-cp310-linux_x86_64.whl size=3565114 sha256=ec561f6f7dd4100c9ad3a0506f546fc245f8650eaa878d4c5593f789ea33177f
  Stored in directory: /root/.cache/pip/wheels/bf/79/13/b298372e5d63a582d489b5460a519423e4f5da3d0efcf7fe83
Successfully built cppyy CPyCppyy
Installing collected packages: cppyy-cling, cppyy-backend, CPyCppyy, cppyy
Successfully installed CPyCppyy-1.12.11 cppyy-2.4.0 cppyy-backend-1.14.9 cppyy-cling-6.27.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/python/cp310-cp310/lib/python3.10/site-packages/cppyy/__init__.py", line 80, in <module>
    from ._cpython_cppyy import *
  File "/opt/python/cp310-cp310/lib/python3.10/site-packages/cppyy/_cpython_cppyy.py", line 20, in <module>
    c = loader.load_cpp_backend()
  File "/opt/python/cp310-cp310/lib/python3.10/site-packages/cppyy_backend/loader.py", line 92, in load_cpp_backend
    raise RuntimeError("could not load cppyy_backend library, details:\n%s" %
RuntimeError: could not load cppyy_backend library, details:
  /opt/python/cp310-cp310/lib/python3.10/site-packages/cppyy_backend/lib/libcppyy_backend.so: undefined symbol: _ZN11CppyyLegacy10TClassEdit9CleanTypeEPKciPS2_
  libcppyy_backend.cpython-310-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
  /opt/python/cp310-cp310/lib/python3.10/site-packages/cppyy_backend/lib/libcppyy_backend.cpython-310-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
  libcppyy_backend.so: cannot open shared object file: No such file or directory
wlav commented 2 years ago

Is b/c that docker image comes with a g++ that builds for the old version ABI (pre gcc 5.1) by default. Not sure why; probably b/c this is fine if no C++ headers are ever exposed.

Certainly adding --no-binary=cppyy-cling to the pip install will fix the install, but will also take a lot more time.

Alternatively, adding -D_GLIBCXX_USE_CXX11_ABI=1 to the build should work as well, but I'm not sure how to do that w/o patching up the setup.py files. Perhaps it suffices to add it to cling-config. In theory, I could also build wheels that enable both ABIs to post on PyPI. Just that I've never tried that and "in the real world" use of pre 5.1 gcc is pretty minimal.

N-Coder commented 2 years ago

I tried it with the more recent manylinux_2_24 and got the following error. Everything works using manylinux_2_28.

$ docker run -ti --rm quay.io/pypa/manylinux_2_24_x86_64:latest /bin/bash -c '/opt/python/cp310-cp310/bin/pip install cppyy; /opt/python/cp310-cp310/bin/python -c "import cppyy; cppyy.gbl.gSystem.ListLibraries()"'
Trying to pull quay.io/pypa/manylinux_2_24_x86_64:latest...
[...]
Collecting cppyy
  Downloading cppyy-2.4.0.tar.gz (24 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> [44 lines of output]
      Collecting cppyy-cling==6.27.0
        Downloading cppyy_cling-6.27.0-py2.py3-none-manylinux2014_x86_64.whl (32.4 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 32.4/32.4 MB 38.0 MB/s eta 0:00:00
      Collecting cppyy-backend==1.14.9
        Downloading cppyy-backend-1.14.9.tar.gz (34 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Building wheels for collected packages: cppyy-backend
        Building wheel for cppyy-backend (pyproject.toml): started
        Building wheel for cppyy-backend (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error

        × Building wheel for cppyy-backend (pyproject.toml) did not run successfully.
        │ exit code: 1
        ╰─> [19 lines of output]
            /tmp/pip-build-env-kw8mbth7/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
              warnings.warn(msg, warning_class)
            running bdist_wheel
            running build
            running build_ext
            checking for build/temp.linux-x86_64-cpython-310
            creating build/temp.linux-x86_64-cpython-310
            creating build/temp.linux-x86_64-cpython-310/src
            gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-build-env-kw8mbth7/overlay/lib/python3.10/site-packages/cppyy_backend/include -I/opt/_internal/cpython-3.10.5/include/python3.10 -c src/clingwrapper.cxx -o build/temp.linux-x86_64-cpython-310/src/clingwrapper.o -O2 -pthread -std=c++1z -m64
            In file included from /tmp/pip-build-env-kw8mbth7/overlay/lib/python3.10/site-packages/cppyy_backend/include/TString.h:28:0,
                             from /tmp/pip-build-env-kw8mbth7/overlay/lib/python3.10/site-packages/cppyy_backend/include/TNamed.h:25,
                             from /tmp/pip-build-env-kw8mbth7/overlay/lib/python3.10/site-packages/cppyy_backend/include/TDictionary.h:41,
                             from /tmp/pip-build-env-kw8mbth7/overlay/lib/python3.10/site-packages/cppyy_backend/include/TBaseClass.h:23,
                             from src/clingwrapper.cxx:14:
            /tmp/pip-build-env-kw8mbth7/overlay/lib/python3.10/site-packages/cppyy_backend/include/ROOT/RStringView.hxx:19:23: fatal error: string_view: No such file or directory
             #include <string_view>
                                   ^
            compilation terminated.
            error: command '/usr/bin/gcc' failed with exit code 1
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for cppyy-backend
      Failed to build cppyy-backend
      ERROR: Could not build wheels for cppyy-backend, which is required to install pyproject.toml-based projects
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cppyy'

So that might be due to somehow compiling for a C++ stdlib version less than 17 although -std=c++1z (which is a synonym for 17) is passed?

N-Coder commented 2 years ago

Note: I'm not fixed on the manylinux2014 image - it's just the oldest still supported one and thus the default for cibuildwheel. It corresponds to glibc 2.17, but it seems that all the major distributions now support a glibc greater than 2.24 anyways. Thus, supporting at least manylinux_2_24 (and thus having the second bug fixed) would be nice.

wlav commented 2 years ago

What version of gcc is in that wheel? (I was under the impression that current manylinux's all have gcc > 7.3, but I guess I'm wrong there.) Anyway, it's the picking up of the binary for cppyy-cling that make it default to C++17. You can still control which version of C++ is used by setting the STDCXX envar (see: https://cppyy.readthedocs.io/en/latest/installation.html#install-from-source). But also here, if building from source (using --no-binary=cppyy-cling), then it should do the right thing by ratcheting down STDCXX to the best the compiler can support.

N-Coder commented 2 years ago

Ah, the Debian 9 used in manylinux_2_24 is using gcc 6.3 (which I could also verify in the container). Screenshot 2022-07-23 at 10-18-00 DistroWatch com Debian But anyways Debian 9 LTS has ended by now, so that distro is also not really supported anymore - I don't know why manylinux still keeps it around. So this issue probably doesn't really need fixing as we can simply wait until manylinux deprecation catches up with the distro lifetimes (which is probably a lot more important as that defines which systems are / should be actually in use). If anybody runs into this problem again, this issue should be here to document that images < manylinux_2_28 are not really supported (at least not without a self-built cppyy-cling), which is probably also easier to find than a note in the docs.