search

rticommunity / connextdds-py

Connext DDS Python API
Other
27 stars 12 forks source link

Building on Debian 12 x86_64 with Python 3.11.2 fails #53

Open kruegerrobotics opened 1 year ago

kruegerrobotics commented 1 year ago

There seems to an issue with pybind11 and/or PyFrameObject.

Following the instructions clone (main branch) and python3 configure.py x64Linux4gcc7.3.0 execute successfully.

But when executing 

pip wheel .

the result is the following: (maybe it could be related to (https://github.com/sirfz/tesserocr/issues/298)[Python3.11 support #298]

Processing /home/tkrueger/projects/connextdds-py
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: rti.connext
  Building wheel for rti.connext (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for rti.connext (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [152 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/rti
      copying rti_pkg/__init__.py -> build/lib.linux-x86_64-cpython-311/rti
      copying rti_pkg/idl.py -> build/lib.linux-x86_64-cpython-311/rti
      copying rti_pkg/asyncio.py -> build/lib.linux-x86_64-cpython-311/rti
      creating build/lib.linux-x86_64-cpython-311/rti/logging
      copying rti_pkg/logging/__init__.py -> build/lib.linux-x86_64-cpython-311/rti/logging
      copying rti_pkg/logging/handler.py -> build/lib.linux-x86_64-cpython-311/rti/logging
      creating build/lib.linux-x86_64-cpython-311/rti/request
      copying rti_pkg/request/_util_async.py -> build/lib.linux-x86_64-cpython-311/rti/request
      copying rti_pkg/request/_async.py -> build/lib.linux-x86_64-cpython-311/rti/request
      copying rti_pkg/request/__init__.py -> build/lib.linux-x86_64-cpython-311/rti/request
      copying rti_pkg/request/_util.py -> build/lib.linux-x86_64-cpython-311/rti/request
      copying rti_pkg/request/_basic.py -> build/lib.linux-x86_64-cpython-311/rti/request
      creating build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/unions.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/__init__.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/decorators.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/csequence.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/type_plugin.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/type_hints.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/sample_interpreter.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/type_utils.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/annotations.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/test_utils.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      copying rti_pkg/idl_impl/reflection_utils.py -> build/lib.linux-x86_64-cpython-311/rti/idl_impl
      creating build/lib.linux-x86_64-cpython-311/rti/types
      copying rti_pkg/types/builtin.py -> build/lib.linux-x86_64-cpython-311/rti/types
      copying rti_pkg/types/__init__.py -> build/lib.linux-x86_64-cpython-311/rti/types
      copying rti_pkg/types/type_registry.py -> build/lib.linux-x86_64-cpython-311/rti/types
      running egg_info
      writing rti.connext.egg-info/PKG-INFO
      writing dependency_links to rti.connext.egg-info/dependency_links.txt
      writing requirements to rti.connext.egg-info/requires.txt
      writing top-level names to rti.connext.egg-info/top_level.txt
      reading manifest file 'rti.connext.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file 'rti.connext.egg-info/SOURCES.txt'
      running build_ext
      -- The C compiler identification is GNU 12.2.0
      -- The CXX compiler identification is GNU 12.2.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Performing Test HAVE_FLAG_STD_CXX17
      -- Performing Test HAVE_FLAG_STD_CXX17 - Success
      -- Performing Test HAVE_FLAG_STD_CXX14
      -- Performing Test HAVE_FLAG_STD_CXX14 - Success
      -- RTI Connext DDS installation directory: /opt/rti_connext_dds
      -- RTI Connext DDS architecture: x64Linux4gcc7.3.0
      -- Found RTIConnextDDS: /opt/rti_connext_dds (found suitable version "7.1.0.0", minimum required is "7.1.0") found components: core
      -- Found Python3: /usr/bin/python3.11 (found suitable exact version "3.11.2") found components: Interpreter Development.Module
      -- Performing Test HAS_FLTO
      -- Performing Test HAS_FLTO - Success
      -- Found pybind11: /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include (found version "2.9.0")
      -- RTI Connext DDS installation directory: /opt/rti_connext_dds
      -- RTI Connext DDS architecture: x64Linux4gcc7.3.0
      -- Found RTIConnextDDS: /opt/rti_connext_dds (found suitable version "7.1.0.0", minimum required is "7.1.0") found components: core distributed_logger
      -- Found Python3: /usr/bin/python3.11 (found suitable exact version "3.11.2") found components: Interpreter Development.Module
      -- Found pybind11: /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include (found version "2.9.0")
      -- RTI Connext DDS installation directory: /opt/rti_connext_dds
      -- RTI Connext DDS architecture: x64Linux4gcc7.3.0
      -- Found RTIConnextDDS: /opt/rti_connext_dds (found suitable version "7.1.0.0", minimum required is "7.1.0") found components: core messaging_api
      -- Found pybind11: /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include (found version "2.9.0")
      -- Configuring done (5.7s)
      -- Generating done (0.1s)
      -- Build files have been written to: /home/tkrueger/projects/connextdds-py/build/temp.linux-x86_64-cpython-311/connext-py
      [  1%] Building CXX object connextdds/CMakeFiles/connextdds.dir/src/misc/Constants.cpp.o
      In file included from /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include/pybind11/cast.h:16,
                       from /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include/pybind11/attr.h:13,
                       from /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include/pybind11/pybind11.h:13,
                       from /home/tkrueger/projects/connextdds-py/modules/connextdds/src/misc/Constants.cpp:12:
      /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include/pybind11/detail/type_caster_base.h: In function ‘std::string pybind11::detail::error_string()’:
      /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include/pybind11/detail/type_caster_base.h:482:26: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
        482 |             frame = frame->f_back;
            |                          ^~
      In file included from /usr/include/python3.11/Python.h:42,
                       from /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include/pybind11/detail/common.h:209,
                       from /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include/pybind11/pytypes.h:12,
                       from /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include/pybind11/cast.h:13:
      /usr/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
         22 | typedef struct _frame PyFrameObject;
            |                ^~~~~~
      gmake[2]: *** [connextdds/CMakeFiles/connextdds.dir/build.make:76: connextdds/CMakeFiles/connextdds.dir/src/misc/Constants.cpp.o] Error 1
      gmake[1]: *** [CMakeFiles/Makefile2:134: connextdds/CMakeFiles/connextdds.dir/all] Error 2
      gmake: *** [Makefile:91: all] Error 2
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 416, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 401, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 488, in run_setup
          self).run_setup(setup_script=setup_script)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 260, in <module>
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/wheel/bdist_wheel.py", line 343, in run
          self.run_command("build")
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 225, in run
        File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/cmake/data/bin/cmake', '--build', '.', '--config', 'Release', '--parallel', '1']' returned non-zero exit status 2.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for rti.connext
Failed to build rti.connext
ERROR: Failed to build one or more wheels
tb-rti commented 1 year ago

Hi @kruegerrobotics,

The build / installation guide states that Python 3.6 or newer is required to build the Connext DDS Python API. So I'm not sure this would be accepted as an 'Issue'.

Hope this helps.

Cheers, Thijs

kruegerrobotics commented 1 year ago

Thanks, maybe just to cross check. Before on Debian 11 with Python 3.9 all was good. Now Debian 12 with almost the newest Python 3.11.2 fails. My guess it has more to do with the change of the API of pybind11 and thus the compilation fail.

I hope this helps. I basically upgraded connextdds to 7.1.0 and Debian from 11 to 12 and thus Python 3.9 to 3.11.

Kind regards,

tb-rti commented 1 year ago

Oh, sorry, I somehow thought 3.11 was smaller than 3.6... Time for the weekend I guess....

tb-rti commented 1 year ago

Ok, I get the same error when I try to build on debian:latest with docker.

kruegerrobotics commented 1 year ago

I think I have an idea and get it to build

Python 3.11 release notes under PyFrameObject states that functions have been changed.

When executing pip wheel . I see that CMake is getting pybind (strangely from my /tmp dir, not sure why)

-- Found RTIConnextDDS: /opt/rti_connext_dds (found suitable version "7.1.0.0", minimum required is "7.1.0") found components: core distributed_logger
-- Found Python3: /usr/bin/python3.11 (found suitable exact version "3.11.2") found components: Interpreter Development.Module
-- Found pybind11: /tmp/pip-build-env-verwekpf/overlay/local/lib/python3.11/dist-packages/pybind11/include (found version "2.9.0")

As a quick fix:

from: 

find_package(
    pybind11
    REQUIRED
)

to

find_package(
    pybind11 2.10
    REQUIRED
)

I think more checks are required and not ready for a pull request since this might break backward compatibility. Maybe a conditional Python version mechanism for the pybind version - or Why does this create a pybind 2.9.0 in the temp folder?