eclipse-cyclonedds / cyclonedds-python

Other
54 stars 44 forks source link

Code portability issue, can't locate libraries on aarch64 #212

Open zatevakhin opened 11 months ago

zatevakhin commented 11 months ago

It seems there might be some challenges with the portability of the code.

Please refer to buildhelp/cyclone_search.py#L48-L52 and buildhelp/oxidize_library.py#L49-L53 for reference. Distro: Docker container with Debian Bookworm.

List of performed actions to reproduce:

cyclonedds-doc/stable 0.10.2-2 all Eclipse Cyclone DDS library - API documentation

cyclonedds-tools/stable 0.10.2-2 arm64 Eclipse Cyclone DDS tools

libcycloneddsidl0/stable 0.10.2-2 arm64 Eclipse Cyclone DDS IDL library


- Install CycloneDDS-dev

root@ababded08176:/# apt install -qq -y cyclonedds-dev The following additional packages will be installed: cyclonedds-tools libcycloneddsidl0 libddsc0debian libiceoryx-binding-c-dev libiceoryx-binding-c2 libiceoryx-hoofs-dev libiceoryx-hoofs2 libiceoryx-platform2 libiceoryx-posh-config2 libiceoryx-posh-dev libiceoryx-posh-gateway2 libiceoryx-posh-roudi2 libiceoryx-posh2 ... Unpacking cyclonedds-tools (0.10.2-2) ... Setting up libcycloneddsidl0:arm64 (0.10.2-2) ... Setting up libiceoryx-platform2:arm64 (2.0.3+dfsg-1) ... Setting up libiceoryx-hoofs2:arm64 (2.0.3+dfsg-1) ... Setting up libiceoryx-hoofs-dev:arm64 (2.0.3+dfsg-1) ... Setting up libiceoryx-posh2:arm64 (2.0.3+dfsg-1) ... Setting up libiceoryx-posh-roudi2:arm64 (2.0.3+dfsg-1) ... Setting up libiceoryx-posh-config2:arm64 (2.0.3+dfsg-1) ... Setting up libiceoryx-posh-gateway2:arm64 (2.0.3+dfsg-1) ... Setting up libiceoryx-binding-c2:arm64 (2.0.3+dfsg-1) ... Setting up libddsc0debian:arm64 (0.10.2-2) ... Setting up cyclonedds-tools (0.10.2-2) ... Setting up libiceoryx-posh-dev:arm64 (2.0.3+dfsg-1) ... Setting up libiceoryx-binding-c-dev:arm64 (2.0.3+dfsg-1) ... Setting up cyclonedds-dev:arm64 (0.10.2-2) ... Processing triggers for libc-bin (2.36-9+deb12u1) ...


- Required libraries are already present in the system.

root@ababded08176:/# ls /lib/aarch64-linux-gnu/ | grep cyclonedds libcycloneddsidl.so libcycloneddsidl.so.0 libcycloneddsidl.so.0.10.2 root@ababded08176:/#


- Install CycloneDDS Python package.

root@ababded08176:/# pip install --break-system-packages cyclonedds Collecting cyclonedds Downloading cyclonedds-0.10.2.tar.gz (156 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.8/156.8 kB 696.0 kB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [1 lines of output] Could not locate cyclonedds. Try to set CYCLONEDDS_HOME or CMAKE_PREFIX_PATH [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.


- Install CycloneDDS Python package (verbose mode).

root@ababded08176:/# pip install -vvv --break-system-packages cyclonedds Using pip 23.0.1 from /usr/lib/python3/dist-packages/pip (python 3.11) Non-user install because site-packages writeable Created temporary directory: /tmp/pip-build-tracker-faskk29k Initialized build tracking at /tmp/pip-build-tracker-faskk29k Created build tracker: /tmp/pip-build-tracker-faskk29k Entered build tracker: /tmp/pip-build-tracker-faskk29k Created temporary directory: /tmp/pip-install-hjq_sjib Created temporary directory: /tmp/pip-ephem-wheel-cache-q4h_v101 1 location(s) to search for versions of cyclonedds:

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. Exception information: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper status = run_func(*args) ^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/cli/req_command.py", line 247, in wrapper return func(self, options, args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 419, in run requirement_set = resolver.resolve( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve result = self._result = resolver.resolve( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve state = resolution.resolve(requirements, max_rounds=max_rounds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve self._add_to_criteria(self.state.criteria, r, parent=None) File "/usr/lib/python3/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria if not criterion.candidates: File "/usr/lib/python3/dist-packages/pip/_vendor/resolvelib/structs.py", line 151, in bool return bool(self._sequence) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in bool return any(self) ^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in return (c for c in iterator if id(c) not in self._incompatible_ids) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built candidate = func() ^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link self._link_candidate_cache[link] = LinkCandidate( ^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in init super().init( File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in init self.dist = self._prepare() ^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare dist = self._prepare_distribution() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement return self._prepare_linked_requirement(req, parallel_builds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/operations/prepare.py", line 577, in _prepare_linked_requirement dist = _get_prepared_distribution( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution abstract_dist.prepare_distribution_metadata( File "/usr/lib/python3/dist-packages/pip/_internal/distributions/sdist.py", line 48, in prepare_distribution_metadata self._install_build_reqs(finder) File "/usr/lib/python3/dist-packages/pip/_internal/distributions/sdist.py", line 118, in _install_build_reqs build_reqs = self._get_build_requires_wheel() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/distributions/sdist.py", line 95, in _get_build_requires_wheel return backend.get_requires_for_build_wheel() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_internal/utils/misc.py", line 725, in get_requires_for_build_wheel return super().get_requires_for_build_wheel(config_settings=cs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel return self._call_hook('get_requires_for_build_wheel', { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_impl.py", line 311, in _call_hook self._subprocess_runner( File "/usr/lib/python3/dist-packages/pip/_internal/utils/subprocess.py", line 252, in runner call_subprocess( File "/usr/lib/python3/dist-packages/pip/_internal/utils/subprocess.py", line 224, in call_subprocess raise error pip._internal.exceptions.InstallationSubprocessError: Getting requirements to build wheel exited with 1 Removed cyclonedds from https://files.pythonhosted.org/packages/62/88/4b440f5916976234838989c3214222a5abc6fd48009a992c4dc22a86c04f/cyclonedds-0.10.2.tar.gz from build tracker '/tmp/pip-build-tracker-faskk29k' Removed build tracker: '/tmp/pip-build-tracker-faskk29k'

eboasson commented 11 months ago

Yes, I agree, it definitely looks like that doesn't allow for the installation paths used by apt install cyclonedds-dev. I suppose the list of locations it checks could be extended fairly easily. However, with all the platform-specific bits in the directory names it is always going to be dicey if we don't figure out the "proper" and portable way to do so.

I suspect you know your way around python setup scripts better than I do[^1], is there any chance you know what the proper way here would be? Otherwise I'll try my hand at it, but that'll take some time ...

If you can't wait, I'd recommend building Cyclone from source, all you need are cmake and build-essentials and it builds in a few seconds. You can take either the heads of the 0.10.x branches for cyclonedds, cyclonedds-cxx (if you want it) and cyclonedds-python, or the heads of master (that would be my personal choice, I have no indication the master suffers from development-related issues).

[^1]: That is almost a given: I never really used Python. That we have such a nice binding is entirely thanks to @thijsmie and I'd hate to go rummaging around these bits and probably causing trouble by trying to fix this the wrong way.

zatevakhin commented 11 months ago

@eboasson Thank you for your answer! Install from sources cyclonedds-python#via-pypi, and cyclonedds-python#via-git works well for me.

Btw. Got the same problem when trying to build in debian:bookworm x86_64, when trying to force library binaries to build from the source.

root@f3a9b81c55cc:/cyclonedds-python# uname -a
Linux f3a9b81c55cc 6.4.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 03 Aug 2023 16:02:01 +0000 x86_64 GNU/Linux

root@f3a9b81c55cc:/cyclonedds-python# pip3 install --use-feature=no-binary-enable-wheel-cache --break-system-packages cyclonedds 
Collecting cyclonedds
  Using cached cyclonedds-0.10.2.tar.gz (156 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      Could not locate cyclonedds. Try to set CYCLONEDDS_HOME or CMAKE_PREFIX_PATH
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Can make more verbose output if needed, but it looks similar to verbose output in my previous comment .

This issue basically can be closed, but it is nice to have a way how to install cyclonedds package on aarch64, without building it from the source.

I can create another issue in which we can continue the discussion about how we can make aarch64 packages (whl) available for cyclonedds, because there are no releases for those packages in PyPi (https://pypi.org/project/cyclonedds/#files).

zatevakhin commented 11 months ago

@eboasson Btw. I implemented fixes for buildhelp/cyclone_search.py#L48-L52 and buildhelp/oxidize_library.py#L49-L53, but because installation of cyclonedds from apt (apt install cyclonedds-dev) provides only libraries without header files there's missing some includes when I trying to build it on aarch64 or x86_64, using pip.

Think there should be a proper way to provide at least source distribution on PyPi (https://pypi.org/project/cyclonedds/#files) which can be built regardless of the system.