BUG: Fails to build on termux

narodnik commented 1 month ago

This is a pyzmq bug

[X] This is a pyzmq-specific bug, not an issue of zmq socket behavior. Don't worry if you're not sure! We'll figure it out together.

What pyzmq version?

master

What libzmq version?

master

Python version (and how it was installed)

3.11

OS

termux/android

What happened?

When I try to install via pip, it spews out cmake related errors.

~ $ pip install zmq
Collecting zmq
  Using cached zmq-0.0.0.zip (2.2 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyzmq (from zmq)
  Using cached pyzmq-26.0.3.tar.gz (267 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> [88 lines of output]
      Collecting cmake>=3.14
        Using cached cmake-3.29.3.tar.gz (30 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'error'
        error: subprocess-exited-with-error

        × pip subprocess to install backend dependencies did not run successfully.
        │ exit code: 2
        ╰─> [65 lines of output]
            Collecting cmake
              Using cached cmake-3.29.3.tar.gz (30 kB)
            ERROR: Exception:
            Traceback (most recent call last):
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 169, in exc_logging_wrapper
                status = run_func(*args)
                         ^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
                return func(self, options, args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 391, in run
                requirement_set = resolver.resolve(
                                  ^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
                result = self._result = resolver.resolve(
                                        ^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
                state = resolution.resolve(requirements, max_rounds=max_rounds)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
                self._add_to_criteria(self.state.criteria, r, parent=None)
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
                if not criterion.candidates:
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
                return bool(self._sequence)
                       ^^^^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
                return any(self)
                       ^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
                return (c for c in iterator if id(c) not in self._incompatible_ids)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
                candidate = func()
                            ^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
                self._link_candidate_cache[link] = LinkCandidate(
                                                   ^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
                super().__init__(
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
                self.dist = self._prepare()
                            ^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
                dist = self._prepare_distribution()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
                return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
                return self._prepare_linked_requirement(req, parallel_builds)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 631, in _prepare_linked_requirement
                dist = _get_prepared_distribution(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 68, in _get_prepared_distribution
                with build_tracker.track(req):
              File "/data/data/com.termux/files/usr/lib/python3.11/contextlib.py", line 137, in __enter__
                return next(self.gen)
                       ^^^^^^^^^^^^^^
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/build/build_tracker.py", line 122, in track
                self.add(req)
              File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/build/build_tracker.py", line 92, in add
                raise LookupError(message)
            LookupError: https://files.pythonhosted.org/packages/9e/cf/fbd014f36d3407f04fcbd4d216a1431ef05f2ec69a25a2ad3af1de75e835/cmake-3.29.3.tar.gz (from https://pypi.org/simple/cmake/) (requires-python:>=3.7) is already being built: cmake>=3.14 from https://files.pythonhosted.org/packages/9e/cf/fbd014f36d3407f04fcbd4d216a1431ef05f2ec69a25a2ad3af1de75e835/cmake-3.29.3.tar.gz
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: subprocess-exited-with-error

      × pip subprocess to install backend dependencies did not run successfully.
      │ exit code: 2
      ╰─> See above for output.

      note: This error originates from a subprocess, and is likely not a problem with pip.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I tried following the old advice from here, but it seems pyzmq was recently upgraded to cmake infra so it no longer is true.

Code to reproduce bug

No response

Traceback, if applicable

No response

More info

No response

minrk commented 1 month ago

building pyzmq from source requires cmake. Try installing cmake first. scikit-build-core tries to install cmake via a wheel if it's not present, but it doesn't work everywhere.

narodnik commented 1 month ago

Thanks, actually I already did that, but it seems cmake required libxml2 to also be installed. Installing that fixed the current error, but now I get another one:

~ $ pip install zmq
Collecting zmq
  Using cached zmq-0.0.0.zip (2.2 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyzmq (from zmq)
  Using cached pyzmq-26.0.3.tar.gz (267 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: zmq, pyzmq
  Building wheel for zmq (pyproject.toml) ... done
  Created wheel for zmq: filename=zmq-0.0.0-py3-none-any.whl size=1262 sha256=6873696fe49cc5eca00f165831e304a886bfd8a0899c4f33cbd11f0ccb050318
  Stored in directory: /data/data/com.termux/files/home/.cache/pip/wheels/16/9e/c7/d497825227491fa00cca08b88ae37f9bcc14809233db76342c
  Building wheel for pyzmq (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pyzmq (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      *** scikit-build-core 0.9.5 using CMake 3.29.4 (wheel)
      *** Configuring CMake...
      loading initial cache file /data/data/com.termux/files/usr/tmp/tmp_avl48zb/build/CMakeInit.txt
      /home/builder/.termux-build/libuv/src/src/unix/process.c:957: assertion "!(options->flags & ~(UV_PROCESS_DETACHED | UV_PROCESS_SETGID | UV_PROCESS_SETUID | UV_PROCESS_WINDOWS_HIDE | UV_PROCESS_WINDOWS_HIDE_CONSOLE | UV_PROCESS_WINDOWS_HIDE_GUI | UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS))" failed

      *** CMake configuration failed
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyzmq
Successfully built zmq
Failed to build pyzmq
ERROR: Could not build wheels for pyzmq, which is required to install pyproject.toml-based projects

minrk commented 1 month ago

Hm, it looks like the cmake wheel built, but it doesn't work. Can you install cmake with your package manager and check the version? I don't know anything about termux.

narodnik commented 1 month ago

I already installed cmake (and libxml2) otherwise we get the first error. cmake version is 3.29.4

It seems the cmake error is related to process flags.

minrk commented 1 month ago

Something's fishy, since it says

scikit-build-core 0.9.5 using CMake 3.29.4 (wheel)

which suggests scikit-build-core is not finding an acceptable cmake, and instead using the cmake wheel, which appears to not be built properly. What's are which -a cmake and cmake --version in the shell before you call pip install?

Also note, the package is not called zmq, it's pyzmq. zmq is a typo-squatting package that only depends on pyzmq.

narodnik commented 1 month ago

which shows the path as /data/data/com.termux/files/usr/bin/cmake. The cmake version is 3.29.4.

I tried building the docker image, and got a pyzmq wheel file for aarch64. Installed that, but it couldn't find libzmq.so which is in /data/data/com.termux/files/usr/lib, so then I set LD_LIBRARY_PATH. Now when trying to import zmq, I get ImportError: dlopen failed: cannot locate symbol "PyExc_ImportError" referenced by "/data/data/com.termux/files/usr/lib/python3.11/site-packages/zmq/backend/cython/_zmq.cpython-311.so"...

I guess it's not to be... 😁

zeromq / pyzmq