Build dependencies not found when using Pip >= 21.3 with Python 3.10

[velovix]

When attempting to install Zivid via Pip on Ubuntu 22.04, the build does not find Ninja even though build dependencies appear to have been installed.

It seems the issue only happens when using Pip >= 21.3 and Python 3.10 at the same time. Using later versions of Pip with Python 3.8 or Python 3.10 with earlier versions of Pip works fine. This leads me to believe that the issue is related to this change in Pip v21.3:

On Python 3.10 or later, the installation scheme backend has been changed to use sysconfig. This is to anticipate the deprecation of distutils in Python 3.10, and its scheduled removal in 3.12. For compatibility considerations, pip installations running on Python 3.9 or lower will continue to use distutils. (#10358)

Pulled from their changelist here

  Downloading zivid-2.1.2.2.4.0.tar.gz (46 kB)
     |████████████████████████████████| 46 kB 1.2 MB/s             
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/local/lib/python3.10/dist-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqgs2h86j
       cwd: /tmp/pip-install-tg5v9xt3/zivid_6f1ec3e7dcdd4a97b39b08c7f8ed3791
  Complete output (54 lines):
  cmake version 3.22.1

  CMake suite maintained and supported by Kitware (kitware.com/cmake).
  CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
  CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
  -- Configuring incomplete, errors occurred!
  See also "/tmp/zivid-python-build-gdriord4/CMakeFiles/CMakeOutput.log".
  Traceback (most recent call last):
    File "setup.py", line 73, in run_process
      raise RuntimeError("Wait failed with exit code {}".format(exit_code))
  RuntimeError: Wait failed with exit code 1

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "setup.py", line 94, in _check_cpp17_compiler
      run_process(("cmake", "-GNinja", "."), cwd=temp_dir)
    File "setup.py", line 75, in run_process
      raise type(ex)("Process failed: '{}'.".format(" ".join(args))) from ex
  RuntimeError: Process failed: 'cmake -GNinja .'.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 143, in _get_build_requires
      self.run_setup()
    File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 267, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 148, in <module>
      _main()
    File "setup.py", line 118, in _main
      _check_cpp17_compiler()
    File "setup.py", line 99, in _check_cpp17_compiler
      raise RuntimeError(
  RuntimeError:

  +-----------------------------------------------------------+
  | !! Module setup failed !!                                 |
  |                                                           |
  | Make sure you have a working C++17 compiler installed     |
  |                                                           |
  | Refer to Readme.md for detailed installation instructions |
  +-----------------------------------------------------------+

  ----------------------------------------

[eskaur]

Thanks for reporting. To be clear: It does work for you in the end, if you use a non-default version of pip?

[velovix]

Yes, that's right! No issues after a downgrade.

pip3 install --upgrade pip==21.2.4

[eskaur]

We will look into this soon once we start officially supporting Ubuntu 22.04. For now I hope you can life with the pip-version workaround.

[eskaur]

I was recently able to just do pip install zivid on a fresh Ubuntu 22.04 with the latest pip, so this problem may have been resolved. Could you perhaps check on your system, @velovix ?

[velovix]

@eskaur After doing some testing on my host machine (which is also Ubuntu 22.04), I noticed that the build dependencies resolve fine only when I use a virtual environment. We don't use virtual environments since we deploy in Docker, so maybe that's why we're seeing different things.

Here's a minimal Dockerfile I made to demonstrate the problem:

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y python3-pip python3-dev g++ cmake curl ca-certificates
RUN pip3 install --upgrade pip==22.2.2

WORKDIR /zivid
RUN update-ca-certificates && \
    curl -sSL https://www.zivid.com/hubfs/softwarefiles/releases/2.7.0+e31dcbe2-1/u20/zivid-telicam-driver_3.0.1.1-3_amd64.deb --output zivid-telicam-driver.deb && \
    curl -sSL https://www.zivid.com/hubfs/softwarefiles/releases/2.7.0+e31dcbe2-1/u20/zivid_2.7.0+e31dcbe2-1_amd64.deb --output zivid.deb && \
    curl -sSL https://www.zivid.com/hubfs/softwarefiles/releases/2.7.0+e31dcbe2-1/u20/zivid-tools_2.7.0+e31dcbe2-1_amd64.deb --output zivid-tools.deb
RUN apt install -y ./zivid.deb ./zivid-tools.deb ./zivid-telicam-driver.deb

RUN pip3 install zivid

[eskaur]

Thanks for the minimum reproducing example. :+1: We will look into it.

[apockill]

Hi @eskaur I was wondering if there's been any progress on this? We're still successfully able to use the latest version of the SDK if we use pip 21.2.4, but eventually using older pip versions might become a blocker for installing newer libraries.

[eskaur]

Hi @apockill !

The latest version of ZIvid SDK (2.8) was the first one to officially support Ubuntu 22.04. Unfortunately the python package is lagging a bit behind, but it is on our short-term roadmap to also officially support Ubuntu 22.04 with the python package. As part of that process I am confident that this issue will be resolved.

In the meantime I think using virtual environment is a decent workaround. Using venv is good practice anyway :)

[SebastianGrans]

@apockill An alternative workaround that doesn't require downgrading pip is to add the following to your Dockerfile.

RUN apt-get install cmake ninja-build --assume-yes
RUN pip install conan

[eskaur]

Update: We recently added Ubuntu 22.04 to the zivid-python CI: https://github.com/zivid/zivid-python/pull/206 . The build and tests happen in a Ubuntu 22.04 Docker container with no virtual environment, and it works fine. I suspect something got improved in pip so that it now works.

FYI @apockill and @velovix

[velovix]

Thanks for the heads up @eskaur. I can confirm that it works with pip==23.0.1 and zivid==2.7.1.2.8.1!