search

scikit-build / cmake-python-distributions

This project provides the infrastructure to build CMake Python wheels.
https://cmake-python-distributions.readthedocs.io
Apache License 2.0
112 stars 34 forks source link

Passing CMake variables when building Python wheel with pip #172

Closed tpimh closed 3 years ago

tpimh commented 3 years ago

I need to build Python wheel using pip on a machine with no network connectivity. I have downloaded the source archives and using pip3 install cmake.

The libuv-v1.23.0.tar.gz and cmake-3.21.1.tar.gz are at a known path, but I need to tell the setup where to find them. I tried adding --install-option="-- -DCMakePythonDistributions_ARCHIVE_DOWNLOAD_DIR:PATH=/my/known/path", but it didn't seem to do anything. Is there an option to supply CMake variables to setup like this? If there is, I couldn't find any relevant documentation.

Here is the error log in case it contains valuable information:

  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmp_aj3mvrc
       cwd: /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea
  Complete output (314 lines):
  CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
    Compatibility with CMake < 2.8.12 will be removed from a future version of
    CMake.

    Update the VERSION argument <min> value or use a ...<max> suffix to tell
    CMake that the project does not need compatibility with older versions.

  Not searching for unused variables given on the command line.

  -- The C compiler identification is GNU 10.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The CXX compiler identification is GNU 10.2.0
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_cmake_test_compile/build
  -- The CXX compiler identification is GNU 10.2.0
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- ***************************************************
  -- Build CMake from source: ON
  -- ***************************************************
  -- SuperBuild - CMakeProject-src-download
  -- SuperBuild - CMakeProject-src-download - URL: https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz
  -- SuperBuild - CMakeProject-src-download - CMakeProject_SOURCE_DIR: /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/CMake-src
  -- SuperBuild -   CMakeProject-build
  -- SuperBuild -   CMakeProject-build - CMakeProject_BINARY_DIR: /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build
  -- SuperBuild -   CMakePythonDistributions
  -- Configuring done
  -- Generating done
  CMake Warning:
    Manually-specified variables were not used by the project:

      PYTHON_EXECUTABLE
      PYTHON_INCLUDE_DIR
      PYTHON_LIBRARY
      PYTHON_VERSION_STRING
      SKBUILD

  -- Build files have been written to: /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build
  [1/34] Creating directories for 'CMakeProject-src-download'
  [1/34] Performing download step (download, verify and extract) for 'CMakeProject-src-download'
  -- Downloading...
     dst='/tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.21.1.tar.gz'
     timeout='none'
     inactivity timeout='none'
  -- Using src='https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz'
  -- Retrying...
  -- Using src='https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz'
  -- Retry after 5 seconds (attempt #2) ...
  -- Using src='https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz'
  -- Retry after 5 seconds (attempt #3) ...
  -- Using src='https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz'
  -- Retry after 15 seconds (attempt #4) ...
  -- Using src='https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz'
  -- Retry after 60 seconds (attempt #5) ...
  -- Using src='https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz'
  CMake Error at _skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-src-download-prefix/src/CMakeProject-src-download-stamp/download-CMakeProject-src-download.cmake:170 (message):
    Each download failed!

      error: downloading 'https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for github.com:443

    Could not resolve host: github.com

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for github.com:443

    Could not resolve host: github.com

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for github.com:443

    Could not resolve host: github.com

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for github.com:443

    Could not resolve host: github.com

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for github.com:443

    Could not resolve host: github.com

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for github.com:443

    Could not resolve host: github.com

    Closing connection 0

            --- LOG END ---

  [2/34] Creating directories for 'LibUV'
  FAILED: CMakeProject-src-download-prefix/src/CMakeProject-src-download-stamp/CMakeProject-src-download-download /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-src-download-prefix/src/CMakeProject-src-download-stamp/CMakeProject-src-download-download
  cd /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea && /app/bin/cmake -P /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-src-download-prefix/src/CMakeProject-src-download-stamp/download-CMakeProject-src-download.cmake && /app/bin/cmake -P /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-src-download-prefix/src/CMakeProject-src-download-stamp/verify-CMakeProject-src-download.cmake && /app/bin/cmake -P /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-src-download-prefix/src/CMakeProject-src-download-stamp/extract-CMakeProject-src-download.cmake && /app/bin/cmake -E touch /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-src-download-prefix/src/CMakeProject-src-download-stamp/CMakeProject-src-download-download
  [4/34] Performing download step (download, verify and extract) for 'LibUV'
  FAILED: LibUV-prefix/src/LibUV-stamp/LibUV-download /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/LibUV-prefix/src/LibUV-stamp/LibUV-download
  cd /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build && /app/bin/cmake -P /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/LibUV-prefix/src/LibUV-stamp/download-LibUV.cmake && /app/bin/cmake -P /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/LibUV-prefix/src/LibUV-stamp/verify-LibUV.cmake && /app/bin/cmake -P /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/LibUV-prefix/src/LibUV-stamp/extract-LibUV.cmake && /app/bin/cmake -E touch /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/LibUV-prefix/src/LibUV-stamp/LibUV-download
  -- Downloading...
     dst='/tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build/LibUV-prefix/src/libuv-v1.23.0.tar.gz'
     timeout='none'
     inactivity timeout='none'
  -- Using src='https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz'
  -- Retrying...
  -- Using src='https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz'
  -- Retry after 5 seconds (attempt #2) ...
  -- Using src='https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz'
  -- Retry after 5 seconds (attempt #3) ...
  -- Using src='https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz'
  -- Retry after 15 seconds (attempt #4) ...
  -- Using src='https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz'
  -- Retry after 60 seconds (attempt #5) ...
  -- Using src='https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz'
  CMake Error at LibUV-prefix/src/LibUV-stamp/download-LibUV.cmake:170 (message):
    Each download failed!

      error: downloading 'https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for dist.libuv.org:443

    Could not resolve host: dist.libuv.org

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for dist.libuv.org:443

    Could not resolve host: dist.libuv.org

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for dist.libuv.org:443

    Could not resolve host: dist.libuv.org

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for dist.libuv.org:443

    Could not resolve host: dist.libuv.org

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for dist.libuv.org:443

    Could not resolve host: dist.libuv.org

    Closing connection 0

            --- LOG END ---
            error: downloading 'https://dist.libuv.org/dist/v1.23.0/libuv-v1.23.0.tar.gz' failed
            status_code: 6
            status_string: "Couldn't resolve host name"
            log:
            --- LOG BEGIN ---
            getaddrinfo(3) failed for dist.libuv.org:443

    Could not resolve host: dist.libuv.org

    Closing connection 0

            --- LOG END ---

  ninja: build stopped: subcommand failed.
    File "/tmp/pip-build-env-iutpwbmj/overlay/lib/python3.8/site-packages/skbuild/setuptools_wrap.py", line 589, in setup
      cmkr.make(make_args, env=env)
    File "/tmp/pip-build-env-iutpwbmj/overlay/lib/python3.8/site-packages/skbuild/cmaker.py", line 496, in make
      raise SKBuildError(

  --------------------------------------------------------------------------------
  -- Trying "Ninja" generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying "Ninja" generator - success
  --------------------------------------------------------------------------------

  Configuring Project
    Working directory:
      /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build
    Command:
      cmake /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea -G Ninja -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-install/src/cmake/data -DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3 -DPYTHON_VERSION_STRING:STRING=3.8.8 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:FILEPATH=/usr/lib/x86_64-linux-gnu/libpython3.8.so -DSKBUILD:BOOL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-iutpwbmj/overlay/lib/python3.8/site-packages/skbuild/resources/cmake -DCMAKE_BUILD_TYPE:STRING=Release

  Traceback (most recent call last):

  An error occurred while building with CMake.
    Command:
      cmake --build . --target install --config Release --
    Source directory:
      /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea
    Working directory:
      /tmp/pip-install-zq30kt44/cmake_113aba04b77e4e6bba7c7a6e24405aea/_skbuild/linux-x86_64-3.8/cmake-build
  Please see CMake's output for more information.
  ----------------------------------------
  ERROR: Failed building wheel for cmake
mayeut commented 3 years ago

You can use SKBUILD_CONFIGURE_OPTIONS environment variables to pass down options to cmake. I'm not familiar enough with scikit-build to know if there are other ways to do this.

e.g.

SKBUILD_CONFIGURE_OPTIONS="-DCMakePythonDistributions_ARCHIVE_DOWNLOAD_DIR:PATH=/my/known/path" pip wheel -v cmake-3.21.1.tar.gz

however, up-to 3.21.1, CMakePythonDistributions_ARCHIVE_DOWNLOAD_DIR is not used for libuv. This will be fixed by #173

tpimh commented 3 years ago

Thank you very much for your reply! Yesterday, I have built it successfully with this small modification: sed -i '216i DOWNLOAD_DIR ${CMakePythonDistributions_ARCHIVE_DOWNLOAD_DIR}' CMakeLists.txt using this command python3 setup.py bdist_wheel -- -DBUILD_CMAKE_FROM_SOURCE:BOOL=ON -DCMakePythonDistributions_ARCHIVE_DOWNLOAD_DIR:PATH=/my/known/path. I will try to build with pip with your suggested command as well.

mayeut commented 3 years ago

cmake 3.21.1.post1 is now available on PyPI with the fix for libuv respecting CMakePythonDistributions_ARCHIVE_DOWNLOAD_DIR