search

scikit-build / ninja-python-distributions

This project provides the infrastructure to build Ninja Python wheels.
Apache License 2.0
61 stars 16 forks source link

Installation of ninja failed due to access https://github.com/Kitware/ninja #217

Closed luoqishuai closed 10 months ago

luoqishuai commented 1 year ago

My network environment is within the company's internal network and I am unable to access external networks

When I install deepspeed, it needs to install ninja.ninja, which will be accessed during installation https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz , resulting in installation failure

The following is the installation log

If you need any other information, please contact me

[root@ai-app-2-46-msxf build]# DS_BUILD_FUSED_ADAM=1 pip3 --no-cache-dir install deepspeed --global-option="build_ext" --global-option="-j64"
DEPRECATION: --build-option and --global-option are deprecated. pip 23.3 will enforce this behaviour change. A possible replacement is to use --config-settings. Discussion can be found at https://github.com/pypa/pip/issues/11859
WARNING: Implying --no-binary=:all: due to the presence of --build-option / --global-option.
Looking in indexes: http://nexus3.msxf.com/repository/pypi/simple
Collecting deepspeed
 Downloading http://nexus3.msxf.com/repository/pypi/packages/deepspeed/0.10.3/deepspeed-0.10.3.tar.gz (867 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 867.3/867.3 kB 114.6 MB/s eta 0:00:00
 Preparing metadata (setup.py) ... done
Collecting hjson (from deepspeed)
 Downloading http://nexus3.msxf.com/repository/pypi/packages/hjson/3.1.0/hjson-3.1.0.tar.gz (40 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.5/40.5 kB 67.5 MB/s eta 0:00:00
 Preparing metadata (setup.py) ... done
Collecting ninja (from deepspeed)
 Downloading http://nexus3.msxf.com/repository/pypi/packages/ninja/1.11.1/ninja-1.11.1.tar.gz (27 kB)
 Installing build dependencies ... done
 Getting requirements to build wheel ... done
 Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy in /usr/local/python3/lib/python3.9/site-packages (from deepspeed) (1.25.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/python3/lib/python3.9/site-packages (from deepspeed) (23.1)
Requirement already satisfied: psutil in /usr/local/python3/lib/python3.9/site-packages (from deepspeed) (5.9.5)
Collecting py-cpuinfo (from deepspeed)
 Downloading http://nexus3.msxf.com/repository/pypi/packages/py-cpuinfo/9.0.0/py-cpuinfo-9.0.0.tar.gz (104 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.7/104.7 kB 99.6 MB/s eta 0:00:00
 Installing build dependencies ... done
 Getting requirements to build wheel ... done
 Preparing metadata (pyproject.toml) ... done
Collecting pydantic<2.0.0 (from deepspeed)
 Downloading http://nexus3.msxf.com/repository/pypi/packages/pydantic/1.10.13/pydantic-1.10.13.tar.gz (348 kB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 348.3/348.3 kB 124.8 MB/s eta 0:00:00
 Preparing metadata (setup.py) ... done
Requirement already satisfied: torch in /usr/local/python3/lib/python3.9/site-packages (from deepspeed) (2.0.1+cu118)
Requirement already satisfied: tqdm in /usr/local/python3/lib/python3.9/site-packages (from deepspeed) (4.66.1)
Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/python3/lib/python3.9/site-packages (from pydantic<2.0.0->deepspeed) (4.7.1)
Requirement already satisfied: filelock in /usr/local/python3/lib/python3.9/site-packages (from torch->deepspeed) (3.12.4)
Requirement already satisfied: sympy in /usr/local/python3/lib/python3.9/site-packages (from torch->deepspeed) (1.12)
Requirement already satisfied: networkx in /usr/local/python3/lib/python3.9/site-packages (from torch->deepspeed) (3.1)
Requirement already satisfied: jinja2 in /usr/local/python3/lib/python3.9/site-packages (from torch->deepspeed) (3.1.2)
Requirement already satisfied: triton==2.0.0 in /usr/local/python3/lib/python3.9/site-packages (from torch->deepspeed) (2.0.0)
Requirement already satisfied: cmake in /usr/local/python3/lib/python3.9/site-packages (from triton==2.0.0->torch->deepspeed) (3.27.5)
Requirement already satisfied: lit in /usr/local/python3/lib/python3.9/site-packages (from triton==2.0.0->torch->deepspeed) (17.0.1)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/python3/lib/python3.9/site-packages (from jinja2->torch->deepspeed) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in /usr/local/python3/lib/python3.9/site-packages (from sympy->torch->deepspeed) (1.3.0)
Building wheels for collected packages: deepspeed, pydantic, hjson, ninja, py-cpuinfo
 Building wheel for deepspeed (setup.py) ... done
 Created wheel for deepspeed: filename=deepspeed-0.10.3-cp39-cp39-linux_x86_64.whl size=4152088 sha256=98453be7391547ca3526e4fedfce22af8f48072c8b31b9ce931938927b386160
 Stored in directory: /tmp/pip-ephem-wheel-cache-6cr62s4z/wheels/49/35/db/aae5b7b65d41ded8e3236c80173246396d29dfbe8d74ae6c53
 Building wheel for pydantic (setup.py) ... done
 Created wheel for pydantic: filename=pydantic-1.10.13-cp39-cp39-linux_x86_64.whl size=3156349 sha256=e2aee2b425bd43c2674b6053659f19d91646902d8de4ff4ffbeae18f136a7bd4
 Stored in directory: /tmp/pip-ephem-wheel-cache-6cr62s4z/wheels/a3/76/da/eed6b1d795f6597060e4bf898f74b7c51cc6fe2e60f8ec4afa
 Building wheel for hjson (setup.py) ... done
 Created wheel for hjson: filename=hjson-3.1.0-py3-none-any.whl size=54019 sha256=798a1235ffdb1a55ac4051bd80796c4795d0b8ec51f8732ecd226cce01d95e82
 Stored in directory: /tmp/pip-ephem-wheel-cache-6cr62s4z/wheels/91/36/58/11e6ef4dd170250e99ce88cd34ca44f848d0bf0c52dbb30fd4
 WARNING: Ignoring --global-option when building ninja using PEP 517
 Building wheel for ninja (pyproject.toml) ... error
 error: subprocess-exited-with-error

 × Building wheel for ninja (pyproject.toml) did not run successfully.
 │ exit code: 1
 ╰─> [250 lines of output]

 --------------------------------------------------------------------------------
 -- Trying 'Ninja' generator
 --------------------------------
 ---------------------------
 ----------------------
 -----------------
 ------------
 -------
 --
 CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
 Compatibility with CMake < 3.5 will be removed from a future version of
 CMake.

 Update the VERSION argument <min> value or use a ...<max> suffix to tell
 CMake that the project does not need compatibility with older versions.

 Not searching for unused variables given on the command line.

 -- The C compiler identification is GNU 9.5.0
 -- Detecting C compiler ABI info
 -- Detecting C compiler ABI info - done
 -- Check for working C compiler: /usr/bin/cc - skipped
 -- Detecting C compile features
 -- Detecting C compile features - done
 -- The CXX compiler identification is GNU 9.5.0
 -- Detecting CXX compiler ABI info
 -- Detecting CXX compiler ABI info - done
 -- Check for working CXX compiler: /usr/bin/c++ - skipped
 -- Detecting CXX compile features
 -- Detecting CXX compile features - done
 -- Configuring done (0.8s)
 -- Generating done (0.0s)
 -- Build files have been written to: /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_cmake_test_compile/build
 --
 -------
 ------------
 -----------------
 ----------------------
 ---------------------------
 --------------------------------
 -- Trying 'Ninja' generator - success
 --------------------------------------------------------------------------------

 Configuring Project
 Working directory:
 /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build
 Command:
 /usr/local/bin/cmake /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-install -DPYTHON_VERSION_STRING:STRING=3.9.17 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-q0kpnrhc/overlay/lib/python3.9/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/local/python3/include/python3.9 -DPYTHON_LIBRARY:PATH=/usr/local/python3/lib/libpython3.9.a -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr/local/python3 -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/local/python3/include/python3.9 -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr/local/python3 -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/local/python3/include/python3.9 -DCMAKE_BUILD_TYPE:STRING=Release

 Not searching for unused variables given on the command line.
 -- The C compiler identification is GNU 9.5.0
 -- The CXX compiler identification is GNU 9.5.0
 -- Detecting C compiler ABI info
 -- Detecting C compiler ABI info - done
 -- Check for working C compiler: /usr/bin/cc - skipped
 -- Detecting C compile features
 -- Detecting C compile features - done
 -- Detecting CXX compiler ABI info
 -- Detecting CXX compiler ABI info - done
 -- Check for working CXX compiler: /usr/bin/c++ - skipped
 -- Detecting CXX compile features
 -- Detecting CXX compile features - done
 -- *********************************************
 -- Ninja Python Distribution
 --
 -- BUILD_VERBOSE : OFF
 -- RUN_NINJA_TEST : ON
 --
 -- ARCHIVE_DOWNLOAD_DIR : /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build
 --
 -- src_archive : unix_source
 -- <src_archive>_url : https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz
 -- <src_archive>_sha256 : 7ba84551f5b315b4270dc7c51adef5dff83a2154a3665a6c9744245c122dd0db
 -- *********************************************
 CMake Warning (dev) at /usr/local/share/cmake-3.27/Modules/ExternalProject.cmake:3136 (message):
 The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
 not set. The policy's OLD behavior will be used. When using a URL
 download, the timestamps of extracted files should preferably be that of
 the time of extraction, otherwise code that depends on the extracted
 contents might not be rebuilt if the URL changes. The OLD behavior
 preserves the timestamps from the archive instead, but this is usually not
 what you want. Update your project to the NEW behavior or specify the
 DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
 robustness issue.
 Call Stack (most recent call first):
 /usr/local/share/cmake-3.27/Modules/ExternalProject.cmake:4345 (_ep_add_download_command)
 CMakeLists.txt:65 (ExternalProject_add)
 This warning is for project developers. Use -Wno-dev to suppress it.

 -- download_ninja_source - URL: https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz
 -- SuperBuild - CMAKE_BUILD_TYPE: Release
 -- Configuring done (0.9s)
 -- Generating done (0.0s)
 -- Build files have been written to: /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build
 [1/19] Creating directories for 'download_ninja_source'
 [1/19] Performing download step (download, verify and extract) for 'download_ninja_source'
 -- Downloading...
 dst='/tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build/v1.11.1.g95dee.kitware.jobserver-1.tar.gz'
 timeout='none'
 inactivity timeout='none'
 -- Using src='https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz'
 -- Retrying...
 -- Using src='https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz'
 -- Retry after 5 seconds (attempt #2) ...
 -- Using src='https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz'
 -- Retry after 5 seconds (attempt #3) ...
 -- Using src='https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz'
 -- Retry after 15 seconds (attempt #4) ...
 -- Using src='https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz'
 -- Retry after 60 seconds (attempt #5) ...
 -- Using src='https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz'
 CMake Error at _skbuild/linux-x86_64-3.9/cmake-build/download_ninja_source-prefix/src/download_ninja_source-stamp/download-download_ninja_source.cmake:170 (message):
 Each download failed!

 error: downloading 'https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz' failed
 status_code: 28
 status_string: "Timeout was reached"
 log:
 --- LOG BEGIN ---
 timeout on name lookup is not supported
 Trying 20.205.243.166:443...

 connect to 20.205.243.166 port 443 failed: Connection timed out

 Failed to connect to github.com port 443 after 127217 ms: Couldn't connect
 to server

 Closing connection 0

 --- LOG END ---
 error: downloading 'https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz' failed
 status_code: 28
 status_string: "Timeout was reached"
 log:
 --- LOG BEGIN ---
 timeout on name lookup is not supported
 Trying 20.205.243.166:443...

 connect to 20.205.243.166 port 443 failed: Connection timed out

 Failed to connect to github.com port 443 after 127227 ms: Couldn't connect
 to server

 Closing connection 0

 --- LOG END ---
 error: downloading 'https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz' failed
 status_code: 28
 status_string: "Timeout was reached"
 log:
 --- LOG BEGIN ---
 timeout on name lookup is not supported
 Trying 20.205.243.166:443...

 connect to 20.205.243.166 port 443 failed: Connection timed out

 Failed to connect to github.com port 443 after 127335 ms: Couldn't connect
 to server

 Closing connection 0

 --- LOG END ---
 error: downloading 'https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz' failed
 status_code: 28
 status_string: "Timeout was reached"
 log:
 --- LOG BEGIN ---
 timeout on name lookup is not supported
 Trying 20.205.243.166:443...

 connect to 20.205.243.166 port 443 failed: Connection timed out

 Failed to connect to github.com port 443 after 127336 ms: Couldn't connect
 to server

 Closing connection 0

 --- LOG END ---
 error: downloading 'https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz' failed
 status_code: 28
 status_string: "Timeout was reached"
 log:
 --- LOG BEGIN ---
 timeout on name lookup is not supported
 Trying 20.205.243.166:443...

 connect to 20.205.243.166 port 443 failed: Connection timed out

 Failed to connect to github.com port 443 after 127325 ms: Couldn't connect
 to server

 Closing connection 0

 --- LOG END ---
 error: downloading 'https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz' failed
 status_code: 28
 status_string: "Timeout was reached"
 log:
 --- LOG BEGIN ---
 timeout on name lookup is not supported
 Trying 20.205.243.166:443...

 connect to 20.205.243.166 port 443 failed: Connection timed out

 Failed to connect to github.com port 443 after 127380 ms: Couldn't connect
 to server

 Closing connection 0

 --- LOG END ---

 FAILED: download_ninja_source-prefix/src/download_ninja_source-stamp/download_ninja_source-download /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build/download_ninja_source-prefix/src/download_ninja_source-stamp/download_ninja_source-download
 cd /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee && /usr/local/bin/cmake -P /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build/download_ninja_source-prefix/src/download_ninja_source-stamp/download-download_ninja_source.cmake && /usr/local/bin/cmake -P /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build/download_ninja_source-prefix/src/download_ninja_source-stamp/verify-download_ninja_source.cmake && /usr/local/bin/cmake -P /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build/download_ninja_source-prefix/src/download_ninja_source-stamp/extract-download_ninja_source.cmake && /usr/local/bin/cmake -E touch /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build/download_ninja_source-prefix/src/download_ninja_source-stamp/download_ninja_source-download
 ninja: build stopped: subcommand failed.
 Traceback (most recent call last):
 File "/tmp/pip-build-env-q0kpnrhc/overlay/lib/python3.9/site-packages/skbuild/setuptools_wrap.py", line 674, in setup
 cmkr.make(make_args, install_target=cmake_install_target, env=env)
 File "/tmp/pip-build-env-q0kpnrhc/overlay/lib/python3.9/site-packages/skbuild/cmaker.py", line 697, in make
 self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
 File "/tmp/pip-build-env-q0kpnrhc/overlay/lib/python3.9/site-packages/skbuild/cmaker.py", line 742, in make_impl
 raise SKBuildError(msg)

 An error occurred while building with CMake.
 Command:
 /usr/local/bin/cmake --build . --target install --config Release --
 Install target:
 install
 Source directory:
 /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee
 Working directory:
 /tmp/pip-install-v_53kpbe/ninja_0d385c4eca7949fd95dd1fd48807f7ee/_skbuild/linux-x86_64-3.9/cmake-build
 Please check the install target is valid and see CMake's output for more information.

 [end of output]

 note: This error originates from a subprocess, and is likely not a problem with pip.
 ERROR: Failed building wheel for ninja
 WARNING: Ignoring --global-option when building py-cpuinfo using PEP 517
 Building wheel for py-cpuinfo (pyproject.toml) ... done
 Created wheel for py-cpuinfo: filename=py_cpuinfo-9.0.0-py3-none-any.whl size=22315 sha256=8b9156706a97208c31e0f1df4bb6775e06422ac57b18622380cba4f97fa05d8f
 Stored in directory: /tmp/pip-ephem-wheel-cache-6cr62s4z/wheels/38/9d/b3/e4ec03bd0ce395add8ee687545908f5d98c446402d4f19d79d
Successfully built deepspeed pydantic hjson py-cpuinfo
Failed to build ninja
ERROR: Could not build wheels for ninja, which is required to install pyproject.toml-based projects
henryiii commented 1 year ago

I'd recommend either making the wheel available on your internal network (download it manually, then set pip to look at your downloaded file with extra index url), or installing ninja and using that instead of the Python package (if deepspeed supports it).

jhance commented 1 year ago

IMO a reasonable solution would be to add a --vendor-ninja-sources option to ninja. There is precedent for this with how setuptools_rust works, one can pass --vendor-crates to avoid hitting the network during the build.

Or just default to vendoring the sources into the pip sdist. I don't think its reasonable behavior for sdists to talk to the internet in general.

The recommendation to "just make the wheel available on the internal network" isn't the best option because the ideal is to build everything from source reproducibility and relying on a binary blob is not great when doing system updates etc.

Personally for my use-case I am planning on manually vendoring and modifying the CMakeLists to simply use the vendored copy rather than downloading. I can't see it being too hard to make that gated behind a flag though. Or the behavior could be to check if the directory already exists in the sdist, and if so use that.

The existing solution has some other problems where it is using a github archive and verifying it by sha256 however github archives are not guaranteed to be stable and have in the past changed (e.g. github decides to change their compression options or whatever).

luoqishuai commented 10 months ago

I ran pip3 install ninja before installing DS_BUILD_FUSED_ADAM=1 deepspeed and then the installation was successful. Thank you so much.