Closed luoqishuai closed 10 months ago
I'd recommend either making the wheel available on your internal network (download it manually, then set pip to look at your downloaded file with extra index url), or installing ninja and using that instead of the Python package (if deepspeed supports it).
IMO a reasonable solution would be to add a --vendor-ninja-sources
option to ninja. There is precedent for this with how setuptools_rust works, one can pass --vendor-crates
to avoid hitting the network during the build.
Or just default to vendoring the sources into the pip sdist. I don't think its reasonable behavior for sdists to talk to the internet in general.
The recommendation to "just make the wheel available on the internal network" isn't the best option because the ideal is to build everything from source reproducibility and relying on a binary blob is not great when doing system updates etc.
Personally for my use-case I am planning on manually vendoring and modifying the CMakeLists to simply use the vendored copy rather than downloading. I can't see it being too hard to make that gated behind a flag though. Or the behavior could be to check if the directory already exists in the sdist, and if so use that.
The existing solution has some other problems where it is using a github archive and verifying it by sha256 however github archives are not guaranteed to be stable and have in the past changed (e.g. github decides to change their compression options or whatever).
I ran
pip3 install ninja
before installing
DS_BUILD_FUSED_ADAM=1 deepspeed
and then the installation was successful.
Thank you so much.
My network environment is within the company's internal network and I am unable to access external networks
When I install deepspeed, it needs to install ninja.ninja, which will be accessed during installation https://github.com/Kitware/ninja/archive/v1.11.1.g95dee.kitware.jobserver-1.tar.gz , resulting in installation failure
The following is the installation log
If you need any other information, please contact me