Closed hongyi-zhao closed 1 year ago
You can pass arbitrary options to setup.py install
with the --install-option
flag (run pip install --help
to learn more). Would this work for you?
Tzu-ping Chung notifications@github.com 于2020年3月18日周三 下午9:29写道:
You can pass arbitrary options to setup.py install with the --install-option flag (run pip install --help to learn more). Would this work for you?
Thanks, but I failed to do the trick for compiling the kwant: https://github.com/kwant-project/kwant. See my following info:
$ git clone https://github.com/kwant-project/kwant.git kwant.git $ cd kwant.git
$ python setup.py build_ext -i -j4
$ pip install -e . --isolated --install-option="--parallel=4" -vvv (kwant) werner@ubuntu-01:~/Public/hpc/tools/kwant.git$ pip install -e . --isolated --install-option="build_ext -i -j4" -vvv /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option. cmdoptions.check_install_build_global(options) Non-user install because site-packages writeable Created temporary directory: /tmp/pip-ephem-wheel-cache-bbybdoxc Created temporary directory: /tmp/pip-req-tracker-lvmy850w Initialized build tracking at /tmp/pip-req-tracker-lvmy850w Created build tracker: /tmp/pip-req-tracker-lvmy850w Entered build tracker: /tmp/pip-req-tracker-lvmy850w Created temporary directory: /tmp/pip-install-4pibnrpj Obtaining file:///home/werner/Public/hpc/tools/kwant.git Added file:///home/werner/Public/hpc/tools/kwant.git to build tracker '/tmp/pip-req-tracker-lvmy850w' Running setup.py (path:/home/werner/Public/hpc/tools/kwant.git/setup.py) egg_info for package from file:///home/werner/Public/hpc/tools/kwant.git Running command python setup.py egg_info Compiling kwant/_system.pyx because it changed. Compiling kwant/operator.pyx because it changed. Compiling kwant/graph/core.pyx because it changed. Compiling kwant/graph/dijkstra.pyx because it changed. Compiling kwant/linalg/lapack.pyx because it changed. Compiling kwant/linalg/_mumps.pyx because it changed. [1/6] Cythonizing kwant/_system.pyx [2/6] Cythonizing kwant/graph/core.pyx [3/6] Cythonizing kwant/graph/dijkstra.pyx [4/6] Cythonizing kwant/linalg/_mumps.pyx [5/6] Cythonizing kwant/linalg/lapack.pyx [6/6] Cythonizing kwant/operator.pyx running egg_info creating kwant.egg-info writing kwant.egg-info/PKG-INFO writing dependency_links to kwant.egg-info/dependency_links.txt writing requirements to kwant.egg-info/requires.txt writing top-level names to kwant.egg-info/top_level.txt writing manifest file 'kwant.egg-info/SOURCES.txt' reading manifest file 'kwant.egg-info/SOURCES.txt' writing manifest file 'kwant.egg-info/SOURCES.txt' Source in /home/werner/Public/hpc/tools/kwant.git has version 1.5.0a0.dev65+g307a54f, which satisfies requirement kwant==1.5.0a0.dev65+g307a54f from file:///home/werner/Public/hpc/tools/kwant.git Removed kwant==1.5.0a0.dev65+g307a54f from file:///home/werner/Public/hpc/tools/kwant.git from build tracker '/tmp/pip-req-tracker-lvmy850w' Requirement already satisfied: numpy>=1.13.3 in /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.18.1) Requirement already satisfied: scipy>=0.19.1 in /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.4.1) Requirement already satisfied: tinyarray>=1.2 in /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.2.2) Installing collected packages: kwant Attempting uninstall: kwant Found existing installation: kwant 1.5.0a0.dev65+g307a54f Uninstalling kwant-1.5.0a0.dev65+g307a54f: Created temporary directory: /tmp/pip-uninstall-z3uj5tsr Removing file or directory /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/kwant.egg-link Removing pth entries from /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/easy-install.pth : Removing entry: /home/werner/Public/hpc/tools/kwant.git Successfully uninstalled kwant-1.5.0a0.dev65+g307a54f Running setup.py develop for kwant Running command /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"'; file='"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --no-user-cfg develop --no-deps 'build_ext -i -j4' invalid command name 'build_ext -i -j4' Rolling back uninstall of kwant Moving to /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/kwant.egg-link from /tmp/pip-uninstall-z3uj5tsr/kwant.egg-link Replacing /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/kwant.egg-link from /tmp/pip-uninstall-z3uj5tsr/kwant.egg-link Rolling /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/easy-install.pth back to previous state Cleaning up... Removed build tracker: '/tmp/pip-req-tracker-lvmy850w' ERROR: Command errored out with exit status 1: /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"'; file='"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --no-user-cfg develop --no-deps 'build_ext -i -j4' Check the logs for full command output. Exception information: Traceback (most recent call last): File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main status = self.run(options, args) File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 404, in run use_user_site=options.use_user_site, File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/req/init.py", line 71, in install_given_reqs **kwargs File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 802, in install unpacked_source_directory=self.unpacked_source_directory, File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/operations/install/editable_legacy.py", line 51, in install_editable cwd=unpacked_source_directory, File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess raise InstallationError(exc_msg) pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"'; file='"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --no-user-cfg develop --no-deps 'build_ext -i -j4' Check the logs for full command output.
Any hints for this issue?
Regards
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pypa/pip/issues/7870#issuecomment-600622788, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACVDTDV3KWJQBNOYPYADG4TRIDEC7ANCNFSM4LOPUYCQ .
-- Hongyi Zhao hongyi.zhao@gmail.com
Based on the help of pip install:
--global-option <options> Extra global options to be supplied to the
setup.py call before the install command.
I also tried the following, but still failed:
$ pip install . --isolated --global-option build_ext --global-option -j4 --global-option -i -vvv
/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
cmdoptions.check_install_build_global(options)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-p62tgz4r
Created temporary directory: /tmp/pip-req-tracker-nlagv2re
Initialized build tracking at /tmp/pip-req-tracker-nlagv2re
Created build tracker: /tmp/pip-req-tracker-nlagv2re
Entered build tracker: /tmp/pip-req-tracker-nlagv2re
Created temporary directory: /tmp/pip-install-3x5io1qv
Processing /home/werner/Public/hpc/tools/kwant.git
Created temporary directory: /tmp/pip-req-build-1fuli4d6
Added file:///home/werner/Public/hpc/tools/kwant.git to build tracker '/tmp/pip-req-tracker-nlagv2re'
Running setup.py (path:/tmp/pip-req-build-1fuli4d6/setup.py) egg_info for package from file:///home/werner/Public/hpc/tools/kwant.git
Running command python setup.py egg_info
Compiling kwant/_system.pyx because it changed.
Compiling kwant/operator.pyx because it changed.
Compiling kwant/graph/core.pyx because it changed.
Compiling kwant/graph/dijkstra.pyx because it changed.
Compiling kwant/linalg/lapack.pyx because it changed.
Compiling kwant/linalg/_mumps.pyx because it changed.
[1/6] Cythonizing kwant/_system.pyx
[2/6] Cythonizing kwant/graph/core.pyx
[3/6] Cythonizing kwant/graph/dijkstra.pyx
[4/6] Cythonizing kwant/linalg/_mumps.pyx
[5/6] Cythonizing kwant/linalg/lapack.pyx
[6/6] Cythonizing kwant/operator.pyx
running egg_info
creating /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info
writing /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/dependency_links.txt
writing requirements to /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/requires.txt
writing top-level names to /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/top_level.txt
writing manifest file '/tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/SOURCES.txt'
writing manifest file '/tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/SOURCES.txt'
Source in /tmp/pip-req-build-1fuli4d6 has version 1.5.0a0.dev65+g307a54f, which satisfies requirement kwant==1.5.0a0.dev65+g307a54f from file:///home/werner/Public/hpc/tools/kwant.git
Removed kwant==1.5.0a0.dev65+g307a54f from file:///home/werner/Public/hpc/tools/kwant.git from build tracker '/tmp/pip-req-tracker-nlagv2re'
Requirement already satisfied: numpy>=1.13.3 in /home/werner/.repo/github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.18.1)
Requirement already satisfied: scipy>=0.19.1 in /home/werner/.repo/github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.4.1)
Requirement already satisfied: tinyarray>=1.2 in /home/werner/.repo/github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.2.2)
Skipping wheel build for kwant, due to binaries being disabled for it.
Installing collected packages: kwant
Created temporary directory: /tmp/pip-record-bxvseifj
Running command /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext -j4 -i --no-user-cfg install --record /tmp/pip-record-bxvseifj/install-record.txt --single-version-externally-managed --compile --install-headers /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/include/python3.7m/kwant
running build_ext
error: error in command line: command 'build_ext' has no such option 'no_user_cfg'
Running setup.py install for kwant ... error
Cleaning up...
Removing source in /tmp/pip-req-build-1fuli4d6
Removed build tracker: '/tmp/pip-req-tracker-nlagv2re'
ERROR: Command errored out with exit status 1: /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext -j4 -i --no-user-cfg install --record /tmp/pip-record-bxvseifj/install-record.txt --single-version-externally-managed --compile --install-headers /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/include/python3.7m/kwant Check the logs for full command output.
Exception information:
Traceback (most recent call last):
File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
status = self.run(options, args)
File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 404, in run
use_user_site=options.use_user_site,
File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 71, in install_given_reqs
**kwargs
File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 829, in install
scheme=scheme,
File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/operations/install/legacy.py", line 72, in install
cwd=install_req.unpacked_source_directory,
File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 275, in runner
spinner=spinner,
File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext -j4 -i --no-user-cfg install --record /tmp/pip-record-bxvseifj/install-record.txt --single-version-externally-managed --compile --install-headers /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/include/python3.7m/kwant Check the logs for full command output.
Related question. I would like to build BTrees as fast as I can.
python setup.py build --parallel 8
works well, but --install-option
or --global-option
fail:
$ python -m pip install --install-option='--parallel 8' .
/home/azmeuk/dev/zodb/BTrees/env/lib/python3.8/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
cmdoptions.check_install_build_global(options)
Processing /home/azmeuk/dev/zodb/BTrees
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting zope.interface
Using cached zope.interface-5.0.0.tar.gz (214 kB)
Collecting persistent>=4.1.0 Using cached persistent-4.6.3.tar.gz (119 kB)Requirement already satisfied: setuptools in ./env/lib/python3.8/site-packages (from zope.interface->BTrees==4.7.1.dev0) (46.0.0)
Collecting cffi
Using cached cffi-1.14.0.tar.gz (463 kB)
Collecting pycparser
Using cached pycparser-2.20.tar.gz (161 kB)
Skipping wheel build for zope.interface, due to binaries being disabled for it.
Skipping wheel build for persistent, due to binaries being disabled for it.
Skipping wheel build for cffi, due to binaries being disabled for it.
Skipping wheel build for pycparser, due to binaries being disabled for it.
Building wheels for collected packages: BTrees
Building wheel for BTrees (PEP 517) ... done Created wheel for BTrees: filename=BTrees-4.7.1.dev0-cp38-cp38-linux_x86_64.whl size=3310012 sha256=8d7ede730f4c7eb1a56b71124a1956fc6381f2f42c2e64be82d0fd040785bc74 Stored in directory: /tmp/pip-ephem-wheel-cache-xvmeibjl/wheels/6e/6d/be/8acde01b09d9c1158ca46a22021870cad200ef2325bf167cc8
Successfully built BTrees
Installing collected packages: zope.interface, pycparser, cffi, persistent, BTrees
Running setup.py install for zope.interface ... error
ERROR: Command errored out with exit status 1:
command: /home/azmeuk/dev/zodb/BTrees/env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n5jo5jsl/zope.interface/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n5jo5jsl/zope.interface/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-byf2qhbs/install-record.txt --single-version-externally-managed --compile --install-headers /home/azmeuk/dev/zodb/BTrees/env/include/site/python3.8/zope.interface '--parallel 8'
cwd: /tmp/pip-install-n5jo5jsl/zope.interface/
Complete output (6 lines):
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: option --parallel 8 not recognized
----------------------------------------
ERROR: Command errored out with exit status 1: /home/azmeuk/dev/zodb/BTrees/env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n5jo5jsl/zope.interface/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n5jo5jsl/zope.interface/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-byf2qhbs/install-record.txt --single-version-externally-managed --compile --install-headers /home/azmeuk/dev/zodb/BTrees/env/include/site/python3.8/zope.interface '--parallel 8' Check the logs for full command output.
@azmeuk pip propagates build options to child dependencies. Use --no-deps
to install .
independently, and then pip install .
again without the build-option flag to pull in dependencies.
Tzu-ping Chung notifications@github.com 于2020年3月20日周五 下午5:33写道:
@azmeuk https://github.com/azmeuk pip propagates build options to child dependencies. Use --no-deps to install . independently, and then pip install . again without the build-option flag to pull in dependencies.
Still no so clear about what should I use the pip for this job. I tried the following but still failed:
$ pip install --global-option=build_ext --global-option="-j4 --no-deps" -vvv . --isolated or $ pip install --global-option=build_ext --global-option="-j4" -vvv . --isolated --no-deps
Regards —
You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pypa/pip/issues/7870#issuecomment-601607931, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACVDTDXPVYNE67QJ76FGULTRIMZ7BANCNFSM4LOPUYCQ .
-- Hongyi Zhao hongyi.zhao@gmail.com
Closing this out since this has been sitting for a while now and the use of --global-option
to directly interact with setup.py is on its way out. See https://pip.pypa.io/en/stable/reference/build-system/ for the details on the second part.
Hi,
The ``python setup.py install'' has the following option:
--parallel (-j) number of parallel build jobs
On the other hand, I noted that the
``pip install''
will invoke the``python setup.py''
if necessary. So, I want to know whether there is the``pip install''
's option corresponding to the python parallel building option -j?Regards