search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.51k stars 3.02k forks source link

``pip install'' 's option corresponding to the python parallel building option -j. #7870

Closed hongyi-zhao closed 1 year ago

hongyi-zhao commented 4 years ago

Hi,

The ``python setup.py install'' has the following option:

--parallel (-j) number of parallel build jobs

On the other hand, I noted that the ``pip install'' will invoke the ``python setup.py'' if necessary. So, I want to know whether there is the ``pip install'' 's option corresponding to the python parallel building option -j?

Regards

uranusjr commented 4 years ago

You can pass arbitrary options to setup.py install with the --install-option flag (run pip install --help to learn more). Would this work for you?

hongyi-zhao commented 4 years ago

Tzu-ping Chung notifications@github.com 于2020年3月18日周三 下午9:29写道：

You can pass arbitrary options to setup.py install with the --install-option flag (run pip install --help to learn more). Would this work for you?

Thanks, but I failed to do the trick for compiling the kwant: https://github.com/kwant-project/kwant. See my following info:

$ git clone https://github.com/kwant-project/kwant.git kwant.git $ cd kwant.git

Then install all of dependencies for compiling this project.

And at this point, I can run the following command successfully:

$ python setup.py build_ext -i -j4

Then I try to use pip to the above job, but failed:

$ pip install -e . --isolated --install-option="--parallel=4" -vvv (kwant) werner@ubuntu-01:~/Public/hpc/tools/kwant.git$ pip install -e . --isolated --install-option="build_ext -i -j4" -vvv /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option. cmdoptions.check_install_build_global(options) Non-user install because site-packages writeable Created temporary directory: /tmp/pip-ephem-wheel-cache-bbybdoxc Created temporary directory: /tmp/pip-req-tracker-lvmy850w Initialized build tracking at /tmp/pip-req-tracker-lvmy850w Created build tracker: /tmp/pip-req-tracker-lvmy850w Entered build tracker: /tmp/pip-req-tracker-lvmy850w Created temporary directory: /tmp/pip-install-4pibnrpj Obtaining file:///home/werner/Public/hpc/tools/kwant.git Added file:///home/werner/Public/hpc/tools/kwant.git to build tracker '/tmp/pip-req-tracker-lvmy850w' Running setup.py (path:/home/werner/Public/hpc/tools/kwant.git/setup.py) egg_info for package from file:///home/werner/Public/hpc/tools/kwant.git Running command python setup.py egg_info Compiling kwant/_system.pyx because it changed. Compiling kwant/operator.pyx because it changed. Compiling kwant/graph/core.pyx because it changed. Compiling kwant/graph/dijkstra.pyx because it changed. Compiling kwant/linalg/lapack.pyx because it changed. Compiling kwant/linalg/_mumps.pyx because it changed. [1/6] Cythonizing kwant/_system.pyx [2/6] Cythonizing kwant/graph/core.pyx [3/6] Cythonizing kwant/graph/dijkstra.pyx [4/6] Cythonizing kwant/linalg/_mumps.pyx [5/6] Cythonizing kwant/linalg/lapack.pyx [6/6] Cythonizing kwant/operator.pyx running egg_info creating kwant.egg-info writing kwant.egg-info/PKG-INFO writing dependency_links to kwant.egg-info/dependency_links.txt writing requirements to kwant.egg-info/requires.txt writing top-level names to kwant.egg-info/top_level.txt writing manifest file 'kwant.egg-info/SOURCES.txt' reading manifest file 'kwant.egg-info/SOURCES.txt' writing manifest file 'kwant.egg-info/SOURCES.txt' Source in /home/werner/Public/hpc/tools/kwant.git has version 1.5.0a0.dev65+g307a54f, which satisfies requirement kwant==1.5.0a0.dev65+g307a54f from file:///home/werner/Public/hpc/tools/kwant.git Removed kwant==1.5.0a0.dev65+g307a54f from file:///home/werner/Public/hpc/tools/kwant.git from build tracker '/tmp/pip-req-tracker-lvmy850w' Requirement already satisfied: numpy>=1.13.3 in /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.18.1) Requirement already satisfied: scipy>=0.19.1 in /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.4.1) Requirement already satisfied: tinyarray>=1.2 in /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.2.2) Installing collected packages: kwant Attempting uninstall: kwant Found existing installation: kwant 1.5.0a0.dev65+g307a54f Uninstalling kwant-1.5.0a0.dev65+g307a54f: Created temporary directory: /tmp/pip-uninstall-z3uj5tsr Removing file or directory /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/kwant.egg-link Removing pth entries from /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/easy-install.pth : Removing entry: /home/werner/Public/hpc/tools/kwant.git Successfully uninstalled kwant-1.5.0a0.dev65+g307a54f Running setup.py develop for kwant Running command /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"'; file='"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --no-user-cfg develop --no-deps 'build_ext -i -j4' invalid command name 'build_ext -i -j4' Rolling back uninstall of kwant Moving to /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/kwant.egg-link from /tmp/pip-uninstall-z3uj5tsr/kwant.egg-link Replacing /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/kwant.egg-link from /tmp/pip-uninstall-z3uj5tsr/kwant.egg-link Rolling /home/werner/.repo/ github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/easy-install.pth back to previous state Cleaning up... Removed build tracker: '/tmp/pip-req-tracker-lvmy850w' ERROR: Command errored out with exit status 1: /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"'; file='"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --no-user-cfg develop --no-deps 'build_ext -i -j4' Check the logs for full command output. Exception information: Traceback (most recent call last): File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main status = self.run(options, args) File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 404, in run use_user_site=options.use_user_site, File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/req/init.py", line 71, in install_given_reqs **kwargs File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 802, in install unpacked_source_directory=self.unpacked_source_directory, File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/operations/install/editable_legacy.py", line 51, in install_editable cwd=unpacked_source_directory, File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess raise InstallationError(exc_msg) pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"'; file='"'"'/home/werner/Public/hpc/tools/kwant.git/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --no-user-cfg develop --no-deps 'build_ext -i -j4' Check the logs for full command output.

Any hints for this issue?

Regards

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pypa/pip/issues/7870#issuecomment-600622788, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACVDTDV3KWJQBNOYPYADG4TRIDEC7ANCNFSM4LOPUYCQ .

-- Hongyi Zhao hongyi.zhao@gmail.com

hongyi-zhao commented 4 years ago

Based on the help of pip install:

  --global-option <options>   Extra global options to be supplied to the
                              setup.py call before the install command.

I also tried the following, but still failed:

$ pip install . --isolated --global-option build_ext --global-option -j4 --global-option -i -vvv
/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-p62tgz4r
Created temporary directory: /tmp/pip-req-tracker-nlagv2re
Initialized build tracking at /tmp/pip-req-tracker-nlagv2re
Created build tracker: /tmp/pip-req-tracker-nlagv2re
Entered build tracker: /tmp/pip-req-tracker-nlagv2re
Created temporary directory: /tmp/pip-install-3x5io1qv
Processing /home/werner/Public/hpc/tools/kwant.git
  Created temporary directory: /tmp/pip-req-build-1fuli4d6
  Added file:///home/werner/Public/hpc/tools/kwant.git to build tracker '/tmp/pip-req-tracker-nlagv2re'
    Running setup.py (path:/tmp/pip-req-build-1fuli4d6/setup.py) egg_info for package from file:///home/werner/Public/hpc/tools/kwant.git
    Running command python setup.py egg_info
    Compiling kwant/_system.pyx because it changed.
    Compiling kwant/operator.pyx because it changed.
    Compiling kwant/graph/core.pyx because it changed.
    Compiling kwant/graph/dijkstra.pyx because it changed.
    Compiling kwant/linalg/lapack.pyx because it changed.
    Compiling kwant/linalg/_mumps.pyx because it changed.
    [1/6] Cythonizing kwant/_system.pyx
    [2/6] Cythonizing kwant/graph/core.pyx
    [3/6] Cythonizing kwant/graph/dijkstra.pyx
    [4/6] Cythonizing kwant/linalg/_mumps.pyx
    [5/6] Cythonizing kwant/linalg/lapack.pyx
    [6/6] Cythonizing kwant/operator.pyx
    running egg_info
    creating /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info
    writing /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/requires.txt
    writing top-level names to /tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/top_level.txt
    writing manifest file '/tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-req-build-1fuli4d6/pip-egg-info/kwant.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-1fuli4d6 has version 1.5.0a0.dev65+g307a54f, which satisfies requirement kwant==1.5.0a0.dev65+g307a54f from file:///home/werner/Public/hpc/tools/kwant.git
  Removed kwant==1.5.0a0.dev65+g307a54f from file:///home/werner/Public/hpc/tools/kwant.git from build tracker '/tmp/pip-req-tracker-nlagv2re'
Requirement already satisfied: numpy>=1.13.3 in /home/werner/.repo/github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.18.1)
Requirement already satisfied: scipy>=0.19.1 in /home/werner/.repo/github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.4.1)
Requirement already satisfied: tinyarray>=1.2 in /home/werner/.repo/github.com/pyenv/versions/linux-x86_64/miniconda3-latest/envs/kwant/lib/python3.7/site-packages (from kwant==1.5.0a0.dev65+g307a54f) (1.2.2)
Skipping wheel build for kwant, due to binaries being disabled for it.
Installing collected packages: kwant
  Created temporary directory: /tmp/pip-record-bxvseifj
    Running command /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext -j4 -i --no-user-cfg install --record /tmp/pip-record-bxvseifj/install-record.txt --single-version-externally-managed --compile --install-headers /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/include/python3.7m/kwant
    running build_ext
    error: error in command line: command 'build_ext' has no such option 'no_user_cfg'
    Running setup.py install for kwant ... error
Cleaning up...
  Removing source in /tmp/pip-req-build-1fuli4d6
Removed build tracker: '/tmp/pip-req-tracker-nlagv2re'
ERROR: Command errored out with exit status 1: /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext -j4 -i --no-user-cfg install --record /tmp/pip-record-bxvseifj/install-record.txt --single-version-externally-managed --compile --install-headers /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/include/python3.7m/kwant Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 404, in run
    use_user_site=options.use_user_site,
  File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 71, in install_given_reqs
    **kwargs
  File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 829, in install
    scheme=scheme,
  File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/operations/install/legacy.py", line 72, in install
    cwd=install_req.unpacked_source_directory,
  File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 275, in runner
    spinner=spinner,
  File "/home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1fuli4d6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext -j4 -i --no-user-cfg install --record /tmp/pip-record-bxvseifj/install-record.txt --single-version-externally-managed --compile --install-headers /home/werner/.pyenv/versions/miniconda3-latest/envs/kwant/include/python3.7m/kwant Check the logs for full command output.
azmeuk commented 4 years ago

Related question. I would like to build BTrees as fast as I can.

python setup.py build --parallel 8 works well, but --install-option or --global-option fail:

$ python -m pip install --install-option='--parallel 8' .
/home/azmeuk/dev/zodb/BTrees/env/lib/python3.8/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Processing /home/azmeuk/dev/zodb/BTrees
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting zope.interface
  Using cached zope.interface-5.0.0.tar.gz (214 kB)
Collecting persistent>=4.1.0  Using cached persistent-4.6.3.tar.gz (119 kB)Requirement already satisfied: setuptools in ./env/lib/python3.8/site-packages (from zope.interface->BTrees==4.7.1.dev0) (46.0.0)
Collecting cffi
  Using cached cffi-1.14.0.tar.gz (463 kB)
Collecting pycparser
  Using cached pycparser-2.20.tar.gz (161 kB)
Skipping wheel build for zope.interface, due to binaries being disabled for it.
Skipping wheel build for persistent, due to binaries being disabled for it.
Skipping wheel build for cffi, due to binaries being disabled for it.
Skipping wheel build for pycparser, due to binaries being disabled for it.
Building wheels for collected packages: BTrees
  Building wheel for BTrees (PEP 517) ... done  Created wheel for BTrees: filename=BTrees-4.7.1.dev0-cp38-cp38-linux_x86_64.whl size=3310012 sha256=8d7ede730f4c7eb1a56b71124a1956fc6381f2f42c2e64be82d0fd040785bc74  Stored in directory: /tmp/pip-ephem-wheel-cache-xvmeibjl/wheels/6e/6d/be/8acde01b09d9c1158ca46a22021870cad200ef2325bf167cc8
Successfully built BTrees
Installing collected packages: zope.interface, pycparser, cffi, persistent, BTrees
    Running setup.py install for zope.interface ... error
    ERROR: Command errored out with exit status 1:
     command: /home/azmeuk/dev/zodb/BTrees/env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n5jo5jsl/zope.interface/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n5jo5jsl/zope.interface/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-byf2qhbs/install-record.txt --single-version-externally-managed --compile --install-headers /home/azmeuk/dev/zodb/BTrees/env/include/site/python3.8/zope.interface '--parallel 8'
         cwd: /tmp/pip-install-n5jo5jsl/zope.interface/
    Complete output (6 lines):
    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help

    error: option --parallel 8 not recognized
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/azmeuk/dev/zodb/BTrees/env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n5jo5jsl/zope.interface/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n5jo5jsl/zope.interface/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-byf2qhbs/install-record.txt --single-version-externally-managed --compile --install-headers /home/azmeuk/dev/zodb/BTrees/env/include/site/python3.8/zope.interface '--parallel 8' Check the logs for full command output.
uranusjr commented 4 years ago

@azmeuk pip propagates build options to child dependencies. Use --no-deps to install . independently, and then pip install . again without the build-option flag to pull in dependencies.

hongyi-zhao commented 4 years ago

Tzu-ping Chung notifications@github.com 于2020年3月20日周五 下午5:33写道：

@azmeuk https://github.com/azmeuk pip propagates build options to child dependencies. Use --no-deps to install . independently, and then pip install . again without the build-option flag to pull in dependencies.

Still no so clear about what should I use the pip for this job. I tried the following but still failed:

$ pip install --global-option=build_ext --global-option="-j4 --no-deps" -vvv . --isolated or $ pip install --global-option=build_ext --global-option="-j4" -vvv . --isolated --no-deps

Regards —

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pypa/pip/issues/7870#issuecomment-601607931, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACVDTDXPVYNE67QJ76FGULTRIMZ7BANCNFSM4LOPUYCQ .

-- Hongyi Zhao hongyi.zhao@gmail.com

pradyunsg commented 1 year ago

Closing this out since this has been sitting for a while now and the use of --global-option to directly interact with setup.py is on its way out. See https://pip.pypa.io/en/stable/reference/build-system/ for the details on the second part.