Allow `Pipenv install --system` to install project for arbitrary site installations of python

[parsons-kyle-89]

Feature request

It would be good if you could specify which site installation of python you install your project for using pipenv install --system. It would also make sense for pipenv to consult and respect the version of python specified in your Pipfile when performing a site installation.

Issue

In cases where there are multiple site installations of python that have different versions, pipenv can only install your project for the default version of python. It always installs your project for the version of python that the site installation of pip installs packages for.

In my particular case I have a site installation of python 2.7. The site installation of pip installs packages for python 2.7. I also have a site installation of python 3.5 and a Pipfile that requires python 3.5. However

$ pipenv install --system

installs my project for python 2.7 (which is an unsuitable version of python for my project). The commands

$ pipenv install --system --python=/usr/bin/python3

and

$ python3 -m pipenv install --system

seem to have the same effect.

Workarounds

As a workaround, performing

$ pipenv lock -r > requirements.txt
$ python3 -m pip install -r requirements.txt
$ python3 -m pip install .

does a passable job at installing my project's dependencies; however, it does not ensure that the sub-dependencies match the versions specified in the Pipfile.lock.

In order to better copy the virtual env that would be produced by pipenv the following commands can be used:

$ pipenv install
$ pipenv lock -r | head -n1 > requirements.txt
$ pipenv run pip freeze >> requirements.txt
$ python3 -m pip install -r requirements.txt
$ python3 -m pip install .

This is a undesirable workaround both because of it's kludgyness and because it requires creating a virtual env just to learn what versions of packages are specified by the lock file.

Pipenv's purpose is to allow you to manage your project's dependencies. It's disappointing that when using the --system flag it has no way to manage the most important dependency, the version of python itself.

Additional context

My particular deployment attempt is on an amazon ec2 instance

Linux version 4.14.33-51.37.amzn1.x86_64 (mockbuild@gobi-build-60006) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC)) #1 SMP Thu May 3 20:07:43 UTC 2018

pipenv --support for a sample project I made to reproduce the issue yields:

$ pipenv --support

Pipenv version: `'2018.11.26'` Pipenv location: `'/usr/lib/python2.7/dist-packages/pipenv'` Python location: `'/usr/bin/python'` Python installations found: - `3.5.6`: `/usr/bin/python3.5` - `3.5.6`: `/usr/bin/python3.5m` - `3.4.9`: `/usr/bin/python3.4m` - `2.7.15`: `/usr/bin/python2.7` - `3.4.9`: `/usr/bin/python3.4` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '0', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '4.14.33-51.37.amzn1.x86_64', 'platform_system': 'Linux', 'platform_version': '#1 SMP Thu May 3 20:07:43 UTC 2018', 'python_full_version': '2.7.15', 'python_version': '2.7', 'sys_platform': 'linux2'} ``` System environment variables: - `PYTHONDONTWRITEBYTECODE` - `LESSOPEN` - `LOGNAME` - `USER` - `HOME` - `PATH` - `AWS_PATH` - `LANG` - `TERM` - `SHELL` - `EC2_AMITOOL_HOME` - `LESS_TERMCAP_me` - `LESS_TERMCAP_md` - `LESS_TERMCAP_mb` - `HISTSIZE` - `PYTHONFINDER_IGNORE_UNSUPPORTED` - `AWS_ELB_HOME` - `SUDO_USER` - `JAVA_HOME` - `EC2_HOME` - `USERNAME` - `SUDO_UID` - `SUDO_GID` - `AWS_AUTO_SCALING_HOME` - `PIP_PYTHON_PATH` - `_` - `SUDO_COMMAND` - `LESS_TERMCAP_ue` - `AWS_CLOUDWATCH_HOME` - `PIP_DISABLE_PIP_VERSION_CHECK` - `OLDPWD` - `HOSTNAME` - `LESS_TERMCAP_se` - `SHLVL` - `PWD` - `LESS_TERMCAP_us` - `PIP_SHIMS_BASE_MODULE` - `MAIL` - `LS_COLORS` - `PYTHON_INSTALL_LAYOUT` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin` - `SHELL`: `/bin/bash` - `LANG`: `en_US.UTF-8` - `PWD`: `/root/proj` --------------------------- Contents of `Pipfile` ('/root/proj/Pipfile'): ```toml [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] requests = "==2.13" [requires] python_version = "3.5" ``` Contents of `Pipfile.lock` ('/root/proj/Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "ca5a46fc7af31eedc15e601dc12842c49b1ff07500bd167ef04ad1929f2b7b29" }, "pipfile-spec": 6, "requires": { "python_version": "3.5" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "requests": { "hashes": [ "sha256:1a720e8862a41aa22e339373b526f508ef0c8988baf48b84d3fc891a8e237efb", "sha256:5722cd09762faa01276230270ff16af7acf7c5c45d623868d9ba116f15791ce8" ], "index": "pypi", "version": "==2.13" } }, "develop": {} } ```

[frostming]

This is a good feature proposal that we don't support yet. /cc @techalchemy @jxltom @uranusjr

[jxltom]

I think the workaround does ensure that the sub-dependencies are matching Pipfile.lock? Is there any difference between pipenv run pip freeze >> requirements.txt and pipenv lock -r > requirements.txt + pipenv lock -r -d > requirements-dev.txt?

[frostming]

@jxltom I think he wants to install dependencies into non-venv python locations which isn't the default one(/usr/bin/python or what is detected from PATH)

[uranusjr]

The usage is reasonable, but --system --python=... is awkward. Maybe it is better to allow pipenv install --python=..., and make --system an alias of pipenv install --python=$(which python).

[frostming]

Pipenv installs dependencies by choosing the proper pip(or via python -m pip). So what if the selected python doesn't have pip installed? Are we going to force it to have one?

[uranusjr]

IMO we should assume it has one, and simply emit an exception (or maybe a prettier error saying pip is required) if it does not.

[techalchemy]

Also on the first point @jxltom we do not test to ensure that our output matches pip freeze as both pipenv and pip perform transformations on the path metadata before presenting it.

With regard to the aliasing question, --system doesn’t technically search $PATH as far as I remember, it should use sys.executable. That’s how it is aware of the system paths for certain, since they align with the running interpreter. I don’t think it makes sense to change this behavior and I definitely don’t think it makes sense to rely on which or even looking for things called python at this point. which is not reliably available and python isn’t always named that.

The simplest non-breaking approach (and seriously this doesn’t need to be complicated with breaking changes — this is the first time this has been requested so I’m confident folks can type the extra words) is to just allow the existing options to work on this command. I’m not looking to break this functionality for no reason so overall it seems high risk low reward