Extraneous virtualenv created when installing VCS dependencies using `--system`

[sburns]

I noticed an issue in my docker builds this morning using this Dockerfile:

FROM python:3.6

WORKDIR /app

COPY . .
COPY docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh
RUN pip install -U pip
RUN pip install pipenv
RUN pipenv install --deploy --system
RUN python -m pipenv.help
ENV PYTHONPATH /app
ENTRYPOINT ["/docker-entrypoint.sh"]

(Pretty standard stuff, I don't believe Docker is the issue here)

Here is a log of the docker build:

Step 1/11 : FROM python:3.6
 ---> 336d482502ab
Step 2/11 : WORKDIR /app
 ---> Using cache
 ---> 485f81cfa840
Step 3/11 : COPY . .
 ---> 278a192259f0
Step 4/11 : COPY docker-entrypoint.sh /
 ---> 1210932b20d0
Step 5/11 : RUN chmod +x /docker-entrypoint.sh
 ---> Running in e48c96b56ad0
Removing intermediate container e48c96b56ad0
 ---> 81d6e6e219bf
Step 6/11 : RUN pip install -U pip
 ---> Running in 55d86b67eef8
Collecting pip
  Downloading pip-9.0.3-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-9.0.3
Removing intermediate container 55d86b67eef8
 ---> 6c30f77d7fc0
Step 7/11 : RUN pip install pipenv
 ---> Running in 13e33223bd42
Collecting pipenv
  Downloading pipenv-11.10.0-py3-none-any.whl (5.6MB)
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.6/site-packages (from pipenv)
Requirement already satisfied: pip>=9.0.1 in /usr/local/lib/python3.6/site-packages (from pipenv)
Collecting virtualenv (from pipenv)
  Downloading virtualenv-15.2.0-py2.py3-none-any.whl (2.6MB)
Collecting certifi (from pipenv)
  Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)
Collecting virtualenv-clone>=0.2.5 (from pipenv)
  Downloading virtualenv_clone-0.3.0-py2.py3-none-any.whl
Installing collected packages: virtualenv, certifi, virtualenv-clone, pipenv
Successfully installed certifi-2018.1.18 pipenv-11.10.0 virtualenv-15.2.0 virtualenv-clone-0.3.0
Removing intermediate container 13e33223bd42
 ---> 2ee25119966e
Step 8/11 : RUN pipenv install --deploy --system
 ---> Running in f541b2c43311
Installing -e git+https://github.com/stratasan/stratatilities.git#egg=stratatilities…
Obtaining stratatilities from git+https://github.com/stratasan/stratatilities.git#egg=stratatilities
  Cloning https://github.com/stratasan/stratatilities.git to ./src/stratatilities
Collecting Click>=6.0 (from stratatilities)
  Downloading click-6.7-py2.py3-none-any.whl (71kB)
Collecting boto3 (from stratatilities)
  Downloading boto3-1.7.4-py2.py3-none-any.whl (128kB)
Collecting requests (from stratatilities)
  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
Collecting hvac (from stratatilities)
  Downloading hvac-0.5.0.tar.gz
Collecting jmespath<1.0.0,>=0.7.1 (from boto3->stratatilities)
  Downloading jmespath-0.9.3-py2.py3-none-any.whl
Collecting botocore<1.11.0,>=1.10.4 (from boto3->stratatilities)
  Downloading botocore-1.10.4-py2.py3-none-any.whl (4.2MB)
Collecting s3transfer<0.2.0,>=0.1.10 (from boto3->stratatilities)
  Downloading s3transfer-0.1.13-py2.py3-none-any.whl (59kB)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests->stratatilities)
Collecting urllib3<1.23,>=1.21.1 (from requests->stratatilities)
  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests->stratatilities)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting idna<2.7,>=2.5 (from requests->stratatilities)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
Collecting docutils>=0.10 (from botocore<1.11.0,>=1.10.4->boto3->stratatilities)
  Downloading docutils-0.14-py3-none-any.whl (543kB)
Collecting python-dateutil<2.7.0,>=2.1 (from botocore<1.11.0,>=1.10.4->boto3->stratatilities)
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
Collecting six>=1.5 (from python-dateutil<2.7.0,>=2.1->botocore<1.11.0,>=1.10.4->boto3->stratatilities)
  Downloading six-1.11.0-py2.py3-none-any.whl
Building wheels for collected packages: hvac
  Running setup.py bdist_wheel for hvac: started
  Running setup.py bdist_wheel for hvac: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/8c/2f/a0/ffeca10bbc3c674c2423c61c9037ec5d5a573a70a29ee423e3
Successfully built hvac
Installing collected packages: Click, jmespath, docutils, six, python-dateutil, botocore, s3transfer, boto3, urllib3, chardet, idna, requests, hvac, stratatilities
  Running setup.py develop for stratatilities
Successfully installed Click-6.7 boto3-1.7.4 botocore-1.10.4 chardet-3.0.4 docutils-0.14 hvac-0.5.0 idna-2.6 jmespath-0.9.3 python-dateutil-2.6.1 requests-2.18.4 s3transfer-0.1.13 six-1.11.0 stratatilities urllib3-1.22

Adding -e git+https://github.com/stratasan/stratatilities.git#egg=stratatilities to Pipfile's [packages]…
Creating a virtualenv for this project…
Using /usr/local/bin/python (3.6.4) to create virtualenv…
Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/app-4PlAip0Q/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /root/.local/share/virtualenvs/app-4PlAip0Q
Installing dependencies from Pipfile.lock (6f84a9)…
Ignoring contextlib2: markers 'python_version < "3.2"' don't match your environment
Removing intermediate container f541b2c43311
 ---> ef23e9d6838c
Step 9/11 : RUN python -m pipenv.help
 ---> Running in 55fe7525e82e
<details><summary>$ python -m pipenv.help output</summary>

Pipenv version: `'11.10.0'`

Pipenv location: `'/usr/local/lib/python3.6/site-packages/pipenv'`

Python location: `'/usr/local/bin/python'`

Other Python installations in `PATH`:

  - `2.7`: `/usr/bin/python2.7`
  - `2.7`: `/usr/bin/python2.7`
  - `3.6`: `/usr/local/bin/python3.6m`
  - `3.6`: `/usr/local/bin/python3.6`
  - `3.6`: `/usr/local/bin/python3.6`

  - `3.6.4`: `/usr/local/bin/python`
  - `3.6.4`: `/usr/local/bin/python`
  - `2.7.9`: `/usr/bin/python`
  - `2.7.9`: `/usr/bin/python2`
  - `3.6.4`: `/usr/local/bin/python3`
  - `3.6.4`: `/usr/local/bin/python3`

PEP 508 Information:

{'implementation_name': 'cpython', 'implementation_version': '3.6.4', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '4.9.87-linuxkit-aufs', 'platform_system': 'Linux', 'platform_version': '#1 SMP Wed Mar 14 15:12:16 UTC 2018', 'python_full_version': '3.6.4', 'python_version': '3.6', 'sys_platform': 'linux'}

System environment variables:

  - `HOSTNAME`
  - `PYTHON_PIP_VERSION`
  - `HOME`
  - `GPG_KEY`
  - `PATH`
  - `LANG`
  - `PYTHON_VERSION`
  - `PWD`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_PYTHON_PATH`
  - `PYTHONUNBUFFERED`

Pipenv–specific environment variables:

Debug–specific environment variables:

  - `PATH`: `/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`
  - `LANG`: `C.UTF-8`
  - `PWD`: `/app`

---------------------------

Contents of `Pipfile` ('/app/Pipfile'):

```toml
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
boto = "*"
"boto3" = "*"
hvac = "*"
luigi = "*"
"psycopg2-binary" = "*"
raven = "*"
stratatilities = {git = "https://github.com/stratasan/stratatilities.git", editable = true}
click = "*"
botocore = "*"
structlog = "*"

[dev-packages]
ipython = {extras = ["notebook"]}
nbformat = "*"
awscli = "*"
colorama = "*"
pytest = "*"
"flake8" = "*"

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/app/Pipfile.lock'):

{
       <snip>
        "stratatilities": {
            "editable": true,
            "git": "https://github.com/stratasan/stratatilities.git",
            "ref": "0027bc89c005d5842560d856c59b2bcb72605a71"
        },
        </snip>

Removing intermediate container 55fe7525e82e ---> 3e730a1c2fa6 Step 10/11 : ENV PYTHONPATH /app ---> Running in 5eeb6a675f18 Removing intermediate container 5eeb6a675f18 ---> d2fb733b0987 Step 11/11 : ENTRYPOINT ["/docker-entrypoint.sh"] ---> Running in d6af7ad2592b Removing intermediate container d6af7ad2592b ---> 4fd18ed472d9 Successfully built 4fd18ed472d9 Successfully tagged peach:latest

##### Expected result

A new virtualenv should not be created when editable dependencies are encountered using `--system`.

##### Actual result

If you look in the logs above, there's a new virtualenv created within the container that only contains `stratatilities` (look for `Virtualenv location: /root/.local/share/virtualenvs/app-4PlAip0Q`). I believe this is related to recent 11.10.0 changes (potentially #1812?) because this behavior does not occur when pinning pipenv to 11.9.0.

##### Steps to replicate

A similar `Dockerfile` that installs w/ a Pipfile that installs django via vcs has the same behavior

$ cat Dockerfile FROM python:3.6

WORKDIR /app

COPY . . RUN pip install -U pip RUN pip install pipenv RUN pipenv install --deploy --system ENV PYTHONPATH /app $ cat Pipfile [[source]]

url = "https://pypi.python.org/simple" verify_ssl = true name = "pypi"

[packages]

Django = {git = "https://github.com/django/django.git", editable = true}

[dev-packages]

[requires]

python_version = "3.6" $ docker build . ~/dev/pipenv-test Sending build context to Docker daemon 4.608kB Step 1/7 : FROM python:3.6 ---> 336d482502ab Step 2/7 : WORKDIR /app ---> Using cache ---> 485f81cfa840 Step 3/7 : COPY . . ---> 49c882489ae1 Step 4/7 : RUN pip install -U pip ---> Running in 4e655cc62b67 Collecting pip Downloading pip-9.0.3-py2.py3-none-any.whl (1.4MB) Installing collected packages: pip Found existing installation: pip 9.0.1 Uninstalling pip-9.0.1: Successfully uninstalled pip-9.0.1 Successfully installed pip-9.0.3 Removing intermediate container 4e655cc62b67 ---> 5477835d99bd Step 5/7 : RUN pip install pipenv ---> Running in 16c2f32f55e4 Collecting pipenv Downloading pipenv-11.10.0-py3-none-any.whl (5.6MB) Collecting virtualenv-clone>=0.2.5 (from pipenv) Downloading virtualenv_clone-0.3.0-py2.py3-none-any.whl Requirement already satisfied: pip>=9.0.1 in /usr/local/lib/python3.6/site-packages (from pipenv) Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.6/site-packages (from pipenv) Collecting virtualenv (from pipenv) Downloading virtualenv-15.2.0-py2.py3-none-any.whl (2.6MB) Collecting certifi (from pipenv) Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB) Installing collected packages: virtualenv-clone, virtualenv, certifi, pipenv Successfully installed certifi-2018.1.18 pipenv-11.10.0 virtualenv-15.2.0 virtualenv-clone-0.3.0 Removing intermediate container 16c2f32f55e4 ---> 0d6b971f8cad Step 6/7 : RUN pipenv install --deploy --system ---> Running in 1c8c621383f8 Installing -e git+https://github.com/django/django.git#egg=Django… Obtaining Django from git+https://github.com/django/django.git#egg=Django Cloning https://github.com/django/django.git to ./src/django Collecting pytz (from Django) Downloading pytz-2018.4-py2.py3-none-any.whl (510kB) Installing collected packages: pytz, Django Running setup.py develop for Django Successfully installed Django pytz-2018.4

Adding -e git+https://github.com/django/django.git#egg=Django to Pipfile's [packages]… Creating a virtualenv for this project… Using /usr/local/bin/python (3.6.4) to create virtualenv… Already using interpreter /usr/local/bin/python Using base prefix '/usr/local' New python executable in /root/.local/share/virtualenvs/app-4PlAip0Q/bin/python Installing setuptools, pip, wheel...done.

Virtualenv location: /root/.local/share/virtualenvs/app-4PlAip0Q Installing dependencies from Pipfile.lock (504886)… Removing intermediate container 1c8c621383f8 ---> 8e08221d59d2 Step 7/7 : ENV PYTHONPATH /app ---> Running in b2dbe9c39856 Removing intermediate container b2dbe9c39856 ---> f65135fd2dbf Successfully built f65135fd2dbf

Sorry if this is super verbose but I hope it helps everyone involved track it down.

[techalchemy]

per #1939 and #1979 you need to include a lockfile -- see https://github.com/pypa/pipenv/issues/1002#issuecomment-381128601

editable vcs dependencies require you to do dependency resolution which always requires a virtualenv

[joejuzl]

@techalchemy I am a little confused by the statement: "editable vcs dependencies require you to do dependency resolution which always requires a virtualenv". We have editable dependencies in our Pipfile, and when we run pipenv install --system --deploy --verbose in our dockerfile it is creating a virtualenv and a lock file from scratch (even though the lock file is in the image). Is this unavoidable? (it takes an extremely long time...).

This is the output - it happens after it's installed all the editable/local dependencies, and can take 20mins or so.

Creating a virtualenv for this project…
Using /usr/local/bin/python3.6 (3.6.2) to create virtualenv…
Already using interpreter /usr/local/bin/python3.6
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/slurmfactory-vhvnywio/bin/python3.6
Also creating executable in /root/.local/share/virtualenvs/slurmfactory-vhvnywio/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /root/.local/share/virtualenvs/slurmfactory-vhvnywio
Pipfile.lock (d9d4e3) out of date, updating to (cc6154)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

[techalchemy]

@joejuzl the —deploy flag is supposed to fail if the lockfile is out of date. If this is not happening please file an issue on that.

No, —system is not meant to create virtualenvs. The problem is you showed me your output but not your input. If you’re experiencing unexpected behavior I need all of your inputs

[joejuzl]

So the contents of my Pipfileis:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]

"0527503" = {path = "../diomedes/", editable = true}
"502a27f" = {path = "../tube_transport/", editable = true}
"6f2d9c9" = {path = "../base/", editable = true}
"91927d2" = {path = "../ml_runner/", editable = true}
"7e3d267" = {path = "../xb_logging/", editable = true}
"ecc5d41" = {path = "../mail_utils/", editable = true}
"d4b7ac4" = {path = "../xb_activity_report/", editable = true}
"argon2" = "==0.1.10"
"argon2-cffi" = "==16.3.0"
"jinja2" = "==2.9.6"
aioredis = "==0.3.0"
arrow = "==0.10.0"
click = "==6.7"
flask = "==0.12.2"
itsdangerous = "==0.24"
keras = "==2.0.3"
markupsafe = "==1.0"
pyowm = "==2.7.1"
python-dateutil = "==2.6.0"
python-gmaps = "==0.3.1"
redlock-py = "==1.0.8"
rq = "==0.8.0"
rq-dashboard = "==0.3.8"
rq-scheduler = "==0.7.0"
scikit-learn = "==0.18.1"
scipy = "==0.19.0"
six = "==1.10.0"
uvloop = "==0.8.0"
werkzeug = "==0.12.2"
sklearn = "*"

[dev-packages]

coverage = "*"
pylint = "*"
pytest = "*"
pytest-cov = "*"
pytest-env = "*"
pytest-mock = "*"
pytest-sugar = "*"

I have attached the Pipfile.lock as it's quite large (changed to .txt so I can upload).

Pipfile_lock.txt

They are both in the directory slurmfactory

This is my docker file:

FROM eu.gcr.io/io-xbird-milkyway/base:1.0

COPY . /opt/app/
WORKDIR /opt/app/slurmfactory
RUN pipenv install --system --deploy --verbose
ENV PYTHONUNBUFFERED 1

I have attached the output of my docker build up until it hangs for 20mins creating a lock file. After that point it just continues to install all the non-editable dependencies.

docker_build_output.txt

If you need anything else, let me know!

This only happens on 11.10.0 not 11.9.0

Thanks!

[joejuzl]

Hi @techalchemy while PR #2006 does seem like a bug, it doesn't fix the issue I am referring too! I have tried the same steps with that commit - and it still tries to create a new virtualenv... I can provide more output if necessary.

[techalchemy]

@joejuzl try pip install —upgrade —pre pipenv and let me know if this is resolved

[joejuzl]

I get this error when running pipenv lock

Locking [dev-packages] dependencies…
odule>
    from .pypi import PyPIRepository
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 11, in <module>
    from notpip.index import PackageFinder
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 33, in <module>
    from notpip._vendor import html5lib, requests, six
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/_vendor/requests/packages.py", line 8, in <module>
    locals()[package] = __import__(vendored_package)
ModuleNotFoundError: No module named 'pip._vendor.urllib3'

[techalchemy]

@joejuzl the comment directly above yours describes a temporary fix, I’ll have a release out in an hour or so

[fronbasal]

This error is still occuring, is there any recommended fix?