Multiple indices with private source using pipenv uses private source for all dependencies

[breneser]

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

{
   "name": "",
   "build": "",
   "support": "*****",
   "version": 0,
   "description": "",
   "authorization_endpoint": "*****"
   "token_endpoint": "*****",
   "min_cli_version": null,
   "min_recommended_cli_version": null,
   "app_ssh_endpoint": "*****:2222",
   "app_ssh_host_key_fingerprint": "****",
   "app_ssh_oauth_client": "ssh-proxy",
   "doppler_logging_endpoint": "endpoint",
   "api_version": "2.146.0",
   "osbapi_version": "2.15",
   "user": "****"
}
cf version 6.49.0+d0dfa93bb.2020-01-07

What version of the buildpack you are using? Tried 1.7.8 and 1.7.12

If you were attempting to accomplish a task, what was it you were attempting to do?

Install private packages from private repository and the rest from pypi

What did you expect to happen? Only private packages to be installed from private source

What was the actual behavior?

Log output suggests only private source is used:

Generating 'requirements.txt' from Pipfile.lock
   -----> Running Pip Install
          Looking in indexes: https://pypi.viewflow.io/***********/simple/    # license number hidden
          Collecting django-recaptcha==2.0.6 (from -r /tmp/app/requirements.txt (line 3))
            ERROR: Could not find a version that satisfies the requirement django-recaptcha==2.0.6 (from -r /tmp/app/requirements.txt (line 3)) (from versions: none)
          ERROR: No matching distribution found for django-recaptcha==2.0.6 (from -r /tmp/app/requirements.txt (line 3))
          WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
          You should consider upgrading via the 'pip install --upgrade pip' command.
          **ERROR** Could not install pip packages: could not run pip: exit status 1
   Failed to compile droplet: Failed to run all supply scripts: exit status 14

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "viewflow"
url = "https://pypi.viewflow.io/${LICENSE}/simple/"
verify_ssl = true

[packages]
django = "==2.2.10"
django-environ = "*"
django-recaptcha = "*"
psycopg2 = "*"
whitenoise = "*"
requests = "*"
django-compressor = "*"
django-viewflow-pro= {version="*", index="viewflow"}
django-material-pro= {version="*", index="viewflow"}

[requires]
python_version = "3.7.6"

Please confirm where necessary:

• [x] I have included a log output
• [x] My log includes an error message
• [x] I have included steps for reproduction

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172331884

The labels on this github issue will be updated when the story is started.

[breneser]

Further information:

Trying locally by generating requirements.txt using pipenv lock -r > requirements.txt with pipenv version pipenv, version 2018.11.26 :

-i https://pypi.org/simple
--extra-index-url https://pypi.viewflow.io/*****/simple/
amqp==2.5.2
babel==2.8.0
billiard==3.6.3.0
boto3==1.12.39
boto==2.49.0
botocore==1.15.39
celery==4.4.2
certifi==2020.4.5.1
chardet==3.0.4
django-annoying==0.10.6
django-appconf==1.0.4
django-celery-results==1.2.1
django-compressor==2.4
django-environ==0.4.5
django-filter==2.2.0
django-ipware==2.1.0
django-jsonstore==0.4.1
django-material-pro==1.6.5
django-phonenumber-field==4.0.0
django-recaptcha==2.0.6
django-ses==0.8.14
django-storages==1.9.1
django-structlog==1.5.2
django-viewflow-pro==1.6.0
django==2.2.10
docutils==0.15.2
future==0.18.2
humanize==2.3.0
idna==2.9
importlib-metadata==1.6.0 ; python_version < '3.8'
jmespath==0.9.5
kombu==4.6.8
markupsafe==1.1.1
packaging==20.3
psycopg2==2.8.5
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2019.3
rcssmin==1.0.6
requests==2.23.0
rjsmin==1.1.0
six==1.14.0
sqlparse==0.3.1
urllib3==1.25.8 ; python_version != '3.4'
whitenoise==5.0.1

Using this requirements file with pip pip install -r requirements.txt works fine:

Looking in indexes: https://pypi.org/simple, https://pypi.viewflow.io/*****/simple/   ### BOTH indexes listed unlike cloud foundry 
Requirement already satisfied: amqp==2.5.2 in /home/brn/.pyenv/versions/3.7.6/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (2.5.2)
Requirement already satisfied: babel==2.8.0 in /home/brn/.pyenv/versions/3.7.6/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (2.8.0)
Requirement already satisfied: billiard==3.6.3.0 in /home/brn/.pyenv/versions/3.7.6/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (3.6.3.0)
Collecting boto3==1.12.39 (from -r requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/27/87/de75e5a24584d82cca60b86f95d06e56412ed9e23807dcf23896f206f58e/boto3-1.12.39-py2.py3-none-any.whl
Collecting boto==2.49.0 (from -r requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl

My guess is however requirements.txt is generated seems to be the issue, I am not sure what version of pipenv is installed with this buildpack.

[breneser]

It could be reproduced using this sample app https://github.com/breneser/django_sample

Pushing using cf push django_sample will result in the same error.

[fg-j]

Thank you for providing a sample app. We were able to reproduce your problem using the buildpack version 1.7.12 and with the latest version, 1.7.20.

We have identified the root cause of your issue: that the buildpack adds an instance of the -i flag to requirements.txt for each index in the sources section of the Pipfile.lock, as you can see here. Meanwhile pip install can only accept one -i argument and additional indices must be added with the extra-index-url flag. This is a bug in the buildpack.

For now, to work around this issue, you can pre-generate the requirements.txt file as you described above: pipenv lock -r > requirements.txt. The buildpack will then use that file instead of generating an incorrect one based on the Pipfile.lock.

We have filed an issue (#212 ) that specifically calls out this bug. There's a Tracker story linked to that issue that will track the progress of the bug fix.

[sophiewigmore]

Closing this due to inactivity and the creation of #212 . @breneser feel free to re-open if there is anything else to add.