search

cloudfoundry / python-buildpack

Cloud Foundry buildpack for the Python Language
http://docs.cloudfoundry.org/buildpacks/
Apache License 2.0
121 stars 279 forks source link

pip version? #168

Closed mhadi813 closed 4 years ago

mhadi813 commented 4 years ago

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

"name": "Pivotal Application Service",
   "build": "2.7.14-build.8",
   "support": "https://support.pivotal.io",
   "version": 0,
   "description": "https://docs.pivotal.io/pivotalcf/2-7/pcf-release-notes/runtime-rn.html",
   "min_cli_version": "6.23.0",
   "min_recommended_cli_version": "6.23.0",
   "app_ssh_oauth_client": "ssh-proxy",

   "api_version": "2.139.0",
   "osbapi_version": "2.15",

What version of the buildpack you are using?

Python Buildpack version 1.7.13

If you were attempting to accomplish a task, what was it you were attempting to do? deploy an app for which requirements includes pantab (which requires tableauhyperapi)

What did you expect to happen?

successful deployment

What was the actual behavior?

deployment fails. below is log

   Downloading python_buildpack...
   Downloaded python_buildpack (490.1M)
   Cell 38bce7eb-8d00-4670-9195-1f8b4e0708b7 creating container for instance 73e310ab-32be-4ffa-a7f4-a545a50b7f03
   Cell 38bce7eb-8d00-4670-9195-1f8b4e0708b7 successfully created container for instance 73e310ab-32be-4ffa-a7f4-a545a50b7f03
   Downloading app package...
   Downloaded app package (4.2M)
   -----> Python Buildpack version 1.7.13
   -----> Supplying Python
   -----> Installing python 3.7.7
          Copy [/tmp/buildpacks/709809d9dc9feba36497a15c04e12a93/dependencies/9bf93ca7119de6195b691afe25353efc/python-3.7.7-linux-x64-cflinuxfs3-50fae53c.tgz]
   -----> Installing pip-pop 0.1.4
          Copy [/tmp/buildpacks/709809d9dc9feba36497a15c04e12a93/dependencies/1cd0187e0e714e99bef932d4c22f6515/pip-pop-0.1.4-0a3b0f1b.tar.gz]
   -----> Noticed dependency requiring libffi. Bootstrapping libffi.
   -----> Installing libffi 3.2.1
          Copy [/tmp/buildpacks/709809d9dc9feba36497a15c04e12a93/dependencies/0841b343acf6839a9aa6d0f42315930e/libffi-3.2.1-linux-x64-5f5bf32c.tgz]
   -----> Running Pip Install
          Collecting pip==20.1 (from -r /tmp/app/requirements.txt (line 2))
            Downloading https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl (1.5MB)
          Collecting wheel==0.34.2 (from -r /tmp/app/requirements.txt (line 3))
            Downloading https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
          Collecting pantab (from -r /tmp/app/requirements.txt (line 4))
            Downloading https://files.pythonhosted.org/packages/73/c4/12ecbef7b2666dd3eab4588099fa92913a55975ef61fce8ded1982e1ab0f/pantab-1.1.0.tar.gz
            Installing build dependencies: started
            Installing build dependencies: finished with status 'error'
            ERROR: Command errored out with exit status 1:
             command: /tmp/contents686067334/deps/0/bin/python /tmp/contents686067334/deps/0/python/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_d1tw2v4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel tableauhyperapi
                 cwd: None
            Complete output (9 lines):
            Collecting setuptools
              Downloading https://files.pythonhosted.org/packages/09/27/98ef5f13ca9c63d9d8601e739817839dde55baeb105a6c32fc8f24daddc2/setuptools-46.3.0-py3-none-any.whl (582kB)
            Collecting wheel
              Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
            Collecting tableauhyperapi
              ERROR: Could not find a version that satisfies the requirement tableauhyperapi (from versions: none)
            ERROR: No matching distribution found for tableauhyperapi
            WARNING: You are using pip version 19.2.3, however version 20.1 is available.
            You should consider upgrading via the 'pip install --upgrade pip' command.
            ----------------------------------------
          ERROR: Command errored out with exit status 1: /tmp/contents686067334/deps/0/bin/python /tmp/contents686067334/deps/0/python/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_d1tw2v4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel tableauhyperapi Check the logs for full command output.
          WARNING: You are using pip version 19.2.3, however version 20.1 is available.
          You should consider upgrading via the 'pip install --upgrade pip' command.
          **ERROR** Could not install pip packages: could not run pip: exit status 1
   Failed to compile droplet: Failed to run all supply scripts: exit status 14
Error staging application: App staging failed in the buildpack compile phase
FAILED

Step to reproduce: runtime.txt > 

python-3.7.x

requirements.txt > pantab

main.py > 

import pantab
print('successfully imported required package')

cf push command

cf push tableau_pydb -b python_buildpack --health-check-type none -c "python main.py" -m 2GB -k 2GB --no-start --no-route

Please confirm where necessary:

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172829824

The labels on this github issue will be updated when the story is started.

mhadi813 commented 4 years ago

tableau hyperAPI note about installation:


Linux installations require pip version 19.3 or newer. Note that pip versions 20.0 and 20.1 are not working because of issues with pip.

https://help.tableau.com/current/api/hyper_api/en-us/docs/hyper_api_installing.html#using-the-tableau-hyper-api-nuget-package

Alternatively, you can download the Python Hyper API package file (.whl file) for your operating system. See Hyper API - Products Release and Download.
    • Use pip to install the .whl file you downloaded.
        ○ On Windows: Scripts\pip install [path_to_whl_file]
On Linux/macOS: bin/pip install [path_to_whl_file] ```
mhaas commented 4 years ago

FYI, if you are interested in running a custom version of pip, there is a similar issue here: https://github.com/cloudfoundry/pip-cnb/issues/2

mhadi813 commented 4 years ago

Thanks mhaas, I couldn't find solution how to install the requirement with specific version of pip.

As a work around, I used mini conda to install pip==19.3.1 and passed packages in environment.yml as below:

name: my_app
channels:
  - defaults
dependencies:
  - pip=19.3.1
  - pip:
      - pantab
      - tableauserverclient
      - google-cloud-bigquery==1.24.0
      - google-cloud-storage==1.27.0
      - pyarrow==0.16.0
      - pandas==1.0.0
mhaas commented 4 years ago

@mhadi813 Thanks a lot, that is a pretty cool trick! Do you know by any chance if the environment.yml can be used to command-line arguments to pip? I checked the conda docs, but couldn't find anything there.

mhadi813 commented 4 years ago

I didn’t think so, environment.yml is for conda command. In this set up, Once conda has installed required version of python and pip, pip takes the driver seat and installs rest of the requirements listed in environment.yml

ryanmoran commented 4 years ago

I am having some trouble reproducing this. I did the following with the latest buildpack (v1.7.17).

I have an application with the following source:

runtime.txt

python-3.7.x

requirements.txt

pantab

main.py

import pantab
print("successfully imported required package")

I then perform the following push command:

cf push myapp -p /tmp/myapp -b python_buildpack --health-check-type none -c "python main.py" --no-route

And I see the following in my logs:

 2020-08-06T11:42:23.80-0400 [STG/0] OUT Downloading app package...
 2020-08-06T11:42:23.83-0400 [STG/0] OUT Downloaded app package (516B)
 2020-08-06T11:42:24.09-0400 [STG/0] OUT -----> Python Buildpack version 1.7.17
 2020-08-06T11:42:24.09-0400 [STG/0] OUT -----> Supplying Python
 2020-08-06T11:42:24.09-0400 [STG/0] OUT -----> Installing python 3.7.8
 2020-08-06T11:42:24.09-0400 [STG/0] OUT        Download [https://buildpacks.cloudfoundry.org/dependencies/python/python_3.7.8_linux_x64_cflinuxfs3_e98a931c.tgz]
 2020-08-06T11:42:28.10-0400 [STG/0] OUT -----> Installing pip-pop 0.1.5
 2020-08-06T11:42:28.10-0400 [STG/0] OUT        Download [https://buildpacks.cloudfoundry.org/dependencies/manual-binaries/pip-pop/pip-pop-0.1.5-b32efe86.tar.gz]
 2020-08-06T11:42:33.13-0400 [STG/0] OUT -----> Running Pip Install
 2020-08-06T11:42:34.48-0400 [STG/0] OUT        Collecting pantab
 2020-08-06T11:42:35.10-0400 [STG/0] OUT          Downloading pantab-1.1.0-cp37-cp37m-manylinux2014_x86_64.whl (3.7 MB)
 2020-08-06T11:42:36.03-0400 [STG/0] OUT        Collecting pandas
 2020-08-06T11:42:36.04-0400 [STG/0] OUT          Downloading pandas-1.1.0-cp37-cp37m-manylinux1_x86_64.whl (10.5 MB)
 2020-08-06T11:42:37.43-0400 [STG/0] OUT        Collecting tableauhyperapi
 2020-08-06T11:42:37.98-0400 [STG/0] OUT          Downloading tableauhyperapi-0.0.11249-py3-none-manylinux2014_x86_64.whl (48.0 MB)
 2020-08-06T11:42:41.97-0400 [STG/0] OUT        Collecting python-dateutil>=2.7.3
 2020-08-06T11:42:41.99-0400 [STG/0] OUT          Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
 2020-08-06T11:42:42.71-0400 [STG/0] OUT        Collecting numpy>=1.15.4
 2020-08-06T11:42:42.73-0400 [STG/0] OUT          Downloading numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)
 2020-08-06T11:42:44.15-0400 [STG/0] OUT        Collecting pytz>=2017.2
 2020-08-06T11:42:44.17-0400 [STG/0] OUT          Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
 2020-08-06T11:42:44.74-0400 [STG/0] OUT        Collecting cffi<2,>=1.12.2
 2020-08-06T11:42:44.75-0400 [STG/0] OUT          Downloading cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
 2020-08-06T11:42:44.86-0400 [STG/0] OUT        Collecting six>=1.5
 2020-08-06T11:42:44.87-0400 [STG/0] OUT          Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
 2020-08-06T11:42:44.92-0400 [STG/0] OUT        Collecting pycparser
 2020-08-06T11:42:44.94-0400 [STG/0] OUT          Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
 2020-08-06T11:42:45.06-0400 [STG/0] OUT        Installing collected packages: six, python-dateutil, numpy, pytz, pandas, pycparser, cffi, tableauhyperapi, pantab
 2020-08-06T11:42:47.76-0400 [STG/0] ERR          WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/tmp/contents192244616/deps/0/python/bin' which is not on PATH.
 2020-08-06T11:42:47.76-0400 [STG/0] ERR          Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
 2020-08-06T11:42:55.32-0400 [STG/0] OUT        Successfully installed cffi-1.14.1 numpy-1.19.1 pandas-1.1.0 pantab-1.1.0 pycparser-2.20 python-dateutil-2.8.1 pytz-2020.1 six-1.15.0 tableauhyperapi-0.0.11249
 2020-08-06T11:42:55.53-0400 [STG/0] ERR        WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
 2020-08-06T11:42:55.53-0400 [STG/0] ERR        You should consider upgrading via the '/tmp/contents192244616/deps/0/bin/python -m pip install --upgrade pip' command.
 2020-08-06T11:42:56.44-0400 [STG/0] ERR No start command specified by buildpack or via Procfile.
 2020-08-06T11:42:56.44-0400 [STG/0] ERR App will not start unless a command is provided at runtime.
 2020-08-06T11:43:28.11-0400 [STG/0] OUT Exit status 0
 2020-08-06T11:43:28.11-0400 [STG/0] OUT Uploading droplet, build artifacts cache...
 2020-08-06T11:43:28.11-0400 [STG/0] OUT Uploading droplet...
 2020-08-06T11:43:28.11-0400 [STG/0] OUT Uploading build artifacts cache...
 2020-08-06T11:43:31.21-0400 [API/0] OUT Creating droplet for app with guid 1cbb2295-46c5-4be6-944b-f3624d5e2c6b
 2020-08-06T11:43:31.59-0400 [STG/0] OUT Uploaded build artifacts cache (118.7M)
 2020-08-06T11:43:36.78-0400 [STG/0] OUT Uploaded droplet (131.7M)
 2020-08-06T11:43:36.80-0400 [STG/0] OUT Uploading complete
 2020-08-06T11:43:36.97-0400 [STG/0] OUT Cell decdb167-c781-4a8c-95d2-27f35d7eadc3 stopping instance 03f829df-ad36-4b07-8ea1-47ad7dba1209
 2020-08-06T11:43:36.97-0400 [STG/0] OUT Cell decdb167-c781-4a8c-95d2-27f35d7eadc3 destroying container for instance 03f829df-ad36-4b07-8ea1-47ad7dba1209
 2020-08-06T11:43:37.24-0400 [CELL/0] OUT Cell decdb167-c781-4a8c-95d2-27f35d7eadc3 creating container for instance 560261b5-2ece-4937-57f0-a027
 2020-08-06T11:43:38.36-0400 [STG/0] OUT Cell decdb167-c781-4a8c-95d2-27f35d7eadc3 successfully destroyed container for instance 03f829df-ad36-4b07-8ea1-47ad7dba1209
 2020-08-06T11:43:38.53-0400 [CELL/0] OUT Cell decdb167-c781-4a8c-95d2-27f35d7eadc3 successfully created container for instance 560261b5-2ece-4937-57f0-a027
 2020-08-06T11:43:38.75-0400 [CELL/0] OUT Downloading droplet...
 2020-08-06T11:43:45.72-0400 [CELL/0] OUT Downloaded droplet (131.7M)
 2020-08-06T11:43:46.68-0400 [APP/PROC/WEB/0] OUT successfully imported required package
 2020-08-06T11:43:46.73-0400 [APP/PROC/WEB/0] OUT Exit status 0

This looks pretty successful. I'm not sure what I am missing.

kvedurmu commented 4 years ago

Closing this due to inactivity. Feel free to open this back up if you have more details.