search

cloudfoundry / python-buildpack

Cloud Foundry buildpack for the Python Language
http://docs.cloudfoundry.org/buildpacks/
Apache License 2.0
121 stars 279 forks source link

Unable to install pip-pop during cf push #140

Closed Manish-Mj closed 4 years ago

Manish-Mj commented 5 years ago

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

root@9e20e273fde0:/mnt# cf curl /v2/info { "name": "Pivotal Application Service", "build": "2.4.5-build.25", "support": "https://support.pivotal.io", "version": 0, "description": "https://docs.pivotal.io/pivotalcf/2-3/pcf-release-notes/runtime-rn.html", "authorization_endpoint": "https://login.scfd.isus.emc.com", "token_endpoint": "https://uaa.scfd.isus.emc.com", "min_cli_version": "6.23.0", "min_recommended_cli_version": "6.23.0", "app_ssh_endpoint": "ssh.scfd.isus.emc.com:2222", "app_ssh_host_key_fingerprint": "58:46:d4:e1:f2:34:37:0a:37:73:4e:da:c8:9e:7d:ee", "app_ssh_oauth_client": "ssh-proxy", "doppler_logging_endpoint": "wss://doppler.scfd.isus.emc.com:443", "api_version": "2.125.0", "osbapi_version": "2.14", "routing_endpoint": "https://api.scfd.isus.emc.com/routing", "user": "6150f31c-9fc4-41bd-a3df-9507c2dc3fa9" }

What version of the buildpack you are using? Python Buildpack version 1.6.29

If you were attempting to accomplish a task, what was it you were attempting to do?

Trying to push the app with "vendor" deps & also 'lib' folder containing all the dependencies pre-installed. I had set PIP_TARGET as the env variable in the manifest.yml. If I remove this variable things go fine. But I just wanted to install dependencies in a 'lib' folder within my app contents.

What did you expect to happen? Successful push. Ideally expect no pip install to happen because I had also pushed 'lib' folder with pre-installed libs.

What was the actual behavior?

pip-pop installation only fails & never proceeds further with this manifest.yml file.

Manifest.yml:

applications:

CF PUSH Error logs:

13:25:11 Staging app and tracing logs...
13:25:12    Downloading dicf_python_buildpack_1629_cflinuxfs3...
13:25:23    Downloaded dicf_python_buildpack_1629_cflinuxfs3 (570.6M)
13:25:24    Cell 4b85e19a-22ee-49ab-8b02-7561218bb0b2 successfully created container for instance ce145cab-ddcc-442a-b2e7-adbaa9bad561
13:25:24    Cell 4b85e19a-22ee-49ab-8b02-7561218bb0b2 creating container for instance ce145cab-ddcc-442a-b2e7-adbaa9bad561
13:25:24    Downloading app package...
13:25:24    -----> Python Buildpack version 1.6.29
13:25:25    Downloaded app package (34.2M)
13:25:25           **WARNING** [DEPRECATION WARNING]:
13:25:26           **WARNING** Please use AppDynamics extension buildpack for Python Application instrumentation
13:25:26           **WARNING** for more details: https://docs.pivotal.io/partners/appdynamics/multibuildpack.html
13:25:26    -----> Supplying Python
13:25:26    -----> Installing python 3.6.8
13:25:26           Copy [/tmp/buildpacks/703a7157440391c89c38d3653019a1fc/dependencies/60105c6a75577797577d4015bc219e5c/python-3.6.8-linux-x64-cflinuxfs3-0e8b91a8.tgz]
13:25:27    -----> Installing pip-pop 0.1.3
13:25:27           Copy [/tmp/buildpacks/703a7157440391c89c38d3653019a1fc/dependencies/859523d4d2137906b68eb4c8951d56b3/pip-pop-0.1.3-fc106ef6.tar.gz]
13:25:31           **ERROR** Could not install pip pop: exit status 1
13:25:31    Failed to compile droplet: Failed to run all supply scripts: exit status 14
13:25:31    Cell 4b85e19a-22ee-49ab-8b02-7561218bb0b2 stopping instance ce145cab-ddcc-442a-b2e7-adbaa9bad561
13:25:31    Cell 4b85e19a-22ee-49ab-8b02-7561218bb0b2 destroying container for instance ce145cab-ddcc-442a-b2e7-adbaa9bad561
13:25:31    Cell 4b85e19a-22ee-49ab-8b02-7561218bb0b2 successfully destroyed container for instance ce145cab-ddcc-442a-b2e7-adbaa9bad561
13:25:33 FAILED
13:25:33 Error message:
13:25:33 Error staging application: App staging failed in the buildpack compile phase

Please confirm where necessary:

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/167110969

The labels on this github issue will be updated when the story is started.

kardolus commented 5 years ago

@Manish-Mj hi there! Quickly scanned your logs and configuration. The only thing I can think of right now is perhaps a problem with that PIP_TARGET setting. But before we dive into that, can you perhaps provide us with a test fixture? Some simple app we could use to reproduce this issue would be amazing. Also, do you have the same issue with the latest version of the buildpack? You could try and push your app with -b https://github.com/cloudfoundry/python-buildpack to grab the buildpack from our edge. Just as a test.

Manish-Mj commented 5 years ago

@kardolus Thanks for looking into this, I'll test this & also provide a test fixture as well if that doesn't work.

dfreilich commented 5 years ago

@Manish-Mj Thanks! We also use pip-pop for scanning requirements.txt, and we have a distribution of it in our manifests. Is that binary in the lib/ folder as well?

Manish-Mj commented 5 years ago

@kardolus I did try the latest buildpack but still got the error. Log snippet below.

C:\Users\Administrator\Documents\Manish\Dev_Code\Test-app>cf push -b https://github.com/cloudfoundry/python-buildpack

Pushing from manifest to org Test / space zero as jawarm... Using manifest file C:\Users\Administrator\Documents\Manish\Dev_Code\Test-app\manifest.yml Deprecation warning: Use of 'buildpack' attribute in manifest is deprecated in favor of 'buildpacks'. Please see http:// docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#deprecated for alternatives and other app manifest deprecations . This feature will be removed in the future.

Getting app info... Updating app with these attributes... name: test-app path: C:\Users\Administrator\Documents\Manish\Dev_Code\Test-app buildpacks:

Updating app test-app... Mapping routes... Comparing local files to remote cache... Packaging files to upload... Uploading files... 1.45 MiB / 1.45 MiB [=====================================================================================] 100.00% 4s

Waiting for API to complete processing files...

Staging app and tracing logs... Downloaded app package (4.4M) -----> Download go 1.12.4 -----> Running go build supply /tmp/buildpackdownloads/fe158333cd0331a0073936bf6f26e45e ~ ~ -----> Python Buildpack version 1.6.36 WARNING buildpack version changed from 1.6.29 to 1.6.36 -----> Supplying Python -----> Installing python 3.6.9 Download [https://buildpacks.cloudfoundry.org/dependencies/python/python-3.6.9-linux-x64-cflinuxfs3-f30bc832.t gz] Download [https://buildpacks.cloudfoundry.org/dependencies/manual-binaries/pip-pop/pip-pop-0.1.3-fc106ef6.tar. gz] -----> Installing pip-pop 0.1.3 ERROR Could not install pip pop: exit status 1 Error staging application: App staging failed in the buildpack compile phase FAILED

Manish-Mj commented 5 years ago

Attaching a sample app to test, please let me know if am missing something here. Test-app.zip

Manish-Mj commented 5 years ago

@dfreilich Thanks for the response and sorry for my delayed response. You are right the lib/ folder also had pip_pop but the test app which I attached doesn't have pip pop in the lib/ folder and I see the same error when I try to "cf push" with PIP_TARGET env variable.

Manish-Mj commented 5 years ago

I also tried installing pip on a Ubuntu host with PIP_TARGET set and it failed. Attached console logs with PIP_TARGET & without PIP_TARGET set.

Manish-Mj commented 5 years ago

PIP_Install_Failed_With_PIPTARGET.docx

ndon55555 commented 5 years ago

Hi @Manish-Mj, I just tried out your fixture using the same version of this buildpack. Unfortunately, I can't seem to reproduce your error even when I update the fixture's manifest.yml's environment variable to use PIP_TARGET instead of PYTHONPATH (no errors occur and runnable app is created).

After glancing through the buildpack, I don't think your PIP_TARGET variable would be respected. We run the following command when you have a vendor folder present in the app root: https://github.com/cloudfoundry/python-buildpack/blob/af99a1c95bbc90347e295783a272e52138356072/src/python/supply/supply.go#L627

Would it be possible for you to vendor your application following these steps without setting the PIP_TARGET variable?

Manish-Mj commented 5 years ago

@ndon55555, Thanks for looking into this. It is strange that you didn't get the errors which am stuck with and get it every time. Let me explain my requirement in brief and the current problems with buildpack.

My requirement: I don't want any install to happen in PCF so basically I install all my dependent libs into a folder "lib" under my app's root directory and present the same to PCF along with other app contents. Ideally if I let PCF to do the install then it would have installed the libs sourced in "vendor" folder to following path: '/home/vcap/deps/0/python/lib/python3.6/site-packages' but since I have it pre-installed under 'lib/', I set the PYTHONPATH to '/home/vcap/app/lib' in the manifest file. But that doesn't work. Below I have attached error logs.

As PYTHONPATH didn't work I thought of setting PIP_TARGET to '/home/vcap/app/lib' and was hoping if libs are already installed then pip wouldn't install it again in PCF, but thanks to @ndon55555 for sharing the code it clearly says "--ignore-installed", so PIP_TARGET wouldn't be of use now.

Attaching the manifest file again with PYTHONPATH set to '/home/vcap/app/lib' but it is not prepended to sys.path at all, logs below

Error logs from PCF for Test-app: 2019-07-31T02:20:30.546-04:00 [CELL/0] [OUT] Starting health monitoring of container 2019-07-31T02:20:30.718-04:00 [APP/PROC/WEB/0] [OUT] sys.path: ['/home/vcap/app', '/home/vcap/deps/0', '/home/vcap/deps/0/python/lib/python36.zip', '/home/vcap/deps/0/python/lib/python3.6', '/home/vcap/deps/0/python/lib/python3.6/lib-dynload', '/home/vcap/deps/0/python/lib/python3.6/site-packages'] 2019-07-31T02:20:30.718-04:00 [APP/PROC/WEB/0] [ERR] Traceback (most recent call last): 2019-07-31T02:20:30.718-04:00 [APP/PROC/WEB/0] [ERR] File "main.py", line 13, in 2019-07-31T02:20:30.718-04:00 [APP/PROC/WEB/0] [ERR] from flask import Flask 2019-07-31T02:20:30.718-04:00 [APP/PROC/WEB/0] [ERR] ModuleNotFoundError: No module named 'flask'

Modified .yml to .txt for successful upload manifest.txt

Hope I made myself clear. I'm open for a quick zoom meeting to discuss as well, please let me know.

Manish-Mj commented 5 years ago

Hello Team, Any update on this issue..? Is the information / logs provided good enough..?

Manish-Mj commented 5 years ago

Hello Team, Could you anyone look into this & confirm if it is a bug or by design..?

dfreilich commented 5 years ago

Hey @Manish-Mj , thanks for checking in. We're a bit busy, and haven't had a chance to look at it more, but it's in our backlog, and we'll take a look at it when we have a chance. Sorry about the holdup!

ndon55555 commented 5 years ago

@sclevine Can I get your thoughts on this? I'm not sure that we support what @Manish-Mj is attempting to do

Manish-Mj commented 5 years ago

@dfreilich Thanks for the response. Sure will wait for the team's response.

sclevine commented 5 years ago

In the (soon to be replaced) v2 buildpack API, all of the file paths change between staging and launch. So none of the values used so far (that start with /home/vcap) will successfully make modules available during staging. They may make modules available during launch, but this isn’t really a supported use case for the buildpack.

The buildpack supports the vendoring workflow described here: https://docs.cloudfoundry.org/buildpacks/python/index.html#vendoring

This is so that native extensions are rebuilt on the stack. Otherwise, your app could fail to start unless you’re pushing from a Linux machine that is similar to cflinuxfs3.

Why do you need to bypass this behavior? Can you describe your use case in more detail?

sclevine commented 5 years ago

If you are sure that all the native extensions in the lib folder are runnable on cflinuxfs3 and don't require any recompilation, you might be able to get this working by appending to (not overriding) PYTHONPATH in a .profile file in the app directory:

export PYTHONPATH=$PYTHONPATH:/home/vcap/app/lib

Unlike manifest.yml, .profile will apply after the buildpack modifies the runtime environment., so your changes to PYTHONPATH would stick. Again though, this is not a supported use case.

Manish-Mj commented 5 years ago

@ndon55555 Thanks for the follow up.

@sclevine Thanks for looking into this. We were following vendoring workflow in our product which worked fine until we came across a situation where one of the app had some dependency which required minimum 4 GB of RAM to be built and also took more than 12 to 15 minutes to be pushed to PCF. To cut down this high RAM usage and PCF push time I thought of pre-compiling it in similar linux environment in our pipeline and making it available in 'lib'. This worked fine after tweeking sys.path in python file but was looking for setting this as an env variable rather than in code. I had tried it with manifest.yml file but now understand that it wouldn't work with manifest.yml and could get it working through .profile.

What other alternatives would you suggest in this case..?

ryanmoran commented 4 years ago

@Manish-Mj We think that appending to the PYTHONPATH in the application's .profile file is the best solution for what you are describing. Did that work? Can we close out this issue?

Manish-Mj commented 4 years ago

Thanks @ryanmoran for looking into this, for now I have used other solution, building wheel and uploading to lib folder, so am good. Yes please close this issue. Would one if required in future.