search

cloudfoundry / python-buildpack

Cloud Foundry buildpack for the Python Language
http://docs.cloudfoundry.org/buildpacks/
Apache License 2.0
121 stars 279 forks source link

Data generated by pre_compile in /app is cleaned up #124

Closed sergeybk closed 5 years ago

sergeybk commented 5 years ago

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

{
   "name": "",
   "build": "",
   "support": "...",
   "version": 0,
   "description": "...",
   "authorization_endpoint": "...",
   "token_endpoint": "...",
   "min_cli_version": "6.0.0",
   "min_recommended_cli_version": "6.10.0",
   "app_ssh_endpoint": "...",
   "app_ssh_host_key_fingerprint": "...",
   "app_ssh_oauth_client": "ssh-proxy",
   "doppler_logging_endpoint": "...",
   "api_version": "2.126.0",
   "osbapi_version": "2.14",
   "routing_endpoint": "..."
}
cf version 6.29.0+ff886fa.2017-07-24

What version of the buildpack you are using?

v1.6.21

If you were attempting to accomplish a task, what was it you were attempting to do?

I'm using _bin/precompile script to download, build, and install an unmanaged library used by one of Python packages (couchbase==2.0.9):

#!/usr/bin/env bash

_LIBCOUCHBASE_VER=2.10.2
_PROJECT_ROOT=/app

# Build and install Couchbase C SDK (core library only).
# Couchbase make code runs some Git commands, so requires making a clone.
git clone --depth 1 --branch $_LIBCOUCHBASE_VER \
    https://github.com/couchbase/libcouchbase.git /tmp/libcouchbase_$$
cd /tmp/libcouchbase_$$
./configure.pl \
    --prefix $_PROJECT_ROOT \
    --disable-tools \
    --disable-tests \
    --disable-plugins \
    --disable-couchbasemock
make
make install

What did you expect to happen?

make install deploys the binaries into the /app folder (e.g. /app/lib, /app/share, /app/include), and the application can use them in runtime. Here is how the manifest.yml looks like:

buildpack: https://github.com/cloudfoundry/python-buildpack.git#v1.6.21
stack: cflinuxfs3
env:
  C_INCLUDE_PATH: /app/include
  LIBRARY_PATH: /app/lib
  LD_LIBRARY_PATH: /app/lib

What was the actual behavior?

After cf push command completes, all the binaries generated by make install are deleted, and the application crashes due to missing dependencies. This was not the case with buildpack v1.5.13, but since my organization moves to cflinuxfs3 I have to upgrade to 1.6.x

The workaround I found working: add _bin/postcompile script to explicitly copy the contents of the /app folder into ${PYTHONPATH}, which is preserved by the buildpack:

#!/usr/bin/env bash

cp -r /app/* ${PYTHONPATH}
ls -l ${PYTHONPATH} > /dev/null

I'm not particularly proud of this implementation, so appreciate a better solution. Also, notice the non-functional second line: if I comment it out, the workaround stops working.

Please confirm where necessary:

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/162570710

The labels on this github issue will be updated when the story is started.

kardolus commented 5 years ago

Tried to reproduce the issue real quick- having the logs would be very helpful. Could you post them here?

sergeybk commented 5 years ago

Here you go: cf-logs.txt

kardolus commented 5 years ago

In the staging container LD_LIBRARY_PATH=/app/lib. In the launch container LD_LIBRARY_PATH=/home/vcap/deps/0/lib:/app/lib. The /app folder does not carry over from the staging container to the launch container. Interesting work around though; moving everything from the app folder to the PYTHONPATH in the post_compile hook. When post_compile runs in the staging container, the PYTHONPATH is set to /tmp/contents<some_guid>/deps/0. This folder does carry over and is available in the launch container as /home/vcap/deps/0/.

I think we could write a better solution on the buildpacks side. If the root folder of deps/0 is known during pre_compile then I don’t think we need a post_compile fix (?). I’m fairly new on the team though. @sclevine what do you think?

sclevine commented 5 years ago

/app has never pointed to the app dir during staging. Instead of copying the dependencies to /app, you can use the current working directory of the pre_compile or post_compile script.

sergeybk commented 5 years ago

@sclevine thanks for confirming what I suspected was the case. Let me just reiterate that it worked in the older buildpack v1.5.13, so the change in behavior was quite significant.

Your suggestion makes sense, however I couldn't get it working. Here is how I modified my _bin/precompile script (and completely deleted _bin/postcompile), I hope that's what you meant:

#!/usr/bin/env bash

_LIBCOUCHBASE_VER=2.10.2
_PROJECT_ROOT=${PWD}

echo "Use $_PROJECT_ROOT as the output directory"
export C_INCLUDE_PATH="${C_INCLUDE_PATH}:$_PROJECT_ROOT/include"
export LIBRARY_PATH="${LIBRARY_PATH}:$_PROJECT_ROOT/lib"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:$_PROJECT_ROOT/lib"

env

# Build and install Couchbase C SDK (core library only).
# Couchbase make code runs some Git commands, so requires making a clone.
git clone --depth 1 --branch $_LIBCOUCHBASE_VER \
    https://github.com/couchbase/libcouchbase.git /tmp/libcouchbase_$$
cd /tmp/libcouchbase_$$
./configure.pl \
    --prefix $_PROJECT_ROOT \
    --disable-tools \
    --disable-tests \
    --disable-plugins \
    --disable-couchbasemock
make
make install

Later on couchbase headers cannot be found.

Logs attached: cf-logs.txt

The reason seems to be that environment variables defined in _precompile script aren't carried over to the next stage. In order for the pip install stage to pass, I had to make quite an ugly change in manifest.yaml:

buildpack: https://github.com/cloudfoundry/python-buildpack.git#v1.6.21
stack: cflinuxfs3
env:
  C_INCLUDE_PATH: /tmp/app/include
  LIBRARY_PATH: /tmp/app/lib
  LD_LIBRARY_PATH: /tmp/app/lib

But then it still fails on startup, unable to find the couchbase library: cf-logs.txt

djoyahoy commented 5 years ago

Hi @sergeybk, I was able to get this working using your pre_compile script and manifest.yml with minor changes:

#!/usr/bin/env bash

_LIBCOUCHBASE_VER=2.10.2
_PROJECT_ROOT=${PWD}

# Build and install Couchbase C SDK (core library only).
# Couchbase make code runs some Git commands, so requires making a clone.
git clone --depth 1 --branch $_LIBCOUCHBASE_VER \
    https://github.com/couchbase/libcouchbase.git /tmp/libcouchbase_$$
cd /tmp/libcouchbase_$$
./configure.pl \
    --prefix $_PROJECT_ROOT \
    --disable-tools \
    --disable-tests \
    --disable-plugins \
    --disable-couchbasemock
make
make install

and

buildpack: https://github.com/cloudfoundry/python-buildpack.git#v1.6.21
stack: cflinuxfs3
env:
  C_INCLUDE_PATH: /tmp/app/include
  LIBRARY_PATH: /tmp/app/lib
  LD_LIBRARY_PATH: /tmp/app/lib

Keep in mind that the env set by manifest.yml is "sticky". It will persist between pushes of your application. Try deleting your app with cf delete <app_name> and then pushing the application again.

It may also be worth ditching the pre_compile script and attempting to use the apt-buildpack to supply libcouchbase https://github.com/cloudfoundry/apt-buildpack

sergeybk commented 5 years ago

The application indeed starts, unless you actually try using couchbase library :) Once you do, the following pops up:

[APP/PROC/WEB/0] ERR [2018-12-16 05:56:38 +0000] [8] [INFO] Starting gunicorn 19.6.0
[APP/PROC/WEB/0] ERR [2018-12-16 05:56:38 +0000] [8] [INFO] Listening at: http://0.0.0.0:8080 (8)
[APP/PROC/WEB/0] ERR [2018-12-16 05:56:38 +0000] [8] [INFO] Using worker: sync
[APP/PROC/WEB/0] ERR [2018-12-16 05:56:38 +0000] [61] [INFO] Booting worker with pid: 61
[APP/PROC/WEB/0] ERR [2018-12-16 05:56:38 +0000] [61] [ERROR] Exception in worker process
[APP/PROC/WEB/0] ERR Traceback (most recent call last):
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
[APP/PROC/WEB/0] ERR     worker.init_process()
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
[APP/PROC/WEB/0] ERR     self.load_wsgi()
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
[APP/PROC/WEB/0] ERR     self.wsgi = self.app.wsgi()
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
[APP/PROC/WEB/0] ERR     self.callable = self.load()
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
[APP/PROC/WEB/0] ERR     return self.load_wsgiapp()
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
[APP/PROC/WEB/0] ERR     return util.import_app(self.app_uri)
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app
[APP/PROC/WEB/0] ERR     __import__(module)
[APP/PROC/WEB/0] ERR   File "/home/vcap/app/run.py", line 6, in <module>
[APP/PROC/WEB/0] ERR     from couchbase.bucket import Bucket
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/couchbase/__init__.py", line 28, in <module>
[APP/PROC/WEB/0] ERR     from couchbase.user_constants import *
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/couchbase/user_constants.py", line 21, in <module>
[APP/PROC/WEB/0] ERR     import couchbase._bootstrap
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/couchbase/_bootstrap.py", line 34, in <module>
[APP/PROC/WEB/0] ERR     import couchbase.exceptions as E
[APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/couchbase/exceptions.py", line 18, in <module>
[APP/PROC/WEB/0] ERR     import couchbase._libcouchbase as C
[APP/PROC/WEB/0] ERR ImportError: libcouchbase.so.2: cannot open shared object file: No such file or directory
[APP/PROC/WEB/0] ERR [2018-12-16 05:56:38 +0000] [61] [INFO] Worker exiting (pid: 61)
[APP/PROC/WEB/0] ERR [2018-12-16 05:56:38 +0000] [8] [INFO] Shutting down: Master
[APP/PROC/WEB/0] ERR [2018-12-16 05:56:38 +0000] [8] [INFO] Reason: Worker failed to boot.

To reproduce, please add the following code to your main method:

from couchbase.bucket import Bucket
bucket = Bucket('connection_string', password='password')
djoyahoy commented 5 years ago

Oh no! Sorry @sergeybk. Try adding the following dir and file to your apps root, .profile.d/setup_ld_path.sh with the contents:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/lib
sergeybk commented 5 years ago

Yeah, that did the trick, the app is running now!

Indeed, this workaround is much better than mine. Still, I'm calling it a workaround since in the _bin/precompile script I'm using ${PWD} environment variable to install couchbase lib, while in manifest.yaml there is a hardcoded /tmp/app path.

Is there a cleaner solution?

djoyahoy commented 5 years ago

@sergeybk I'm glad to hear that this worked for you. I agree, this is a silly workaround. @sclevine @shanks3012 is there something we can do here to make this experience less kludgy?

@sergeybk you can try to use the https://github.com/cloudfoundry/apt-buildpack to supply libcouchbase if this solution is not sustainable.

sclevine commented 5 years ago

Let me just reiterate that it worked in the older buildpack v1.5.13, so the change in behavior was quite significant.

v1.5.13 is over 2 years old, so I would not be surprised if it behaves differently from the current release. That said, /app has never been preserved after staging on Cloud Foundry, so this particular behavior should not have changed.

Try adding the following dir and file to your apps root, .profile.d/setup_ld_path.sh

This should be in a <app>/.profile file and not the <app>/.profile.d/ directory. While placing files to be sourced in .profile.d in the app directory happens to work in some cases, that directory is actually reserved for (Heroku-style) buildpacks.

is there something we can do here to make this experience less kludgy?

The new CNB buildpack API v3 (https://buildpacks.io) will make tasks like this much easier. Many of these environment variables will be set by the platform automatically, and the staging and launch directories will always be the same. Additionally, we're working on a compatibility layer that will bring many of the benefits of v3 to existing CF foundations.