search

cloudfoundry / docs-buildpacks

http://docs.cloudfoundry.org/buildpacks/
Apache License 2.0
21 stars 118 forks source link

Python vendoring: note pitfalls #291

Closed vanschelven closed 7 months ago

vanschelven commented 2 years ago

Vendoring python dependencies is not trivial, and this should at least be reflected in the documentation, preferably while presenting a solution that "mostly works".

The problem is that python packages may contain arbitrary build steps (compilation), and that:

  1. compilation may require certain other files (not provided by pip) to be present in the build environment.
  2. the results of compilation are not generally transferable from one environment to the next. In particular: they are not generally transferable from arbitrary build environments to the python-buildpack.

The present solution (pip download -r requirements.txt --no-binary=:none: -d vendor) will, when no extra steps are taken, at least fail for packages which are not trivially buildable, and for which no wheels are provided on pypi or some other repo. This is because for such packages what will be vendored will not be a wheel, but a source package that remains to be built (in the python-buildpack itself). If the python-buildpack does not contain the necessary requirements for building (which it generally does not) building will fail. An example of such a package is python-ldap==3.4.2 because it depends on bdist_wheel to be available while building.

A solution with a higher chance of working is probably to build wheels for such packages yourself in the build environment (using pip wheel). However, if that road is taken great care must be taken to sufficiently match the build and target environments, because wheels are not generally transferable. That, in turn, depends on a careful specification of the python-buildpack's python version, and includes such details as the version of glibc on the target platform.

See also #289 and #290; but the issue described at present is much more general.

cf-gitbot commented 2 years ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

anita-flegg commented 9 months ago

@ryanmoran , is any action planned for this issue? If not, I will close the issue. Thanks.

anita-flegg commented 7 months ago

No activity - closing.