search

paketo-buildpacks / pip

Apache License 2.0
5 stars 6 forks source link

Support manylinux10 ABI in 3.6.x buildpack by updating to pip 19.x #2

Closed ghost closed 3 years ago

ghost commented 4 years ago

Versions

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

{
   "name": "",
   "build": "",
   "support": "",
   "version": 0,
   "description": "Cloud Foundry at SAP Cloud Platform",
   "min_cli_version": null,
   "min_recommended_cli_version": null,
   "api_version": "2.141.0",
   "osbapi_version": "2.15",

}
➜  ~ cf version
cf version 6.46.0+29d6257f1.2019-07-09

What version of the buildpack you are using?

1.6.36

If you were attempting to accomplish a task, what was it you were attempting to do?

I am trying to cf push an app with vendored dependencies. These vendored dependencies were downloaded using pip version 19.x. Among these vendored dependencies was cryptography-2.8-cp34-abi3-manylinux2010_x86_64.whl

What did you expect to happen?

I expected the push to succeed.

What was the actual behavior?

The push failed because the vendored dependencies could not be installed:

[2019-10-20T21:56:08.834Z]        Collecting cryptography (from sap-xssec==2.0.6->meh-commons==2.0.26->-r /tmp/app/requirements.txt (line 14))
[2019-10-20T21:56:08.834Z]          Could not find a version that satisfies the requirement cryptography (from sap-xssec==2.0.6->meh-commons==2.0.26->-r /tmp/app/requirements.txt (line 14)) (from versions: )
[2019-10-20T21:56:08.834Z]        No matching distribution found for cryptography (from sap-xssec==2.0.6->meh-commons==2.0.26->-r /tmp/app/requirements.txt (line 14))

I believe this problem occurs because the buildpack mentioned uses an older version of pip (18.x), which does not have support for the manylinux2010 ABI (see PEP 571. Support for this ABI was introduced in pip 19.0 according to the pip changelog.

Thus, packages vendored with a newer version of pip cannot be used with the current version of the Python buildpack. This recently started occuring ion my project due to 2.8 release of the cryptography package, which now provides manylinux2010 along the old manylinux1 wheels.

Steps to reproduce:

Suggested resolution

Update pip to 19.x.

Please confirm where necessary:

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/169262822

The labels on this github issue will be updated when the story is started.

dfreilich commented 4 years ago

Hey @mhaas-work , thanks for posting the issue! Python 3.8 comes with pip 19, so that should hopefully solve your problem. I'm closing it for now, but feel free to reopen it if it doesn't help!

ghost commented 4 years ago

Hi @dfreilich, the problem is that I cannot upgrade to Python 3.7 or 3.8 due to some binary dependencies - so the ability to update to a newer pip would be useful.

dfreilich commented 4 years ago

Hmm ok. We're focusing more on our development of the CNBs, and we can take that as a feature request in our pip-cnb, but I don't think that this will be done in this buildpack. I'll transfer it there.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/169680227

The labels on this github issue will be updated when the story is started.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/169680253

The labels on this github issue will be updated when the story is started.

ryanmoran commented 4 years ago

The pip-cnb currently includes the latest version of pip in the buildpack.toml file, but doesn't allow the user to configure the usage of that version in any way. The version of pip that is used is the version included in the python runtime that is installed.

We should either remove the pip dependencies that are included currently in the buildpack.toml as they are not used, or we should enable the user to configure the use of the latest version of pip.

mhaas commented 4 years ago

@ryanmoran Just checking in - do you have any updates here? Or even a possible workaround?

My goal is to use Tensorflow 2.x with Python 3.6. Tensorflow only supports manylinux2010, so I have no way of installing this.

Is there some hook how I can install my own version of pip before staging the virtualenv itself? There is some mention of hooks, but I am not sure if I could leverage that. I didn't dig deep enough.

ryanmoran commented 4 years ago

@mhaas are you using CNBs, or Cloud Foundry buildpacks?

For Cloud Foundry buildpacks, you can run arbitrary code in a hook like can be seen in this fixture app: https://github.com/cloudfoundry/python-buildpack/blob/master/fixtures/with_hooks. Check out the bin/pre_compile file for an example.

As for CNBs, the Python CNB is not anywhere near supported yet. We have only just started working on the python CNBs over in https://github.com/paketo-community/python-runtime. This is only the first buildpack to start the move over to the CNB spec. Ultimately, these new CNBs will replace this existing python-buildpack, but that is not likely to happen very soon.

mhaas commented 4 years ago

Hi @ryanmoran, I am indeed using the older CF buildpacks - and this issue was moved to pip-cnb back in the day. I will investigate the use of the pre_compile hooks again, but it was my impression that this was run BEFORE the deploy stage of the buildpack. This would mean that I would have a hard time installing an updated version of pip, because python would not yet be available. When I was looking into the code, I was under the impression that I would need some kind of hook inside the deploy stage.

Also perhaps consider my thoughts in https://github.com/cloudfoundry/python-buildpack/issues/172 - I believe it is in a general a great idea to have either a user-provided version of pip or simply the latest, similar to how setuptools is currently handled.

arjun024 commented 3 years ago

This issue seems to be about cloudfoundry buildpacks and not paketo buildpacks. I'm closing this for now. Feel free to reopen the issue if it's still relevant to the paketo pip or python CNB