pip 18: pep518 support causing issues on PyPy

[alex]

Environment

• pip version: 18.0
• Python version: pypy2.7 5.10.0
• OS: Ubuntu Trusty

Description

Now that pip supports pep518 installing packages from source (🎉), we're attempting to integrate it into pyca/cryptography: https://github.com/pyca/cryptography/pull/4363

This works great on all platforms, except for PyPy, where there's apparently some sort of issue that causes our cffi extensions to not get built (or perhaps not get build in the right location).

Expected behavior

It works as smoothely on PyPy as it does on CPython!

How to Reproduce

https://github.com/pyca/cryptography/pull/4363 + Travis

Output

https://travis-ci.org/pyca/cryptography/jobs/406816583

[pradyunsg]

Wohooo! First pip 18.0 issue. ;)

Looking at https://github.com/pyca/cryptography/issues/3548#issuecomment-302824577, it seems to me you shouldn't be using cffi on PyPy...

[alex]

Depending on cffi on PyPy should be fine; modern versions of cffi simply have their setup.py install nothing if they're running on PyPy -- https://bitbucket.org/cffi/cffi/src/68933f0ea99adb777121f0f3763cc5ae072a64ee/setup.py?at=default&fileviewer=file-view-default#setup.py-180

Is this somehow incomaptible with pep 518?

[pradyunsg]

Is this somehow incomaptible with pep 518?

Nope.

---

I'm not sure what's happening here; @dstufft or @jaraco might have more clue...

[daa]

When pip installs cffi into build environment under PyPy cffi-1.11.5.dist-info directory is created without entry_points.txt file which is source for setuptools to find setup keywords, cffi_modules namely in this case. So cffi_modules keyword is not found and consequently cffi extensions are not built.

[alex]

Nice debugging!

What's the right fix? Should we just use env markers to make it conditional?

On Mon, Jul 23, 2018, 11:48 AM daa notifications@github.com wrote:

When pip installs cffi into build environment under PyPy cffi-1.11.5.dist-info directory is created without entry_points.txt file which is source for setuptools to find setup keywords, cffi_modules namely in this case. So cffi_modules keyword is not found and consequently cffi extensions are not built.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pypa/pip/issues/5631#issuecomment-407105429, or mute the thread https://github.com/notifications/unsubscribe-auth/AAADBJyf5tEfu4w0GhEWsoUqMOq8Yuv4ks5uJfBKgaJpZM4VaAmS .

[daa]

Forgot to mention that described behaviour is triggered by the way pip uses build environment - it sets PYTHONPATH variable and that path becomes one of the first for pkg_resources to search distributions in, so it finds installed cffi without entry points and does not look further. Conversely if we install cffi into a virtualenv, that env's site-packages is the last entry and pkg_resources correctly find cffi distribution inside of lib_pypy.

And for the fix, I see at least 3 ways to deal with issue:

• use environment marker and make cffi build dependency conditional
• make cffi install entry points even if it is run under PyPy
• somehow rework pip's work with build environment to be closer to virtualenv or venv environments. Actually I don't know which would be better, but the easiest is making cffi conditional and I'd prefer it.

[pradyunsg]

Nice debugging @daa! ^>^

ISTM the simplest fix for cryptography is indeed making cffi conditional; slightly longer term, I do feel cffi should change to install entry_points.txt under PyPy as well.

Then again, those are cases where the work someone else has to do so, maybe I don't feel those costs. :)

[chrahunt]

Is there anything for pip to do here?

[jaraco]

make cffi install entry points even if it is run under PyPy

That seems like the most correct solution to me.

[daa]

make cffi install entry points even if it is run under PyPy

That seems like the most correct solution to me.

This was already done: https://bitbucket.org/cffi/cffi/pull-requests/88, https://bitbucket.org/cffi/cffi/issues/371/cffi-should-specify-entry-points-for-pypy. So pip can't and shouldn'd do anything here.

[chrahunt]

Thanks both of you. I will close this now as resolved.