Compatibility with PyPy

[jamadden]

I notice that setup.py specifies both RelStorage[postgresql] and also psycopg2. The former will work correctly on PyPy, but the latter will not (you want psycopg2cffi).

There doesn't seem to be any actual usage of psycopg2 in the code itself, only in a test case:

newtdb$ ack psycopg2
setup.py
3:install_requires = ['setuptools', 'RelStorage[postgresql]', 'psycopg2', 'six']

src/newt/db/_search.py
6:import psycopg2

src/newt/db/tests/base.py
1:import psycopg2
13:        self.base_conn = psycopg2.connect('')

If you need to connect to a database in a test case, you could try this:

import relstorage.adapters.postgresql.adapter as adapter
self.base_conn = adapter.select_driver().connect('')

Then in theory you should Just Work with any supported driver, assuming you stick to the DB-API. Or at least, psyc2pg2 and psycopg2cffi pretty much implement the same extensions.

Adding - pypy-5.4.1 to .travis.yml should run your tests on a new-ish pypy.

[jamadden]

Something in the test suite is leaking an open connection (that gets reclaimed by garbage collection), hence the failure. I'm trying to track it down, but it's proving elusive.

[jamadden]

ZODB.tests.TransactionalUndoStorage definitely leaks open databases and connections but I don't see how that could be related

[jamadden]

I'm pretty sure I've tracked the leaked objects that cause this failure down to relstorage.tests.RecoveryStorage:UndoableRecoveryStorage.checkRestoreAcrossPack (and possibly checkRestoreWithMultipleObjectsInUndoRedo). It takes some weird manipulations but I can reproduce the errors now.

Running the GC here will be the simplest solution, I think, until there's an updated RelStorage release that closes things up properly.

[jamadden]

Thoughts @jimfulton ?

[jimfulton]

Hm, I wasn't notified of this PR. I wonder what I need to do to fix that... sigh

[jimfulton]

Thanks for this! I knew I had some debt in this area. :)

I'll look at this shortly.

[jimfulton]

Oh, I needed to watch this repo. :)

[jamadden]

Thanks. Sorry about the unpleasantness of needing to call gc.collect() due to the leaking test case in RelStorage. But since there's also a leaking test case in ZODB itself, I'm not too sorry 😉 I'll try to submit a PR against ZODB.

[jimfulton]

Gaaaa, it looks like pypy and cpython eggs get named the same thing. This is a pain for buildlut because of it's shared eggs directory. whimper.

[jamadden]

There's no binary (native) code here, so why does it matter? Don't both interpreters ignore bad .pyc files if there's a .py file too?

Which reminds me, there should probably be a setup.cfg file with

[bdist_wheel]
universal = 1

So that creating wheels and uploading them to PyPI does the right thing.

[jamadden]

Oh, or is it the native code in the RelStorage/persistent/BTree eggs that causing the problems? That I've definitely run into before and had to resort to separate egg directories.

[jimfulton]

idk what's going on. I have to remove RelStorage and cffi eggs from my egg cache when switching between cpython and pypy.

For example, switching to pypy without removing those eggs:

ImportError: unable to load extension module '/Users/jim/.buildout/eggs/cffi-1.9.1-py2.7-macosx-10.10-x86_64.egg/_cffi_backend.so': dlopen(/Users/jim/.buildout/eggs/cffi-1.9.1-py2.7-macosx-10.10-x86_64.egg/_cffi_backend.so, 6): Symbol not found: _PyBuffer_Type
  Referenced from: /Users/jim/.buildout/eggs/cffi-1.9.1-py2.7-macosx-10.10-x86_64.egg/_cffi_backend.so
  Expected in: flat namespace
 in /Users/jim/.buildout/eggs/cffi-1.9.1-py2.7-macosx-10.10-x86_64.egg/_cffi_backend.so

Also, RelStorage's egg name is: RelStorage-2.0.0-py2.7-macosx-10.10-x86_64.egg, which suggests something thinks it has binary bits :) Oh, cache_ring.

If I remove the cffi egg and rerun buildout, I get:

ImportError: unable to load extension module '/Users/jim/.buildout/eggs/RelStorage-2.0.0-py2.7-macosx-10.10-x86_64.egg/relstorage/cache/_cache_ring.so': dlopen(/Users/jim/.buildout/eggs/RelStorage-2.0.0-py2.7-macosx-10.10-x86_64.egg/relstorage/cache/_cache_ring.so, 6): Symbol not found: __Py_NoneStruct
  Referenced from: /Users/jim/.buildout/eggs/RelStorage-2.0.0-py2.7-macosx-10.10-x86_64.egg/relstorage/cache/_cache_ring.so
  Expected in: flat namespace
 in /Users/jim/.buildout/eggs/RelStorage-2.0.0-py2.7-macosx-10.10-x86_64.egg/relstorage/cache/_cache_ring.so

[jimfulton]

I don't really understand how pypy handles extensions. When building with pypy, I get the same egg names as with cpython. I wonder if there's something else going on here.

[jamadden]

No, that's what's happening. I've seen the same thing with persistent and other things that use CFFI or optional native code. Wheels use a different wheel "ABI tag" for PyPy and CPython (pypy_41 and cp27-cp27m[u] respectively), but traditional setuptools doesn't know about ABI tags, just the architecture...you'd have the same problem, for example, switching between wide and narrow unicode builds of CPython 2.

[jimfulton]

Yup. But I'd have no reason to switch back and forth between those. sigh

[jamadden]

This is a large part of why I now typically prefer to avoid buildout in favor of virtualenvs for individual projects (reserving buildout for integration of applications). I work on a lot of projects that use CFFI/native code, all of which support PyPy 😄

[jimfulton]

https://github.com/buildout/buildout/pull/325 :-)