search

chtd / psycopg2cffi

Port to cffi with some speed improvements
Other
177 stars 43 forks source link

Installation problems #35

Closed ludovic-gasc closed 9 years ago

ludovic-gasc commented 9 years ago

For now, psycopg2cffi from master branch works pretty well on CPython 3.4, but the release on PyPI has some issues. If you have few time, it should be useful for new corners to publish a new release, everybody doesn't have the reflex to test also with the development version when you have some issues with latest stable version.

lopuhin commented 9 years ago

Latest release on pypi (2.6.1) should match current state of the master branch. Perhaps the wrong version is picked by pip, or the is some difference between them. Could you describe some way to reproduce the problem? I am glad you've found psycopg2cffi useful.

ludovic-gasc commented 9 years ago

Will you be present to the PyPy sprint code at PyCON-US ? It would be quicker to show you that directly.

lopuhin commented 9 years ago

No, unfortunately not, I am really far away from PyCon)

ludovic-gasc commented 9 years ago

Ok, no problems, I'll give you all details this week-end.

ludovic-gasc commented 9 years ago

It needs some work to create for you an all-in-one Python script easy to reproduce the bug.

Nevertheless, I already give you the stacktrace I have with pyscopg2cffi on PyPI that I don't have with the Git repository, maybe you'll have an idea to fix that:

(pyvenv) lg@steroids:~/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour$ api_hour -ac --chdir=hello/ --config_dir=hello/etc/hello/ hello:Container -w 1
gunicorn.error:ERROR 2015-04-11 12:28:22,728 glogging.py:224 => Exception in worker process:
Traceback (most recent call last):
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/vengine_cpy.py", line 150, in load_library
    self.verifier.modulefilename)
ImportError: /home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/__pycache__/_cffi__x78541fd6xba1dd32f.cpython-34m.so: undefined symbol: PQfmod

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
    worker.init_process()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/api_hour/worker.py", line 27, in init_process
    super().init_process()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/workers/base.py", line 114, in init_process
    self.wsgi = self.app.wsgi()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/api_hour/application.py", line 72, in load
    return util.import_app(self.app_uri)
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/util.py", line 356, in import_app
    __import__(module)
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/hello/hello/__init__.py", line 5, in <module>
    import aiopg
File "/home/lg/Documents/IDEA/aiopg/aiopg/__init__.py", line 5, in <module>
    from .connection import connect, Connection, TIMEOUT as DEFAULT_TIMEOUT
File "/home/lg/Documents/IDEA/aiopg/aiopg/connection.py", line 4, in <module>
    import psycopg2cffi as psycopg2
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/__init__.py", line 4, in <module>
    from psycopg2cffi import extensions
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/extensions.py", line 39, in <module>
    from psycopg2cffi._impl import connection as _connection
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/connection.py", line 10, in <module>
    from psycopg2cffi._impl import exceptions
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/exceptions.py", line 12, in <module>
    from psycopg2cffi._impl.libpq import libpq, ffi
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/libpq.py", line 257, in <module>
    ext_package='psycopg2cffi')
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/api.py", line 367, in verify
    lib = self.verifier.load_library()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/verifier.py", line 97, in load_library
    return self._load_library()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/verifier.py", line 207, in _load_library
    return self._vengine.load_library()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/vengine_cpy.py", line 153, in load_library
    raise ffiplatform.VerificationError(error)
cffi.ffiplatform.VerificationError: importing '/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/__pycache__/_cffi__x78541fd6xba1dd32f.cpython-34m.so': /home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/__pycache__/_cffi__x78541fd6xba1dd32f.cpython-34m.so: undefined symbol: PQfmod
Traceback (most recent call last):
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/vengine_cpy.py", line 150, in load_library
    self.verifier.modulefilename)
ImportError: /home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/__pycache__/_cffi__x78541fd6xba1dd32f.cpython-34m.so: undefined symbol: PQfmod

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
    worker.init_process()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/api_hour/worker.py", line 27, in init_process
    super().init_process()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/workers/base.py", line 114, in init_process
    self.wsgi = self.app.wsgi()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/api_hour/application.py", line 72, in load
    return util.import_app(self.app_uri)
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/util.py", line 356, in import_app
    __import__(module)
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/hello/hello/__init__.py", line 5, in <module>
    import aiopg
File "/home/lg/Documents/IDEA/aiopg/aiopg/__init__.py", line 5, in <module>
    from .connection import connect, Connection, TIMEOUT as DEFAULT_TIMEOUT
File "/home/lg/Documents/IDEA/aiopg/aiopg/connection.py", line 4, in <module>
    import psycopg2cffi as psycopg2
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/__init__.py", line 4, in <module>
    from psycopg2cffi import extensions
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/extensions.py", line 39, in <module>
    from psycopg2cffi._impl import connection as _connection
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/connection.py", line 10, in <module>
    from psycopg2cffi._impl import exceptions
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/exceptions.py", line 12, in <module>
    from psycopg2cffi._impl.libpq import libpq, ffi
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/libpq.py", line 257, in <module>
    ext_package='psycopg2cffi')
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/api.py", line 367, in verify
    lib = self.verifier.load_library()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/verifier.py", line 97, in load_library
    return self._load_library()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/verifier.py", line 207, in _load_library
    return self._vengine.load_library()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/vengine_cpy.py", line 153, in load_library
    raise ffiplatform.VerificationError(error)
cffi.ffiplatform.VerificationError: importing '/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/__pycache__/_cffi__x78541fd6xba1dd32f.cpython-34m.so': /home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi/_impl/__pycache__/_cffi__x78541fd6xba1dd32f.cpython-34m.so: undefined symbol: PQfmod
lopuhin commented 9 years ago

Hmm, nothing obvious so far, it is strange that PQfmod is the missing symbol. Perhaps you have a link to a failed Travis CI build? Otherwise pip version and you platform description would be helpful.

ludovic-gasc commented 9 years ago

Hmm, nothing obvious so far, it is strange that PQfmod is the missing symbol. Perhaps you have a link to a failed Travis CI build?

No travis-CI link, it's on my laptop.

Otherwise pip version and you platform description would be helpful.

I use a Kubuntu 14.04 with latest upgrades. I use Python 3.4.2, compiled locally with Pythonz. I use pip 6.1.1.

The very strange stuff is why it works via git master and not via PyPI if it's the same source code ? If you don't have a magic clue on that, I will test on a Debian Jessie. If it works on Debian Jessie, I'll admit it's only on my computer. If I reproduce, maybe something is wrong.

lopuhin commented 9 years ago

One thing I can suggest is to try to download the package from pypi by hand, extract it and run python setup.py install, if you did not try this already - it might behave in a different way from just running pip install psycopg2cffi.

ludovic-gasc commented 9 years ago

It's worse, cf. below. Maybe something is wrong on my laptop, I'll test on a clean Debian instance to be sure.

(pyvenv) lg@steroids:~/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour$ api_hour -ac --chdir=hello/ --config_dir=hello/etc/hello/ hello:Container -w 1
gunicorn.error:ERROR 2015-04-13 14:29:58,590 glogging.py:224 => Exception in worker process:
Traceback (most recent call last):
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
    worker.init_process()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/api_hour/worker.py", line 27, in init_process
    super().init_process()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/workers/base.py", line 114, in init_process
    self.wsgi = self.app.wsgi()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/api_hour/application.py", line 72, in load
    return util.import_app(self.app_uri)
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/gunicorn/util.py", line 356, in import_app
    __import__(module)
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/hello/hello/__init__.py", line 5, in <module>
    import aiopg
File "/home/lg/Documents/IDEA/aiopg/aiopg/__init__.py", line 5, in <module>
    from .connection import connect, Connection, TIMEOUT as DEFAULT_TIMEOUT
File "/home/lg/Documents/IDEA/aiopg/aiopg/connection.py", line 4, in <module>
    import psycopg2cffi as psycopg2
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi-2.6.1-py3.4.egg/psycopg2cffi/__init__.py", line 4, in <module>
    from psycopg2cffi import extensions
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi-2.6.1-py3.4.egg/psycopg2cffi/extensions.py", line 39, in <module>
    from psycopg2cffi._impl import connection as _connection
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi-2.6.1-py3.4.egg/psycopg2cffi/_impl/connection.py", line 10, in <module>
    from psycopg2cffi._impl import exceptions
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi-2.6.1-py3.4.egg/psycopg2cffi/_impl/exceptions.py", line 12, in <module>
    from psycopg2cffi._impl.libpq import libpq, ffi
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi-2.6.1-py3.4.egg/psycopg2cffi/_impl/libpq.py", line 257, in <module>
    ext_package='psycopg2cffi')
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/api.py", line 367, in verify
    lib = self.verifier.load_library()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/verifier.py", line 95, in load_library
    self._write_source()
File "/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/cffi/verifier.py", line 183, in _write_source
    with open(self.sourcefilename, "w") as fp:
NotADirectoryError: [Errno 20] Not a directory: '/home/lg/Documents/IDEA/FrameworkBenchmarks/frameworks/Python/API-Hour/pyvenv/lib/python3.4/site-packages/psycopg2cffi-2.6.1-py3.4.egg/psycopg2cffi/_impl/__pycache__/_cffi__x78541fd6xba1dd32f.c'
ludovic-gasc commented 9 years ago

Ok, I've found something: When I install psycopg2cffi via setup.py develop or pip install -e . from master branch or PyPI release, it works. But, with setup.py install or pip install . I've theses stacktraces.

lopuhin commented 9 years ago

Great, so this looks like a problem with how setup.py is written - I think I will be able to work out the reason from the details you have given, thank you! I think I will look into it in a couple of days.

ludovic-gasc commented 9 years ago

@lopuhin You're welcome.

stefankoegl commented 9 years ago

I seem to have a very similar problem.

I've pasted my output for installing / importing to https://gist.github.com/stefankoegl/cfd6be00dc25a1627326 (Ubuntu 15.04 with latest packages, using CPython 3.4)

When installing with

pip install -e git+https://github.com/chtd/psycopg2cffi.git#egg=psycopg2cffi-dev

everything is working fine. More interestingly, though, my travis builds (which install from PyPI) are working fine.

lopuhin commented 9 years ago

Reproduced on CPython 3.4 on Ubuntu 14.04 too (creating venv using this https://gist.github.com/damienstanton/f63c8aed8f4a432cfcf2 workaround).

lopuhin commented 9 years ago

The plan is to use cffi 1.0, that makes installation more robust. I made the required changes in cffi-1.0 branch, the way to test is (first installing the dependencies, cause we will use a custom index server):

pip install six
pip install -e hg+ssh://hg@bitbucket.org/kostialopuhin/cffi@cffi-1.0#egg=cffi   # a little fix over default cffi
pip install -i https://testpypi.python.org/pypi psycopg2cffi==2.7.0.dev2

I am not sure yet how to handle the transition to cffi 1.0 (especially for pypy) - maybe a proper release will have to wait for some time.

lopuhin commented 9 years ago

Just released 2.7.0 version with cffi 1.0 support - everything should install fine now, will leave the issue open for a while.

ludovic-gasc commented 9 years ago

I'll test that as soon as possible.

thedrow commented 9 years ago

@lopuhin Can you please tag it in the repository?

lopuhin commented 9 years ago

@thedrow just pushed 2.7.0 tag (forgot to push it with the other changes)

thedrow commented 9 years ago

Thanks!