search

aerospike / aerospike-client-python

Aerospike Python Client
Apache License 2.0
133 stars 110 forks source link

Aerospike + PyPy. ImportError. Mac OS X. #80

Open asyne opened 9 years ago

asyne commented 9 years ago 
$ pypy main.py

Traceback (most recent call last):
  File "main.py", line 2, in <module>
    import aerospike
  File "/Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.9-x86_64.egg/aerospike.py", line 7, in <module>
    __bootstrap__()
  File "/Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.9-x86_64.egg/aerospike.py", line 6, in __bootstrap__
    imp.load_dynamic(__name__,__file__)
ImportError: unable to load extension module '/Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.9-x86_64.egg/aerospike.so': dlopen(/Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.9-x86_64.egg/aerospike.so, 6): Symbol not found: _PyBool_Type
  Referenced from: /Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.9-x86_64.egg/aerospike.so
  Expected in: flat namespace
 in /Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.9-x86_64.egg/aerospike.so

__Also, there is an error during the installation of Aerospike using pip with PyPy. Btw easy_install doesn't show any error, but result is the same -- ImportError.__

$ pypy -m pip install aerospike
Collecting aerospike
  Using cached aerospike-1.0.53.tar.gz
Building wheels for collected packages: aerospike
  Running setup.py bdist_wheel for aerospike
  Complete output from command /usr/local/bin/pypy -c "import setuptools;__file__='/private/var/folders/1t/5_908kvd373ctwcsxgqp49hw0000gq/T/pip-build-sQ97fo/aerospike/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/1t/5_908kvd373ctwcsxgqp49hw0000gq/T/tmpkH7Mgbpip-wheel-:
  running bdist_wheel
  running build
  running build_ext
  building 'aerospike' extension
  creating build
  creating build/temp.macosx-10.11-x86_64-2.7
  creating build/temp.macosx-10.11-x86_64-2.7/src
  creating build/temp.macosx-10.11-x86_64-2.7/src/main
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/client
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/key
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/query
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/scan
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/lstack
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/lset
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/llist
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/lmap
  creating build/temp.macosx-10.11-x86_64-2.7/src/main/geospatial
  cc -O2 -fPIC -Wimplicit -arch x86_64 -Isrc/include -I/Users/food/Applications/pypy-2.6.1/include -c src/main/aerospike.c -o build/temp.macosx-10.11-x86_64-2.7/src/main/aerospike.o -std=gnu99 -g -Wall -fPIC -O1 -fno-common -fno-strict-aliasing -Wno-strict-prototypes -march=nocona -D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_GNU_SOURCE -D_DARWIN_UNLIMITED_SELECT -DMARCH_x86_64
  In file included from src/main/aerospike.c:22:
  In file included from src/include/client.h:21:
  src/include/types.h:22:10: fatal error: 'aerospike/aerospike.h' file not found
  #include < aerospike/aerospike.h>
           ^
  1 error generated.
  error: command 'cc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for aerospike
Failed to build aerospike
Installing collected packages: aerospike
  Running setup.py install for aerospike
Successfully installed aerospike-1.0.53

System info: ProductName: Mac OS X ProductVersion: 10.11 BuildVersion: 15A284

LLVM info: Apple LLVM version 7.0.0 (clang-700.0.72) Target: x86_64-apple-darwin15.0.0 Thread model: posix

PyPy & pip info: Python 2.7.10 (f3ad1e1e1d62, Aug 28 2015, 09:36:42) [PyPy 2.6.1 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin

dmitriy-tkalich commented 9 years ago

+

rbotzer commented 9 years ago

The client compiles with the CPython interpreter versions 2.6 and 2.7. See the PyPI repo for more details, and the README.

asyne commented 9 years ago

Ok. So I have built aerospike-client-python using CPython: python setup.py build --force . And then tried installing it to PyPy's site-packages with pypy setup.py install --force. Is it correct?

The result, by the way, is the same, but error have changed in some places:

Traceback (most recent call last):
  File "main.py", line 2, in <module>
    import aerospike
  File "/Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.11-x86_64.egg/aerospike.py", line 7, in <module>
    __bootstrap__()
  File "/Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.11-x86_64.egg/aerospike.py", line 6, in __bootstrap__
    imp.load_dynamic(__name__,__file__)
ImportError: unable to load extension module '/Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.11-x86_64.egg/aerospike.pypy-26.so': dlopen(/Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.11-x86_64.egg/aerospike.pypy-26.so, 6): Symbol not found: _PyByteArray_AsString
  Referenced from: /Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.11-x86_64.egg/aerospike.pypy-26.so
  Expected in: flat namespace
 in /Users/food/Applications/pypy-2.6.1/site-packages/aerospike-1.0.53-py2.7-macosx-10.11-x86_64.egg/aerospike.pypy-26.so

BTW, I also tried to build and install aerospike client using CPython and then move package from CPython to PyPy site-packages. Still have same problem.

rbotzer commented 9 years ago

Well, it's a C extension for CPython, meaning it uses its C API. PyPy runs pure Python code, so it cannot use our module. I'm looking into ways of getting our module to work with PyPy.

asyne commented 9 years ago

I see. Maybe there is some way for me to call CPython package directly from PyPy code? Something like running through cmd, but native, e.g. execfile()?

asyne commented 9 years ago

The main problem that PyPy is missing some types from CPython like PyBool, PyByteArray, etc. That is kinda obvious. Temporary solving for this is to call CPython from PyPy using PyExt or PyMetabiosis (which was suggested to use on PyPy's IRC channel) extension. This is how it works in PyPy with PyMetabiosis:

import pymetabiosis
import aspike_config

aerospike = pymetabiosis.module.import_module("aerospike")

client = aerospike.client(aspike_config.get_config())
client.connect()
...
rbotzer commented 9 years ago

If you can get such a project going I'd be happy to promote it. I was looking into CFFI as a more generic solution, since it works with more runtimes than just PyPy.

arthurprs commented 9 years ago

A few weeks a go I hacked the driver to avoid using the ByteArray functions/types, very hacky but it worked well enough for our application (except for the callback based parts like query and scan).

Final performance was bad though (as expected from the CPython emulation layer). I'm almost sure it had some memory leaks too. You can check the code here https://github.com/arthurprs/aerospike-client-python/tree/pypy

Finally, unless PYPY makes the rest of you application fly I recommend avoiding it unless you can find an wrapper that adds less overhead (pymetabiosis is probably even slower).