search

douban / greenify

Make blocking C library work with gevent
BSD 3-Clause "New" or "Revised" License
430 stars 58 forks source link

Release to PyPi #9

Closed thedrow closed 9 years ago

thedrow commented 9 years ago

There's no need to install from the repository unless you consider this project not ready for consumption.

mckelvin commented 9 years ago

@thedrow Actually it's running in our production environment already. The current version of greenify doesn't support ELF on OS X yet. After that part of work is done, we'll release to PyPi.

thedrow commented 9 years ago

Running in production? Any side effects? We'd really like to have a greenlet friendly version of pylibmc.

mckelvin commented 9 years ago

@thedrow No side effect found yet. We are using https://github.com/douban/python-libmemcached instead of pylibmc currently. Both of the 2 Python libraries are just wrappers, which are based on libmemcached. I guess you just need to patch the "/usr/lib/libmemcached.so" using greenify and then the pylibmc should work without blocking in gevent.

thedrow commented 9 years ago

Why aren't you guys using pylibmc?

mckelvin commented 9 years ago

@thedrow We have already started using python-libmemcached before the birth of pylibmc. These 2 libraries don't keep the same behavior for some functions (like set_multi, after they do the same work successfully, one return True and one return [] ). That actually matters little. Main reason is that the feature of large object support has already been added in python-libmemcached before libmemcached start to support it. These 2 features are not compatible. Since data have been persisted in memcached servers. it's hard for us to do any changement.

Though pylibmc is the fastest among all these libmemcached-based mc client libraries in python, the codebase of libmemcached is quite complex, just like a blackbox. The branch of libmemcached we use is old and even not be possible to sync with upstream. So we have to reinvent a new wheel named libmc, written in C++/Cython with fully benchmark and without extra dependency. which can be seen as a alternative to (libmemcached + python-libmemcached). Benchmark shows it's not slower than pylibmc in almost all cases. Main development work is almost done. We're currently testing it and will consider open source it in the near feature.

thedrow commented 9 years ago

Do you have some example code I can benchmark against? For example 1000 reads with threads vs. 1000 reads with greenify+gevent?

mckelvin commented 9 years ago

@thedrow

One benchmark result and the code: https://gist.github.com/mckelvin/5423501360db27d7bc50 , though we haven't benched greenify+libmc yet.

(I'm afraid the issue is off the topic now. :smile: )

windreamer commented 9 years ago

It's available on pypi now: https://pypi.python.org/pypi/greenify

thedrow commented 9 years ago

Woot!