Upgrade the messaging system

[MarioVilas]

Right now we're using Python's multiprocessing module for everything, but the way it uses pipes has proven to be somewhat inefficient. I think we would benefit from using a proper message queue.

Main options are:

• RabbitMQ: I've used it in the past and it's great. However it requires a server to work, and it's written in C (with Python bindings of course).
• ZeroMQ: I've no experience with it, but it's supposed to be much more lightweight than RabbitMQ.
• ActiveMQ: Written in Java. I've used it from Java, but not from Python, although it seems possible: https://activemq.apache.org/python.html

However we need a pure-Python fallback to keep the core platform independent. A pure-Python MQ implementation would be ideal for this.

[MarioVilas]

This could be of use too, we should check it out: http://gearman.org/download/

[jekkay]

What about ActiveMQ, I use ActiveMQ currently. http://activemq.apache.org/

[MarioVilas]

Good idea, I've added it to the ticket description.

[MarioVilas]

@cr0hn proposes using kombu, and I think he's right. It's a pure-Python module that wraps over other messaging libraries when available and falls back to a pure-Python implementation of an AMQP client when missing, so it pretty much gives us the best of both worlds.

https://github.com/celery/kombu

[MarioVilas]

In fact, with any luck we could just use Celery directly to replace both the messaging system and the process pool. https://github.com/celery/celery

[cr0hn]

Yes, celery can be the better option and, if we want (in the future), also can be integrated with django:

http://ask.github.io/celery/tutorials/clickcounter.html

El 10/12/2013, a las 18:31, Mario Vilas notifications@github.com escribió:

In fact, with any luck we could just use Celery directly to replace both the messaging system and the process pool. https://github.com/celery/celery

— Reply to this email directly or view it on GitHub.

[MarioVilas]

Relevant: http://taotetek.net/2011/02/02/python-multiprocessing-with-zeromq/

[MarioVilas]

Closing as duplicate of #228.