collective / collective.taskqueue

Asyncore-based asynchronous task queue for Plone
https://pypi.org/project/collective.taskqueue
8 stars 7 forks source link

doesn't support redis failover #11

Open djay opened 8 years ago

djay commented 8 years ago

for instance by using http://redis.io/topics/sentinel

datakurre commented 8 years ago

Some implementation notes below, but at first: is Redis sentinel multi master? What I understood from redis python client documentation is that slaves are read only, which makes c.taskqueue always require to use master.

I have no experience of sentinel yet. By reading https://pypi.python.org/pypi/redis it seems that it would be enough to add a single new redis configuration argument

    <taskqueue>
      type redis
      sentinel mymaster
    </taskqueue>

and then create connection with sentinel.master_for('mymaster', socket_timeout=0.1).

What remains to be checked while implementing this, is, does asyncore loop's socket map survive for redis connection change.

djay commented 8 years ago

Sentinal handles switching master for you. So I think it would work where upon you get a disconnection you go back to sentinal to workout where the master is. Some interesting detail in https://aphyr.com/posts/283-call-me-maybe-redis Just a quick look but doesn't seem to me.like cellery and rabbitmq is much better.

On Thu, 23 Jul 2015 08:06 Asko Soukka notifications@github.com wrote:

Some implementation notes below, but at first: is Redis sentinel multi master? What I understood from redis python client documentation is that slaves are read only, which makes c.taskqueue always require to use master.

I have no experience of sentinel yet. By reading https://pypi.python.org/pypi/redis it seems that it would be enough to add a single new redis configuration argument

<taskqueue>
  type redis
  sentinel mymaster
</taskqueue>

and then create connection with sentinel.master_for('mymaster', socket_timeout=0.1).

What remains to be checked while implementing this, is, does asyncore loop's socket map survive for redis connection change.

— Reply to this email directly or view it on GitHub https://github.com/collective/collective.taskqueue/issues/11#issuecomment-124001207 .