Yelp / python-gearman

Gearman API - Client, worker, and admin client interfaces
http://github.com/Yelp/python-gearman/
Other
242 stars 124 forks source link

Better handling of heterogenous server/task pairings #17

Open josharian opened 12 years ago

josharian commented 12 years ago

The following (simplified) setup goes awry:

The outcome is based on a race condition: Whichever server (A or B) replies first gets the task. If it is A, the task succeeds; if it is B, the task fails.

In general, unless all servers have the exact same set of worker tasks available, clients can't rely on requests succeeding.

This makes it difficult to rely on multiple gearmand servers for load balancing and redundancy, even if the general intent is to have all workers connect to all servers (config errors, connection flukes, etc. can get in the way -- exactly what the redundancy is there to help with).

josharian commented 12 years ago

I'd be happy to take a crack at fixing this, but before I start, I just wanted to make sure that I wasn't Doing It Wrong, and that there wasn't a different intended way to use multiple gearmand servers for redundancy.