[question] Proving a failover beanstalkd server

[bgallagher]

I have a need to provide failover for the beanstalkd queue.

Has it been considered to handle this in code, or is there a more standard was of proxying the ip address and switching to a back up server in the event of failure?

[wandersonwhcr]

Hi @bgallagher,

I don't know, but I think you need an exception event, like #49, to handle it?

[bgallagher]

@juriansluiman do you have any suggestions whether or not this belong in a code solution?

[juriansluiman]

@bgallagher SlmQueueBeanstalkd utilizes for the connection the Pheanstalk package. Afaik, Pheanstalk does not support connection pools, so this is atm unsupported for SlmQueue. However, I think it is "fairly" easy to accomplish.

The Beanstalkd queue calls the Pheanstalk\Pheanstalk object. This object creates automatically a Pheanstalk\Connection. Inside the connection, a socket is set which is performing all the messaging work.

The dirty solution is to have an array of possible connections. You listen early in the SlmQueue process and calls $pheanstalk->getConnection()->isServiceListening(). If false, replace the connection with another one you instantiate yourself ($conn = new Connection(); $pheanstalk->setConnection($conn);).

The alternative is you create a ConnectionPool class, extending Pheanstalk\Connection. You set a random socket from your pool and test the connection. If unavailable, you set a new socket. This is the most clean solution, but there is unfortunately one catch: https://github.com/pda/pheanstalk/blob/master/src/Pheanstalk.php#L408-L434. So when dispatching fails, the Pheanstalk object itself instantiates a new Connection object and retries. Actually you want to retry on another host in the pool, but Pheanstalk does not allow you to do it.

TL;DR: I do not think this should (now) be a part of SlmQueueBeanstalkd. If you need it, you can either do this in userland or try to propose the feature to pda for a connection pool. Note you have solved the "job entry" problem with beanstalkd, but the failover solution doesn't provide any guarantee about retrieval of jobs, or the job priority between several hosts: it's the same problem Gearman faces.