forking question

[yashh]

Hey I been using pyres for quite some time but only as a secondary queuing infrastructure. I have a couple of questions I want to raise since I am giving serious thoughts on using pyres as our primary queue.

• Forking, I see that every worker forks himself and the forked worker performs the task. But in our situation we have a lot of cache connections, sql connections. We pool all these connections to reuse the open connection, since we are facing high traffic. I want to know whats the advantage of forking ? Can we just make workers BLPOP on redis and pick up task from redis
  process and perform BLPOP ?
• Restarting workers. Is there a graceful way of restarting workers? Currently we are using supervisors to manage workers. We are killing all workers and then restarting the supervisor. Can we make sure kill pid will not abruptly kill the task but kill after completing one?

Pretty much forking has been slow in my opinion. I see that tasks which are blazing fast take about a second on pyres since it has to open a bunch of connections etc. Wht do u think?

[binarymatt]

• forking. originally, the jobs fork themselves for every job in order to recover from failure/hung jobs. Because the parent process doesn't actually run the job, it can still be cleaned up if need be. However, like you mentioned, forking incurs a performance penalty, especially for stuff that is creating connections or something like compiling regular expressions. Because of these drawbacks with forking, I created a pre-fork style worker in the horde module, called pyres_manager. Instead of forking every jobs, the manager forks n children and those children continually pull jobs off of the queue and process them.
• restarting jobs - for workers, if you use the SIGQUIT signal, they will gracefully shut down. For the pyres_manager script if you send SIGTERM, SIGQUIT, or SIGINT it will try to gracefully shutdown it's pool of child subprocesses.

Let me know if you have any other questions.