Forking bug

[rajibahmed]

Hi, I am not sure why this is happening. I have custom script for running resque workers. The script below logs to that 3 worker is created. But on the process list there is only one worker.

#!/usr/bin/env php
<?php
require_once 'vendor/autoload.php';
$app = require __DIR__.'/../services.php';
use Monolog\Logger;
use Monolog\Formatter\LogstashFormatter ,
    Monolog\Handler\StreamHandler;

// This variable gets passed to resque script
$logger = new Logger('ALAN_WORKER');

$handler = new StreamHandler(__DIR__.'/../log/worker.log', Logger::DEBUG);
$handler->setFormatter($app['log.formatter']);
$logger->pushHandler($handler);

putenv('QUEUE=*');
putenv('COUNT=3');
include 'vendor/bin/resque';

but when I do like this,

QUEUE=* COUNT=3 ./vendor/bin/resque

This works flawlessly.

Can anyone tell me what am I doing wrong?

[Rockstar04]

Im not sure if its a typo, but your not calling the same script vendor/bin/resque vs /vendor/bin/phpresque

[rajibahmed]

@Rockstar04 u are right , I had give the wrong example the problem still persists. The problem seems to me is. Resque::fork() for creating child don't get the time to fork. Ends up as only one process. But weird thing is I have echo statement in side of COUNT loop in the bin/resque script. It enters the loop with count env var, loops properly but dont create the processes.

@Rockstar04 I am trying to use logstash logger for worker logs. Do you have any examples, in any other way I can inject the logger to bin/resque ? //Best

[danhunsaker]

Is the pcntl extension installed? It needs to be for forking to work. I suspect this isn't the issue, but I have to ask.

Are the pcntl_* functions disabled in your CLI php.ini? Many distros do this by default for security reasons - Ubuntu is the one we see most often causing problems in this way. Again, don't think this is the problem, but have to check.

Is __DIR__/../log/worker.log writable by the user you're starting the workers with? If not, the new workers may be aborting with an unhandled exception from the logger. In fact, since the logger is the only visible difference between the two startup methods you have here, I'm fairly sure the problem lies with it, or at least how it's configured.

The answers to these questions will probably point the correct direction for further questions, if the problem isn't solved already.

[rajibahmed]

Firstly, can see the pcntl from my command line, 'php -m'. And it I can run the /vendor/bin/resque with forking. So, no problem with pcntl forking. BTW we are using gentoo, with PHP 5.4.27 (cli) (built: Apr 10 2014 14:13:55)

Secondly, bin/script is generating the worker notice log of starting 5 new worker. And it is on the worker.log file. So, the bin/script has access to write to log file. And putenv is sending the QUEUE and COUNT properly to 'bin/resque' script. echoed and checked.

Changed the file permission on worker.log to 777 that didn't work either.

Seems like an OS issue. But then why would it work when I just use ./vendor/bin/resque script. Does not make any sense to me.

[danhunsaker]

Ah, Gentoo. They don't make assumptions like that, so no, it's not pcntl.

If the logger is logging, it's not a permissions issue, so that's out, too.

Which version of PHP-Resque are you using? I suspect you're using the latest master, since you're setting a logger.

The only thing is, the logger is the only difference I can see, so it has to be related to the logger in some way. The trick is going to be figuring out how the logger and Resque are interfering with each other.

[rajibahmed]

Hi @danhunsaker

now the script looks like this

<?php

putenv('QUEUE=*');

putenv('COUNT=5');

require_once 'vendor/bin/resque';

But only creates one process :) . BTW, I was using resque-web to check the process.

Bug Found.

But now checked with ps -el | grep php which show five processes, resque-web picks up only one process. The last one. Only problem is now , it hinders the our visual monitoring.

Do you know any other project that can monitor php-resque properly or do you guys have success with default resque-web.

[danhunsaker]

Oh! I know what that bug is!

There's a mechanism that registers workers in a Redis key when they start, but it also clears out entries for other workers if it doesn't see their PIDs still active in a ps response. This is a tricky issue to solve, actually, but there is a potential workaround. Create a beforeFork hook that will re-register the current worker PID, and your workers will reappear in resque-web as soon as each starts performing jobs. Something like:

$QUEUE = getenv('QUEUE');
$queues = explode(',', $QUEUE);
$worker = new Resque_Worker($queues);
$worker->registerWorker();

Unfortunately, the worker isn't assigned as a property of the Resque_Job object until after the beforeFork hook is called, so you'll have to temporarily create a new one long enough to register it properly.

Another, better approach to this requires that the bin/resque script continue running and monitoring the forked workers. With this approach, bin/resque takes on responsibility for maintaining the worker list in Redis. The problem with this approach is that it needs a lot of changes to the way PHP-Resque is built to make it work. The beforeFork method above doesn't require any of these changes to be made.

Hope that helps!

[rajibahmed]

Should we document this as a common problem that people may run into as a wiki page or something ?

[danhunsaker]

The project isn't using the Wiki feature at the moment, but I'd rather build in the fix anyway. We'll see what happens when @chrisboulton has time to start reviewing changes again.

[rajibahmed]

I think php-resque should break down the README to smaller digestible pieces. And an full example of using a PSR logger with APP_INCLUDE which would be great . Just an idea.

[danhunsaker]

We already broke out the internals walkthrough into a separate file, so I'm sure this would be easy enough. Create a new issue for it so it doesn't get forgotten.