New Relic Extension not Available in beforeFork

[KeKs0r]

Hi,

I am trying to add some monitoring to my Jobs but somehow the new Relic Extension is not available. In my System, New Relic is generally available what I tested with the following script:

echo "This is a worker->";
if (extension_loaded('newrelic')) {
    echo "New relic is loaded!!";
}

$pid = pcntl_fork();
if ($pid == -1) {
    die("Couldn't Fork");
} else if ($pid) {
    if (extension_loaded('newrelic')) {
        echo "New Relic in Parent Process working";
    } else {
        echo 'New Relic in Parent not working';
    }
    pcntl_wait($status); //Schützt uns vor Zombie Kindern
} else {
    if (extension_loaded('newrelic')) {
        echo "New Relic in child working";
    } else {
        echo 'New Relic in child not working';
    }
}

But as soon as I try to access it within the php-resque hooks, its not available:

Resque_Event::listen('beforeFork', function(Resque_Job $job) use ($logger) {
    if (extension_loaded('newrelic')) {
        $logger->info('NEWRELIC: AVAILABLE on beforeFork');
    } else {
        $logger->info('NEWRELIC: NOT available on beforeFork');
    }
});

It is already not available in my "bootstrap" file which is included via the "APP_INCLUDE". All other php extensions are still available.

Has anybody figured out how to monitor runtimes with new relic?

[danhunsaker]

One of the hardest things for PHP Resque users to do, at least at first, is to remember that the workers run via the CLI SAPI, in a completely separate environment from their web interfaces. This means that a lot of data available to the web side isn't available to the workers, and vice versa, but also that (on almost all systems) the web side and the worker side use different php.ini files. This is a really common gotcha, and something we intend to highlight more strongly in the README at some point soon.

Check your CLI INI. I'd bet the NewRelic extension isn't in there. If you use the php-redis extension, check for it there, too. Basically, make sure everything your workers need from PHP is configured in your CLI version of php.ini. Then restart everything and you should be good to go.

[KeKs0r]

Hi Dan,

the test script I ran was in CLI. (I am using Symfony2 and when starting my worker, via CLI I have the new Relic extension.) But when it comes to the Job execution its not available anymore. So all my other tests should be using the same php.ini (from CLI)

[danhunsaker]

That...doesn't make any sense whatsoever. You're right that they should all be using the same php.ini file - but somehow, they aren't, because the extension is missing. Which makes no sense. Unless you have more than one PHP installation on your system. Which I've seen happen, but rarely on purpose, and never well.

At this point, all I can guarantee is that PHP-Resque itself isn't the culprit, here. It doesn't unload any extensions or have some magic mechanism for switching INI files, so the config differences aren't caused there. Are you using the BCC Resque Bundle? You mentioned using Symfony, so that would make sense. I'm not personally familiar with Symfony, so neither am I familiar with the inner workings of the bundle itself. It might help to have the bundle maintainers' input on this as well. I try to keep an eye on any issues opened there, in case they're actually problems with Resque itself rather than the bundle implementation, and I don't recall seeing this mentioned there, but I may have missed it. I doubt the bundle has anything to do with it, but it doesn't hurt to check.

This is a hard one. Hopefully we can figure it out soon!

[KeKs0r]

I am adding some debug stuff to my jobs and workers in order to see if they are using different configuration files.

Just as Reference Information:

• I am using BCC Resque Bundle
• I am running site on CloudControl

I will investigate further, where this issue is coming from. I don't think its php-resque itself either, but it might be some "integration" point or inner workings with regards to the process forking. I am really puzzled myself with this.

[danhunsaker]

Forking wouldn't do it, as that just means a full copy of process memory into a new process space, down to the current instruction. No chance for anything fancy there.

Not familiar with CloudControl, either. I'll see what I can do to fix that shortly.

[KeKs0r]

I just run some more tests.

In my plain worker I am using the same php.ini file as in my app include. And within the app_include the extension is not available.

I got the php.ini via:

php_ini_loaded_file();

Which means they are both times using the same configuration and should therefore have the same extensions.

[danhunsaker]

And that's the same file shown by php -r 'echo php_ini_loaded_file(), "\n";'?

[KeKs0r]

Yes always the same CLI config file: /etc/php5/cli/php.ini

[danhunsaker]

Just had to confirm. I'd assume the return value of php_ini_scanned_files() is also identical, because there's no reason it wouldn't be. This is all very odd.

[KeKs0r]

Found something: The difference is indeed the php_ini_scanned_files. The question is just, why are they different if the main php.ini is the same.

[danhunsaker]

The directory to scan is set at compile time, as I understand it. If you have multiple PHP CLI binaries, and they were compiled to scan different directories, that would do it. If the same directory is scanned in all cases, however, a completely different PHP bug exists that we'll have to track down. So, are the paths to the scanned files different, or just the specific files in the list?

[KeKs0r]

The whole paths are different: e.g. in the Working Worker:

/app/php/conf/000_20-apc.ini,
/app/php/conf/000_20-curl.ini,
.....
/app/php/conf/011_newrelic.ini,
/app/php/conf/011_newrelic_ext.ini

In comparison to the setup in the job execution:

/etc/php5/cli/conf.d/20-apc.ini,
/etc/php5/cli/conf.d/20-curl.ini,
....
(no new relic)

As I understand php-resque tries to find the php executable when creating a worker process, maybe this is not using the same executable as my main process would.

[danhunsaker]

Resque doesn't look for anything, as it needs to already be running in PHP to do so, so there's no reason to bother. However, it does have a "shebang" first line comment in the bin/resque script that tells the shell what binary to execute it with. If that binary has a different path than, say, which php, it would start a different instance. The differences in scan paths definitely confirm you're using different executables, though.