[DEPLOYMENT] Starting workers via ssh or php hangs shell

[mfn]

Trying to start workers from other processes hangs the shell:

$ php -r 'passthru("Console/cake CakeResque.CakeResque start --queue test; echo foo");'
Creating workers
Starting worker ... Done

foo
^C

What's happening here is that the shell is never returned to the user, although executing Console/cake CakeResque.CakeResque start --queue test does.

Another observation is that there is actually another command in my test which gets executed (echo foo) -> but the whole command, executed in a subshell through php, never returns.

This is the most simplified case I came up with; the higher goal is automating of deployment via e.g. ssh and phing. I have both setups already running, but all hang when starting workers.

I looked at CakeResque and saw quite some sophisticated usage of subshells, nohup, pcntl; but it doesn't dawn me what could lead to the hang.

I tried strace but it floods my screen and I'm unable to pinpoint anything obvious:

$ strace -f php -r 'passthru("Console/cake CakeResque.CakeResque start --queue test");'
... snipped thousands of lines
[pid 28293] rt_sigprocmask(SIG_SETMASK, 0x7fffed271030, NULL, 8) = 0
[pid 28293] nanosleep({...}, ^CProcess 28276 detached

I was able to reproduce this behaviour in three environments: one VM, two physical machines (two Ubuntu (precise and trusty) and Debian (squeeze). I tested this with cake-resque 4.1.0 in environments with php 5.4 and 5.5 .

I tested using other means of starting them, e.g. via ruby: this works locally but also hangs when executed via ssh: ruby -e 'system("Console/cake CakeResque.CakeResque start --queue test")'.

I could work around this by starting them in the background, but I need to see the output and verify they properly started.

I've never seen such a behaviour before, any pointers/hints much appreciated.

[wa0x6e]

Does running other cake console command hanging too ? or is the issue specific to CakeResque ?

[mfn]

It's specific; any other cake command runs normally through. My theory is that it has something to do with the special signal handling regarding parent/child which is going on.

[mfn]

Trying to deduce a minimal testcase going from Plugin/CakeResque/Console/Command/CakeResqueShell.php:

#!/usr/bin/env php
<?php
$cmd = 'nohup bash -c \'QUEUE=test <YOURPATH>/app/Vendor/kamisama/php-resque-ex/bin/resque\' >/dev/null 2>/dev/null &';
$f = fopen('php://stdout', 'w');
passthru($cmd);

Wondering why the fopen() is there?

Because this is what Cakes \Shell::__construct() is essentially doing:

    public function __construct($stdout = null, $stderr = null, $stdin = null) {
    # ...
    $this->stdout = $stdout ? $stdout : new ConsoleOutput('php://stdout');

This leads to \ConsoleOutput::__construct():

    public function __construct($stream = 'php://stdout') {
        $this->_output = fopen($stream, 'w');

By having a file handle open to the output stream, the process will not return properly when either executing by something like php -r 'passthru("./processtest.php");' or when ran via ssh like ssh host <PATH>/processtest.php.

Digging deeper

I've limited C-foo knowledge but to me it seems this is some unix/posix behaviour; I can reproduce this with a simplistic C program. Reading the php-src what is going on when calling fopen('php://stdout'); under the hood performs dup():

#include <stdio.h>
#include <unistd.h> 

int main() {
  char *cmd = "nohup bash -c 'QUEUE=test <PATH>/app/Vendor/kamisama/php-resque-ex/bin/resque\' >/dev/null 2>/dev/null &\0";
  FILE *fcmd;
  int out;

  out = dup(STDOUT_FILENO);
  // close(out);

  fcmd = popen(cmd, "r");
}

Compiling with gcc process_test.c and running with php -r 'passthru("./a.out"); it also hangs. If I comment in the close(out) call, it runs through.

This whole behavior matching exactly what I see in PHP.

[dilab]

Facing the same issue when trying to start a worker using Phing.

[mfn]

@dilab See https://github.com/wa0x6e/Cake-Resque/pull/76 and try if it works for you

[jbourassa]

I've had the same issue, my workaround has been to redirect all output to /dev/null. That's obviously not a good solution since errors are no longer reported, but meh, at least I can deploy.

[wa0x6e]

Does your system support nohup ?

Seems like the shell is hanging because it's waiting for a response from resque, but resque goes in an infinite loop internally, and does not return anything.

There's 2 thing in place to prevent: nohup and the output redirect to null.

[jbourassa]

@wa0x6e I would assume so, I'm deploying to Ubuntu 14.04 (from OSX, but I don't think that matters).

[wa0x6e]

Does the command works hang only when run via a php ?

If yes, I do recommend a deployment tool such as Capistrano, that does not use php to execute system command. There's already a passthru inside CakeResque, and calling it on top of another passthru seems weird, and hard to debug.

[jbourassa]

No, this also hangs:

$ ssh user@server 'cd /path/to/cake; bin/cake CakeResque.CakeResque start'

Creating workers
Starting worker ..
.
 Done
[hanging forever]

PS: Thanks for your help, I appreciate.

[wa0x6e]

Can you try again by loging to ssh first, then calling the rest of the command ?

It's sadly not the same environment.

[jbourassa]

Oh, that works. It only breaks when calling it from ssh as a command, not in an interactive shell.

[wa0x6e]

Interactive shell has been causing issue for a long time.

http://capistranorb.com/documentation/faq/why-does-something-work-in-my-ssh-session-but-not-in-capistrano/#

[mfn]

Closing, no longer using CakeResque