Hooks not working when running deploy command for multiple connections / Current release doesn't exist yet on other servers

[dmaphy]

There is an issue with the hooks when running a deploy command for more than just one server. Lets say, for example, there are the following connections defined in config.php:

    // The various connections you defined
    // You can leave all of this empty or remove it entirely if you don't want
    // to track files with credentials : Rocketeer will prompt you for your credentials
    // and store them locally
    'connections'      => [
        'integration' => [
            'host'      => 'blade12',
            'username'  => 'appuser',
            'password'  => '',
            'key'       => '/home/appuser/.ssh/id_rsa',
            'keyphrase' => '',
            'agent'     => '',
            'db_role'   => true,
        ],
        'testing' => [
            'host'      => 'blade13',
            'username'  => 'appuser',
            'password'  => '',
            'key'       => '/home/appuser/.ssh/id_rsa',
            'keyphrase' => '',
            'agent'     => '',
            'db_role'   => true,
        ],
        'staging' => [
            'host'      => 'blade14',
            'username'  => 'appuser',
            'password'  => '',
            'key'       => '/home/appuser/.ssh/id_rsa',
            'keyphrase' => '',
            'agent'     => '',
            'db_role'   => true,
        ],
        'production' => array(
          'servers' => array(
            [
                'host'      => 'blade1',
                'username'  => 'appuser',
                'password'  => '',
                'key'       => '/home/appuser/.ssh/id_rsa',
                'keyphrase' => '',
                'agent'     => '',
                'db_role'   => false,
            ],
            [
                'host'      => 'blade2',
                'username'  => 'appuser',
                'password'  => '',
                'key'       => '/home/appuser/.ssh/id_rsa',
                'keyphrase' => '',
                'agent'     => '',
                'db_role'   => false,
            ],
            [
                'host'      => 'blade3',
                'username'  => 'appuser',
                'password'  => '',
                'key'       => '/home/appuser/.ssh/id_rsa',
                'keyphrase' => '',
                'agent'     => '',
                'db_role'   => false,
            ],
            [
                'host'      => 'blade4',
                'username'  => 'appuser',
                'password'  => '',
                'key'       => '/home/appuser/.ssh/id_rsa',
                'keyphrase' => '',
                'agent'     => '',
                'db_role'   => false,
            ],
          ),
        ),
    ],

The hooks.php contains some systemd commands to stop services before deployment and start services after deployment:

<?php

return [

    // Tasks
    //
    // Here you can define in the `before` and `after` array, Tasks to execute
    // before or after the core Rocketeer Tasks. You can either put a simple command,
    // a closure which receives a $task object, or the name of a class extending
    // the Rocketeer\Abstracts\AbstractTask class
    //
    // In the `custom` array you can list custom Tasks classes to be added
    // to Rocketeer. Those will then be available in the command line
    // with all the other tasks
    //////////////////////////////////////////////////////////////////////

    // Tasks to execute before the core Rocketeer Tasks
    'before' => [
        'setup'   => [],
        'deploy'  => ['systemctl stop nginx.service','systemctl stop php5-fpm.service','service php5-gearman stop'],
        'cleanup' => [],
    ],

    // Tasks to execute after the core Rocketeer Tasks
    'after'  => [
        'setup'   => [],
        'deploy'  => ['systemctl start php5-fpm.service', 'service php5-gearman start', 'systemctl start nginx.service'],
        'cleanup' => [],
        'rollback' => ['systemctl restart php5-fpm.service','service php5-gearman restart', 'systemctl restart nginx.service'],
    ],

    // Custom Tasks to register with Rocketeer
    'custom' => [],

];

Those commands are allowed for appuser to be executed via sudo on every server and this basically works when running a simple deploy command to deploy stuff on one server:

rocketeer deploy --on="testing" --branch="master"

The issue takes effect as soon as you try to run this in any form for multiple of the above mentioned connections (or the production connection which contains four servers), e.g.:

rocketeer deploy --on="testing,integration" --branch="master"

You then get something like the following output:

$ rocketeer deploy --on "testing,integration" --branch="master"
No username is set for [repository], please provide one:
No password is set for [repository], please provide one:
testing/0 | Deploy (Deploys the website)
testing/0 |-- Closure (systemctl stop nginx.service) fired by deploy.before
$ cd /home/appuser/app/releases/20160317123838
$ sudo -u root systemctl stop nginx.service
testing/0 |-- Closure (systemctl stop php5-fpm.service) fired by deploy.before [~1.45s]
$ cd /home/appuser/app/releases/20160317123838
$ sudo -u root systemctl stop php5-fpm.service
testing/0 |-- Closure (systemctl stop php5-gearman.service) fired by deploy.before [~0.82s]
$ cd /home/appuser/app/releases/20160317123838
$ sudo -u root systemctl stop php5-gearman.service
testing/0 |-- Primer (Run local checks to ensure deploy can proceed)
testing/0 |-- CreateRelease (Creates a new release on the server)
testing/0 |---- Deploy/Clone (Clones a fresh instance of the repository by SCM)
testing/0 |===> Cloning repository in "/home/appuser/app/releases/20160317124049"
$ git clone "git@gitlab.companydomain.com:app.git" "/home/appuser/app/releases/20160317124049" --branch="master"
[appuser@blade13] (testing) Cloning into '/home/appuser/app/releases/20160317124049'...
testing/0 |===> Initializing submodules if any
$ cd /home/appuser/app/releases/20160317124049
$ git submodule update --init --recursive
testing/0 |-- Dependencies (Installs or update the dependencies on server)
testing/0 |=> Sharing file /home/appuser/app/releases/20160317124049/storage/logs
testing/0 |=> Sharing file /home/appuser/app/releases/20160317124049/storage/sessions
$ ln -s /home/appuser/app/releases/20160317124049 /home/appuser/app/current-temp
$ mv -Tf /home/appuser/app/current-temp /home/appuser/app/current
testing/0 |=> Successfully deployed release 20160317124049
testing/0 |-- Closure (systemctl start php5-fpm.service) fired by deploy.after [~1.65s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl start php5-fpm.service
testing/0 |-- Closure (systemctl start php5-gearman.service) fired by deploy.after [~1.5s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl start php5-gearman.service
testing/0 |-- Closure (systemctl start nginx.service) fired by deploy.after [~1.26s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl start nginx.service
testing/0 | Cleanup (Clean up old releases from the server)
$ rm -rf /home/appuser/app/releases/20160316175344
testing/0 |=> Removing 1 release from the server
integration/0 | Deploy (Deploys the website) [~10.26s]
integration/0 |-- Closure (systemctl stop nginx.service) fired by deploy.before [~1.19s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl stop nginx.service
[appuser@blade12] (integration) bash: line 0: cd: /home/appuser/app/releases/20160317124049: No such file or directory
integration/0 |-- Closure (systemctl stop php5-fpm.service) fired by deploy.before [~1.16s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl stop php5-fpm.service
[appuser@blade12] (integration) bash: line 0: cd: /home/appuser/app/releases/20160317124049: No such file or directory
integration/0 |-- Closure (systemctl stop php5-gearman.service) fired by deploy.before [~1.02s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl stop php5-gearman.service
[appuser@blade12] (integration) bash: line 0: cd: /home/appuser/app/releases/20160317124049: No such file or directory
integration/0 |-- Primer (Run local checks to ensure deploy can proceed)
integration/0 |-- CreateRelease (Creates a new release on the server) [~9.55s]
integration/0 |---- Deploy/Clone (Clones a fresh instance of the repository by SCM)
integration/0 |===> Cloning repository in "/home/appuser/app/releases/20160317124049"
$ git clone "git@gitlab.companydomain.com:app.git" "/home/appuser/app/releases/20160317124049" --branch="master"
[appuser@blade12] (integration) Cloning into '/home/appuser/app/releases/20160317124049'...
integration/0 |===> Initializing submodules if any
$ cd /home/appuser/app/releases/20160317124049
$ git submodule update --init --recursive
integration/0 |-- Dependencies (Installs or update the dependencies on server) [~0.04s]
integration/0 |=> Sharing file /home/appuser/app/releases/20160317124049/storage/logs
integration/0 |=> Sharing file /home/appuser/app/releases/20160317124049/storage/sessions
$ ln -s /home/appuser/app/releases/20160317124049 /home/appuser/app/current-temp
$ mv -Tf /home/appuser/app/current-temp /home/appuser/app/current
integration/0 |=> Successfully deployed release 20160317124049
integration/0 |-- Closure (systemctl start php5-fpm.service) fired by deploy.after [~0.91s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl start php5-fpm.service
integration/0 |-- Closure (systemctl start php5-gearman.service) fired by deploy.after [~0.82s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl start php5-gearman.service
integration/0 |-- Closure (systemctl start nginx.service) fired by deploy.after [~0.75s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl start nginx.service
integration/0 | Cleanup (Clean up old releases from the server) [~0.3s]
$ rm -rf /home/appuser/app/releases/20160316175344
integration/0 |=> Removing 1 release from the server
Execution time: 29.2839s
Saved logs to /home/appuser/.rocketeer/logs/testing--20160317.log
Saved logs to /home/appuser/.rocketeer/logs/integration--20160317.log

Now, note that, fully independent of the connstellation or what servers you list, from the second server on always this happens before the deploy:

integration/0 |-- Closure (systemctl stop nginx.service) fired by deploy.before [~1.19s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl stop nginx.service
[appuser@blade12] (integration) bash: line 0: cd: /home/appuser/app/releases/20160317124049: No such file or directory
integration/0 |-- Closure (systemctl stop php5-fpm.service) fired by deploy.before [~1.16s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl stop php5-fpm.service
[appuser@blade12] (integration) bash: line 0: cd: /home/appuser/app/releases/20160317124049: No such file or directory
integration/0 |-- Closure (systemctl stop php5-gearman.service) fired by deploy.before [~1.02s]
$ cd /home/appuser/app/releases/20160317124049
$ sudo -u root systemctl stop php5-gearman.service
[appuser@blade12] (integration) bash: line 0: cd: /home/appuser/app/releases/20160317124049: No such file or directory

This leads to two issues:

• rocketeer presumes the release which has just been created on the first server is also present on the second server, in a moment where the deploy just didn't take place yet on the second server (and every following server)
• associated with this or not: the stop commands defined in hooks.php just do not work from the second server on

For us, this results in old APC caches which have not been cleared because PHP5-FPM has not been restarted. This is, in particular, suboptimal when deploying to the production connection.

Any ideas how to solve this?

Thanks very much in advance & Best Regards, Dominic

[dmaphy]

Oh, and one additional thing I forgot to mention: this issue is not present when calling the rocketeer deploy command with --parallel. This isn't an option for our deployment, though. Since it would lead to suboptimal situations as well when having all php5-fpm.services down at once.

[leanmac]

Having the same problem here. Did you find out how to solve it?

[dmaphy]

No, unfortunately not. Currently we're living with the --parallel switch as a workaround for this.