Deployer stopped working for one repository only

[loranger]

Hello, I use deployer on a daily basis for years now. I used to use deployer for deploying on one server only but for 3 differents websites (all using laravel recipe), having two stages each.

All the three websites are deployed from a private gitlab server and everything worked flawlessly, but suddenly this morning, one of the three websites refused to deploy.

It always fails at the git clone steps, even if the clone successfully completes on the server-side. I can't figure out what happens, but I really do appreciate a little help…

Deployer version : Deployer v6.5.0 PHP version : PHP 7.2.21 (cli) (built: Aug 6 2019 22:48:46) ( NTS ) Deployment target : Debian 8.11

Deploy.php (same as the two other websites, which still working)

```php hostname($server) ->user('deploy') ->forwardAgent() ->stage('production') ->set('deploy_path', $path . '/prod'); host('stage') ->hostname($server) ->user('deploy') ->forwardAgent() ->set('deploy_path', $path . '/stage'); set('repository', `git config --get remote.origin.url`); set('shared_dirs', [ 'storage/app', 'storage/framework/cache', 'storage/framework/sessions', 'storage/framework/views', 'storage/logs', 'public/media', ]); set('writable_dirs', [ 'bootstrap/cache', 'public/media', 'resources/lang', 'storage', 'storage/app', 'storage/framework/views', 'storage/framework/cache', 'storage/framework/cache/data', 'storage/framework/sessions', 'storage/logs', ]); set('slack_webhook', 'https://hooks.slack.com/services/XXXXXX/YYYY/ZZZ'); set('slack_text', '_{{user}}_ déploie `{{branch}}` en *{{target}}*'); set('slack_success_color', '#7ABA70'); set('slack_success_text', 'Déploiement en *{{target}}* réussi !'); set('slack_failure_text', 'Echec du déploiement en *{{target}}*'); set('newrelic', [ 'license' => '123456789012345678901234567890', 'application_id' => '0987654321', 'app_name' => $name, ]); /* |-------------------------------------------------------------------------- | Tasks |-------------------------------------------------------------------------- */ task('env:create', function () { $output = run('if [ ! -f {{deploy_path}}/current/.env ]; then printf "APP_ENV=' . get('stage', 'stage') . '\nAPP_KEY=\nAPP_LOG_LEVEL=debug" >> {{deploy_path}}/current/.env && echo "Minimal .env file created"; fi'); if (trim($output) != '') { run('{{bin/php}} {{deploy_path}}/current/artisan key:generate'); writeln('' . $output . ''); } })->desc('Create env file if missing'); task('fix:permissions', function () { cd('{{deploy_path}}'); run('sudo chown -R `whoami`:www-data .'); run('sudo chmod -R g+rwX .'); run('sudo chown -R www-data:www-data current/storage/app/laravel-google-analytics'); })->desc('Apply right permissions on deployments folder'); task('fix:dotenv', function () { $lines = run('cat {{deploy_path}}/current/.env | wc -l'); if ((int) trim($lines) < 5) { upload('.env', '{{deploy_path}}/shared/.env'); run('sed -i "s/APP_ENV=.*/APP_ENV=' . get('stage', 'stage') . '/gm" {{deploy_path}}/shared/.env'); run('sed -i "s/APP_DEBUG=.*/APP_DEBUG=false/gm" {{deploy_path}}/shared/.env'); run('sed -i "s/DB_HOST=.*/DB_HOST=localhost/gm" {{deploy_path}}/shared/.env'); run('php {{deploy_path}}/current/artisan key:generate'); write(' -> review .env shortly! '); } }); /** * Git operations */ task('deploy:git', function () { set('previous_commit', run('cd {{deploy_path}}/current && git rev-parse HEAD')); }); task('deploy:changelog', function () { set('current_commit', run('cd {{deploy_path}}/current && git rev-parse HEAD')); set('current_commit', run('cd {{deploy_path}}/current && git fetch --depth 20')); set('changelog', run("cd {{deploy_path}}/current && git log {{previous_commit}}...{{current_commit}} --pretty=format:'• ' --reverse")); set('slack', [ 'token' => get('slack')['token'], 'icon' => get('slack')['icon'], 'username' => get('slack')['username'], 'team' => get('slack')['team'], 'channel' => get('slack')['channel'], 'message' => "{{user}} successfully deployed to `{{host}}` on *{{stage}}*\n" . get('changelog'), ]); }); /** * Database migrations */ task('migrate', function () { if (askConfirmation("Application in " . get('stage', 'stage') . "! Do you really want to migrate?")) { run('php {{deploy_path}}/current/artisan migrate --force'); } })->desc('Run the database migrations'); task('migrate:ingredients', function () { if (askConfirmation("Application in " . get('stage', 'stage') . "! Do you really want to migrate?")) { $out = run('php {{deploy_path}}/current/artisan migrate:ingredients --force'); if (trim($out) != '') { writeln('' . $out . ''); } } })->desc('Run the ingredients database migrations'); task('migrate:refresh', function () { if (askConfirmation("Application in " . get('stage', 'stage') . "! Do you really want to RESET and re-run all migrations?")) { run('php {{deploy_path}}/current/artisan migrate:refresh --force'); } })->desc('Reset and re-run all migrations'); task('migrate:refreshandseed', function () { if (askConfirmation("Application in " . get('stage', 'stage') . "! Do you really want to RESET and re-run all migrations and seed?")) { run('php {{deploy_path}}/current/artisan migrate:refresh --force --seed'); } })->desc('Reset and re-run all migrations and seed'); task('migrate:rollback', function () { if (askConfirmation("Application in " . get('stage', 'stage') . "! Do you really want to rollback latest migrations?")) { run('php {{deploy_path}}/current/artisan migrate:rollback --force'); } })->desc('Rollback the last database migration'); task('db:seed', function () { if (askConfirmation("Application in " . get('stage', 'stage') . "! Do you really want to rollback latest migrations?")) { run('php {{deploy_path}}/current/artisan db:seed --force'); } })->desc('Seed the database with records'); task('monitor:beanstalkd', function () { write(run('php {{deploy_path}}/current/artisan monitor:beanstalkd')); })->desc('Monitor Beanstalkd Server'); /** * Queue management */ task('beans', function () { run('php {{deploy_path}}/current/artisan beans:tube'); })->desc('Display current beanstalkd tubes usage (do not forget to add -vvv to display output)'); /** * Services managment */ task('reload:php-fpm', function () { run('sudo /usr/sbin/service php7.2-fpm reload'); })->desc('Reload PHP7 FPM configuration'); task('reload:nginx', function () { run('sudo /usr/sbin/service nginx reload'); })->desc('Reload Nginx configuration'); task('reload:queue', function () { run('{{bin/php}} {{deploy_path}}/current/artisan queue:restart'); })->desc('Reload queue handler'); /** * Files managment */ task('flush:logs', function () { run('sudo rm {{deploy_path}}/current/storage/logs/*.log'); })->desc('Flush logs'); task('flush:views', function () { run('php {{deploy_path}}/current/artisan view:clear'); })->desc('Clear views'); task('update:translations', function () { run('php {{deploy_path}}/current/artisan translations:export all'); })->desc('Update translations files'); /* |-------------------------------------------------------------------------- | Hooks |-------------------------------------------------------------------------- */ after('cleanup', 'env:create'); after('cleanup', 'reload:php-fpm'); after('cleanup', 'reload:queue'); before('deploy', 'slack:notify'); after('success', 'slack:notify:success'); after('deploy:failed', 'slack:notify:failure'); ```

Output log

```shell $ ./vendor/bin/dep deploy production -vvv [localhost] > git config --get user.name [localhost] > git rev-parse --abbrev-ref HEAD [localhost] < master • done on [production] ✈︎ Deploying master on production • done on [production] ➤ Executing task deploy:prepare [production] > echo $0 [production] < bash [production] > if [ ! -d /home/web/project/admin/prod ]; then mkdir -p /home/web/project/admin/prod; fi [production] > if [ ! -L /home/web/project/admin/prod/current ] && [ -d /home/web/project/admin/prod/current ]; then echo 'true'; fi [production] > cd /home/web/project/admin/prod && if [ ! -d .dep ]; then mkdir .dep; fi [production] > cd /home/web/project/admin/prod && if [ ! -d releases ]; then mkdir releases; fi [production] > cd /home/web/project/admin/prod && if [ ! -d shared ]; then mkdir shared; fi • done on [production] ✔ Ok [1s 431ms] ➤ Executing task deploy:lock [production] > if [ -f /home/web/project/admin/prod/.dep/deploy.lock ]; then echo 'true'; fi [production] > touch /home/web/project/admin/prod/.dep/deploy.lock • done on [production] ✔ Ok [457ms] ➤ Executing task deploy:release [production] > cd /home/web/project/admin/prod && (if [ -h release ]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (rm -rf "$(readlink release)") [production] > cd /home/web/project/admin/prod && (rm release) [production] > cd /home/web/project/admin/prod && (if [ -d releases ] && [ "$(ls -A releases)" ]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (cd releases && ls -t -1 -d */) [production] < 2/ [production] < 1/ [production] < 1083_old/ [production] < 1082/ [production] < 1081/ [production] < 1080/ [production] < 1079/ [production] < 1078/ [production] < 20170215165255/ [production] < 20170208093858/ [production] < 20170208085735/ [production] > cd /home/web/project/admin/prod && (if [ -f .dep/releases ]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (tail -n 15 .dep/releases) [production] < 20190827145447,1083 [production] < 20190827145628,1083 [production] < 20190827145716,1083 [production] < 20190827145817,1083 [production] < 20190827145943,1083 [production] < 20190827150140,1083 [production] < 20190827150721,1 [production] < 20190827150946,1 [production] < 20190827151858,2 [production] < 20190827151917,3 [production] < 20190827193646,3 [production] < 20190827200556,3 [production] < 20190827201316,3 [production] < 20190827201439,3 [production] < 20190827201718,3 [production] > cd /home/web/project/admin/prod && (if [ -d /home/web/project/admin/prod/releases/3 ]; then echo 'true'; fi) [production] > cd /home/web/project/admin/prod && (date +"%Y%m%d%H%M%S") [production] < 20190827222128 [production] > cd /home/web/project/admin/prod && (echo '20190827222128,3' >> .dep/releases) [production] > cd /home/web/project/admin/prod && (mkdir /home/web/project/admin/prod/releases/3) [production] > cd /home/web/project/admin/prod && (if [[ $(man ln 2>&1 || ln -h 2>&1 || ln --help 2>&1) =~ '--relative' ]]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (ln -nfs --relative /home/web/project/admin/prod/releases/3 /home/web/project/admin/prod/release) • done on [production] ✔ Ok [3s 69ms] ➤ Executing task deploy:update_code [production] > command -v 'git' || which 'git' || type -p 'git' [production] < /usr/bin/git [production] > /usr/bin/git version [production] < git version 2.1.4 [production] > cd /home/web/project/admin/prod && (if [ -h /home/web/project/admin/prod/release ]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (readlink /home/web/project/admin/prod/release) [production] < releases/3 [production] > cd /home/web/project/admin/prod && (/usr/bin/git clone -b master --depth 1 --recursive git@git.du-jour.fr:du-jour/menu/admin.git /home/web/project/admin/prod/releases/3 2>&1) [production] < Clonage dans 'admin'... [production] < bash: ligne 2: /home/web/project/admin/prod/releases/3 : est un dossier ➤ Executing task deploy:failed • done on [production] ✔ Ok [0ms] • done on [production] In Client.php line 103: [Deployer\Exception\RuntimeException (126)] The command "cd /home/web/project/admin/prod && (/usr/bin/git clone -b maste r --depth 1 --recursive git@git.du-jour.fr:du-jour/menu/admin.git /home/web/project/admin/prod/releases/3 2>&1)" failed. Exit Code: 126 (Invoked command cannot execute) Host Name: production ================ Clonage dans 'admin'... Exception trace: at /Users/loranger/project/admin/vendor/deployer/deployer/src/Ssh/Client.php:103 Deployer\Ssh\Client->run() at /Users/loranger/project/admin/vendor/deployer/deployer/src/functions.php:304 Deployer\run() at /Users/loranger/project/admin/vendor/deployer/deployer/recipe/deploy/update_code.php:96 Deployer\Deployer::Deployer\{closure}() at n/a:n/a call_user_func() at /Users/loranger/project/admin/vendor/deployer/deployer/src/Task/Task.php:105 Deployer\Task\Task->run() at /Users/loranger/project/admin/vendor/deployer/deployer/src/Executor/SeriesExecutor.php:60 Deployer\Executor\SeriesExecutor->run() at /Users/loranger/project/admin/vendor/deployer/deployer/src/Console/TaskCommand.php:142 Deployer\Console\TaskCommand->execute() at /Users/loranger/project/admin/vendor/symfony/console/Command/Command.php:255 Symfony\Component\Console\Command\Command->run() at /Users/loranger/project/admin/vendor/symfony/console/Application.php:915 Symfony\Component\Console\Application->doRunCommand() at /Users/loranger/project/admin/vendor/deployer/deployer/src/Console/Application.php:133 Deployer\Console\Application->doRunCommand() at /Users/loranger/project/admin/vendor/symfony/console/Application.php:272 Symfony\Component\Console\Application->doRun() at /Users/loranger/project/admin/vendor/symfony/console/Application.php:148 Symfony\Component\Console\Application->run() at /Users/loranger/project/admin/vendor/deployer/deployer/src/Deployer.php:326 Deployer\Deployer::run() at /Users/loranger/project/admin/vendor/deployer/deployer/bin/dep:124 deploy [-p|--parallel] [-l|--limit LIMIT] [--no-hooks] [--log LOG] [--roles ROLES] [--hosts HOSTS] [-o|--option OPTION] [--] [] ```

Whereas deployer considers git clone has failed, the repository has been successfully cloned one the remote side :

git status (in french, sorry)

```shell $ cd /home/web/dujour/admin/prod/admin $ git status Sur la branche master Votre branche est à jour avec 'origin/master'. rien à valider, la copie de travail est propre ```

Do you have any clue, please ?

[antonmedv]

Fo you use ssh-agent or deploykeys? Try to use deploy key.

[loranger]

I did remove the ->forwardAgent() from the host definitions and now use a deployment key for the deploy user but the result remains the same :

Output log

```shell $ ./vendor/bin/dep deploy production -vvvv [localhost] > git config --get user.name [localhost] > git rev-parse --abbrev-ref HEAD [localhost] < master • done on [production] ✈︎ Deploying master on production • done on [production] ➤ Executing task deploy:prepare [production] > echo $0 [production] < bash [production] > if [ ! -d /home/web/project/admin/prod ]; then mkdir -p /home/web/project/admin/prod; fi [production] > if [ ! -L /home/web/project/admin/prod/current ] && [ -d /home/web/project/admin/prod/current ]; then echo 'true'; fi [production] > cd /home/web/project/admin/prod && if [ ! -d .dep ]; then mkdir .dep; fi [production] > cd /home/web/project/admin/prod && if [ ! -d releases ]; then mkdir releases; fi [production] > cd /home/web/project/admin/prod && if [ ! -d shared ]; then mkdir shared; fi • done on [production] ✔ Ok [666ms] ➤ Executing task deploy:lock [production] > if [ -f /home/web/project/admin/prod/.dep/deploy.lock ]; then echo 'true'; fi [production] > touch /home/web/project/admin/prod/.dep/deploy.lock • done on [production] ✔ Ok [227ms] ➤ Executing task deploy:release [production] > cd /home/web/project/admin/prod && (if [ -h release ]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (rm -rf "$(readlink release)") [production] > cd /home/web/project/admin/prod && (rm release) [production] > cd /home/web/project/admin/prod && (if [ -d releases ] && [ "$(ls -A releases)" ]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (cd releases && ls -t -1 -d */) [production] < 2/ [production] < 1/ [production] < 1083_old/ [production] < 1082/ [production] < 1081/ [production] < 1080/ [production] < 1079/ [production] < 1078/ [production] < 20170215165255/ [production] < 20170208093858/ [production] < 20170208085735/ [production] > cd /home/web/project/admin/prod && (if [ -f .dep/releases ]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (tail -n 15 .dep/releases) [production] < 20190827151858,2 [production] < 20190827151917,3 [production] < 20190827193646,3 [production] < 20190827200556,3 [production] < 20190827201316,3 [production] < 20190827201439,3 [production] < 20190827201718,3 [production] < 20190827222128,3 [production] < 20190828071435,3 [production] < 20190828092243,3 [production] < 20190828110446,3 [production] < 20190828110537,3 [production] < 20190828145614,3 [production] < 20190828151053,3 [production] < 20190828151459,3 [production] > cd /home/web/project/admin/prod && (if [ -d /home/web/project/admin/prod/releases/3 ]; then echo 'true'; fi) [production] > cd /home/web/project/admin/prod && (date +"%Y%m%d%H%M%S") [production] < 20190828151615 [production] > cd /home/web/project/admin/prod && (echo '20190828151615,3' >> .dep/releases) [production] > cd /home/web/project/admin/prod && (mkdir /home/web/project/admin/prod/releases/3) [production] > cd /home/web/project/admin/prod && (if [[ $(man ln 2>&1 || ln -h 2>&1 || ln --help 2>&1) =~ '--relative' ]]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (ln -nfs --relative /home/web/project/admin/prod/releases/3 /home/web/project/admin/prod/release) • done on [production] ✔ Ok [1s 527ms] ➤ Executing task deploy:update_code [production] > command -v 'git' || which 'git' || type -p 'git' [production] < /usr/bin/git [production] > /usr/bin/git version [production] < git version 2.1.4 [production] > cd /home/web/project/admin/prod && (if [ -h /home/web/project/admin/prod/release ]; then echo 'true'; fi) [production] < true [production] > cd /home/web/project/admin/prod && (readlink /home/web/project/admin/prod/release) [production] < releases/3 [production] > cd /home/web/project/admin/prod && (/usr/bin/git clone -b master --depth 1 --recursive git@git.du-jour.fr:du-jour/menu/admin.git /home/web/project/admin/prod/releases/3 2>&1) [production] < Clonage dans 'admin'... [production] < bash: ligne 2: /home/web/project/admin/prod/releases/3 : est un dossier ➤ Executing task deploy:failed • done on [production] ✔ Ok [0ms] • done on [production] In Client.php line 103: [Deployer\Exception\RuntimeException (126)] The command "cd /home/web/project/admin/prod && (/usr/bin/git clone -b maste r --depth 1 --recursive git@git.du-jour.fr:du-jour/menu/admin.git /home/web/project/admin/prod/releases/3 2>&1)" failed. Exit Code: 126 (Invoked command cannot execute) Host Name: production ================ Clonage dans 'admin'... Exception trace: at /Users/loranger/Desktop/admin/vendor/deployer/deployer/src/Ssh/Client.php:103 Deployer\Ssh\Client->run() at /Users/loranger/Desktop/admin/vendor/deployer/deployer/src/functions.php:304 Deployer\run() at /Users/loranger/Desktop/admin/vendor/deployer/deployer/recipe/deploy/update_code.php:96 Deployer\Deployer::Deployer\{closure}() at n/a:n/a call_user_func() at /Users/loranger/Desktop/admin/vendor/deployer/deployer/src/Task/Task.php:105 Deployer\Task\Task->run() at /Users/loranger/Desktop/admin/vendor/deployer/deployer/src/Executor/SeriesExecutor.php:60 Deployer\Executor\SeriesExecutor->run() at /Users/loranger/Desktop/admin/vendor/deployer/deployer/src/Console/TaskCommand.php:142 Deployer\Console\TaskCommand->execute() at /Users/loranger/Desktop/admin/vendor/symfony/console/Command/Command.php:255 Symfony\Component\Console\Command\Command->run() at /Users/loranger/Desktop/admin/vendor/symfony/console/Application.php:915 Symfony\Component\Console\Application->doRunCommand() at /Users/loranger/Desktop/admin/vendor/deployer/deployer/src/Console/Application.php:133 Deployer\Console\Application->doRunCommand() at /Users/loranger/Desktop/admin/vendor/symfony/console/Application.php:272 Symfony\Component\Console\Application->doRun() at /Users/loranger/Desktop/admin/vendor/symfony/console/Application.php:148 Symfony\Component\Console\Application->run() at /Users/loranger/Desktop/admin/vendor/deployer/deployer/src/Deployer.php:326 Deployer\Deployer::run() at /Users/loranger/Desktop/admin/vendor/deployer/deployer/bin/dep:124 deploy [-p|--parallel] [-l|--limit LIMIT] [--no-hooks] [--log LOG] [--roles ROLES] [--hosts HOSTS] [-o|--option OPTION] [--] [] ```

When I launch the command manually (as the deploy user on the server), everything works fine. I can't understand why deployer believe the command failed as it did not

[antonmedv]

It found Exit Code: 126 Can you try to run in manually? cd /home/web/project/admin/prod && (/usr/bin/git clone -b maste r --depth 1 --recursive git@git.du-jour.fr:du-jour/menu/admin.git /home/web/project/admin/prod/releases/3 2>&1) via same user?

[loranger]

Absolutely no problem :

Merlin:admin loranger$ ./vendor/bin/dep ssh production

deploy@server$ /home/web/project/admin/prod/current

$ cd /home/web/project/admin/prod && (/usr/bin/git clone -b master --depth 1 --recursive  git@git.du-jour.fr:du-jour/menu/admin.git /home/web/project/admin/prod/releases/3 2>&1)

Clonage dans '/home/web/project/admin/prod/releases/3'...
remote: Enumerating objects: 1304, done.
remote: Counting objects: 100% (1304/1304), done.
remote: Compressing objects: 100% (1038/1038), done.
remote: Total 1304 (delta 301), reused 843 (delta 185)
Réception d'objets: 100% (1304/1304), 6.37 MiB | 0 bytes/s, fait.
Résolution des deltas: 100% (301/301), fait.
Vérification de la connectivité... fait.

[antonmedv]

Really difficult to tell why. Maybe this one server has different git version?

[loranger]

It's a git version 2.1.4 It used to work until Tuesday (where no upgrade occurred on this server or on the gitlab), and on the same server, the two other repositories are still working fine… I cannot understand what happened, but we can't deploy anymore :(

[antonmedv]

This is really strange. But I don't know what else can be done. As workaraund use rsync for this server for now.