Cannot use Deployer with submodules

Before submitting your issue, please make sure that you've checked all of the checkboxes below.

[X] You're running the latest release version of Deployer.
[X] Ensure that you're running at least PHP 7.0.8, you can check this by running php -v
[X] You've ran composer install --no-dev from the root of your installation.
[X] You've ran npm install --production from the root of your installation.
[x] You've ran rm -rf bootstrap/cache/* from the root of your installation.
[X] You have restarted the queue listener and node socket server.

Describe the bug

When trying to use deployer with a repository that has submodules. The deploy fails with the following error: permission denied (publickey) while cloning the submodule.

Expected behaviour

It would properly deploy the project.

Actual behaviour

The deploy fails due to the error mentioned above without event entering the first step.

Steps to reproduce

Include a project with submodule in deployer.
Configure the same ssh key in the master and submodule projects.
Try to deploy a version of the project.

Environment info

Laravel Environment

Laravel Version: 5.5.41
- Timezone: America/Sao_Paulo
- Debug Mode: ✔
- Storage Dir Writable: ✔
- Cache Dir Writable: ✔
- Decomposer Version: ^1.0
- App Size: 2 GB
Server Environment
- PHP Version: 7.0.33-0+deb9u3
- Server Software: nginx/1.10.3
- Server OS: Linux magneto 3.16.0-6-amd64 #1 SMP Debian 3.16.57-2 (2018-07-14) x86_64
- Database: pgsql
- SSL Installed: ✘
- Cache Driver: file
- Session Driver: file
- Openssl Ext: ✔
- PDO Ext: ✔
- Mbstring Ext: ✔
- Tokenizer Ext: ✔
- XML Ext: ✘
- Curl Ext: ✘
- GD Ext: ✔
- JSON Ext: ✔
Installed Packages & their version numbers
- andywer/js-localization : dev-laravel-5
- backup-manager/laravel : ^1.1
- creativeorange/gravatar : ~1.0
- doctrine/dbal : 2.5.13
- fideloper/proxy : ^3.3
- graham-campbell/binput : ~4.0
- guzzlehttp/guzzle : ^6.2
- htmlmin/htmlmin : ~5.0
- intervention/image : ^2.3
- laracademy/interactive-make : ^1.1
- laravel-notification-channels/hipchat : ~1.0
- laravel-notification-channels/twilio : ~2.0.1
- laravel-notification-channels/webhook : ~1.1.0
- laravel/framework : ^5.5
- laravel/tinker : 1.0.*
- lubusin/laravel-decomposer : ^1.0
- mccool/laravel-auto-presenter : ^6.2
- melihovv/laravel-log-viewer : ^5.5
- michele-angioni/multi-language : 0.4
- pda/pheanstalk : ~3.1
- pragmarx/google2fa : ^1.0.0
- predis/predis : ~1.0
- rebelinblue/laravel5-zxcvbn : ^1.0.0
- tymon/jwt-auth : ~0.5
- version/version : ^2.2
Extra Information
- proc_open enabled : ✔

Logs (see storage/logs/) or other output that would be helpful

[2019-09-05 20:26:38] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Mirror git repository
chmod +x /var/www/deployer/storage/app/tmp/ssh5W8wC8
export GIT_SSH=/var/www/deployer/storage/app/tmp/ssh5W8wC8

if [ ! -d /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git ]; then
    git clone --mirror git@bitbucket.org:VE3/colplan.git /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git
fi

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git fetch --all --prune
EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Get commit details - 1365
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git log master -n1 --pretty=format:"%H%x09%an%x09%ae"
EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### Create release archive - 1365
git clone --depth 1 --recursive /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git /var/www/deployer/storage/app/tmp/clone_17_20190905202638
cd /var/www/deployer/storage/app/tmp/clone_17_20190905202638
git checkout 36cc87b2cbb01f96473c80abce76c85081a871f0
/var/www/deployer/resources/scripts/tools/GitArchiveAll.sh --tree-ish 36cc87b2cbb01f96473c80abce76c85081a871f0 --format tar.gz --verbose /var/www/deployer/storage/app/17_20190905202636.tar.gz
cd -
rm -rf /var/www/deployer/storage/app/tmp/clone_17_20190905202638
EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### List git references
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git tag --list --no-column
EOF

[2019-09-05 20:26:43] development.DEBUG: ### Run script locally
bash -s << 'EOF'
    # Turn on quit on non-zero exit
    set -e
    ### List git references
cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git branch --list --no-column
EOF

[2019-09-05 20:26:45] development.ERROR: Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638'...
warning: --depth is ignored in local clones; use file:// instead.
done.
Submodule 'forecaster' (git@bitbucket.org:VE3/forecaster.git) registered for path 'forecaster'
Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@bitbucket.org:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed
Failed to clone 'forecaster'. Retry scheduled
Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@bitbucket.org:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed
Failed to clone 'forecaster' a second time, aborting

When nailing down the issue I found that cloning a submodule from a mirror requires authentication. So the ssh configs script should be passed as an argument to the CreateReleaseArchive.sh which should export GIT_SSH as you do when you clone the mirror.

If you do not need a submodule just remove --recursive from the git clone command.

Have you made sure the deploy key is added to the sub module repository?

Stephen Ball stephen@rebelinblue.com On 6 Sep 2019, 21:19 +0100, FelipeEmerim notifications@github.com, wrote:

Before submitting your issue, please make sure that you've checked all of the checkboxes below.

• [ X ] You're running the latest release version of Deployer. • [ X ] Ensure that you're running at least PHP 7.0.8, you can check this by running php -v • [ X ] You've ran composer install --no-dev from the root of your installation. • [ X ] You've ran npm install --production from the root of your installation. • [ X ] You've ran rm -rf bootstrap/cache/* from the root of your installation. • [ X ] You have restarted the queue listener and node socket server.

Describe the bug When trying to use deployer with a repository that has submodules. The deploy fails with the following error: permission denied (publickey) while cloning the submodule. Expected behaviour It would properly deploy the project. Actual behaviour The deploy fails due to the error mentioned above without event entering the first step. Steps to reproduce

• Include a project with submodule in deployer. • Configure the same ssh key in the master and submodule projects. • Try to deploy a version of the project.

Environment info Laravel Environment

• Laravel Version: 5.5.41 • Timezone: America/Sao_Paulo • Debug Mode: ✔ • Storage Dir Writable: ✔ • Cache Dir Writable: ✔ • Decomposer Version: ^1.0 • App Size: 2 GB

Server Environment

• PHP Version: 7.0.33-0+deb9u3 • Server Software: nginx/1.10.3 • Server OS: Linux magneto 3.16.0-6-amd64 #1 SMP Debian 3.16.57-2 (2018-07-14) x86_64 • Database: pgsql • SSL Installed: ✘ • Cache Driver: file • Session Driver: file • Openssl Ext: ✔ • PDO Ext: ✔ • Mbstring Ext: ✔ • Tokenizer Ext: ✔ • XML Ext: ✘ • Curl Ext: ✘ • GD Ext: ✔ • JSON Ext: ✔

Installed Packages & their version numbers

• andywer/js-localization : dev-laravel-5 • backup-manager/laravel : ^1.1 • creativeorange/gravatar : ~1.0 • doctrine/dbal : 2.5.13 • fideloper/proxy : ^3.3 • graham-campbell/binput : ~4.0 • guzzlehttp/guzzle : ^6.2 • htmlmin/htmlmin : ~5.0 • intervention/image : ^2.3 • laracademy/interactive-make : ^1.1 • laravel-notification-channels/hipchat : ~1.0 • laravel-notification-channels/twilio : ~2.0.1 • laravel-notification-channels/webhook : ~1.1.0 • laravel/framework : ^5.5 • laravel/tinker : 1.0.* • lubusin/laravel-decomposer : ^1.0 • mccool/laravel-auto-presenter : ^6.2 • melihovv/laravel-log-viewer : ^5.5 • michele-angioni/multi-language : 0.4 • pda/pheanstalk : ~3.1 • pragmarx/google2fa : ^1.0.0 • predis/predis : ~1.0 • rebelinblue/laravel5-zxcvbn : ^1.0.0 • tymon/jwt-auth : ~0.5 • version/version : ^2.2

Extra Information

• proc_open enabled : ✔

Logs (see storage/logs/) or other output that would be helpful [2019-09-05 20:26:38] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

Mirror git repository

chmod +x /var/www/deployer/storage/app/tmp/ssh5W8wC8

export GIT_SSH=/var/www/deployer/storage/app/tmp/ssh5W8wC8

if [ ! -d /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git ]; then

git clone --mirror git@bitbucket.org:VE3/colplan.git /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

fi

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git fetch --all --prune

EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

Get commit details - 1365

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git log master -n1 --pretty=format:"%H%x09%an%x09%ae"

EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

Create release archive - 1365

git clone --depth 1 --recursive /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git /var/www/deployer/storage/app/tmp/clone_17_20190905202638

cd /var/www/deployer/storage/app/tmp/clone_17_20190905202638

git checkout 36cc87b2cbb01f96473c80abce76c85081a871f0

/var/www/deployer/resources/scripts/tools/GitArchiveAll.sh --tree-ish 36cc87b2cbb01f96473c80abce76c85081a871f0 --format tar.gz --verbose /var/www/deployer/storage/app/17_20190905202636.tar.gz

cd -

rm -rf /var/www/deployer/storage/app/tmp/clone_17_20190905202638

EOF

[2019-09-05 20:26:41] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

List git references

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git tag --list --no-column

EOF

[2019-09-05 20:26:43] development.DEBUG: ### Run script locally

bash -s << 'EOF'

Turn on quit on non-zero exit

set -e

List git references

cd /var/www/deployer/storage/app/mirrors/git_bitbucket.org_VE3_colplan.git

git branch --list --no-column

EOF

[2019-09-05 20:26:45] development.ERROR: Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638'...

warning: --depth is ignored in local clones; use file:// instead.

done.

Submodule 'forecaster' (git@bitbucket.org:VE3/forecaster.git) registered for path 'forecaster'

Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...

Permission denied (publickey).

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

fatal: clone of 'git@bitbucket.org:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed

Failed to clone 'forecaster'. Retry scheduled

Cloning into '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster'...

Permission denied (publickey).

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

fatal: clone of 'git@bitbucket.org:VE3/forecaster.git' into submodule path '/var/www/deployer/storage/app/tmp/clone_17_20190905202638/forecaster' failed

Failed to clone 'forecaster' a second time, aborting

When nailing down the issue I found that cloning a submodule from a mirror requires authentication. So the ssh configs script should be passed as an argument to the CreateReleaseArchive.sh which should export GIT_SSH as you do when you clone the mirror. If you do not need a submodule just remove --recursive from the git clone command. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Yes, same private key on the two repos. I can clone them as long as I pass the ssh key. It seems that the deployer does not use the wrapper ssh script when cloning from the mirror which raises the error.

A simple test:

#!/usr/bin/env sh

ssh -o CheckHostIP=no \
    -o IdentitiesOnly=yes \
    -o StrictHostKeyChecking=no \
    -o PasswordAuthentication=no \
    -o IdentityFile=MyprivateKey $*`

export GIT_SSH=/path/to/script.sh

git clone --mirror git@bitbucket.org:VE3/repoWithSubmodules.git

This is what deployer does to clone the mirror and it is working. Now when it clones from the mirror to the repository to archive, it does not set the git_ssh var so:

unset GIT_SSH

git clone --depth 1 --recursive projectWithSubmdules.git projectWithSubmodules

This last command will raise the mentioned error.

Hi everybody, I have a same problem. Project with submodules, all of them have access keys same as the main project, but deploy keeps failing.

Is there any way to make it work without getting rid of the submodules?


Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.
fatal: clone of 'git@bitbucket.org:xyz.git' into submodule path 'xyz' failed

I tried some workarounds but none of them worked. In our case, since the submodule is actually a SOAP service we deployed it separately.

Hey there 👋🏼

I've encountered similar problem some time ago, and ended up writing a quick tip on how I approached it – https://heydarmen.com/blog/using-deployer-with-git-submodules.html

hey @darmen I think you are looking for https://github.com/deployphp/deployer

REBELinBLUE / deployer