Deployment over 12 hours fails due to expired IdentityFile "no such identity"

dgpro commented 6 years ago

Before submitting your issue, please make sure that you've checked all of the checkboxes below.

[x] You're running the latest release version of Deployer.
[x] Ensure that you're running at least PHP 7.0.8, you can check this by running php -v
[x] You've ran composer install --no-dev from the root of your installation.
[x] You've ran npm install --production from the root of your installation.
[x] You've ran rm -rf bootstrap/cache/* from the root of your installation.
[x] You have restarted the queue listener and node socket server.

Describe the bug

We have a command that runs about 14 hours and when it finishes the next command fails with this error:

no such identity: /var/www/deployer/storage/app/tmp/keyAC3Rrs: No such file or directory
Permission denied (publickey).

Other quick commands run one after each other without issues. I've looked at the log file and looks like the IdentityFile expires if I understood it correctly. Is there any way to make identity file expire time longer at least when deployment is still running?

Expected behaviour

Commands will continue to run after long command is finished.

Actual behaviour

Deployment fails after long command is finished.

Steps to reproduce

Run a command that takes longer than 12 hours to complete

Environment info

Visit /admin/sysinfo on your install and click the "Get System Report" button. Paste the report here, if you can't please provide the following instead.

PHP Version: 7.1.20-1+ubuntu16.04.1+deb.sury.org+1
Server Software: nginx/1.10.3
Server OS: Linux deployer 4.4.0-1065-aws #75-Ubuntu SMP Fri Aug 10 11:14:32 UTC 2018 x86_64
Database: mysql 5.7.1

Logs (see storage/logs/) or other output that would be helpful

deployer-error.log

REBELinBLUE commented 6 years ago

wow, I never thought it would be possible for a deploy to take this long.

The reason is that SSH keys are normally cleaned up when deploy is finished, but if something happens (say the worker crashes) it gets left on the server which isn't great so there is a cronjob to clean up keys older than 12 hours.

For now the simplest fix is to change app/Console/Commands/ClearOldkeys.php and increase the KEEP_FILES_FOR_HOURS constant. Maybe I should make it a config option?

Trying to figure out if the deploy which uses the key is still actually running because much more complicated, although it may be doable.

dgpro commented 6 years ago

Thanks for your response, Stephen. I'll try to update the constant and see if it helps. Maybe a configurable option is a good idea even if it's rarely going to be changed. The command used to take just a couple of hours until we moved our servers to AWS and they reduce RDS instance Write IOPS and CPU when it's heavily used unless you pay a lot. But we need to run it maybe once a month so it's not worth it.

dgpro commented 6 years ago

Hi Stephen, I can confirm that changing KEEP_FILES_FOR_HOURS constant in app/Console/Commands/ClearOldKeys.php to a longer period works and we were able successfully complete the deployment. Thank you!

REBELinBLUE / deployer