Unitech / pm2

Node.js Production Process Manager with a built-in Load Balancer.
https://pm2.keymetrics.io/docs/usage/quick-start/
Other
41.46k stars 2.62k forks source link

Deploy via SSH fails frequently, until it doesn't #5598

Open yadoga opened 1 year ago

yadoga commented 1 year ago

Deploying via SSH often aborts with this following message:

  ○ deploying origin/main
  ○ executing pre-deploy-local
  ○ hook pre-deploy
  ○ fetching updates
  ○ full fetch
Fetching origin
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error: Could not fetch origin

  fetch failed

Deploy failed
Deploy failed with exit code: 1

Checked the rights, they are OK.

Sometime deploys work only after 50 attempts. Sometimes on the very first one. Which makes this very puzzling for me to comprehend.

It feels like a timeout thing or something similar. Again, deploying always works at some point. Ideas?

PM2 report:

--- PM2 report ----------------------------------------------------------------
Date                 : Fri May 05 2023 18:32:46 GMT+0200 (Mitteleuropäische Sommerzeit)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.3.0
node version         : 18.3.0
node path            : /opt/homebrew/bin/pm2
argv                 : /opt/homebrew/Cellar/node/18.3.0/bin/node,/opt/homebrew/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : yadoga
uid                  : 501
gid                  : 20
uptime               : 1min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.3.0
node version         : 18.3.0
node path            : /opt/homebrew/bin/pm2
argv                 : /opt/homebrew/Cellar/node/18.3.0/bin/node,/opt/homebrew/bin/pm2,report
argv0                : node
user                 : yadoga
uid                  : 501
gid                  : 20
===============================================================================
--- System info --------------------------------------------
arch                 : arm64
platform             : darwin
type                 : Darwin
cpus                 : Apple M1 Pro
cpus nb              : 8
freemem              : 120750080
totalmem             : 17179869184
home                 : /Users/yadoga
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬───────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name      │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
└────┴───────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/Users/yadoga/.pm2/pm2.log last 20 lines:
PM2        | 2023-05-05T10:34:50: PM2 log: Concurrent actions   : 2
PM2        | 2023-05-05T10:34:50: PM2 log: SIGTERM timeout      : 1600
PM2        | 2023-05-05T10:34:50: PM2 log: ===============================================================================
PM2        | 2023-05-05T18:31:05: PM2 log: PM2 successfully stopped
PM2        | 2023-05-05T18:31:05: PM2 log: ===============================================================================
PM2        | 2023-05-05T18:31:05: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
PM2        | 2023-05-05T18:31:05: PM2 log: Time                 : Fri May 05 2023 18:31:05 GMT+0200 (Mitteleuropäische Sommerzeit)
PM2        | 2023-05-05T18:31:05: PM2 log: PM2 version          : 5.3.0
PM2        | 2023-05-05T18:31:05: PM2 log: Node.js version      : 18.3.0
PM2        | 2023-05-05T18:31:05: PM2 log: Current arch         : arm64
PM2        | 2023-05-05T18:31:05: PM2 log: PM2 home             : /Users/yadoga/.pm2
PM2        | 2023-05-05T18:31:05: PM2 log: PM2 PID file         : /Users/yadoga/.pm2/pm2.pid
PM2        | 2023-05-05T18:31:05: PM2 log: RPC socket file      : /Users/yadoga/.pm2/rpc.sock
PM2        | 2023-05-05T18:31:05: PM2 log: BUS socket file      : /Users/yadoga/.pm2/pub.sock
PM2        | 2023-05-05T18:31:05: PM2 log: Application log path : /Users/yadoga/.pm2/logs
PM2        | 2023-05-05T18:31:05: PM2 log: Worker Interval      : 30000
PM2        | 2023-05-05T18:31:05: PM2 log: Process dump file    : /Users/yadoga/.pm2/dump.pm2
PM2        | 2023-05-05T18:31:05: PM2 log: Concurrent actions   : 2
PM2        | 2023-05-05T18:31:05: PM2 log: SIGTERM timeout      : 1600
PM2        | 2023-05-05T18:31:05: PM2 log: ===============================================================================
yadoga commented 1 year ago

I have tried changing the SSH port, to no change in behaviour.

yadoga commented 1 year ago

More Log info:

May  5 09:25:53 myserver sshd[3236236]: Accepted publickey for myusername from myip port 58657 ssh2: RSA SHA256:1234/+ABC123/abcdefghijklmnop
May  5 09:25:53 myserver sshd[3236236]: pam_unix(sshd:session): session opened for user myusername by (uid=0)
May  5 09:25:53 myserver systemd-logind[1075]: New session 249598 of user myusername.
May  5 09:25:53 myserver sshd[3236289]: Received disconnect from myip port 58657:11: disconnected by user
May  5 09:25:53 myserver sshd[3236289]: Disconnected from user username myip port 58657
May  5 09:25:53 myserver sshd[3236236]: pam_unix(sshd:session): session closed for user myusername
May  5 09:25:53 myserver systemd-logind[1075]: Session 249598 logged out. Waiting for processes to exit.
May  5 09:25:53 myserver systemd-logind[1075]: Removed session 249598.

Received disconnect from myip could hint to pm2 sending an abort signal.

Could the connection to GitHub be a culprit?