SSH fails to connect after launchpad apply

[th3geekguy]

I receive the following errors and am unable to ever connect via SSH (~/.mirantis-launchpad/cluster/[cluster-name]/apply.log):

time="03 Apr 24 14:36 UTC" level=debug msg="preparing phase 'Check For Upgrades'"
time="03 Apr 24 14:36 UTC" level=info msg="\x1b[32m==> Running phase: Check For Upgrades\x1b[0m"
time="03 Apr 24 14:36 UTC" level=debug msg="phase 'Check For Upgrades' took 0s"
time="03 Apr 24 14:36 UTC" level=debug msg="preparing phase 'Open Remote Connection'"
time="03 Apr 24 14:36 UTC" level=info msg="\x1b[32m==> Running phase: Open Remote Connection\x1b[0m"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using default known_hosts file ~/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"

Instances are up in Openstack and I am able to connect with key pair on my local machine -- but I should also be able to connect using my TerraTrain lab and configuration setup by launchpad 🤷

What am I missing? Thanks.

[mirantisbard]

You may want to check the permissions on the /terraTrain/.ssh/known_hosts file and other files in the .ssh directory.

On Wed, Apr 3, 2024 at 7:52 AM Phillip Benzinger @.***> wrote:

I receive the following errors and am unable to ever connect via SSH (~/.mirantis-launchpad/cluster/[cluster-name]/apply.log):

time="03 Apr 24 14:36 UTC" level=debug msg="preparing phase 'Check For Upgrades'" time="03 Apr 24 14:36 UTC" level=info msg="\x1b[32m==> Running phase: Check For Upgrades\x1b[0m" time="03 Apr 24 14:36 UTC" level=debug msg="phase 'Check For Upgrades' took 0s" time="03 Apr 24 14:36 UTC" level=debug msg="preparing phase 'Open Remote Connection'" time="03 Apr 24 14:36 UTC" level=info msg="\x1b[32m==> Running phase: Open Remote Connection\x1b[0m" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using default known_hosts file ~/.ssh/known_hosts" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out" time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out" time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out" time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out" time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set" time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"

Instances are up in Openstack and I am able to connect with key pair on my local machine -- but I should also be able to connect using my TerraTrain lab and configuration setup by launchpad 🤷

What am I missing? Thanks.

— Reply to this email directly, view it on GitHub https://github.com/Mirantis/launchpad/issues/96, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBIT35YLY62MBNCVZEI6PLY3QJTBAVCNFSM6AAAAABFVPNINGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZDGMJVGMYDMNQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[th3geekguy]

Ok, I checked and it looks fine (I'm also running as root within the container):

[root]-[e345a1b4870b]-[~]-[20:22-Wed Apr 03]
$ ll /terraTrain/.ssh/
total 12
drwx------ 2 root root 4096 Apr  3 20:19 ./
drwxr-xr-x 1 root root 4096 Apr  3 20:19 ../
-rw------- 1 root root    0 Apr  3 15:17 known_hosts

[th3geekguy]

known_hosts is empty -- but I'm not sure that would block making a connection

[mirantisbard]

I wonder what happens if you remove it?

On Wed, Apr 3, 2024 at 1:25 PM Phillip Benzinger @.***> wrote:

known_hosts is empty -- but I'm not sure that would block making a connection

— Reply to this email directly, view it on GitHub https://github.com/Mirantis/launchpad/issues/96#issuecomment-2035514939, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBIT3YC7KSK7ZOZ3X5N2M3Y3RQR7AVCNFSM6AAAAABFVPNINGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZVGUYTIOJTHE . You are receiving this because you commented.Message ID: @.***>

[th3geekguy]

No change:

$ ssh -i key-pair ubuntu@[lab_ip] -vvv
OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "lab_ip" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to lab_ip [lab_ip] port 22.
debug1: connect to address lab_ip port 22: Connection timed out
ssh: connect to host lab_ip port 22: Connection timed out

[mirantisbard]

Okay. So, not a permissions issue at all... I wonder if it is having an issue resolving the "lab_ip" and perhaps if the lab_ip was placed in the "known_hosts" file it might resolve the issue?

On Thu, Apr 4, 2024 at 11:14 AM Phillip Benzinger @.***> wrote:

No change:

$ ssh -i key-pair @.**_ip] -vvv OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n 7 Dec 2017 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for debug2: resolving "lab_ip" port 22 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to lab_ip [lab_ip] port 22. debug1: connect to address lab_ip port 22: Connection timed out ssh: connect to host lab_ip port 22: Connection timed out

— Reply to this email directly, view it on GitHub https://github.com/Mirantis/launchpad/issues/96#issuecomment-2037879712, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBIT33KD6QUBKP6JLYVAB3Y3WKAHAVCNFSM6AAAAABFVPNINGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZXHA3TSNZRGI . You are receiving this because you commented.Message ID: @.***>

[james-nesbitt]

can you confirm something for me: are you running in an environment with ssh-agent running, with at least one key added, but not your ssh key? Rig has a known bug in this case caused by https://github.com/k0sproject/rig/commit/0c0e31130db1f26f31ade67f8d6a49a063142894.

[kke]

The issue title is SSH fails to connect after launchpad apply but the issue text says I receive the following errors and am unable to ever connect via SSH.

I'm not sure if I understood correctly that you have never been able to connect using ssh from that host at all? The title would suggest that launchpad broke something, but reading between the lines makes me think it never worked?

Maybe your client was blacklisted because of multiple failed connection attempts or perhaps there is some firewall. Both launchpad and ssh give connection timed out which should not result from problems with known_hosts file, agent or keys.