no matching host key type found. Their offer: ssh-rsa,ssh-dss

[Nilesh486]

Environment

• OS: CentOS 8
• Kubernetes/K3s: k3s

• AWX Operator: 1.1.4 We are getting below error while Job execution, some of the jobs are running but most of them are getting failed due to below error.

  "unreachable": true, "msg": "Failed to connect to the host via ssh: Unable to negotiate with *** port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss", "changed": false }

[kurokobo]

no matching host key type found. Their offer: ssh-rsa,ssh-dss

This error message is the answer. Have you made any investigation on your side?

[Nilesh486]

In investigation, Jobs which are failing are not connecting from container. Also we have check and found in below path from container, entry is present for failling servers. ~/.ssh/known_hosts

tried to connect the server with below command

ssh jboss@**

Unable to negotiate with ** port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss

[kurokobo]

I mean, you should investigate a bit to know what that error message means. There is a lot of knowledge about that error on the internet.

[Nilesh486]

Hi ,

We are able to connect target server from host. But, we are not able to connect target server from Container which is on the host server.

[Nilesh486]

please provide appropriate link so we can implement and try it

[Nilesh486]

we have tried this resolution as well but not working

https://github.com/iam-orsu/SSH-troubleshoot/blob/main/Host%20key%20algorithm

[Nilesh486]

we have run this debug command , getting below output from container

ssh -v -oHostKeyAlgorithms=+ssh-rsa jboss@****

Connection failed ERROR:

debug1: Connecting to **** [****] port 22. debug1: Connection established. debug1: identity file /var/lib/awx/.ssh/id_rsa type -1 debug1: identity file /var/lib/awx/.ssh/id_rsa-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_dsa type -1 debug1: identity file /var/lib/awx/.ssh/id_dsa-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_ecdsa type -1 debug1: identity file /var/lib/awx/.ssh/id_ecdsa-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_ecdsa_sk type -1 debug1: identity file /var/lib/awx/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_ed25519 type -1 debug1: identity file /var/lib/awx/.ssh/id_ed25519-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_ed25519_sk type -1 debug1: identity file /var/lib/awx/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_xmss type -1 debug1: identity file /var/lib/awx/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.7 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: compat_banner: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000002 debug1: Authenticating to *****:22 as 'jboss' debug1: load_hostkeys: fopen /var/lib/awx/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 debug1: kex: host key algorithm: (no match) Unable to negotiate with ** port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss

[Nilesh486]

Connection established debug log:

debug1: Connecting to *** [****] port 22. debug1: Connection established. debug1: identity file /var/lib/awx/.ssh/id_rsa type -1 debug1: identity file /var/lib/awx/.ssh/id_rsa-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_dsa type -1 debug1: identity file /var/lib/awx/.ssh/id_dsa-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_ecdsa type -1 debug1: identity file /var/lib/awx/.ssh/id_ecdsa-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_ecdsa_sk type -1 debug1: identity file /var/lib/awx/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_ed25519 type -1 debug1: identity file /var/lib/awx/.ssh/id_ed25519-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_ed25519_sk type -1 debug1: identity file /var/lib/awx/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /var/lib/awx/.ssh/id_xmss type -1 debug1: identity file /var/lib/awx/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.7 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0 debug1: compat_banner: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000 debug1: Authenticating to *****:22 as 'jboss' debug1: load_hostkeys: fopen /var/lib/awx/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-ed25519 debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: compression: none debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: compression: none debug1: kex: curve25519-sha256 need=32 dh_need=32 debug1: kex: curve25519-sha256 need=32 dh_need=32 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: SSH2_MSG_KEX_ECDH_REPLY received debug1: Server host key: ssh-ed25519 SHA256:YirMQk8izApUbtagc0M0LZmBhqygXn8wV3GCsGEtm3k debug1: load_hostkeys: fopen /var/lib/awx/.ssh/known_hosts2: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: Host '**' is known and matches the ED25519 host key. debug1: Found key in /var/lib/awx/.ssh/known_hosts:1 debug1: rekey out after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey in after 4294967296 blocks debug1: Will attempt key: /var/lib/awx/.ssh/id_rsa debug1: Will attempt key: /var/lib/awx/.ssh/id_dsa debug1: Will attempt key: /var/lib/awx/.ssh/id_ecdsa debug1: Will attempt key: /var/lib/awx/.ssh/id_ecdsa_sk debug1: Will attempt key: /var/lib/awx/.ssh/id_ed25519 debug1: Will attempt key: /var/lib/awx/.ssh/id_ed25519_sk debug1: Will attempt key: /var/lib/awx/.ssh/id_xmss debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521> debug1: SSH2_MSG_SERVICE_ACCEPT received

---

• !! ONLY AUTHORIZED USERS ARE ALLOWED TO LOGON UNDER PENALTY OF LAW!! *
• This is a private computer network and may be used only by direct *
• permission of its owner(s). The owner(s) reserves the right to *
• monitor use of this network to ensure network security and to respond *
• to specific allegations of misuse. Use of this network shall constitute *
• consent to monitoring for these and any other purposes. In addition, *
• the owner(s) reserves the right to consent to a valid law enforcement *
• request to search the network for evidence of a crime stored within *
• this network. *

  ---

  debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug1: Next authentication method: gssapi-with-mic debug1: No credentials were supplied, or the credentials were unavailable or inaccessible No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: No credentials were supplied, or the credentials were unavailable or inaccessible No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Next authentication method: publickey debug1: Trying private key: /var/lib/awx/.ssh/id_rsa debug1: Trying private key: /var/lib/awx/.ssh/id_dsa debug1: Trying private key: /var/lib/awx/.ssh/id_ecdsa debug1: Trying private key: /var/lib/awx/.ssh/id_ecdsa_sk debug1: Trying private key: /var/lib/awx/.ssh/id_ed25519 debug1: Trying private key: /var/lib/awx/.ssh/id_ed25519_sk debug1: Trying private key: /var/lib/awx/.ssh/id_xmss debug1: Next authentication method: password

[Nilesh486]

atlast we came to know the OS version difference where execution is not working has OS version 6.10 and where execution is working having OS version 8.6.

Should be the reason ?

[kurokobo]

atlast we came to know the OS version difference where execution is not working has OS version 6.10 and where execution is working having OS version 8.6. Should be the reason ?

Of course yes. Current AWX is based on CentOS Stream 9, and for security reasons, OpenSSL in CentOS Stream 9 doesn't allow some legacy algorithms by default that used in RHEL/CentOS 6 by default.

For security reasons, the legacy algorithms should not be used. Therefore, your first choice should be replacing host key on your RHEL 6 host with newly generated one by ECDSA instead of RSA. If this is difficult for you, you can use legacy algotithms by some way.

• If you want to use (latest) CentOS 9 based EE:
  • Customize your EE image with update-crypto-policies to use LEGACY or DEFAULT:SHA1 (by building custom image) and then make your Job Template to use customized EE
  • In addition to customize EE, pass -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa via ansible_ssh_common_args by specifying it in your inventory
• If it's ok for you to use (old) CentOS 8 based EE:
  • Add new EE with tag quay.io/ansible/awx-ee:21.11.0 (this is the latest 8 based EE) and specify it in your Job Template

[Nilesh486]

We are unable to build customize EE image , could you please share steps or give some clarity on how to and which file we have to make changes so that can be used for LEGACY algorithm

[kurokobo]

Sorry, I'm tired, you are asking too much in recent your 5 issues. I am happy to help you develop a better technical understanding, but please don't pass all the costs of solving your issues on to me without doing enough research, trial and error befor asking, providing what you did and what you see, how much you understand and what you do not understand.

Use quay.io/ansible/awx-ee:21.11.0 instead. This is the simplest solution. Or try to start with following Dockerfile.

FROM quay.io/ansible/awx-ee:latest
USER root
RUN update-crypto-policies --set LEGACY
USER 1000

[Nilesh486]

Thanks for the all your help. As we dont have time bcaz the currently multiple activity is going on and also, we have to do it within 2 days hence if there is solution available that we can apply it directly.

Anyways Thanks for all your help again.