search

cockpit-project / cockpit

Cockpit is a web-based graphical interface for servers.
http://www.cockpit-project.org/
GNU Lesser General Public License v2.1
11.36k stars 1.13k forks source link

sshd does not accept 512 bit RSA keys from cockpit-ssh, but does from ssh #15963

Closed rrandecker closed 1 year ago

rrandecker commented 3 years ago

I have an issue that recently, I started having to put in password for all connected servers each time I log into cockpit. previously, It would log me in without needing to enter a password. Am I doing something wrong? I can use ssh to the other servers without needing a password, so I don't understand why cockpit is needing a password. Is there something more I need to setup?

Also previously there was a button on the login screen to re-use the credentials on connected servers, but this check is not there anymore, there is a other options, which seems to let me put in a different ip to connect to other than the one in the URL of the browser. Not sure if that is related.

When I try to run: /usr/libexec/cockpit-ssh id>@<ip It returns: 75 {"command":"authorize","challenge":"*","cookie":"session1501921623853972"}

and waits for input.. any input I give: cockpit-ssh-Message: 14:33:12.330: couldn't read control message: Bad message 79

not sure if that is same that cockpit uses for ssh.

mvollmer commented 3 years ago

I have an issue that recently, [...]

Do you still remember which version of Cockpit did work for you? Connecting to remote hosts has changed significantly in Cockpit 228. Have you used a version of Cockpit later than 228 but earlier than 238? Did it work for you?

See https://cockpit-project.org/blog/cockpit-228.html for the 228 release notes.

I can use ssh to the other servers without needing a password, so I don't understand why cockpit is needing a password. Is there something more I need to setup?

No, Cockpit should ideally behave just like /bin/ssh on the command line. If /bin/ssh connects without prompting, then Cockpit should do the same. However, Cockpit doesn't use the same code as /bin/ssh and there might be unintended differences in behavior. For example, if you have a lot of keys in ~/.ssh, Cockpit might not be able to figure out which one will work with the remote machine, while /bin/ssh manages to do so. But Cockpit will look at ~/.ssh/config and honor the IdentityFile settings there.

There might also be complications with the SSH agent and key passphrases.

Could you show relevant pieces of ~/.ssh/config and the output of /bin/ssh -v <remote>?

rrandecker commented 3 years ago

Thank you for the reply.

rrandecker commented 3 years ago

I have it working on cockpit 224.2. Not sure if other versions were loaded between 224.2 and 238. Here is the result of the /bin/ssh -v : 

[xxx@abc3t-m2mch-a1 ~]$ /bin/ssh -v 10.11.12.13
OpenSSH_8.0p1, OpenSSL 1.1.1g FIPS  21 Apr 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/02-ospp.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/02-ospp.conf
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: FIPS mode initialized
debug1: Connecting to 10.11.12.13 [10.11.12.13] port 22.
debug1: Connection established.
debug1: identity file /home/xxx/.ssh/id_rsa type 0
debug1: identity file /home/xxx/.ssh/id_rsa-cert type -1
debug1: identity file /home/xxx/.ssh/id_dsa type -1
debug1: identity file /home/xxx/.ssh/id_dsa-cert type -1
debug1: identity file /home/xxx/.ssh/id_ecdsa type -1
debug1: identity file /home/xxx/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/xxx/.ssh/id_ed25519 type -1
debug1: identity file /home/xxx/.ssh/id_ed25519-cert type -1
debug1: identity file /home/xxx/.ssh/id_xmss type -1
debug1: identity file /home/xxx/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.11.12.13:22 as 'xxx'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: ecdh-sha2-nistp521
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-512 compression: none
debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-512 compression: none
debug1: kex: ecdh-sha2-nistp521 need=64 dh_need=64
debug1: kex: ecdh-sha2-nistp521 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:/vaSU8uoqqp7R6LSNOYirneyvD5F2hMNWwgT1uNp5QY
debug1: Host '10.11.12.13' is known and matches the ECDSA host key.
debug1: Found key in /home/xxx/.ssh/known_hosts:4
debug1: rekey out after 33554432 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 33554432 blocks
debug1: Will attempt key: /home/xxx/.ssh/id_rsa RSA SHA256:weZetFj36CNWMZF3ccbEAbe+VAKA5me1X4Al/ouhQyA agent
debug1: Will attempt key: /home/xxx/.ssh/id_dsa 
debug1: Will attempt key: /home/xxx/.ssh/id_ecdsa 
debug1: Will attempt key: /home/xxx/.ssh/id_ed25519 
debug1: Will attempt key: /home/xxx/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
#############################################################
#  This system is restricted to authorized users only.      #
#  Unauthorized access or access attempts to this system    #
#  or services are prohibited. All user activity is logged. #
#  Evidence of unauthorized use collected during monitoring #
#  may be provided to appropriate personnel for             #
#  administrative, criminal or other adverse action.        #
#############################################################
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/xxx/.ssh/id_rsa RSA SHA256:weZetFj36CNWMZF3ccbEAbe+VAKA5me1X4Al/ouhQyA agent
debug1: Server accepts key: /home/xxx/.ssh/id_rsa RSA SHA256:weZetFj36CNWMZF3ccbEAbe+VAKA5me1X4Al/ouhQyA agent
debug1: Authentication succeeded (publickey).
Authenticated to 10.11.12.13 ([10.11.12.13]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /home/xxx/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/xxx/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
#############################################################
#  This system is restricted to authorized users only.      #
#  Unauthorized access or access attempts to this system    #
#  or services are prohibited. All user activity is logged. #
#  Evidence of unauthorized use collected during monitoring #
#  may be provided to appropriate personnel for             #
#  administrative, criminal or other adverse action.        #
#############################################################
Web console: https://abc3t-m2mch-a2:9090/

Last login: Thu Jun 17 11:14:35 2021 from 10.11.12.14
OS      : Red Hat Enterprise Linux 8.4 (Ootpa)
Kernel  : 4.18.0-305.3.1.el8_4.x86_64

This user does not have a ~/.ssh/config file.. I assume it would use the /etc/ssh/ssh_config by itself, as it seems that is what /bin/ssh is using.

/etc/ssh/ssh_config is:

Include /etc/ssh/ssh_config.d/*.conf
StrictHostKeyChecking ask

In /etc/ssh/ssh_config.d there are 2 files: 02-ospp.conf which contains: 

Match final all
RekeyLimit 512M 1h
GSSAPIAuthentication no
Ciphers aes256-ctr,aes256-cbc,aes128-ctr,aes128-cbc
PubkeyAcceptedKeyTypes ssh-rsa,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256
MACs hmac-sha2-512,hmac-sha2-256
KexAlgorithms ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group14-sha1

and 05-redhat.conf which contains: 

Match final all
        Include /etc/crypto-policies/back-ends/openssh.config
        GSSAPIAuthentication yes
        ForwardX11Trusted yes
        SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
        SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
        SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE
        SendEnv XMODIFIERS

These are all the lines in the config files that are not commented out.

mvollmer commented 3 years ago

Here is the result of the /bin/ssh -v :

Thank you! I am afraid I don't spot anything that would help us. Let's try cockpit-ssh as you did earlier. Run it like so:

$ G_MESSAGES_DEBUG=all /usr/libexec/cockpit-ssh user@host
75

{"command":"authorize","challenge":"*","cookie":"session8927941624009637"}

[ Hit RET a couple of times until it continues.  This authorize challenge always happens regardless of your SSH setup. ]

cockpit-ssh-Message: 12:47:43.434: couldn't read control message: Bad message
(cockpit-ssh:892794): cockpit-ssh-DEBUG: 12:47:43.434: cockpit-ssh root@dev: host argument 'root@dev', host 'dev', username 'root', port '0'
[2021/06/18 12:47:43.437211, 3] ssh_config_parse_file:  Reading configuration data from /home/mvo/.ssh/config
[2021/06/18 12:47:43.437377, 3] ssh_config_parse_file:  Reading configuration data from /etc/libssh/libssh_client.config
[2021/06/18 12:47:43.437519, 3] local_parse_file:  Reading additional configuration data from /etc/crypto-policies/back-ends/libssh.config
[2021/06/18 12:47:43.437792, 3] local_parse_file:  Reading additional configuration data from /etc/ssh/ssh_config
[2021/06/18 12:47:43.438034, 3] local_parse_file:  Reading additional configuration data from /etc/ssh/ssh_config.d/50-redhat.conf
[2021/06/18 12:47:43.438117, 4] ssh_config_parse_line:  line 3: Processing Match keyword 'final'
[2021/06/18 12:47:43.438164, 1] ssh_config_parse_line:  line 3: Unsupported Match keyword 'final', skipping
[2021/06/18 12:47:43.438200, 4] ssh_config_parse_line:  line 3: Processing Match keyword 'all'
...
mvollmer commented 3 years ago

Oh, one more question: Is there a SSH agent or key passphrases involved here somewhere?

rrandecker commented 3 years ago

No SSH agent or key passphrases that I am aware of..

rrandecker commented 3 years ago

Here is the output of the cockpit-ssh with debug info: 

[xxx@abc3t-m2mch-a1 ~]$ G_MESSAGES_DEBUG=all /usr/libexec/cockpit-ssh xxx@10.11.12.13
75

{"command":"authorize","challenge":"*","cookie":"session8430401624020423"}

cockpit-ssh-Message: 12:47:08.866: couldn't read control message: Bad message
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.866: cockpit-ssh xxx@10.11.12.13: host argument 'xxx@10.11.12.13', host '10.11.12.13', username 'xxx', port '0'
[2021/06/18 12:47:08.866991, 3] ssh_config_parse_file:  Reading configuration data from /etc/libssh/libssh_client.config
[2021/06/18 12:47:08.867041, 3] local_parse_file:  Reading additional configuration data from /etc/crypto-policies/back-ends/libssh.config
[2021/06/18 12:47:08.867117, 3] local_parse_file:  Reading additional configuration data from /etc/ssh/ssh_config
[2021/06/18 12:47:08.867172, 3] local_parse_file:  Reading additional configuration data from /etc/ssh/ssh_config.d/02-ospp.conf
[2021/06/18 12:47:08.867194, 4] ssh_config_parse_line:  line 1: Processing Match keyword 'final'
[2021/06/18 12:47:08.867209, 1] ssh_config_parse_line:  line 1: Unsupported Match keyword 'final', skipping
[2021/06/18 12:47:08.867223, 4] ssh_config_parse_line:  line 1: Processing Match keyword 'all'
[2021/06/18 12:47:08.867254, 3] local_parse_file:  Reading additional configuration data from /etc/ssh/ssh_config.d/05-redhat.conf
[2021/06/18 12:47:08.867274, 4] ssh_config_parse_line:  line 3: Processing Match keyword 'final'
[2021/06/18 12:47:08.867289, 1] ssh_config_parse_line:  line 3: Unsupported Match keyword 'final', skipping
[2021/06/18 12:47:08.867303, 4] ssh_config_parse_line:  line 3: Processing Match keyword 'all'
[2021/06/18 12:47:08.867330, 3] local_parse_file:  Reading additional configuration data from /etc/crypto-policies/back-ends/openssh.config
[2021/06/18 12:47:08.867352, 1] ssh_config_parse_line:  Unknown option: GSSAPIKexAlgorithms, line: 3
[2021/06/18 12:47:08.867370, 1] ssh_config_parse_line:  Unknown option: CASignatureAlgorithms, line: 6
[2021/06/18 12:47:08.867393, 2] ssh_config_parse_line:  Unapplicable option: ForwardX11Trusted, line: 12
[2021/06/18 12:47:08.867410, 2] ssh_config_parse_line:  Unapplicable option: SendEnv, line: 15
[2021/06/18 12:47:08.867426, 2] ssh_config_parse_line:  Unapplicable option: SendEnv, line: 16
[2021/06/18 12:47:08.867442, 2] ssh_config_parse_line:  Unapplicable option: SendEnv, line: 17
[2021/06/18 12:47:08.867457, 2] ssh_config_parse_line:  Unapplicable option: SendEnv, line: 18
[2021/06/18 12:47:08.867502, 1] ssh_session_has_known_hosts_entry:  Cannot access file /etc/ssh/ssh_known_hosts
[2021/06/18 12:47:08.867644, 1] ssh_key_cmp:  key types don't match!
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.867: cockpit-ssh xxx@10.11.12.13: using known hosts file (null); host known: 1; connect to unknown hosts: 1
[2021/06/18 12:47:08.867731, 2] ssh_connect:  libssh 0.9.4 (c) 2003-2019 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
[2021/06/18 12:47:08.867753, 3] getai:  host 10.11.12.13 matches an IP address
[2021/06/18 12:47:08.867822, 2] ssh_socket_connect:  Nonblocking connection socket: 5
[2021/06/18 12:47:08.867839, 2] ssh_connect:  Socket connecting, now waiting for the callbacks to work
[2021/06/18 12:47:08.867853, 3] ssh_connect:  Actual timeout : 10000
[2021/06/18 12:47:08.868037, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2021/06/18 12:47:08.868056, 3] ssh_socket_pollcallback:  Received POLLOUT in connecting state
[2021/06/18 12:47:08.868073, 1] socket_callback_connected:  Socket connection callback: 1 (0)
[2021/06/18 12:47:08.868105, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:47:08.868129, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2021/06/18 12:47:08.940774, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLIN ), out buffer 0
[2021/06/18 12:47:08.940810, 3] callback_receive_banner:  Received banner: SSH-2.0-OpenSSH_8.0
[2021/06/18 12:47:08.940824, 2] ssh_client_connection_callback:  SSH server banner: SSH-2.0-OpenSSH_8.0
[2021/06/18 12:47:08.940841, 2] ssh_analyze_banner:  Analyzing banner: SSH-2.0-OpenSSH_8.0
[2021/06/18 12:47:08.940857, 2] ssh_analyze_banner:  We are talking to an OpenSSH client version: 8.0 (80000)
[2021/06/18 12:47:08.940942, 3] ssh_client_select_hostkeys:  Order of wanted host keys: "ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512"
[2021/06/18 12:47:08.941057, 1] ssh_key_cmp:  key types don't match!
[2021/06/18 12:47:08.941108, 1] ssh_known_hosts_read_entries:  Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
[2021/06/18 12:47:08.941132, 3] ssh_client_select_hostkeys:  Algorithms found in known_hosts files: "rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256"
[2021/06/18 12:47:08.941155, 3] ssh_client_select_hostkeys:  Changing host key method to "ecdsa-sha2-nistp256,rsa-sha2-256,rsa-sha2-512,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521"
[2021/06/18 12:47:08.941174, 4] ssh_list_kex:  kex algos: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c
[2021/06/18 12:47:08.941188, 4] ssh_list_kex:  server host key algo: ecdsa-sha2-nistp256,rsa-sha2-256,rsa-sha2-512,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
[2021/06/18 12:47:08.941203, 4] ssh_list_kex:  encryption client->server: aes256-gcm@openssh.com,aes256-ctr,aes256-cbc
[2021/06/18 12:47:08.941217, 4] ssh_list_kex:  encryption server->client: aes256-gcm@openssh.com,aes256-ctr,aes256-cbc
[2021/06/18 12:47:08.941231, 4] ssh_list_kex:  mac algo client->server: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
[2021/06/18 12:47:08.941245, 4] ssh_list_kex:  mac algo server->client: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
[2021/06/18 12:47:08.941259, 4] ssh_list_kex:  compression algo client->server: none,zlib,zlib@openssh.com
[2021/06/18 12:47:08.941273, 4] ssh_list_kex:  compression algo server->client: none,zlib,zlib@openssh.com
[2021/06/18 12:47:08.941287, 4] ssh_list_kex:  languages client->server: 
[2021/06/18 12:47:08.941301, 4] ssh_list_kex:  languages server->client: 
[2021/06/18 12:47:08.941339, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:47:08.941360, 3] packet_send2:  packet: wrote [type=20, len=636, padding_size=10, comp=625, payload=625]
[2021/06/18 12:47:08.941372, 3] ssh_send_kex:  SSH_MSG_KEXINIT sent
[2021/06/18 12:47:08.941392, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2021/06/18 12:47:08.941406, 4] ssh_socket_pollcallback:  sending control flow event
[2021/06/18 12:47:08.941420, 4] ssh_packet_socket_controlflow_callback:  sending channel_write_wontblock callback
[2021/06/18 12:47:08.942929, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLIN ), out buffer 0
[2021/06/18 12:47:08.942954, 3] ssh_packet_socket_callback:  packet: read type 20 [len=380,padding=6,comp=373,payload=373]
[2021/06/18 12:47:08.942971, 3] ssh_packet_process:  Dispatching handler for packet type 20
[2021/06/18 12:47:08.942989, 4] ssh_list_kex:  kex algos: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
[2021/06/18 12:47:08.943003, 4] ssh_list_kex:  server host key algo: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256
[2021/06/18 12:47:08.943017, 4] ssh_list_kex:  encryption client->server: aes128-ctr,aes256-ctr,aes128-cbc,aes256-cbc
[2021/06/18 12:47:08.943031, 4] ssh_list_kex:  encryption server->client: aes128-ctr,aes256-ctr,aes128-cbc,aes256-cbc
[2021/06/18 12:47:08.943045, 4] ssh_list_kex:  mac algo client->server: hmac-sha2-256,hmac-sha2-512
[2021/06/18 12:47:08.943059, 4] ssh_list_kex:  mac algo server->client: hmac-sha2-256,hmac-sha2-512
[2021/06/18 12:47:08.943073, 4] ssh_list_kex:  compression algo client->server: none,zlib@openssh.com
[2021/06/18 12:47:08.943087, 4] ssh_list_kex:  compression algo server->client: none,zlib@openssh.com
[2021/06/18 12:47:08.943101, 4] ssh_list_kex:  languages client->server: 
[2021/06/18 12:47:08.943115, 4] ssh_list_kex:  languages server->client: 
[2021/06/18 12:47:08.943138, 2] ssh_kex_select_methods:  Negotiated ecdh-sha2-nistp256,ecdsa-sha2-nistp256,aes256-ctr,aes256-ctr,hmac-sha2-256,hmac-sha2-256,none,none,,
[2021/06/18 12:47:08.943486, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:47:08.943512, 3] packet_send2:  packet: wrote [type=30, len=76, padding_size=5, comp=70, payload=70]
[2021/06/18 12:47:08.943528, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2021/06/18 12:47:08.943545, 4] ssh_socket_pollcallback:  sending control flow event
[2021/06/18 12:47:08.943562, 4] ssh_packet_socket_controlflow_callback:  sending channel_write_wontblock callback
[2021/06/18 12:47:08.944263, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLIN ), out buffer 0
[2021/06/18 12:47:08.944286, 3] ssh_packet_socket_callback:  packet: read type 31 [len=292,padding=8,comp=283,payload=283]
[2021/06/18 12:47:08.944302, 3] ssh_packet_process:  Dispatching handler for packet type 31
[2021/06/18 12:47:08.944448, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:47:08.944472, 3] packet_send2:  packet: wrote [type=21, len=12, padding_size=10, comp=1, payload=1]
[2021/06/18 12:47:08.944486, 4] ssh_packet_set_newkeys:  called, direction = OUT 
[2021/06/18 12:47:08.944522, 3] crypt_set_algorithms2:  Set output algorithm to aes256-ctr
[2021/06/18 12:47:08.944538, 3] crypt_set_algorithms2:  Set HMAC output algorithm to hmac-sha2-256
[2021/06/18 12:47:08.944552, 3] crypt_set_algorithms2:  Set input algorithm to aes256-ctr
[2021/06/18 12:47:08.944566, 3] crypt_set_algorithms2:  Set HMAC input algorithm to hmac-sha2-256
[2021/06/18 12:47:08.944600, 2] ssh_init_rekey_state:  Set rekey after 33554432 blocks
[2021/06/18 12:47:08.944615, 2] ssh_init_rekey_state:  Set rekey after 33554432 blocks
[2021/06/18 12:47:08.944629, 2] ssh_packet_set_newkeys:  Set rekey after 3600 seconds
[2021/06/18 12:47:08.944646, 2] ssh_packet_client_ecdh_reply:  SSH_MSG_NEWKEYS sent
[2021/06/18 12:47:08.944660, 3] ssh_packet_socket_callback:  Processing 208 bytes left in socket buffer
[2021/06/18 12:47:08.944675, 3] ssh_packet_socket_callback:  packet: read type 21 [len=12,padding=10,comp=1,payload=1]
[2021/06/18 12:47:08.944689, 3] ssh_packet_process:  Dispatching handler for packet type 21
[2021/06/18 12:47:08.944703, 2] ssh_packet_newkeys:  Received SSH_MSG_NEWKEYS
[2021/06/18 12:47:08.944723, 4] ssh_pki_signature_verify:  Going to verify a ecdsa-sha2-nistp256 type signature
[2021/06/18 12:47:08.944847, 4] pki_verify_data_signature:  Signature valid
[2021/06/18 12:47:08.944865, 2] ssh_packet_newkeys:  Signature verified and valid
[2021/06/18 12:47:08.944884, 4] ssh_packet_set_newkeys:  called, direction = IN 
[2021/06/18 12:47:08.944899, 3] ssh_packet_socket_callback:  Processing 192 bytes left in socket buffer
[2021/06/18 12:47:08.944921, 3] ssh_packet_socket_callback:  packet: read type 7 [len=156,padding=14,comp=141,payload=141]
[2021/06/18 12:47:08.944936, 3] ssh_packet_process:  Dispatching handler for packet type 7
[2021/06/18 12:47:08.944950, 3] ssh_packet_ext_info:  Received SSH_MSG_EXT_INFO
[2021/06/18 12:47:08.944964, 3] ssh_packet_ext_info:  Follows 1 extensions
[2021/06/18 12:47:08.944978, 3] ssh_packet_ext_info:  Extension: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
[2021/06/18 12:47:08.944994, 3] ssh_connect:  current state : 7
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.945: cockpit-ssh xxx@10.11.12.13: connected
[2021/06/18 12:47:08.945144, 1] ssh_key_cmp:  key types don't match!
[2021/06/18 12:47:08.945189, 1] ssh_key_cmp:  key types don't match!
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.945: cockpit-ssh xxx@10.11.12.13: verified host key
[2021/06/18 12:47:08.945261, 3] packet_send2:  packet: wrote [type=5, len=28, padding_size=10, comp=17, payload=17]
[2021/06/18 12:47:08.945277, 3] ssh_service_request:  Sent SSH_MSG_SERVICE_REQUEST (service ssh-userauth)
[2021/06/18 12:47:08.945294, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 64
[2021/06/18 12:47:08.945314, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:47:08.945331, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2021/06/18 12:47:08.945345, 4] ssh_socket_pollcallback:  sending control flow event
[2021/06/18 12:47:08.945359, 4] ssh_packet_socket_controlflow_callback:  sending channel_write_wontblock callback
[2021/06/18 12:47:08.984801, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLIN ), out buffer 0
[2021/06/18 12:47:08.984839, 3] ssh_packet_socket_callback:  packet: read type 6 [len=28,padding=10,comp=17,payload=17]
[2021/06/18 12:47:08.984856, 3] ssh_packet_process:  Dispatching handler for packet type 6
[2021/06/18 12:47:08.984875, 3] ssh_packet_service_accept:  Received SSH_MSG_SERVICE_ACCEPT
[2021/06/18 12:47:08.984922, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:47:08.984944, 3] packet_send2:  packet: wrote [type=50, len=44, padding_size=6, comp=37, payload=37]
[2021/06/18 12:47:08.984963, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2021/06/18 12:47:08.984977, 4] ssh_socket_pollcallback:  sending control flow event
[2021/06/18 12:47:08.984991, 4] ssh_packet_socket_controlflow_callback:  sending channel_write_wontblock callback
[2021/06/18 12:47:08.991812, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLIN ), out buffer 0
[2021/06/18 12:47:08.991852, 3] ssh_packet_socket_callback:  packet: read type 53 [len=524,padding=18,comp=505,payload=505]
[2021/06/18 12:47:08.991875, 3] ssh_packet_process:  Dispatching handler for packet type 53
[2021/06/18 12:47:08.991891, 3] ssh_packet_userauth_banner:  Received SSH_USERAUTH_BANNER packet
[2021/06/18 12:47:08.991905, 3] ssh_packet_socket_callback:  Processing 96 bytes left in socket buffer
[2021/06/18 12:47:08.991924, 3] ssh_packet_socket_callback:  packet: read type 51 [len=60,padding=6,comp=53,payload=53]
[2021/06/18 12:47:08.991939, 3] ssh_packet_process:  Dispatching handler for packet type 51
[2021/06/18 12:47:08.991952, 1] ssh_packet_userauth_failure:  Access denied for 'none'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
[2021/06/18 12:47:08.991964, 2] ssh_packet_userauth_failure:  Access denied for 'none'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
[2021/06/18 12:47:08.992010, 4] agent_talk:  Request length: 1
[2021/06/18 12:47:08.992113, 4] agent_talk:  Response length: 431
[2021/06/18 12:47:08.992137, 1] ssh_agent_get_ident_count:  Answer type: 12, expected answer: 12
[2021/06/18 12:47:08.992152, 3] ssh_agent_get_ident_count:  Agent count: 1
[2021/06/18 12:47:08.992169, 3] ssh_userauth_agent:  Trying identity root@a30ncm
[2021/06/18 12:47:08.992182, 3] ssh_key_algorithm_allowed:  Checking rsa-sha2-512 with list <ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com>
[2021/06/18 12:47:08.992196, 3] ssh_key_algorithm_allowed:  Checking rsa-sha2-512 with list <ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com>
[2021/06/18 12:47:08.992244, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:47:08.992263, 3] packet_send2:  packet: wrote [type=50, len=476, padding_size=5, comp=470, payload=470]
[2021/06/18 12:47:08.992279, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2021/06/18 12:47:08.992290, 4] ssh_socket_pollcallback:  sending control flow event
[2021/06/18 12:47:08.992302, 4] ssh_packet_socket_controlflow_callback:  sending channel_write_wontblock callback
[2021/06/18 12:47:08.999154, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLIN ), out buffer 0
[2021/06/18 12:47:08.999184, 3] ssh_packet_socket_callback:  packet: read type 51 [len=60,padding=6,comp=53,payload=53]
[2021/06/18 12:47:08.999201, 3] ssh_packet_process:  Dispatching handler for packet type 51
[2021/06/18 12:47:08.999216, 1] ssh_packet_userauth_failure:  Access denied for 'publickey'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
[2021/06/18 12:47:08.999231, 2] ssh_packet_userauth_failure:  Access denied for 'publickey'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
[2021/06/18 12:47:08.999247, 3] ssh_userauth_agent:  Public key of root@a30ncm refused by server
[2021/06/18 12:47:08.999276, 1] ssh_pki_import_pubkey_file:  Error opening /home/xxx/.ssh/id_ed25519.pub: No such file or directory
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.999: Public key file /home/xxx/.ssh/id_ed25519.pub doesn't exist or isn't readable
[2021/06/18 12:47:08.999321, 1] ssh_pki_import_privkey_file:  Error opening /home/xxx/.ssh/id_ed25519: No such file or directory
[2021/06/18 12:47:08.999340, 1] ssh_pki_import_pubkey_file:  Error opening /home/xxx/.ssh/id_dsa.pub: No such file or directory
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.999: Public key file /home/xxx/.ssh/id_dsa.pub doesn't exist or isn't readable
[2021/06/18 12:47:08.999375, 1] ssh_pki_import_privkey_file:  Error opening /home/xxx/.ssh/id_dsa: No such file or directory
[2021/06/18 12:47:08.999396, 1] ssh_pki_import_pubkey_file:  Error opening /home/xxx/.ssh/id_ecdsa.pub: No such file or directory
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.999: Public key file /home/xxx/.ssh/id_ecdsa.pub doesn't exist or isn't readable
[2021/06/18 12:47:08.999433, 1] ssh_pki_import_privkey_file:  Error opening /home/xxx/.ssh/id_ecdsa: No such file or directory
[2021/06/18 12:47:08.999451, 1] ssh_pki_import_pubkey_file:  Error opening /home/xxx/.ssh/id_ecdsa_sk.pub: No such file or directory
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.999: Public key file /home/xxx/.ssh/id_ecdsa_sk.pub doesn't exist or isn't readable
[2021/06/18 12:47:08.999485, 1] ssh_pki_import_privkey_file:  Error opening /home/xxx/.ssh/id_ecdsa_sk: No such file or directory
[2021/06/18 12:47:08.999503, 1] ssh_pki_import_pubkey_file:  Error opening /home/xxx/.ssh/id_ed25519_sk.pub: No such file or directory
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:08.999: Public key file /home/xxx/.ssh/id_ed25519_sk.pub doesn't exist or isn't readable
[2021/06/18 12:47:08.999539, 1] ssh_pki_import_privkey_file:  Error opening /home/xxx/.ssh/id_ed25519_sk: No such file or directory
[2021/06/18 12:47:08.999583, 3] ssh_key_algorithm_allowed:  Checking rsa-sha2-512 with list <ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com>
[2021/06/18 12:47:08.999610, 3] ssh_key_algorithm_allowed:  Checking rsa-sha2-512 with list <ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com>
[2021/06/18 12:47:08.999660, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:47:08.999677, 3] packet_send2:  packet: wrote [type=50, len=476, padding_size=5, comp=470, payload=470]
[2021/06/18 12:47:08.999692, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2021/06/18 12:47:08.999707, 4] ssh_socket_pollcallback:  sending control flow event
[2021/06/18 12:47:08.999718, 4] ssh_packet_socket_controlflow_callback:  sending channel_write_wontblock callback
[2021/06/18 12:47:09.006570, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLIN ), out buffer 0
[2021/06/18 12:47:09.006609, 3] ssh_packet_socket_callback:  packet: read type 51 [len=60,padding=6,comp=53,payload=53]
[2021/06/18 12:47:09.006626, 3] ssh_packet_process:  Dispatching handler for packet type 51
[2021/06/18 12:47:09.006641, 1] ssh_packet_userauth_failure:  Access denied for 'publickey'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
[2021/06/18 12:47:09.006656, 2] ssh_packet_userauth_failure:  Access denied for 'publickey'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:09.006: /home/xxx/.ssh/id_rsa isn't accepted by the server
79

{"command":"authorize","challenge":"basic","cookie":"session8430401624020429"}

cockpit-ssh-Message: 12:49:06.740: couldn't read control message: Bad message
[2021/06/18 12:49:06.740500, 2] ssh_userauth_gssapi:  Authenticating with gssapi-with-mic
[2021/06/18 12:49:06.740541, 2] ssh_gssapi_auth_mic:  Authenticating with gssapi to host 10.11.12.13 with user xxx
(cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:49:07.277: cockpit-ssh xxx@10.11.12.13: gssapi auth failed
441

{"command":"init","host-key":"10.11.12.13 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJA+akv7Wf4LLm/DJhKwL/OLLQtCtxC+NvjolyD7id2+bfZ5twY9zBZxr3TQDW9mF4pbVuIyMq0RbhR4dWkJ1Bg=\n","host-fingerprint":"SHA256:/vaSU8uoqqp7R6LSNOYirneyvD5F2hMNWwgT1uNp5QY","problem":"authentication-failed","error":"authentication-failed","auth-method-results":{"password":"not-provided","public-key":"denied","gssapi-mic":"denied"}}[2021/06/18 12:49:07.277656, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2021/06/18 12:49:07.277682, 3] packet_send2:  packet: wrote [type=1, len=28, padding_size=7, comp=20, payload=20]
[xxx@abc3t-m2mch-a1 ~]$
mvollmer commented 3 years ago

Ok, let's see. From /bin/ssh:

debug1: Offering public key: /home/xxx/.ssh/id_rsa RSA SHA256:weZetFj36CNWMZF3ccbEAbe+VAKA5me1X4Al/ouhQyA agent
debug1: Server accepts key: /home/xxx/.ssh/id_rsa RSA SHA256:weZetFj36CNWMZF3ccbEAbe+VAKA5me1X4Al/ouhQyA agent

From cockpit-ssh:

cockpit-ssh:843040): cockpit-ssh-DEBUG: 12:47:09.006: /home/xxx/.ssh/id_rsa isn't accepted by the server

So when id_rsa is offered by cockpit-ssh the server rejects it, but when /bin/ssh offers it, it is accepted. Can you try to find relevant messages by sshd on the server? Maybe put sshd into debug mode.

I can also try to reproduce this; for that I would need some versions. Which versions of which OSes are involved, both where Cockpit runs and the server that you try to log into? How did you make the id_rsa key. How long ago? But our best bet is to figure out why your server rejects id_rsa when offered by cockpit-ssh.

Just to describe what is going on: It was very likely always the case in your setup that /bin/ssh was able to log into the server using keys, but Cockpit never was. This is a bug; Cockpit should also be able to use the keys to log into the server. /bin/ssh and Cockpit (unfortunately) use completely different code to do SSH, so discrepancies re supported algorithms etc are not unexpected.

And thanks a lot for helping us figure this out!

rrandecker commented 3 years ago

I am running RHEL 8.4 and cockpit 238 on both systems.
Looking at the sshd logs, found this: Jun 16 11:59:41 abc3t-m2mch-a2 sshd[28168]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth] When looking at the logs, I found a difference here: SSH: Jun 21 12:54:51 abc3t-m2mch-a2 sshd[1316060]: debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0 Cockpit: Jun 21 12:58:00 abc3t-m2mch-a2 sshd[1316693]: debug1: Remote protocol version 2.0, remote software version libssh_0.9.4 It seems that the remote libssh_0.9.4 is not being accepted. Found that changing to use ecdsa keys instead of rsa lets cockpit connect without needing a password, but thought rsa keys should work.

mvollmer commented 3 years ago

Looking at the sshd logs, found this:

Cool, this helps a lot! I'll do some code reading to try to figure out what the conditions are to reproduce this.

It seems that the remote libssh_0.9.4 is not being accepted.

Yeah, but it is not being rejected just because it reports itself as "libssh 0.9.4" (at least I hope not). There must be something specific that libssh is doing that sshd does not accept, like trying to use a "ssh-rsa" key type.

Found that changing to use ecdsa keys instead of rsa lets cockpit connect without needing a password,

Excellent!

but thought rsa keys should work.

Definitely.

rrandecker commented 3 years ago

Wanted to give an update:
I was able to get rsa keys working if I add "rsa-sha2-512" to the /etc/crypto-policies/backends/opensshserver.config

martinpitt commented 3 years ago

Interesting.. but that is an option that should affect the ssh server side only. I understand that 512 bit keys fell out of fashion, as they are too insecure these days -- but it shouldn't care which particular program/library presents a short key to sshd, surely?

rrandecker commented 3 years ago

On our system, we setup to use ECDSA keys and removed the workaround in /etc/crypto-policies/backends/opensshserver.config. I don't know why it would matter with the rsa keys between openssh and the ssh cockpit uses.

martinpitt commented 1 year ago

This is stale -- rrandecker found a solution, and if this happens again, we'll need to reassign it to libssh. Also, we'll very likely replace libssh with a wrapper around ssh(1) soon.