Open mhtr opened 1 day ago
memetb
commented
1 day ago failed to connect to libvirt on the remote host: ssh: rejected: connect failed (open failed)
This is not a bug at all. The error you are getting is coming directly from the OS. It's saying it wasn't able to open a tcp/ip connection, let alone an ssh one.
I am not sure what your first log output is, but it appears to be a red herring from what I can tell.
mhtr
commented
1 day ago @memetb
The first log output from sshd on libvirt host when command terraform plan is executed. This means that the connection via ssh to the host is established, but then something goes wrong. But at the same time the command virsh -c ... with the same uri as in terraform is executed without problems
memetb
commented
1 day ago The code that is emitting the logs you are showing is here:
log.Printf("[INFO] SSH connecting to '%v' (%v)", target, hostName)
conn, err := ssh.Dial("tcp", net.JoinHostPort(hostName, port), &cfg)
if err != nil {
return nil, fmt.Errorf("failed to connect to remote host '%v': %w", target, err)
}Can you do me a favor and post:
ssh -vvirsh. It has many possible options, it may not be going over ssh at all.mhtr
commented
22 hours ago The code that is emitting the logs you are showing is here
Are you sure? Looks like the code that is emitting the error is here when it trying to connect to unix socket.
address := u.Query().Get("socket")
if address == "" {
address = defaultUnixSock
}
c, err := sshClient.Dial("unix", address)
if err != nil {
return nil, fmt.Errorf("failed to connect to libvirt on the remote host: %w", err) <---- the error message is like the error log from terraform
}Because the error log is
Error: failed to connect: failed to connect to libvirt on the remote host: ssh: rejected: connect failed (open failed)
- the stanza for this host in ~/.ssh/config
Host libvirt_host
Port 12345
- an attempt at connecting to the server via a cli call to
ssh -v
OpenSSH_9.2p1 Debian-2+deb12u3, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/username/.ssh/config
debug1: /home/username/.ssh/config line 1: Applying options for <libvirt_host>
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to <libvirt_host> [<libvirt_host>] port 12345.
debug1: Connection established.
debug1: identity file /home/username/.ssh/id_rsa type -1
debug1: identity file /home/username/.ssh/id_rsa-cert type -1
debug1: identity file /home/username/.ssh/id_ecdsa type -1
debug1: identity file /home/username/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/username/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/username/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/username/.ssh/id_ed25519 type 3
debug1: identity file /home/username/.ssh/id_ed25519-cert type -1
debug1: identity file /home/username/.ssh/id_ed25519_sk type -1
debug1: identity file /home/username/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/username/.ssh/id_xmss type -1
debug1: identity file /home/username/.ssh/id_xmss-cert type -1
debug1: identity file /home/username/.ssh/id_dsa type -1
debug1: identity file /home/username/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u3
debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to <libvirt_host>:12345 as 'username'
debug1: load_hostkeys: fopen /home/username/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:5MvPCke/XBgLCJiNlnMcChV8RS+5P2XwCkSulk+d8Dg
debug1: load_hostkeys: fopen /home/username/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[<libvirt_host>]:12345' is known and matches the ED25519 host key.
debug1: Found key in /home/username/.ssh/known_hosts:5
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/username/.ssh/id_rsa
debug1: Will attempt key: /home/username/.ssh/id_ecdsa
debug1: Will attempt key: /home/username/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/username/.ssh/id_ed25519 ED25519 SHA256:Y9gw3CEhjS6fumcAGw8WPjWsuUkOoZyu6ICcYjjUNBU
debug1: Will attempt key: /home/username/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/username/.ssh/id_xmss
debug1: Will attempt key: /home/username/.ssh/id_dsa
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
#################################################################
# ALERT!!! #
# You are entering into a secured area! Your IP, Login Time, #
# Username has been noted and has been sent to the server #
# administrator! #
# This service is restricted to authorized users only. All #
# activities on this system are logged. #
# Unauthorized access will be fully investigated and reported #
# to the appropriate law enforcement agencies. #
#################################################################
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/username/.ssh/id_rsa
debug1: Trying private key: /home/username/.ssh/id_ecdsa
debug1: Trying private key: /home/username/.ssh/id_ecdsa_sk
debug1: Offering public key: /home/username/.ssh/id_ed25519 ED25519 SHA256:Y9gw3CEhjS6fumcAGw8WPjWsuUkOoZyu6ICcYjjUNBU
debug1: Server accepts key: /home/username/.ssh/id_ed25519 ED25519 SHA256:Y9gw3CEhjS6fumcAGw8WPjWsuUkOoZyu6ICcYjjUNBU
Authenticated to <libvirt_host> ([<libvirt_host>]:12345) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /home/username/.ssh/known_hosts for [<libvirt_host>]:12345 / (none)
debug1: client_input_hostkeys: searching /home/username/.ssh/known_hosts2 for [<libvirt_host>]:12345 / (none)
debug1: client_input_hostkeys: hostkeys file /home/username/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: no new or deprecated keys from server
debug1: Remote: /home/username/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/username/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug1: channel 0: setting env LC_CTYPE = "UTF-8"
debug1: pledge: fork
Linux <libvirt_host> 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Oct 24 09:07:47 2024 from <terraform_host>
- show more details about the call to
virsh. It has many possible options, it may not be going over ssh at all.
virsh -c "qemu+ssh://username@<libvirt_host>/system?sshauth=privkey&keyfile=/home/username/.ssh/id_ed25519&no_verify=1"
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh # @mhtr from all I'm seeing, my gut is saying that this is some sort of proper ssh authentication error that isn't verbose enough: meaning the behaviour is correct, but we're just not seeing why it's failing. The plugin clearly isn't segfaulting, and it seems it is attempting to make a connection with the correct parameters.
It would really help out if you could isolate the issue by carrying over things one by one. That is, take this starting query string:
qemu+ssh://username@host/system?sshauth=privkey&keyfile=/home/username/.ssh/id_ed25519&no_verify=1And start by:
qemu+ssh://host/systemLet me know what you discover.
Fyi: StrictHostKeyChecking <-> no_verify IdentityFile <-> keyfile sshauth=privkey is unnecessary if you specify IdentityFile.
System Information
Linux distribution
Debian 12Terraform version
Provider and libvirt versions
with fix from 1117
Description of Issue/Question
Failed to connect to libvirt on the remote host when run
terraform plan, but with virsh everything is okSetup
Additional information:
In logs on libvirt_host I see that terraform_host is connected via ssh
But I have error with connecting to libvirt debug logs: