Open tuxpeople opened 2 years ago
I was thinking I'm affected by this issue too, however in my case locking the version to an older one does not fix it. After more testing on the current release version I found that using:
qemu+ssh://{USER}@{IP}/system?keyfile={keyPath}&sshauth=privkey
Note it needed to have sshauth=privkey
otherwise would fail.
@dmacvicar FYI, I just confirmed the same issue in v0.6.14 as reported by @tuxpeople.
Tried successfully with v0.6.13, but same failure as above with v0.6.14.
Client machine and target server are both Centos 7.9.2009.
Here's my uri string:
uri = "qemu+ssh://centos@linux08.fqdn/system?keyfile=/home/centos/.ssh/id_rsa_linux08_centos"
Also, tried this from command line successfully:
virsh -c qemu+ssh://centos@linux08.fqdn/system?keyfile=/home/centos/.ssh/id_rsa_linux08_centos
Also tested appending &sshauth=privkey
as suggested @mallardduck with this uri string. but failed with the same error:
uri = "qemu+ssh://centos@linux08.fqdn/system?keyfile=/home/centos/.ssh/id_rsa_linux08_centos&sshauth=privkey"
But, again, works fine from the command line:
virsh -c 'qemu+ssh://centos@linux08.fqdn/system?keyfile=/home/centos/.ssh/id_rsa_linux08_centos&sshauth=privkey'
Here's the error string (same generated from both URI attempts):
╷
│ Error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
│
│ with provider["registry.terraform.io/dmacvicar/libvirt"],
│ on main.tf line 12, in provider "libvirt":
│ 12: provider "libvirt" {
│
╵
[centos@linux04 kvm-tf]$ terraform --version
Terraform v1.2.1
on linux_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.6.14
Let me know if there is any other information that would be helpful.
Hello, no fixed this problem?
@tiknick - you may want to try debugging the client and server side configs. As noted if you are using ssh keys you should try adding &sshauth=privkey
. Or potentially you may even need to add &no_verify=1
too. While attempting to run the terraform plan you should consider SSHing into the target system and watching the system logs. Potentially you could find an error on the server that could give insight.
@mallardduck I have tried all the methods described in this thread or in other questions.
This link in uri "qemu+ssh://root@
"Error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [no publickey], no supported methods remain".
Error from remote server in /var/log/secure:
sshd[17513]: Connection closed by
@tiknick - That's really odd but unfortunately there's not enough info for me to go off of to help more. It sounds as though you've tried a lot more things already so any more suggestions will probably just overlap with things you attempted. Nothing more about what you've shared here.
The only remaining bits of advice that stick out are:
Based on the logs you found on the server it's not even getting an auth from the client - that's generally what that line means. As it's saying it was still in preauth
meaning the connection was closed before the client even sent an auth attempt. That's kinda why I'm wondering about SSH key perms - maybe terraform isn't accessing the key and sending the auth.
Given that this seems just as likely to be a terraform/client bug as potential to be a server bug, you may want to consider trying to capture more logging on the terraform side too. You can follow this resource: https://www.terraform.io/internals/debugging To try and enable more logging on your client before attempting further debugging. I'd suspect these logs could lead you to find more areas to investigate.
@mallardduck thank you for your answer.
Host - ubuntu 22.04
Remote_server - centos 7
Terraform v1.2.5
on linux_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.6.14
+ provider registry.terraform.io/hashicorp/template v2.2.0
SSH key generate is default and correct privileges. Virsh by terminal is work:
virsh
connect qemu+ssh://root@<remote_server>/system
list
and i see all virtual mashines on remote server.
And I use Virtual Machine Manager with this link for connect to remote_server.
And if I want go to remote server via SSH, i use standart SSH-connect: ssh root@
Just curious, but does it work with:
Because in my case, the exact same config and tf works with v0.6.13, but not v0.6.14.
On Fri, Jul 22, 2022 at 08:01 tiknick @.***> wrote:
@mallardduck https://github.com/mallardduck thank you for your answer.
Host - ubuntu 22.04 Remote_server - centos 7
SSH key generate is default and correct privileges. Virsh by terminal is work: virsh -> then "connect qemu+ssh://root@
/system" -> then "list" and i see all virtual mashines on remote server. And I use Virtual Machine Manager with this link for connect to remote_server.
And if I want go to remote server via SSH, i use standart SSH-connect: ssh root@
without problem. `Terraform v1.2.5 on linux_amd64
- provider registry.terraform.io/dmacvicar/libvirt v0.6.14
- provider registry.terraform.io/hashicorp/template v2.2.0`
— Reply to this email directly, view it on GitHub https://github.com/dmacvicar/terraform-provider-libvirt/issues/939#issuecomment-1192607481, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZNOK6MVXZGSEZ3KFS3UZYLVVKSUHANCNFSM5SBI7GMA . You are receiving this because you commented.Message ID: @.***>
-- Kind regards,
Jordan
-- Jordan Olin Senior Solution Engineer 508-982-4872
...It’s all connected
Just curious, but does it work with: - provider registry.terraform.io/dmacvicar/libvirt v0.6.13 Because in my case, the exact same config and tf works with v0.6.13, but not v0.6.14. On Fri, Jul 22, 2022 at 08:01 tiknick @.> wrote: @mallardduck https://github.com/mallardduck thank you for your answer. Host - ubuntu 22.04 Remote_server - centos 7 SSH key generate is default and correct privileges. Virsh by terminal is work: virsh -> then "connect qemu+ssh://root@
/system" -> then "list" and i see all virtual mashines on remote server. And I use Virtual Machine Manager with this link for connect to remote_server. And if I want go to remote server via SSH, i use standart SSH-connect: ssh root@ @.> -- Kind regards, Jordan … -- Jordan Olin Senior Solution Engineer 508-982-4872 ...It’s all connectedwithout problem. Terraform v1.2.5 on linux_amd64 - provider registry.terraform.io/dmacvicar/libvirt v0.6.14 - provider registry.terraform.io/hashicorp/template v2.2.0
— Reply to this email directly, view it on GitHub <#939 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZNOK6MVXZGSEZ3KFS3UZYLVVKSUHANCNFSM5SBI7GMA . You are receiving this because you commented.Message ID:
It doesn't work on 0.6.9-pre3, 0.6.10-0.6.14.
Sorry, I didn't find a spoiler option here :)
TF_LOG=debug terraform apply
.....
2022-07-22T17:42:21.755+0300 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-07-22T17:42:21.755+0300 [DEBUG] No provider meta schema returned
data.template_file.user_data_master1: Read complete after 0s [id=7fbbd83259fbf96330312c7461f0d930e286990d61d185582b6381c72d7f2beb]
2022-07-22T17:42:21.762+0300 [DEBUG] provider.terraform-provider-libvirt_v0.6.13: 2022/07/22 17:42:21 [DEBUG] Configuring provider for 'qemu+ssh://root@192.168.101.68/system?keyfile=/home/nikita/.ssh/id_rsa&sshauth=privkey': &{map[uri:0xc00042e000] <nil> <nil> 0xc0006520c0 map[] <nil> 0xc000652140 0xc00045a1f8 0xc000674230 false map[] {1 {0 0}} false false}
data.template_file.user_data_worker2: Read complete after 0s [id=8e5452d4842975f9c2a17101d65ea4d5b13609c3e906205bb35617bd7768355c]
2022-07-22T17:42:21.769+0300 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4 pid=63066
2022-07-22T17:42:21.770+0300 [DEBUG] provider: plugin exited
2022-07-22T17:42:22.006+0300 [ERROR] vertex "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" error: failed to dial libvirt: failed to connect to libvirt on the remote host: ssh: rejected: administratively prohibited (open failed)
╷
│ Error: failed to dial libvirt: failed to connect to libvirt on the remote host: ssh: rejected: administratively prohibited (open failed)
│
│ with provider["registry.terraform.io/dmacvicar/libvirt"],
│ on main.tf line 10, in provider "libvirt":
│ 10: provider "libvirt" {
│
╵
2022-07-22T17:42:22.008+0300 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.13/linux_amd64/terraform-provider-libvirt_v0.6.13 pid=63076
2022-07-22T17:42:22.008+0300 [DEBUG] provider: plugin exited
Can you make sure you use the URI format I suggested? The SSH client used by this provider needs you to set the keyfile parameter. In the debug logs I see you're just using the raw /system
path with no parameters.
Please use qemu+ssh://root@192.168.101.68/system?keyfile={keyPath}&sshauth=privkey
Just because using the more simple URI works with QEMU client doesn't mean it will work with this. I would suggest you keep this suggested URI consistent thru your further testing.
If you read this providers source code you will find this is required unless you are using "${HOME}/.ssh/id_rsa"
as the SSH key. Given that you said you are using ed25519
you should include this parameter and direct it to the full/absolute path to the key. See here:
https://github.com/dmacvicar/terraform-provider-libvirt/blob/e5bec5d00819d6fe71a66ee022d1e9d9acd4fe5c/libvirt/uri/ssh.go#L19 https://github.com/dmacvicar/terraform-provider-libvirt/blob/e5bec5d00819d6fe71a66ee022d1e9d9acd4fe5c/libvirt/uri/ssh.go#L32-L34
I tried all possible methods and combinations. Now I use the standard id_rsa key, but I also tried id_ed25519 in my link. Both keys are already flushed to remote_server in the authorized_keys file. I have updated the output of the command above.
cat main.tf
terraform {
required_providers {
libvirt = {
source = "dmacvicar/libvirt"
version = "0.6.14"
}
}
}
provider "libvirt" {
uri = "qemu+ssh://root@ip/system?sshauth=privkey"
}
terraform plan
╷
│ Error: Plugin did not respond
│
│ with provider["registry.terraform.io/dmacvicar/libvirt"],
│ on main.tf line 10, in provider "libvirt":
│ 10: provider "libvirt" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ConfigureProvider call. The plugin
│ logs may contain more details.
╵
Stack trace from the terraform-provider-libvirt_v0.6.14 plugin:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x128002a]
goroutine 14 [running]:
golang.org/x/crypto/ssh.publicKeyCallback.auth(0x11, {0xc0005a6220, 0x20, 0x20}, {0xc00003ee4b, 0x4}, {0x1bad3d0, 0xc0005b8180}, {0x1b9dac0, 0xc00007e660})
golang.org/x/crypto@v0.0.0-20220112180741-5e0467b6c7ce/ssh/client_auth.go:235 +0x24a
golang.org/x/crypto/ssh.(*connection).clientAuthenticate(0xc0005b0080, 0xc0005ac000)
golang.org/x/crypto@v0.0.0-20220112180741-5e0467b6c7ce/ssh/client_auth.go:63 +0x3a9
golang.org/x/crypto/ssh.(*connection).clientHandshake(0xc0005b0080, {0xc00020a0a8, 0x11}, 0xc0005ac000)
golang.org/x/crypto@v0.0.0-20220112180741-5e0467b6c7ce/ssh/client.go:113 +0x26a
golang.org/x/crypto/ssh.NewClientConn({0x1bc21c8, 0xc0005a4008}, {0xc00020a0a8, 0x11}, 0xc0000cb800)
golang.org/x/crypto@v0.0.0-20220112180741-5e0467b6c7ce/ssh/client.go:83 +0x12b
golang.org/x/crypto/ssh.Dial({0x1a22411, 0x5}, {0xc00020a0a8, 0x11}, 0xc0000cb800)
golang.org/x/crypto@v0.0.0-20220112180741-5e0467b6c7ce/ssh/client.go:190 +0x59
github.com/dmacvicar/terraform-provider-libvirt/libvirt/uri.(*ConnectionURI).dialSSH(0xc0000cb9c8)
github.com/dmacvicar/terraform-provider-libvirt/libvirt/uri/ssh.go:130 +0x465
github.com/dmacvicar/terraform-provider-libvirt/libvirt/uri.(*ConnectionURI).DialTransport(0xc00003ee40)
github.com/dmacvicar/terraform-provider-libvirt/libvirt/uri/connection_uri.go:83 +0x45
github.com/dmacvicar/terraform-provider-libvirt/libvirt.(*Config).Client(0x1a4b923)
github.com/dmacvicar/terraform-provider-libvirt/libvirt/config.go:35 +0x73
github.com/dmacvicar/terraform-provider-libvirt/libvirt.providerConfigure(0xc00018a150)
github.com/dmacvicar/terraform-provider-libvirt/libvirt/provider.go:68 +0xff
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).Configure(0xc000532000, 0xc00037a6c8)
github.com/hashicorp/terraform-plugin-sdk@v1.9.0/helper/schema/provider.go:275 +0xb4
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).Configure(0xc00037a068, {0xc00037ce80, 0x1111946}, 0xc00037ce80)
github.com/hashicorp/terraform-plugin-sdk@v1.9.0/internal/helper/plugin/grpc_provider.go:487 +0x1d2
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_Configure_Handler({0x19f4140, 0xc00037a068}, {0x1bb4c30, 0xc00001d440}, 0xc0004f9c20, 0x0)
github.com/hashicorp/terraform-plugin-sdk@v1.9.0/internal/tfplugin5/tfplugin5.pb.go:3251 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000ae180, {0x1bc2380, 0xc0000ae300}, 0xc00058c700, 0xc00008e540, 0x215f0b8, 0x0)
google.golang.org/grpc@v1.27.1/server.go:1024 +0xd1e
google.golang.org/grpc.(*Server).handleStream(0xc0000ae180, {0x1bc2380, 0xc0000ae300}, 0xc00058c700, 0x0)
google.golang.org/grpc@v1.27.1/server.go:1313 +0xa56
google.golang.org/grpc.(*Server).serveStreams.func1.1()
google.golang.org/grpc@v1.27.1/server.go:722 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/grpc@v1.27.1/server.go:720 +0xef
Error: The terraform-provider-libvirt_v0.6.14 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
Hello I confirm it's works
example:
terraform {
required_version = ">= 0.14.0"
required_providers {
libvirt = {
source = "dmacvicar/libvirt"
version = "0.6.14"
}
}
}
provider "libvirt" {
uri = "qemu+ssh://USER@IP/system?keyfile=/tmp/id_rsa_sample&sshauth=privkey"
}
Works:
"Apply complete! Resources: 4 added, 0 changed, 0 destroyed."
best regards
Hi @dmacvicar
confirming similar issue, works with "virsh -c" and not with terraform --> but adding more details maybe it helps - would appreciate some help, if some workaround is found or so... <-- kinda tried everything (no other key works, like ed25519, swapped also provider versions - same issue )
terraform: 1.3.2 libvirt-provider: 0.6.10-14 and 0.7.0
Tested on: os: centos 7.9 (2009) and red hat 7.9 maipo
provider "libvirt" {
## Configuration options
uri = "qemu+ssh://root@10.*.*.5/system?keyfile=/root/.ssh/id_rsa"
# uri = "qemu+ssh://root@10.*.*.5/system?keyfile=/root/.ssh/id_rsa?sshauth=privkey"
}
terraform plan
2022-10-14T12:39:45.057+0100 [ERROR] vertex "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" error: failed to dial libvirt: failed to connect to libvirt on the remote host: ssh: rejected: administratively prohibited (open failed)
2022-10-14T12:39:45.057+0100 [INFO] backend/local: plan operation completed
╷
│ Error: failed to dial libvirt: failed to connect to libvirt on the remote host: ssh: rejected: administratively prohibited (open failed)
│
│ with provider["registry.terraform.io/dmacvicar/libvirt"],
│ on provider.tf line 20, in provider "libvirt":
│ 20: provider "libvirt" {
│
╵
Destination server(..*.5) --> key is accepted but does not go forward
Oct 14 12:34:05 hv2 sshd[90019]: Accepted publickey for root from 10.11.48.4 port 35702 ssh2: RSA SHA256:3Q7NpIRujvbsPARh7cp9BVERrw0ApI/7JE8QndprVBo
Oct 14 12:34:05 hv2 sshd[90019]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 14 12:34:05 hv2 sshd[90019]: refused streamlocal port forward: originator port 0, target /var/run/libvirt/libvirt-sock
Oct 14 12:34:05 hv2 sshd[90019]: pam_unix(sshd:session): session closed for user root
SSHD_CONFIG
UsePAM yes
#slagian-libvirt-sock
AllowAgentForwarding yes
AllowTcpForwarding no
PermitOpen any
PermitTunnel yes
#GatewayPorts no
X11Forwarding yes
...
I had the same issue on Centos 7.9.2009. The problem is due to openssh 7.4 on CentOS/RHEL not allowing root user ssh tunnel on a socket.
References: https://bugzilla.redhat.com/show_bug.cgi?id=1527565 https://bugs.centos.org/view.php?id=14291
I have resolved it by creating a new user and adding it to libvirt group:
usermod -a -G libvirt <username>
Hope this does the trick also for you!
I'm not sure if this is related, but I had a similar issue with SSH failing and it ended up being caused by incomplete support for the EtM (Encrypt-then-MAC) HMAC ciphers in the Go ssh libraries.
I came across golang/go#32075 where I found the following comment:
https://github.com/golang/go/commit/84bacda6ede319f5074d43b5d096b7ee7f3f5d77 added support for hmac-sha2-256-etm@openssh.com, however only implemented it for stream ciphers (arcfour*) and not for CBC block ciphers (aes128-cbc and 3des-cbc). This means that the SSH client will advertise support for EtM, the SSH server will select a CBC cipher with EtM, then the client fails to handle the resulting packets correctly
As a work-around, I disabled those HMAC ciphers by adding the following line to the /etc/ssh/sshd_config file on the libvirt host. This way the SSH server will never use EtM.
MACs hmac-sha1,hmac-sha1,hmac-sha1-96,hmac-sha1-96,hmac-sha2-256,hmac-sha2-256,hmac-sha2-512,hmac-sha2-512,hmac-md5,hmac-md5,hmac-md5-96,hmac-md5-96,umac-64@openssh.com,umac-64@openssh.com,umac-128@openssh.com,umac-128@openssh.com
I got the list by running ssh -Q mac
to show all supported ciphers and removed the ones using EtM. Some of these are less secure than others, so use your own discretion and make sure you understand the implications. I have no idea if all of them should be enabled on the server.
I had the same issue on Centos 7.9.2009. The problem is due to openssh 7.4 on CentOS/RHEL not allowing root user ssh tunnel on a socket.
References: https://bugzilla.redhat.com/show_bug.cgi?id=1527565 https://bugs.centos.org/view.php?id=14291
I have resolved it by creating a new user and adding it to libvirt group:
usermod -a -G libvirt <username>
Hope this does the trick also for you!
It's work on 0.7.1 version, thanks. I created other user and added him on libvirt group.
Finally link is: uri = "qemu+ssh://terraform@
Works on Mac and Ubuntu 22.04. No need to specify the key since id_rsa is the default. terraform { required_providers { libvirt = { source = "dmacvicar/libvirt" version = "0.7.1" } } } provider "libvirt" { uri = "qemu+ssh://robin@192.168.1.95/system?sshauth=privkey&no_verify=1" }
System Information
Linux distribution
Terraform is running on MacOS 12.3 Libvirt is running on Red Hat Enterprise Linux Server release 7.9 (Maipo)
Terraform version
Description of Issue/Question
Setup
This is the
.tf
i use:Steps to Reproduce Issue
If I specify
version = "=0.6.13"
in above Terraform file, it works:If I change it to
version = "=0.6.14"
it fails: