search

dmacvicar / terraform-provider-libvirt

Terraform provider to provision infrastructure with Linux's KVM using libvirt
Apache License 2.0
1.61k stars 460 forks source link

Support RSA SHA-2 (RFC8332) signatures for SSH authentication #916

Closed davidalger closed 2 years ago

davidalger commented 2 years ago

System Information

Linux distribution

This affects systems were the remote server is running Fedora 33, 34, 35, CentOS Stream 9 and RHEL 8 with FIPS.

Terraform and provider versions

Terraform v1.1.2
on darwin_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.6.12

Description of Issue/Question

Attempting to connect to any of the affected OpenSSH configurations — in my case, default Fedora 34 ssh config — RSA based ssh authentication will fail with the error failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain and the servers sshd.log will report the following:

Dec 28 11:37:12 hylfing.westmore sshd[843630]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Dec 28 11:37:12 hylfing.westmore sshd[843630]: Connection closed by authenticating user davidalger 172.16.0.138 port 54755 [preauth]

Fedora 33 updated the system-wide crypto policy to disallow SHA-1 hashes in signatures and OpenSSH 8.8 (released on 2021-09-26) disables the ssh-rsa signature scheme by default as well (http://www.openssh.com/txt/release-8.7) so the change will eventually trickle into other Linux families as well if it hasn't already. In particular, "ssh-rsa" keys are capable of signing using "rsa-sha2-256" (RSA/SHA256), "rsa-sha2-512" (RSA/SHA512) and "ssh-rsa" (RSA/SHA1). Only the last of these is being turned off by default.

This explains why it was reported that ed25519 keys worked with ssh-agent support added in 0.6.11 while RSA keys continued to fail (https://github.com/dmacvicar/terraform-provider-libvirt/issues/864#issuecomment-919094286) as the commenter indicated (in previous comment) he was testing with a Fedora 34 server, one which would have ssh-rsa disabled by default.

Setup

terraform {
  required_version = "~> 1.1"

  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "~> 0.6"
    }
  }
}

provider "libvirt" {
  uri = "qemu+ssh://davidalger@hylfing/system?sshauth=agent&socket=/var/run/libvirt/libvirt-sock"
}

resource "libvirt_volume" "fcos35" {
  name   = "fcos35"
  source = "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20211203.3.0/x86_64/fedora-coreos-35.20211203.3.0-qemu.x86_64.qcow2.xz"
}

Steps to Reproduce Issue

Terraform debug output:

2021-12-29T10:38:30.543-0600 [INFO]  Terraform version: 1.1.2
2021-12-29T10:38:30.543-0600 [INFO]  Go runtime version: go1.17.2
2021-12-29T10:38:30.543-0600 [INFO]  CLI args: []string{"/usr/local/Cellar/tfenv/2.2.2/versions/1.1.2/terraform", "plan"}
2021-12-29T10:38:30.543-0600 [DEBUG] Attempting to open CLI config file: /Users/davidalger/.terraformrc
2021-12-29T10:38:30.544-0600 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021-12-29T10:38:30.544-0600 [DEBUG] checking for credentials in "/Users/davidalger/.terraform.d/plugins"
2021-12-29T10:38:30.544-0600 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-12-29T10:38:30.544-0600 [DEBUG] will search for provider plugins in /Users/davidalger/.terraform.d/plugins
2021-12-29T10:38:30.544-0600 [DEBUG] ignoring non-existing provider search directory /Users/davidalger/Library/Application Support/io.terraform/plugins
2021-12-29T10:38:30.544-0600 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021-12-29T10:38:30.544-0600 [INFO]  CLI command args: []string{"plan"}
2021-12-29T10:38:30.545-0600 [DEBUG] New state was assigned lineage "81062f15-d37c-b4e5-9bf2-8aec4bfcc378"
2021-12-29T10:38:30.607-0600 [DEBUG] checking for provisioner in "."
2021-12-29T10:38:30.607-0600 [DEBUG] checking for provisioner in "/usr/local/Cellar/tfenv/2.2.2/versions/1.1.2"
2021-12-29T10:38:30.607-0600 [DEBUG] checking for provisioner in "/Users/davidalger/.terraform.d/plugins"
2021-12-29T10:38:30.609-0600 [INFO]  backend/local: starting Plan operation
2021-12-29T10:38:30.611-0600 [DEBUG] created provider logger: level=debug
2021-12-29T10:38:30.611-0600 [INFO]  provider: configuring client automatic mTLS
2021-12-29T10:38:30.619-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12]
2021-12-29T10:38:30.621-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56983
2021-12-29T10:38:30.622-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12
2021-12-29T10:38:30.636-0600 [INFO]  provider.terraform-provider-libvirt_v0.6.12: configuring server automatic mTLS: timestamp=2021-12-29T10:38:30.636-0600
2021-12-29T10:38:30.676-0600 [DEBUG] provider: using plugin: version=5
2021-12-29T10:38:30.676-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.12: plugin address: address=/var/folders/b5/88xzlq3n61jdf_f30wlkkg580000gn/T/plugin073562578 network=unix timestamp=2021-12-29T10:38:30.676-0600
2021-12-29T10:38:30.706-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2021-12-29T10:38:30.706-0600 [DEBUG] No provider meta schema returned
2021-12-29T10:38:30.709-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56983
2021-12-29T10:38:30.709-0600 [DEBUG] provider: plugin exited
2021-12-29T10:38:30.709-0600 [DEBUG] Building and walking validate graph
2021-12-29T10:38:30.709-0600 [DEBUG] ProviderTransformer: "libvirt_volume.fcos35" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2021-12-29T10:38:30.709-0600 [DEBUG] ReferenceTransformer: "libvirt_volume.fcos35" references: []
2021-12-29T10:38:30.709-0600 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" references: []
2021-12-29T10:38:30.709-0600 [DEBUG] Starting graph walk: walkValidate
2021-12-29T10:38:30.710-0600 [DEBUG] created provider logger: level=debug
2021-12-29T10:38:30.710-0600 [INFO]  provider: configuring client automatic mTLS
2021-12-29T10:38:30.719-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12]
2021-12-29T10:38:30.722-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56984
2021-12-29T10:38:30.723-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12
2021-12-29T10:38:30.736-0600 [INFO]  provider.terraform-provider-libvirt_v0.6.12: configuring server automatic mTLS: timestamp=2021-12-29T10:38:30.736-0600
2021-12-29T10:38:30.772-0600 [DEBUG] provider: using plugin: version=5
2021-12-29T10:38:30.772-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.12: plugin address: network=unix address=/var/folders/b5/88xzlq3n61jdf_f30wlkkg580000gn/T/plugin216229823 timestamp=2021-12-29T10:38:30.772-0600
2021-12-29T10:38:30.802-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2021-12-29T10:38:30.802-0600 [DEBUG] No provider meta schema returned
2021-12-29T10:38:30.806-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56984
2021-12-29T10:38:30.806-0600 [DEBUG] provider: plugin exited
2021-12-29T10:38:30.806-0600 [INFO]  backend/local: plan calling Plan
2021-12-29T10:38:30.807-0600 [DEBUG] Building and walking plan graph for NormalMode
2021-12-29T10:38:30.807-0600 [DEBUG] ProviderTransformer: "libvirt_volume.fcos35 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2021-12-29T10:38:30.808-0600 [DEBUG] ReferenceTransformer: "libvirt_volume.fcos35 (expand)" references: []
2021-12-29T10:38:30.808-0600 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" references: []
2021-12-29T10:38:30.808-0600 [DEBUG] Starting graph walk: walkPlan
2021-12-29T10:38:30.809-0600 [DEBUG] created provider logger: level=debug
2021-12-29T10:38:30.809-0600 [INFO]  provider: configuring client automatic mTLS
2021-12-29T10:38:30.818-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12]
2021-12-29T10:38:30.820-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56985
2021-12-29T10:38:30.820-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12
2021-12-29T10:38:30.834-0600 [INFO]  provider.terraform-provider-libvirt_v0.6.12: configuring server automatic mTLS: timestamp=2021-12-29T10:38:30.834-0600
2021-12-29T10:38:30.870-0600 [DEBUG] provider: using plugin: version=5
2021-12-29T10:38:30.870-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.12: plugin address: address=/var/folders/b5/88xzlq3n61jdf_f30wlkkg580000gn/T/plugin702862148 network=unix timestamp=2021-12-29T10:38:30.870-0600
2021-12-29T10:38:30.894-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2021-12-29T10:38:30.895-0600 [DEBUG] No provider meta schema returned
2021-12-29T10:38:30.896-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.12: 2021/12/29 10:38:30 [DEBUG] Configuring provider for 'qemu+ssh://davidalger@hylfing/system?sshauth=agent&socket=/var/run/libvirt/libvirt-sock': &{map[uri:0xc0000b63c0] <nil> <nil> 0xc0005a6ba0 map[] <nil> 0xc0005a6c20 0xc00019bef0 0xc000150280 false map[] {1 {0 0}} false false}
2021-12-29T10:38:31.162-0600 [ERROR] vertex "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-12-29T10:38:31.163-0600 [INFO]  backend/local: plan operation completed
╷
│ Error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
│ 
│   with provider["registry.terraform.io/dmacvicar/libvirt"],
│   on main.tf line 12, in provider "libvirt":
│   12: provider "libvirt" {
│ 
╵
2021-12-29T10:38:31.166-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.12/darwin_amd64/terraform-provider-libvirt_v0.6.12 pid=56985
2021-12-29T10:38:31.166-0600 [DEBUG] provider: plugin exited

sshd.log from the server showing why the handshake failed

Dec 28 11:37:12 hylfing.westmore sshd[843630]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Dec 28 11:37:12 hylfing.westmore sshd[843630]: Connection closed by authenticating user davidalger 172.16.0.138 port 54755 [preauth]

Additional information:

root@hylfing:~# ssh -Q PubkeyAcceptedAlgorithms | grep rsa
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-rsa-cert-v01@openssh.com
rsa-sha2-256-cert-v01@openssh.com
rsa-sha2-512-cert-v01@openssh.com
root@hylfing:~# sshd -T | grep -i PubkeyAcceptedAlgorithms
pubkeyacceptedalgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com

This may be the root cause for some of the issues noted in #886

Possible solution:

Crypto packages should now support RSA SHA-2 (RFC8332) signatures, so upgrading may be all that is needed to support them. Please see golang/go#37278.

For now I'm going to workaround the issue by adding an ed25519 key to my agent, allowing the plan to succeed:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # libvirt_volume.fcos35 will be created
  + resource "libvirt_volume" "fcos35" {
      + format = (known after apply)
      + id     = (known after apply)
      + name   = "fcos35"
      + pool   = "default"
      + size   = (known after apply)
      + source = "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20211203.3.0/x86_64/fedora-coreos-35.20211203.3.0-qemu.x86_64.qcow2.xz"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
harrisonbc commented 2 years ago

I can confirm that the use of an ed25519 key type resolves the connection issue for me.

I just created a new keypair with ssh-keygen -t ed25519 (no passphrase) and copied it with ssh-copy-id user@sever

You then need to specify the new key in the URI

uri = "qemu+ssh://user@server/system?keyfile=/home/localuser/.ssh/id_ed25519"

tdcook commented 2 years ago

Same problem with a Fedora 35 client to a CentOS Stream 9 server. Using an ed25519 key also fixed the issue for me.

dmacvicar commented 2 years ago

@davidalger thanks for the detailed report. I will look into it.

dmacvicar commented 2 years ago

I have pushed a v0.6.13 with upgraded golang.org/x/crypto

Let me know if upgrading helps. Otherwise I will look deeper in the ssh client settings.

harrisonbc commented 2 years ago

It hasn't changed the behaviour for me on zorin os 16 & Terraform v1.1.4 on linux_amd64

This URI Works uri = "qemu+ssh://root@dl380/system?keyfile=/home/brynn/.ssh/id_ed25519"

This one doesn't uri = "qemu+ssh://root@dl380/system?keyfile=/home/brynn/.ssh/id_rsa"

╷ │ Error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain │ │ with provider["registry.terraform.io/dmacvicar/libvirt"], │ on main.tf line 12, in provider "libvirt": │ 12: provider "libvirt" { │

tdcook commented 2 years ago

I'm also still seeing the error with v0.6.13.

davidalger commented 2 years ago

@dmacvicar Thanks for updating that lib. Unfortunately doesn't seem to resolve it, still get the same err:

│ Error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

with the following showing up in the sshd log on the server side:

Jan 23 21:32:57 hylfing.westmore sshd[3984108]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]

The relevant changes should be included based on the new version pinned in the go.mod file. This comment on the issue I linked previously seems to indicate the merged upstream fix didn't cover all the bases, and I found a tracking ticket which shows there are a few outstanding items it seems: https://github.com/golang/go/issues/49952 So there may or may not be anything you can do here to resolve in the near term. For now, I'll just keep using the ed25519 key. Thanks for all the efforts you put into this.

dmacvicar commented 2 years ago

I was able to reproduce locally and after hours trying to wrap the Signer APIs, I figured out the missing patches that haven't been merged yet.

If you know how to build locally, you can build the ssh-rsa-fix branch of #925 and use a ~/.terraformrc file to override the plugin with your local version (note that this refers to a path in my local filesystem, so adapt to yours):

provider_installation {

  # Use /home/developer/tmp/terraform-null as an overridden package directory
  # for the hashicorp/null provider. This disables the version and checksum
  # verifications for this provider and forces Terraform to look for the
  # null provider plugin in the given directory.
  dev_overrides {
    "dmacvicar/libvirt" = "/space/git/terraform-provider-libvirt"
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

If I can get some feedback, I'd release it as soon as possible.

tdcook commented 2 years ago

I've built the ssh-rsa-fix branch and used it locally following your instructions, and I can confirm the error no longer appears for me when using that branch.

dmacvicar commented 2 years ago

Thanks @tdcook. I will wait for a bit of more feedback and then do a release.

davidalger commented 2 years ago

@dmacvicar Built and tested, issue solved here as well with the patched Crypto lib. It successfully authenticated via SSH using my encrypted RSA key loaded into ssh-agent. Hopefully those patches will be merged upstream soon so you don't have to maintain a patched fork for this to work. 🤞🏻

dmacvicar commented 2 years ago

Thanks everyone for the input:

https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.6.14 should be available in the registry soon.

harrisonbc commented 2 years ago

Thanks Duncan,

I can confirm that this fixes my issue - thanks again.

Brynn

On Mon, 31 Jan 2022 at 22:43, Duncan Mac-Vicar P. @.***> wrote:

Thanks everyone for the input:

https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.6.14 should be available in the registry soon.

— Reply to this email directly, view it on GitHub https://github.com/dmacvicar/terraform-provider-libvirt/issues/916#issuecomment-1026285817, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRYNIRRU2UFVFNVMC7WA7LUY4GCLANCNFSM5K6JC63A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

tomp736 commented 2 years ago

Hi there,

I stumbled upon this when attempting to remote instance with SSH - not sure if related. Using 0.6.14 provider.

The following was failing for me.

provider "libvirt" {
  uri = "qemu+ssh://user@host/system?keyfile=/home/user/.ssh/id_ed25519"
}
│ Error: failed to dial libvirt: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
│ 
│   with provider["registry.terraform.io/dmacvicar/libvirt"],
│   on provider.tf line 8, in provider "libvirt":
│    8: provider "libvirt" {

I needed to add sshauth=privkey in uri and then it was able to connect.

provider "libvirt" {
  uri = "qemu+ssh://user@host/system?keyfile=/home/user/.ssh/id_ed25519&sshauth=privkey"
}
harrisonbc commented 2 years ago

I had to do that too, however I didn't see it as an error - just a required configuration step.

renich commented 2 years ago

I dunno if i'm doing anything wrong. I'm using v0.6.14. Tried all permutations of the URI (with and without sshauth and keyfile). Nothing worked.

I can, definitely, login with my passowrd protected key. Terraform doesn't work, though.

I generated the key with: ssh-keygen -t ed25519 -a 1000.

The key is in the keyring. Fedora 35 here. 

[renich@introdesk centos8]$ ssh-add -l
4096 SHA256:Db3ywQ4RD59bEyD8iMKjlQhSoj+vij00jq0vRlHfFoU /home/renich/.config/gsconnect/private.pem (RSA)
256 SHA256:3DFsKOg4QeEI1qVne55invnOIFiNFd1IjklHa6FdM20 renich@introdesk (ED25519)
8192 SHA256:THti4XQK2aRCcVTt3D65zNWnOYdtg1AQJs8D5j9Tu9A /home/renich/.ssh/id_rsa (RSA)
2048 SHA256:SjrK/fmvShaVldFo53z7ftqCS0NBvvCLTgzbjKYAyek renich@google (RSA)
256 SHA256:dxBuwLNUVEt1dovbFRyUrVjGYf+jH5yAyYl6NK5txqA renich@fedora (ED25519)
8192 SHA256:qvasD/Z6z4oW5PJnVyqJjrOWLUfIQdbgD4whzVLQjyI renich@fedora (RSA)

[renich@introdesk centos8]$ TF_LOG=debug terraform plan
2022-02-09T03:36:42.973-0600 [INFO]  Terraform version: 1.1.0
2022-02-09T03:36:42.973-0600 [INFO]  Go runtime version: go1.17.2
2022-02-09T03:36:42.973-0600 [INFO]  CLI args: []string{"terraform", "plan"}
2022-02-09T03:36:42.973-0600 [DEBUG] Attempting to open CLI config file: /home/renich/.terraformrc
2022-02-09T03:36:42.973-0600 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2022-02-09T03:36:42.973-0600 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2022-02-09T03:36:42.973-0600 [DEBUG] ignoring non-existing provider search directory /home/renich/.terraform.d/plugins
2022-02-09T03:36:42.973-0600 [DEBUG] ignoring non-existing provider search directory /home/renich/.local/share/terraform/plugins
2022-02-09T03:36:42.973-0600 [DEBUG] ignoring non-existing provider search directory /home/renich/.local/share/flatpak/exports/share/terraform/plugins
2022-02-09T03:36:42.973-0600 [DEBUG] ignoring non-existing provider search directory /var/lib/flatpak/exports/share/terraform/plugins
2022-02-09T03:36:42.973-0600 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2022-02-09T03:36:42.973-0600 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2022-02-09T03:36:42.974-0600 [INFO]  CLI command args: []string{"plan"}
2022-02-09T03:36:42.975-0600 [DEBUG] New state was assigned lineage "c89da1a9-c680-186b-821e-4afa5200c3e6"
2022-02-09T03:36:43.042-0600 [DEBUG] checking for provisioner in "."
2022-02-09T03:36:43.042-0600 [DEBUG] checking for provisioner in "/home/renich/bin"
2022-02-09T03:36:43.042-0600 [INFO]  backend/local: starting Plan operation
2022-02-09T03:36:43.043-0600 [DEBUG] created provider logger: level=debug
2022-02-09T03:36:43.043-0600 [INFO]  provider: configuring client automatic mTLS
2022-02-09T03:36:43.054-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14]
2022-02-09T03:36:43.055-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 pid=19586
2022-02-09T03:36:43.055-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14
2022-02-09T03:36:43.064-0600 [INFO]  provider.terraform-provider-libvirt_v0.6.14: configuring server automatic mTLS: timestamp=2022-02-09T03:36:43.064-0600
2022-02-09T03:36:43.080-0600 [DEBUG] provider: using plugin: version=5
2022-02-09T03:36:43.080-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.14: plugin address: address=/tmp/plugin2968674901 network=unix timestamp=2022-02-09T03:36:43.079-0600
2022-02-09T03:36:43.094-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-02-09T03:36:43.095-0600 [DEBUG] No provider meta schema returned
2022-02-09T03:36:43.098-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 pid=19586
2022-02-09T03:36:43.098-0600 [DEBUG] provider: plugin exited
2022-02-09T03:36:43.098-0600 [DEBUG] Building and walking validate graph
2022-02-09T03:36:43.098-0600 [DEBUG] ProviderTransformer: "libvirt_domain.centos8" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2022-02-09T03:36:43.098-0600 [DEBUG] ProviderTransformer: "libvirt_cloudinit_disk.centos-cloudinit" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2022-02-09T03:36:43.098-0600 [DEBUG] ProviderTransformer: "libvirt_volume.centos8-base" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2022-02-09T03:36:43.098-0600 [DEBUG] ProviderTransformer: "libvirt_volume.centos-system" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2022-02-09T03:36:43.099-0600 [DEBUG] ReferenceTransformer: "libvirt_volume.centos-system" references: [libvirt_volume.centos8-base]
2022-02-09T03:36:43.099-0600 [INFO]  ReferenceTransformer: reference not found: "self"
2022-02-09T03:36:43.099-0600 [INFO]  ReferenceTransformer: reference not found: "self"
2022-02-09T03:36:43.099-0600 [INFO]  ReferenceTransformer: reference not found: "self"
2022-02-09T03:36:43.099-0600 [DEBUG] ReferenceTransformer: "libvirt_domain.centos8" references: [libvirt_cloudinit_disk.centos-cloudinit libvirt_volume.centos-system]
2022-02-09T03:36:43.099-0600 [DEBUG] ReferenceTransformer: "var.root_password" references: []
2022-02-09T03:36:43.099-0600 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" references: []
2022-02-09T03:36:43.099-0600 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.centos-cloudinit" references: [var.root_password]
2022-02-09T03:36:43.099-0600 [DEBUG] ReferenceTransformer: "libvirt_volume.centos8-base" references: []
2022-02-09T03:36:43.100-0600 [DEBUG] Starting graph walk: walkValidate
2022-02-09T03:36:43.100-0600 [DEBUG] created provider logger: level=debug
2022-02-09T03:36:43.100-0600 [INFO]  provider: configuring client automatic mTLS
2022-02-09T03:36:43.113-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14]
2022-02-09T03:36:43.113-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 pid=19600
2022-02-09T03:36:43.113-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14
2022-02-09T03:36:43.119-0600 [INFO]  provider.terraform-provider-libvirt_v0.6.14: configuring server automatic mTLS: timestamp=2022-02-09T03:36:43.119-0600
2022-02-09T03:36:43.132-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.14: plugin address: address=/tmp/plugin669123645 network=unix timestamp=2022-02-09T03:36:43.131-0600
2022-02-09T03:36:43.132-0600 [DEBUG] provider: using plugin: version=5
2022-02-09T03:36:43.149-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-02-09T03:36:43.150-0600 [DEBUG] No provider meta schema returned
2022-02-09T03:36:43.160-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 pid=19600
2022-02-09T03:36:43.160-0600 [DEBUG] provider: plugin exited
2022-02-09T03:36:43.160-0600 [INFO]  backend/local: plan calling Plan
2022-02-09T03:36:43.160-0600 [DEBUG] Building and walking plan graph for NormalMode
2022-02-09T03:36:43.160-0600 [DEBUG] ProviderTransformer: "libvirt_volume.centos8-base (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2022-02-09T03:36:43.160-0600 [DEBUG] ProviderTransformer: "libvirt_volume.centos-system (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2022-02-09T03:36:43.160-0600 [DEBUG] ProviderTransformer: "libvirt_domain.centos8 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2022-02-09T03:36:43.160-0600 [DEBUG] ProviderTransformer: "libvirt_cloudinit_disk.centos-cloudinit (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2022-02-09T03:36:43.161-0600 [DEBUG] ReferenceTransformer: "libvirt_volume.centos8-base (expand)" references: []
2022-02-09T03:36:43.161-0600 [DEBUG] ReferenceTransformer: "libvirt_volume.centos-system (expand)" references: [libvirt_volume.centos8-base (expand)]
2022-02-09T03:36:43.161-0600 [INFO]  ReferenceTransformer: reference not found: "self"
2022-02-09T03:36:43.161-0600 [INFO]  ReferenceTransformer: reference not found: "self"
2022-02-09T03:36:43.161-0600 [INFO]  ReferenceTransformer: reference not found: "self"
2022-02-09T03:36:43.161-0600 [DEBUG] ReferenceTransformer: "libvirt_domain.centos8 (expand)" references: [libvirt_cloudinit_disk.centos-cloudinit (expand) libvirt_volume.centos-system (expand)]
2022-02-09T03:36:43.161-0600 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.centos-cloudinit (expand)" references: [var.root_password]
2022-02-09T03:36:43.161-0600 [DEBUG] ReferenceTransformer: "var.root_password" references: []
2022-02-09T03:36:43.161-0600 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" references: []
2022-02-09T03:36:43.161-0600 [DEBUG] Starting graph walk: walkPlan
2022-02-09T03:36:43.161-0600 [DEBUG] created provider logger: level=debug
2022-02-09T03:36:43.161-0600 [INFO]  provider: configuring client automatic mTLS
2022-02-09T03:36:43.173-0600 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14]
2022-02-09T03:36:43.173-0600 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 pid=19612
2022-02-09T03:36:43.173-0600 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14
2022-02-09T03:36:43.185-0600 [INFO]  provider.terraform-provider-libvirt_v0.6.14: configuring server automatic mTLS: timestamp=2022-02-09T03:36:43.184-0600
2022-02-09T03:36:43.199-0600 [DEBUG] provider: using plugin: version=5
2022-02-09T03:36:43.199-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.14: plugin address: network=unix address=/tmp/plugin4282762731 timestamp=2022-02-09T03:36:43.199-0600
2022-02-09T03:36:43.218-0600 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2022-02-09T03:36:43.219-0600 [DEBUG] No provider meta schema returned
2022-02-09T03:36:43.220-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/02/09 03:36:43 [DEBUG] Configuring provider for 'qemu+ssh://root@myhost.mydomain.tld/system?keyfile=/home/renich/.ssh/id_ed25519&sshauth=privkey': &{map[uri:0xc00013e3c0] <nil> <nil> 0xc0006b2d20 map[] <nil> 0xc0006b2da0 0xc000508f30 0xc000252320 false map[] {1 {0 0}} false false}
2022-02-09T03:36:43.220-0600 [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/02/09 03:36:43 [ERROR] Failed to parse ssh key: ssh: this private key is passphrase protected
2022-02-09T03:36:43.441-0600 [ERROR] vertex "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" error: failed to dial libvirt: ssh: handshake failed: knownhosts: key mismatch
2022-02-09T03:36:43.441-0600 [INFO]  backend/local: plan operation completed
╷
│ Error: failed to dial libvirt: ssh: handshake failed: knownhosts: key mismatch
│
│   with provider["registry.terraform.io/dmacvicar/libvirt"],
│   on main.tf line 10, in provider "libvirt":
│   10: provider "libvirt" {
│
╵
2022-02-09T03:36:43.445-0600 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.6.14/linux_amd64/terraform-provider-libvirt_v0.6.14 pid=19612
2022-02-09T03:36:43.445-0600 [DEBUG] provider: plugin exited
dmacvicar commented 2 years ago

@renich can you login with your password protected key without entering a password?

the provider would not work with a password protected key unless you are using the agent.

renich commented 2 years ago

@renich can you login with your password protected key without entering a password?

yes; of course. I've been using this key for years.

the provider would not work with a password protected key unless you are using the agent.

Well, I am using the agent. Using Fedora 35 with GNOME. The agent is configured by default. This is why I listed my keys beforehand.

Let me know if there are any commands you want me to try in order to prove it or help debug this further.

owenthereal commented 2 years ago

Just ran into this issue. I'm wondering whether https://github.com/golang/crypto/compare/master...dmacvicar:master could be contributed upstream so that other projects benefit from it.

MonolithProjects commented 2 years ago

Same here (Fedora 35). As a workaround i ended up by disabling known hosts verification by using no_verify=1

qemu+ssh://<server user>@<server address>/system?keyfile=<path to my key>&no_verify=1
renich commented 2 years ago

Same here (Fedora 35). As a workaround i ended up by disabling known hosts verification by using no_verify=1

qemu+ssh://<server user>@<server address>/system?keyfile=<path to my key>&no_verify=1

This totally works but it's odd. I can ssh to that host without issues. It's present in ~/.ssh/known_hosts.

alpana17 commented 1 year ago

I am using terraform version: 0.12.14 I have created a VM in Azure with RHEL-8 fips enabled Image. Fips does not support rsa-sha keys. So, I have created a key with rsa-sha2-256. With same key, I can manually login to VM. But terraform fails to login. error is: error: SSH authentication failed (rhel@4.193.140.71:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Any guidance is appreciated.