Failed to start SASL negotiation -4. No mechanism available: No worthy mechs found. Towards oVirt setup

larssb commented 3 years ago

System Information

Linux distribution

CentOS 8

Terraform version

v0.13.5

Provider and libvirt versions

v0.6.3

Others

oVirt v4.4.2
- VDSM v4.20

Checklist

[ ] Is your issue/contribution related with enabling some setting/option exposed by libvirt that the plugin does not yet support, or requires changing/extending the provider terraform schema? --> It might be something that it does not support. I don't think so. But I'm not entirely sure. Bare with me.
- [X] Make sure you explain why this option is important to you, why it should be important to everyone. Describe your use-case with detail and provide examples where possible.
- [ ] If it is a very special case, consider using the XSLT support in the provider to tweak the definition instead of opening an issue --> How do I do this?
- [X] Maintainers do not have expertise in every libvirt setting, so please, describe the feature and how it is used. Link to the appropriate documentation
[X] Is it a bug or something that does not work as expected? Please make sure you fill the version information below: --> maybe a bit of both

Description of Issue/Question

Setup

  required_providers {                                                                                                                                                                    
    libvirt = {                                                                                                                                                                           
      source  = "local/providers/libvirt"                                                                                                                                                 
      version = "~> 0.6.3"                                                                                                                                                                
    }                                                                                                                                                                                     
  }                                                                                                                                                                                       
  required_version = ">= 0.13"                                                                                                                                                            
}                                                                                                                                                                                         

provider "libvirt" {                                                                                                                                                                      
    uri = "qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf"                                                                                                                                                                                                                                                      
}                                                                                                                                                                                         

resource "libvirt_domain" "terraform_test" {                                                                                                                                              
  name = "terraform_test"                                                                                                                                                                 
}

Steps to Reproduce Issue

Install CentOS 8
Install oVirt v4.4.2
setup Terraform & this provider
Use the .tf file shown under ###Setup
execute terraform init + terraform plan ... when executing terraform plan the error should be thrown.

Additional information:

It's a std. CentOS 8 installation. I haven't hardened it more or anything of that sort.

I get the error in the subject as long as auth_unix_rw in /etc/libvirt/libvirtd.conf is set to sasl. As soon as I comment that out or set it to none and do a systemctl restart libvirtd on the CentOS host it works.

This is the exact error:

 [user@node-1 ~]# terraform plan                                                                                                                                                    
Refreshing Terraform state in-memory prior to plan...                                                                                                                                     
The refreshed state will be used to calculate this plan, but will not be                                                                                                                  
persisted to local or remote state storage.                                                                                                                                               

------------------------------------------------------------------------                                                                                                                  

Error: virError(Code=45, Domain=7, Message='authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)')                       

  on libvirt.tf line 11, in provider "libvirt":                                                                                                                                           
  11: provider "libvirt" {

The reason I’m using this provider and not the oVirt one is because the oVirt provider does not support ISO’s. See this GitHub issue. And as I’m using K3OS as the OS on the VM’s I’m going to configure on the oVirt platform, in order to setup a K3S cluster. I need ISO support because K3OS will get its cloud-init data from an ISO.

Therefore I’m testing to see whether this provider can be used towards an oVirt setup. Consisting of:

3 hosts
And a self-hosted engine

More on that setup of oVirt here.

As I’ve already stated at the top of this issue I found out how-to work around this issue. However, I’m wondering, there have to be a reason for the oVirt team to have set the auth_unix_rw setting to =sasl. I’m thinking along the lines of:

What are the security implications of setting auth_unix_rw setting to =none or simply commenting it out?
Does this provider not support that setting configured to sasl
1. Or generally not support oVirt at all (oVirt is based on libvirt, so I think should be a chance that it does however)

I can also conclude that with auth_unix_rw set to =none I can execute virsh -c qemu:///system list without authentication. Whereas, on a default oVirt setup would normally have to execute virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list to authenticate.

I’ve read:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-troubleshooting-common_libvirt_errors_and_troubleshooting
https://bobcares.com/blog/virsh-failed-to-connect-to-the-hypervisor/
- But I REALLY DO NOT want to use DIGEST-MD5
https://wiki.libvirt.org/page/Tips (under #The connection fails using md5 digest auth the specific err is mentioned)
https://libvirt.org/uri.html

Any help I can get is highly regarded.

Thank you

larssb commented 3 years ago

I've been reading up a bit on this and maybe this have something to do with whether the authfile query parameter in the connect URI is supported or not. I've read:

https://libvirt.org/uri.html#URI_qemu
https://libvirt.org/html/libvirt-libvirt-host.html#virConnectOpen & the *OpenAuth method as well
https://libvirt.org/auth.html

Thank you

MalloZup commented 3 years ago

hi @larssb thx for your detailed issue. I think this issue would appropriate more on gitlab of libvirt https://gitlab.com/libvirt/libvirt

To me it seems it has to do with the libvirt OS setup rather then the terraform provider.

I don't have the capacity right now to have look at this issue, nevertheless I will keep this open to help the community on this special config.

If I might have some free time I will look at , in order to help you on the investigation thx!

larssb commented 3 years ago

Hi @MalloZup,

Thank you for your reply. Fair enough. It might very well be the OS setup in some way or the other or because of oVirt "closing of access to libvirt" in some way or the other.

larssb commented 3 years ago

Hmmm I tried changing the uri from uri = "qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf" to uri = "qemu://localhost/system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf" - a thought that came to after my readings.

And now I got this error:

Error: virError(Code=45, Domain=7, Message='authentication failed: Failed to verify peer's certificate')

Alrighty then! That lead me to thinking that the credentials defined in the /etc/ovirt-hosted-engine/virsh_auth.conf auth file is now being sent to libvirt as they should. Likely was before as well, however using the wrong URI.

That led me to control the certs on the machine. As this is in the hosts /etc/libvirt/libvirtd.conf file.

auth_unix_rw="sasl"
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
host_uuid="AN_UUID"
keepalive_interval=-1
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"

That lead me to read this one and then to read the PEM file: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -text >> which showed me that localhost of course is not the FQDN that the cert was created for. So I updated the URI again, to match the hostname mentioned in the cert.....and now, wait for it 💣 terraform plan worked with no issues.

So it was a case of me simply needing to learn quite some stuff on libvirt, qemu and what not.

I hope this can help someone else out there.

I'll be closing this one now 🎉

dmacvicar / terraform-provider-libvirt