Permission denied at VM create on remote KVM with non-root user

mocart2 commented 11 months ago

System Information

Linux distribution

Ubuntu 22.04 amd64

Terraform version

Terraform v1.2.9
on linux_amd64

Provider and libvirt versions

.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.7.1/linux_amd64/terraform-provider-libvirt_v0.7.1

Checklist

[ ] Is your issue/contribution related with enabling some setting/option exposed by libvirt that the plugin does not yet support, or requires changing/extending the provider terraform schema?
- [ ] Make sure you explain why this option is important to you, why it should be important to everyone. Describe your use-case with detail and provide examples where possible.
- [ ] If it is a very special case, consider using the XSLT support in the provider to tweak the definition instead of opening an issue
- [ ] Maintainers do not have expertise in every libvirt setting, so please, describe the feature and how it is used. Link to the appropriate documentation
[x] Is it a bug or something that does not work as expected? Please make sure you fill the version information below:

Description of Issue/Question

Setup

I use alias to connect remote KVM server and use non-root ubuntu(with sudo no passwd all) user:

provider "libvirt" {
  uri = "qemu:///system"
}

provider "libvirt" {
    alias = "core"
    uri   = "qemu+ssh://ubuntu@10.81.156.101/system?keyfile=/root/.ssh/ubuntu&known_hosts_verify=ignore"
}

Steps to Reproduce Issue

connect to remote KVM is good, but when TF creates libvirt domain i have error: error on localhost:

Error: error creating libvirt domain: Cannot access storage file '/data/vms/admintestvm/admintestvm-root' (as uid:64055, gid:108): Permission denied
│
│   with module.vm.libvirt_domain.vm[0],
│   on .terraform/modules/vm/main.tf line 47, in resource "libvirt_domain" "vm":
│   47: resource "libvirt_domain" "vm" {

tf debug on localhost:

2023-07-20T22:15:36.117Z [ERROR] provider.terraform-provider-libvirt_v0.7.1: Response contains error diagnostic: tf_provider_addr=provider tf_resource_type=libvirt_domain tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/diag/diagnostics.go:55 diagnostic_detail= diagnostic_severity=ERROR diagnostic_summary="error creating libvirt domain: Cannot access storage file '/data/vms/admintestvm/admintestvm-root' (as uid:64055, gid:108): Permission denied" tf_proto_version=5.3 tf_req_id=266459ac-d72e-a0ec-1fa9-de27d372376f @module=sdk.proto timestamp=2023-07-20T22:15:36.117Z
2023-07-20T22:15:36.118Z [ERROR] vertex "module.vm.libvirt_domain.vm[0]" error: error creating libvirt domain: Cannot access storage file '/data/vms/admintestvm/admintestvm-root' (as uid:64055, gid:108): Permission denied

2023-07-21T06:08:48.198Z [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/07/21 06:08:48 [DEBUG] cleaning up connection for URI: qemu+ssh://ubuntu@10.81.156.101/system?keyfile=/root/.ssh/ubuntu&known_hosts_verify=ignore: timestamp=2023-07-21T06:08:48.198Z

after i get error, i checked dir based storage pool permissions on remote KVM:

root@core:/data/vms# ls -lth
total 4,0K
drwx--x--x 2 root root 4,0K июл 21 06:08 admintestvm

root@core:/data/vms/admintestvm# ls -lth
total 625M
-rw-r--r-- 1 root root 194K июл 21 06:08 admintestvm-root
-rw-r--r-- 1 root root 366K июл 21 06:08 commoninit.iso
-rw-r--r-- 1 root root 625M июл 21 06:08 ubuntu-jammy

root@core:/data/vms/admintestvm# groups ubuntu
ubuntu : ubuntu adm cdrom sudo dip plugdev libvirt

so, i seen, TF creates VMs dir by root (but i connect to remove KVM with ubuntu user) and write permissions only root user, why?

p.s. when i use root user on connection string - i haven't problems.

Additional information:

Do you have SELinux or Apparmor/Firewall enabled? Some special configuration? First of all, i read README.md and:

remove apparmor from my remote KVM server
Add libvirt non-root user (before install KVM)
Enable libvirt group in /etc/libvirt/libvirtd.conf

root@core:/data/vms# cat /etc/libvirt/libvirtd.conf | grep -i unix_sock_group
unix_sock_group = "libvirt"

Set the UNIX socket permissions for the R/W socket in /etc/libvirt/libvirtd.conf

root@core:/data/vms# cat /etc/libvirt/libvirtd.conf | grep -i unix_sock_rw_perms
unix_sock_rw_perms = "0770"

disable security driver:

root@core:/data/vms# cat /etc/libvirt/qemu.conf | grep -i security_driver
#       security_driver = [ "selinux", "apparmor" ]
# value of security_driver cannot contain "dac".  The value "none" is
# a special value; security_driver can be set to that value in
security_driver = "none"

And restart systemctl restart libvirtd

Have you tried to reproduce the issue without them enabled? yes

luispabon commented 5 months ago

Same issue here. Issue persists even when using a root user on the qemu ssh connection string. The disk file on disk has the correct permissions and ownership, same as all of the others on the pool (pre-existing default pool at /var/lib/libvirt/images).

bmeyer71 commented 2 months ago

I am seeing this as well. Has anyone found a workaround at least for now?

laszlo-gyulai commented 1 week ago

Maybe this thread would help, I experienced a similar issue and setfactl -m helped me. See below: https://www.linux.org/threads/kvm-machine-doesnt-start-due-to-permissions.44566/post-188884

bmeyer71 commented 1 week ago

Maybe this thread would help, I experienced a similar issue and setfactl -m helped me. See below: https://www.linux.org/threads/kvm-machine-doesnt-start-due-to-permissions.44566/post-188884

I've actually tried something similar by just granting libvirt-qemu full control of the path. But that didn't make a difference. The thing that is odd, I can use Virtual Machine Manager to connect to the remote host and it will create the VM without any issues. The file permissions are set correctly as well.

dmacvicar / terraform-provider-libvirt