search

dmacvicar / terraform-provider-libvirt

Terraform provider to provision infrastructure with Linux's KVM using libvirt
Apache License 2.0
1.54k stars 457 forks source link

Images are created with root user and I get access denied #978

Open rtacconi opened 1 year ago

rtacconi commented 1 year ago

System Information

AMD Ryzen processor

Linux distribution

Ubuntu 22.04

Terraform version

Terraform v1.3.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/template v2.2.0

Provider and libvirt versions

+ provider registry.terraform.io/dmacvicar/libvirt v0.6.14

Description of Issue/Question

╷
│ Error: error creating libvirt domain: internal error: qemu unexpectedly closed the monitor: 2022-10-09T15:05:21.850163Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/worker1","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/worker1': Permission denied
│ 
│   with libvirt_domain.domain-ubuntu[1],
│   on ubuntu.tf line 60, in resource "libvirt_domain" "domain-ubuntu":
│   60: resource "libvirt_domain" "domain-ubuntu" {
│ 
╵
╷
│ Error: error creating libvirt domain: internal error: qemu unexpectedly closed the monitor: 2022-10-09T15:05:21.850163Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/master1","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/master1': Permission denied
│ 
│   with libvirt_domain.domain-ubuntu[0],
│   on ubuntu.tf line 60, in resource "libvirt_domain" "domain-ubuntu":
│   60: resource "libvirt_domain" "domain-ubuntu" {
│

Setup

terraform {
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
  uri = "qemu+ssh://rtacconi@localhost/system?keyfile=/home/rtacconi/.ssh/id_rsa"
  # same result with uri below
  # "qemu:///system"
}

variable "libvirt_disk_path" {
  description = "path for libvirt pool"
  default     = "/var/lib/libvirt/images/jammy-server-cloudimg-amd64.img"
}

variable "ssh_username" {
  description = "the ssh user to use"
  default     = "ubuntu"
}

variable "ssh_private_key" {
  description = "the private key to use"
  default     = "~/.ssh/id_ed25519"
}

variable "masters" {
  description = "List of masters"
  default = ["master1", "worker1"]
  type = list(string)
}

resource "libvirt_volume" "ubuntu-qcow2" {
  count = length(var.masters)
  name = var.masters[count.index]
  pool = "default"
  source = var.libvirt_disk_path
  format = "qcow2"
}

data "template_file" "user_data" {
  template = file("${path.module}/config/cloud_init.yml")
}

data "template_file" "network_config" {
  template = file("${path.module}/config/network_config.yml")
}

resource "libvirt_cloudinit_disk" "commoninit" {
  name           = "commoninit.iso"
  user_data      = data.template_file.user_data.rendered
  network_config = data.template_file.network_config.rendered
  pool           = "default"
}

resource "libvirt_domain" "domain-ubuntu" {
  count = length(var.masters)
  name   = var.masters[count.index]
  memory = "2048"
  vcpu   = 2

  cloudinit = libvirt_cloudinit_disk.commoninit.id

  network_interface {
    network_name = "default"
    hostname       = var.masters[count.index]
    wait_for_lease = true
  }

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }

  disk {
    volume_id = libvirt_volume.ubuntu-qcow2[count.index].id
  }

  graphics {
    type        = "spice"
    listen_type = "address"
    autoport    = true
  }

}

Steps to Reproduce Issue

It happens in a standard Ubuntu 22.04 desktop or server. I tried disabling Apparmor, specifing user as libvirt-qmeu and kvm as group in /etc/libvirt/libvirtd.conf and /etc/libvirt/qemu.conf. I can create a virtual machine with virt-manager and this is the permission of the images: -rwxrwxrwx 1 libvirt-qemu kvm 374784 Oct 9 11:18 commoninit.iso -rw-rw-r-- 1 libvirt-qemu kvm 400556032 Oct 9 11:49 debian-11.5.0-amd64-netinst.iso -rw------- 1 root root 21478375424 Oct 9 15:49 debian-alus.qcow2 -rwxrwxrwx 1 libvirt-qemu kvm 660996096 Oct 7 03:39 jammy-server-cloudimg-amd64.img drwxrwxrwx 2 libvirt-qemu kvm 16384 Jun 28 11:23 lost+found/ -rw-r--r-- 1 root root 660996096 Oct 9 15:50 master1 -rwxrwxrwx 1 libvirt-qemu kvm 0 Jun 28 11:25 test.txt -rwxrwxrwx 1 libvirt-qemu kvm 127999672320 Jun 28 11:42 win10.img -rwxrwxrwx 1 libvirt-qemu kvm 137460187136 Jun 28 16:49 win10.qcow2* -rw-r--r-- 1 root root 660996096 Oct 9 15:50 worker1

The image created with virt-manager is debian-alus.qcow2 and it is still created with root:root but it works.

I know it is more a Linux configuration problem but I hope someone could still help, sorry about that.

Additional information:

I have Apparmor disable but it seems not to make any difference

kubealex commented 1 year ago

I got the same issue with Ubuntu in the past, for me it was enough to uncomment security_driver = "none" in /etc/libvirt/qemu.conf and restart libvirtd service. (sudo systemctl restart libvirtd) Hope it helps!

Magnitus- commented 1 year ago

I use this workaround in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398/comments/43

Assuming you reuse volume pools a fair bit and don't maintain too many of them, it does the trick.

You can even include the editing of the apparmor file as part of your terraform workflow with the "local_file" provider.

rockmenjack commented 1 year ago

I believe we need to support setting uid/gid, for example:

diff --git a/libvirt/volume_def.go b/libvirt/volume_def.go
index 53809d01..434b85ce 100644
--- a/libvirt/volume_def.go
+++ b/libvirt/volume_def.go
@@ -8,14 +8,16 @@ import (
        "libvirt.org/go/libvirtxml"
 )

-func newDefVolume() libvirtxml.StorageVolume {
+func newDefVolume(uid, gid string) libvirtxml.StorageVolume {
        return libvirtxml.StorageVolume{
                Target: &libvirtxml.StorageVolumeTarget{
                        Format: &libvirtxml.StorageVolumeTargetFormat{
                                Type: "qcow2",
                        },
                        Permissions: &libvirtxml.StorageVolumeTargetPermissions{
-                               Mode: "644",
+                               Mode:  "644",
+                               Owner: uid,
+                               Group: gid,
                        },
                },
                Capacity: &libvirtxml.StorageVolumeSize{

but that would require newDefVolume to take parameters, a series of changes to many its callers

Magnitus- commented 1 year ago

I did that before with the volume pools when I was finicking around previously to resolve my vm image permission issue before finding the solution above.

resource "libvirt_pool" "os" {
  name = "os"
  type = "dir"
  path = "${var.pools_path}/os"
  xml {
    xslt = templatefile(
      "${path.module}/files/os_pool_permissions.xsl.tpl", 
      {
        permission = var.pools_permission
        owner = local.owners.users.libvirt_qemu
        group = local.owners.groups.kvm
      }
    )
  }

  lifecycle {
    prevent_destroy = true
  }
}

files/os_pool_permissions.xsl.tpl:

<?xml version="1.0"?>
  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   <xsl:template match="pool">
     <pool type='dir'>
       <name><xsl:value-of select="name"/></name>
       <target>
         <path><xsl:value-of select="target/path"/></path>
         <permissions>
           <mode>${permission}</mode>
           <owner>${owner}</owner>
           <group>${group}</group>
         </permissions>
       </target>
     </pool>
   </xsl:template>
</xsl:stylesheet>

You can probably adapt it for volumes, though we didn't need to do that.

rockmenjack commented 1 year ago

I did that before with the volume pools when I was finicking around previously to resolve my vm image permission issue before finding the solution above.

resource "libvirt_pool" "os" {
  name = "os"
  type = "dir"
  path = "${var.pools_path}/os"
  xml {
    xslt = templatefile(
      "${path.module}/files/os_pool_permissions.xsl.tpl", 
      {
        permission = var.pools_permission
        owner = local.owners.users.libvirt_qemu
        group = local.owners.groups.kvm
      }
    )
  }

  lifecycle {
    prevent_destroy = true
  }
}

files/os_pool_permissions.xsl.tpl:

<?xml version="1.0"?>
  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   <xsl:template match="pool">
     <pool type='dir'>
       <name><xsl:value-of select="name"/></name>
       <target>
         <path><xsl:value-of select="target/path"/></path>
         <permissions>
           <mode>${permission}</mode>
           <owner>${owner}</owner>
           <group>${group}</group>
         </permissions>
       </target>
     </pool>
   </xsl:template>
</xsl:stylesheet>

You can probably adapt it for volumes, though we didn't need to do that.

base on @Magnitus- 's suggestion, I have come up with a configuration:

<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output omit-xml-declaration="yes" indent="yes"/>

  <xsl:template match="node()|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="/volume/target/permissions">
    <permissions>
      <owner>${owner}</owner>
      <group>${group}</group>
    </permissions>
  </xsl:template>
</xsl:stylesheet>
pgasiorowski commented 4 months ago

Instead of disabling AppArmor it should be possible to add a local profile override to /etc/apparmor.d/local/abstractions/libvirt-qemu

/var/lib/libvirt/images/master1 rwk,
/var/lib/libvirt/images/worker1 rwk,

I am not exactly sure why the write+lock permissions are needed though.