Automatically configure app-armor for non-standard paths (to work around upstream bug)

raoulbhatia commented 2 years ago

System Information

Linux distribution

Ubuntu 20.04.3 LTS

Terraform, provider and libvirt version

Terraform v1.1.2
on darwin_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.6.12
+ provider registry.terraform.io/hashicorp/template v2.2.0

Description of Issue/Question

Setup

Base setup from https://blog.ruanbekker.com/blog/2020/10/08/using-the-libvirt-provisioner-with-terraform-for-kvm/ with the following change to the libvirt_disk_path variable.

variable "libvirt_disk_path" {
  description = "path for libvirt pool"
  default     = "/data/kvm/pool1"
}

(Please provide the full main.tf file for reproducing the issue (Be sure to remove sensitive information)

Steps to Reproduce Issue

Use Ubuntu 20.04 with standard apparmor and libvirt packages

apparmor                       2.13.3-7ubuntu5.1
apparmor-utils                 2.13.3-7ubuntu5.1
libapparmor1:amd64             2.13.3-7ubuntu5.1
libvirt-clients                6.0.0-0ubuntu8.15
libvirt-daemon                 6.0.0-0ubuntu8.15
libvirt-daemon-driver-qemu     6.0.0-0ubuntu8.15
libvirt-daemon-system          6.0.0-0ubuntu8.15
libvirt-daemon-system-systemd  6.0.0-0ubuntu8.15
libvirt0:amd64                 6.0.0-0ubuntu8.15
python3-apparmor               2.13.3-7ubuntu5.1
python3-libapparmor            2.13.3-7ubuntu5.1

The defined libvirtd domain terraform-kvm-ansible will fail to start with the following output:

libvirt_volume.ubuntu-qcow2: Creating...
libvirt_volume.ubuntu-qcow2: Still creating... [10s elapsed]
libvirt_volume.ubuntu-qcow2: Still creating... [20s elapsed]
libvirt_volume.ubuntu-qcow2: Still creating... [30s elapsed]
libvirt_volume.ubuntu-qcow2: Still creating... [40s elapsed]
libvirt_volume.ubuntu-qcow2: Still creating... [50s elapsed]
libvirt_volume.ubuntu-qcow2: Still creating... [1m0s elapsed]
libvirt_volume.ubuntu-qcow2: Creation complete after 1m4s [id=/data/kvm/pool1/ubuntu-qcow2]
libvirt_domain.domain-ubuntu: Creating...
╷
│ Error: error creating libvirt domain: internal error: qemu unexpectedly closed the monitor: 2022-01-01T18:59:57.281099Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/data/kvm/pool1/ubuntu-qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/data/kvm/pool1/ubuntu-qcow2': Permission denied
│
│   with libvirt_domain.domain-ubuntu,
│   on main.tf line 33, in resource "libvirt_domain" "domain-ubuntu":
│   33: resource "libvirt_domain" "domain-ubuntu" {
│

This can also be reproduced by running virsh start terraform-kvm-ansible on the KVM hypervisor.

# virsh start terraform-kvm-ansible
error: Failed to start domain terraform-kvm-ansible
error: internal error: process exited while connecting to monitor: 2022-01-01T19:01:47.326724Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/data/kvm/pool1/ubuntu-qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/data/kvm/pool1/ubuntu-qcow2': Permission denied

apparmor error in dmesg

Sat Jan  1 20:01:47 2022] audit: type=1400 audit(1641063707.322:43): apparmor="DENIED" operation="open" profile="libvirt-7454585d-3a7c-42a9-a822-835b9ca3c1b6" name="/data/kvm/pool1/ubuntu-qcow2" pid=787871 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=109 ouid=109

Solution

Add the non-standard path to /etc/apparmor.d/local/abstractions/libvirt-qemu, i.e.

"/data/kvm/**/*qcow2" rwk,

Proposal

Perhaps there would be an option to either

hint at this issue, or
auto-magically configure apparmor to allow this non-standard path (whilst displaying a warning to the user?)

Related upstream bug reports:

Thanks, Raoul

MadsRC commented 2 years ago

I am experiencing a nearly identical issue - My only difference is that the libvirtd host is running Debian 11 and the guest tries to start from an image whose backing volume is a Debian 11 image stored at the default '/var/lib/libvirt/images' location.

error: internal error: process exited while connecting to monitor: 2022-02-12T12:21:17.463372Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/debian11","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/debian11': Permission denied

As access to the default image store is also denied, I do not believe that the auto-magic configuration for non-standard paths would solve all the issues.

I'll continue my debugging and see if I can find a viable solution.

dmacvicar commented 2 years ago

I don't think the provider should auto-configure this (and can't as we can't assume we run as root).

I'd accept a PR hinting at this in the README and also in the docs.

ajquack commented 1 year ago

@dmacvicar

So what I have discovered now checking the bug within in the libvirt-apparmor thing and the behaviour of the terraform provider:

This Occurs when you create a storage pool with a non-standard dir path and specifiy a disk with "volume_id" in the libvirt_domain disk block. Apparmor does not generate a allow rule for these because it cannot read the dir path of the storage pool (Check XMLs down bellow to see the difference between the two disk arguments) and therefore cannot generate a full path to the image file. This will result in the image not beeing added to the allow rules and the "permission denied" error occurs
It can work with the terraform provider if you specifiy the disk as "file" but this depends on your used file format.

I'am using .qcow2 as my image formats. If I specifiy my volume as following the instance is able to start but not boot. This is because the provider sets the disk format to raw by default. Changing the disk format manually to qcow2 afterwards resolves the boot issue.

Terraform code:

disk { file = libvirt_volume.terminal.id } Generated XML:

`

Apparmor can generate a allow rule for this because the attribute "source file" specifies the full path. The Apparmor virt-aa-helper reads the libvirt domain xml and adds it's image file path to the allow rules.

This is not working if you specifiy the following:

Terraform code:

disk { volume_id = libvirt_volume.terminal.id }

Generated XML:

`

Error after "terraform apply:

Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2023-02-09T21:19:11.930365Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"***/terminal.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Could not open '***/terminal.qcow2': Permission denied

The Apparmor helper cannot generate a allow rule because it cannot read the dir path of the storage pool and therefore cannot generate the absolute path to the image file.

Unfortunately this issue is open since 2017 within libvirt. Therefore I would suggest to modify the disk block within libvirt_domain a litte bit as disabling the libvirt security mechanism should not be a prefered fix.

First: If you specifiy "file" as the disk source there should be a option to set the disk format to something other than raw. This would ensure that custom storage pools with a dir path and a file format other than raw are working with the provider even if apparmor is enabled. Apparmor can read the full disk path and generate a allow rule. So the domain XML should like this

`

Second: What I noticed after "terraform plan" and "terraform apply" with the disk specified as "file" when you do a new "terraform plan" the provider suggests a change:

disk { "- file = "/terminal.qcow2"" "+ volume_id = "/terminal.qcow2"" }

This should also be looked into. Don't know why terraform wants to change something here. If you apply nothing happens as far as I checked it. Non the less this change should not appear. This only occurs the first time after the instance as has been provisioned with the disk as "file.

Initial Libvirt Bug Discussion: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398

Libvirt Gitlab Upstream Bug: https://gitlab.com/libvirt/libvirt/-/issues/135

dmacvicar / terraform-provider-libvirt