Volumes physical files are not removed during destroy

[rascasoft]

System Information

Linux distribution

Ubuntu 22.04

Provider & Terraform version

> terraform -v
Terraform v1.3.9
on linux_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.7.1
+ provider registry.terraform.io/hashicorp/azurerm v2.40.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Libvirt versions

> dpkg -l libvirt* | grep ^ii
ii  libvirt-clients                            8.0.0-1ubuntu7.4 amd64        Programs for the libvirt library
ii  libvirt-daemon                             8.0.0-1ubuntu7.4 amd64        Virtualization daemon
ii  libvirt-daemon-config-network              8.0.0-1ubuntu7.4 all          Libvirt daemon configuration files (default network)
ii  libvirt-daemon-config-nwfilter             8.0.0-1ubuntu7.4 all          Libvirt daemon configuration files (default network filters)
ii  libvirt-daemon-driver-qemu                 8.0.0-1ubuntu7.4 amd64        Virtualization daemon QEMU connection driver
ii  libvirt-daemon-system                      8.0.0-1ubuntu7.4 amd64        Libvirt daemon configuration files
ii  libvirt-daemon-system-systemd              8.0.0-1ubuntu7.4 all          Libvirt daemon configuration files (systemd)
ii  libvirt-glib-1.0-0:amd64                   4.0.0-2          amd64        libvirt GLib and GObject mapping library
ii  libvirt-glib-1.0-data                      4.0.0-2          all          Common files for libvirt GLib library
ii  libvirt0:amd64                             8.0.0-1ubuntu7.4 amd64        library for interfacing with different virtualization systems

---

Checklist

• [ ] Is your issue/contribution related with enabling some setting/option exposed by libvirt that the plugin does not yet support, or requires changing/extending the provider terraform schema?

  • [ ] Make sure you explain why this option is important to you, why it should be important to everyone. Describe your use-case with detail and provide examples where possible.
  • [ ] If it is a very special case, consider using the XSLT support in the provider to tweak the definition instead of opening an issue
  • [ ] Maintainers do not have expertise in every libvirt setting, so please, describe the feature and how it is used. Link to the appropriate documentation

• [X] Is it a bug or something that does not work as expected? Please make sure you fill the version information below:

Description of Issue/Question

Setup

The main.tf I'm using:

variable "libvirt_uri" {
  description = "This is the libvirt daemon url, i.e. qemu:///system"
}

provider "libvirt" {
  uri = var.libvirt_uri
}

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
      version = "0.7.1"
    }
  }
}

# Subnet that will be created:
# default nat "192.168.122.0/24"
resource "libvirt_network" "default" {
    name      = "default"
    mode      = "nat"
    addresses = ["192.168.122.0/24"]

    dhcp {
      enabled = true
    }
}

# Cloud init
resource "libvirt_cloudinit_disk" "commoninit" {
  name      = "commoninit.iso"
  pool      = "kubelab"
  user_data = data.template_file.user_data.rendered
}

data "template_file" "user_data" {
  template = file("${path.module}/cloud_init.cfg")
}

# Libvirt Storage Pools
resource "libvirt_pool" "kubelab" {
  name = "kubelab"
  type = "dir"
  path = "/kubelab"
}

# Libvirt Storage Volumes
resource "libvirt_volume" "almalinux-8" {
  name = "almalinux-8.qcow2"
  pool = "kubelab"
  source = "/images/almalinux-8.qcow2"
  format = "qcow2"
}

resource "libvirt_volume" "kubernetes-1-qcow2" {
  name = "kubernetes-1.qcow2"
  pool = "kubelab"
  base_volume_id = libvirt_volume.almalinux-8.id
  format = "qcow2"
}

# Virtual machine creation blocks for group terraform

# kubernetes-1
resource "libvirt_domain" "kubernetes-1" {
  name                         = "kubernetes-1"
  memory                       = "4096"
  vcpu                         = "4"

  cloudinit = libvirt_cloudinit_disk.commoninit.id

  network_interface {
    network_id     = libvirt_network.default.id
    addresses      = ["192.168.122.31"]
  }

  console {
    type        = "pty"
    target_port = "1"
    target_type = "virtio"
  }

  disk {
    volume_id            = libvirt_volume.kubernetes-1-qcow2.id
  }

  graphics {
    type        = "spice"
    listen_type = "address"
    autoport    = "true"
  }
}

Steps to Reproduce Issue

Simply run:

> TF_LOG=DEBUG terraform/libvirt_kubelab/bin/terraform -chdir=terraform/libvirt_kubelab/ destroy -auto-approve

These are the relevant logs:

...
Plan: 0 to add, 0 to change, 6 to destroy.
...
libvirt_pool.kubelab: Destroying... [id=fc3cfc20-746b-49ac-bd96-e781323b3f72]
2023-02-16T17:50:52.424+0100 [INFO]  Starting apply for libvirt_pool.kubelab
2023-02-16T17:50:52.424+0100 [DEBUG] libvirt_pool.kubelab: applying the planned Delete change
2023-02-16T17:50:52.425+0100 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/02/16 17:50:52 [DEBUG] Locking "kubelab": timestamp=2023-02-16T17:50:52.425+0100
2023-02-16T17:50:52.425+0100 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/02/16 17:50:52 [DEBUG] Locked "kubelab": timestamp=2023-02-16T17:50:52.425+0100
2023-02-16T17:50:52.425+0100 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/02/16 17:50:52 [DEBUG] Unlocking "kubelab": timestamp=2023-02-16T17:50:52.425+0100
2023-02-16T17:50:52.425+0100 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/02/16 17:50:52 [DEBUG] Unlocked "kubelab": timestamp=2023-02-16T17:50:52.425+0100
libvirt_domain.kubernetes-1: Destroying... [id=0b0201e6-07fd-43b2-9005-8b363380b5cd]
2023-02-16T17:50:52.425+0100 [INFO]  Starting apply for libvirt_domain.kubernetes-1
2023-02-16T17:50:52.425+0100 [DEBUG] libvirt_domain.kubernetes-1: applying the planned Delete change
2023-02-16T17:50:52.425+0100 [ERROR] provider.terraform-provider-libvirt_v0.7.1: Response contains error diagnostic: diagnostic_severity=ERROR diagnostic_summary="error deleting storage pool: failed to remove pool '/kubelab': Directory not empty" tf_proto_version=5.3 tf_provider_addr=provider tf_req_id=2403ac33-9905-7bd2-68eb-600b0e011ec3 diagnostic_detail= tf_resource_type=libvirt_pool tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto timestamp=2023-02-16T17:50:52.425+0100
...
2023-02-16T17:50:52.438+0100 [ERROR] vertex "libvirt_pool.kubelab (destroy)" error: error deleting storage pool: failed to remove pool '/kubelab': Directory not empty

Note that the deletion of the volumes seems to work fine:

...
libvirt_cloudinit_disk.commoninit: Destroying... [id=/kubelab/commoninit.iso;0bd9f662-3828-4229-87c5-c70ca7e40156]
2023-02-16T17:50:52.858+0100 [INFO]  Starting apply for libvirt_cloudinit_disk.commoninit
2023-02-16T17:50:52.858+0100 [DEBUG] libvirt_cloudinit_disk.commoninit: applying the planned Delete change
libvirt_volume.kubernetes-1-qcow2: Destroying... [id=/kubelab/kubernetes-1.qcow2]
2023-02-16T17:50:52.858+0100 [INFO]  Starting apply for libvirt_volume.kubernetes-1-qcow2
2023-02-16T17:50:52.858+0100 [DEBUG] libvirt_volume.kubernetes-1-qcow2: applying the planned Delete change
libvirt_network.default: Destroying... [id=838f91cb-03da-4e82-92b8-fbc6dfcf1263]
2023-02-16T17:50:52.859+0100 [INFO]  Starting apply for libvirt_network.default
2023-02-16T17:50:52.859+0100 [DEBUG] libvirt_network.default: applying the planned Delete change
libvirt_cloudinit_disk.commoninit: Destruction complete after 0s
2023-02-16T17:50:52.859+0100 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/02/16 17:50:52 [DEBUG] Deleting network ID 838f91cb-03da-4e82-92b8-fbc6dfcf1263: timestamp=2023-02-16T17:50:52.859+0100
libvirt_volume.kubernetes-1-qcow2: Destruction complete after 0s
libvirt_volume.almalinux-8: Destroying... [id=/kubelab/almalinux-8.qcow2]
2023-02-16T17:50:52.872+0100 [INFO]  Starting apply for libvirt_volume.almalinux-8
2023-02-16T17:50:52.872+0100 [DEBUG] libvirt_volume.almalinux-8: applying the planned Delete change
libvirt_volume.almalinux-8: Destruction complete after 0s
...

But physical files are still there and so the pool deletion cannot be completed:

>  ls /kubelab/
almalinux-8.qcow2  commoninit.iso  kubernetes-1.qcow2

---

Additional information:

• There's an issue identical to this closed one https://github.com/dmacvicar/terraform-provider-libvirt/issues/470, but I can't understand the reasons mentioned in there to close it.
• No SELinux/Apparmor/Firewall enabled.
• No special configuration.
• Issue is 100% reproducible everywhere, it also does not depend from the user, results are the same if you use a user (part of the libvirt group) or root.

[rascasoft]

Digging more around the issue I discovered that the problem is something that can be easily solved by using Terraform dependencies.

Starting from the original main.tf I reported originally these are the additional sections that make everything behave as expected:

...

# Libvirt Storage Volumes                                                       

resource "libvirt_volume" "almalinux-8" {                                       
  name = "almalinux-8.qcow2"                                                    
  pool = "kubelab"                                                              
  source = "/home/rasca/Work/VMS/images/almalinux-8.qcow2"                      
  format = "qcow2"                                                              

  depends_on = [libvirt_pool.kubelab]                                           
}                                                                               

resource "libvirt_volume" "kubernetes-1-qcow2" {                                
  name = "kubernetes-1.qcow2"                                                   
  pool = "kubelab"                                                              
  base_volume_id = libvirt_volume.almalinux-8.id                                
  format = "qcow2"                                                              

  depends_on = [libvirt_pool.kubelab]                                           
} 

...

# kubernetes-1                                                                  
resource "libvirt_domain" "kubernetes-1" {                                      
  name                         = "kubernetes-1"                                 
  memory                       = "4096"                                         
  vcpu                         = "4"                                            

  cloudinit = libvirt_cloudinit_disk.commoninit.id                              

  network_interface {                                                           
    network_id     = libvirt_network.kubelab.id                                 
    addresses      = ["10.0.0.31"]                                              
  }                                                                             

  console {                                                                     
    type        = "pty"                                                         
    target_port = "1"                                                           
    target_type = "virtio"                                                      
  }                                                                             

  disk {                                                                        
    volume_id            = libvirt_volume.kubernetes-1-qcow2.id                 
  }                                                                             

  depends_on = [libvirt_volume.kubernetes-1-qcow2]

...

The depends_on is what makes the trick, which is in the end quite logical. Sorry for the noise.