search

dmacvicar / terraform-provider-libvirt

Terraform provider to provision infrastructure with Linux's KVM using libvirt
Apache License 2.0
1.54k stars 457 forks source link

Creating domain with attached cloudinit iso hangs permanently #1021

Closed NHAS closed 1 year ago

NHAS commented 1 year ago

System Information

Archlinux latest using libvirt 9.4.0

Linux distribution

Archlinux

Version information

Terraform v1.5.0
on linux_amd64
+ provider registry.terraform.io/dmacvicar/libvirt v0.7.1
+ provider registry.terraform.io/hashicorp/template v2.2.0

Checklist

Description of Issue/Question

When trying to do a very simple setup the creation of a single domain hangs forever. With builds taking over 20 minutes. This only happens when a cloud-init image is attached to the configuration, otherwise it takes less than 2.

Attaching a cloud-init image works fine on terraform 1.4.6, but hangs on 1.5.0.

Setup

On archlinux with the most recent version of everything (just after a pacman -Syu and reboot)

terraform {
  required_version = ">= 0.13"
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "0.7.1"
    }
  }
}

# instance the provider
provider "libvirt" {
  uri = "qemu:///system"
}

# We fetch the latest ubuntu release image from their mirrors
resource "libvirt_volume" "rocky9" {
  name   = "rocky9"
  source = "https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2"
  format = "qcow2"
}

data "template_file" "user_data" {
  template = file("${path.module}/cloud_init.cfg")
}

data "template_file" "network_config" {
  template = file("${path.module}/network_config.cfg")
}

# for more info about paramater check this out
# https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/website/docs/r/cloudinit.html.markdown
# Use CloudInit to add our ssh-key to the instance
# you can add also meta_data field
resource "libvirt_cloudinit_disk" "commoninit" {
  name           = "commoninit.iso"
  user_data      = data.template_file.user_data.rendered
  network_config = data.template_file.network_config.rendered
}

# Create the machine
resource "libvirt_domain" "domain-rocky9" {
  name   = "rocky9-terraform"
  memory = "2048"
  vcpu   = 2

  cloudinit = libvirt_cloudinit_disk.commoninit.id

  network_interface {
    network_name   = "default"
    wait_for_lease = true
  }

  cpu {
    mode = "host-passthrough"
  }

  # IMPORTANT: this is a known bug on cloud images, since they expect a console
  # we need to pass it
  # https://bugs.launchpad.net/cloud-images/+bug/1573095
  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }

  disk {
    volume_id = libvirt_volume.rocky9.id
  }

  graphics {
    type     = "vnc"
    autoport = true
  }
}

cloud_init.cfg:

#cloud-config
# vim: syntax=yaml

ssh_pwauth: False
users:
  - name: ansible
    homedir: /home/ansible
    shell: /bin/bash
    sudo: 'ALL=(ALL) NOPASSWD:ALL'
    ssh_authorized_keys:
      - 'ssh-ed25519 <my_public_key>'

network_config.cfg

version: 2
ethernets:
  ens3:
    dhcp4: true

Steps to Reproduce Issue

Using the platform and the provided TF file, 

terraform apply

Debug log:

2023-06-14T09:46:58.535+1200 [INFO]  backend/local: apply calling Apply
2023-06-14T09:46:58.535+1200 [DEBUG] Building and walking apply graph for NormalMode plan
2023-06-14T09:46:58.535+1200 [DEBUG] Resource state not found for node "libvirt_cloudinit_disk.commoninit", instance libvirt_cloudinit_disk.commoninit
2023-06-14T09:46:58.535+1200 [DEBUG] Resource state not found for node "libvirt_volume.rocky9", instance libvirt_volume.rocky9
2023-06-14T09:46:58.535+1200 [DEBUG] Resource state not found for node "libvirt_domain.domain-rocky9", instance libvirt_domain.domain-rocky9
2023-06-14T09:46:58.535+1200 [DEBUG] adding implicit provider configuration provider["registry.terraform.io/hashicorp/template"], implied first by data.template_file.user_data (expand)
2023-06-14T09:46:58.535+1200 [DEBUG] ProviderTransformer: "data.template_file.user_data (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/template"]
2023-06-14T09:46:58.535+1200 [DEBUG] ProviderTransformer: "data.template_file.network_config (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/template"]
2023-06-14T09:46:58.535+1200 [DEBUG] ProviderTransformer: "libvirt_cloudinit_disk.commoninit" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2023-06-14T09:46:58.535+1200 [DEBUG] ProviderTransformer: "libvirt_volume.rocky9" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2023-06-14T09:46:58.535+1200 [DEBUG] ProviderTransformer: "libvirt_domain.domain-rocky9" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2023-06-14T09:46:58.535+1200 [DEBUG] ProviderTransformer: "libvirt_cloudinit_disk.commoninit (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2023-06-14T09:46:58.535+1200 [DEBUG] ProviderTransformer: "libvirt_domain.domain-rocky9 (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2023-06-14T09:46:58.535+1200 [DEBUG] ProviderTransformer: "libvirt_volume.rocky9 (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/dmacvicar/libvirt"]
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "libvirt_volume.rocky9" references: []
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "libvirt_domain.domain-rocky9" references: [libvirt_cloudinit_disk.commoninit libvirt_cloudinit_disk.commoninit libvirt_cloudinit_disk.commoninit (expand) libvirt_volume.rocky9 libvirt_volume.rocky9 libvirt_volume.rocky9 (expand)]
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "data.template_file.user_data (expand)" references: []
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "data.template_file.network_config (expand)" references: []
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.commoninit" references: [data.template_file.network_config (expand) data.template_file.user_data (expand)]
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" references: []
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/template\"]" references: []
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.commoninit (expand)" references: []
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "libvirt_domain.domain-rocky9 (expand)" references: []
2023-06-14T09:46:58.536+1200 [DEBUG] ReferenceTransformer: "libvirt_volume.rocky9 (expand)" references: []
2023-06-14T09:46:58.537+1200 [DEBUG] Starting graph walk: walkApply
2023-06-14T09:46:58.537+1200 [DEBUG] created provider logger: level=debug
2023-06-14T09:46:58.537+1200 [INFO]  provider: configuring client automatic mTLS
2023-06-14T09:46:58.553+1200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4 args=[.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4]
2023-06-14T09:46:58.553+1200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4 pid=5474
2023-06-14T09:46:58.553+1200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4
2023-06-14T09:46:58.557+1200 [INFO]  provider.terraform-provider-template_v2.2.0_x4: configuring server automatic mTLS: timestamp=2023-06-14T09:46:58.557+1200
2023-06-14T09:46:58.569+1200 [DEBUG] provider.terraform-provider-template_v2.2.0_x4: plugin address: address=/tmp/plugin209818447 network=unix timestamp=2023-06-14T09:46:58.569+1200
2023-06-14T09:46:58.569+1200 [DEBUG] provider: using plugin: version=5
2023-06-14T09:46:58.592+1200 [DEBUG] created provider logger: level=debug
2023-06-14T09:46:58.592+1200 [INFO]  provider: configuring client automatic mTLS
2023-06-14T09:46:58.593+1200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
2023-06-14T09:46:58.593+1200 [DEBUG] No provider meta schema returned
2023-06-14T09:46:58.605+1200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.7.1/linux_amd64/terraform-provider-libvirt_v0.7.1 args=[.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.7.1/linux_amd64/terraform-provider-libvirt_v0.7.1]
2023-06-14T09:46:58.605+1200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.7.1/linux_amd64/terraform-provider-libvirt_v0.7.1 pid=5498
2023-06-14T09:46:58.605+1200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.7.1/linux_amd64/terraform-provider-libvirt_v0.7.1
2023-06-14T09:46:58.608+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: configuring server automatic mTLS: timestamp=2023-06-14T09:46:58.608+1200
2023-06-14T09:46:58.621+1200 [DEBUG] provider: using plugin: version=5
2023-06-14T09:46:58.621+1200 [DEBUG] provider.terraform-provider-libvirt_v0.7.1: plugin address: network=unix address=/tmp/plugin3627148650 timestamp=2023-06-14T09:46:58.621+1200
2023-06-14T09:46:58.638+1200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4 pid=5474
2023-06-14T09:46:58.638+1200 [DEBUG] provider: plugin exited
2023-06-14T09:46:58.640+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 [DEBUG] Configuring provider for 'qemu:///system': &{map[uri:0xc00037c3c0] <nil> <nil> 0xc0005d2680 map[] <nil> {{<nil>} <nil>} 0xc000504180 0xc0003263a8 0xc0000be8f0 false {1 {0 0}} false false}: timestamp=2023-06-14T09:46:58.640+1200
2023-06-14T09:46:58.640+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 [INFO] Set up libvirt transport: &{{0xc0005d2a80}}: timestamp=2023-06-14T09:46:58.640+1200
2023-06-14T09:46:58.643+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 [INFO] libvirt client libvirt version: 9004000: timestamp=2023-06-14T09:46:58.643+1200
2023-06-14T09:46:58.646+1200 [WARN]  Provider "registry.terraform.io/dmacvicar/libvirt" produced an invalid plan for libvirt_cloudinit_disk.commoninit, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .pool: planned value cty.StringVal("default") for a non-computed attribute
2023-06-14T09:46:58.646+1200 [WARN]  Provider "registry.terraform.io/dmacvicar/libvirt" produced an invalid plan for libvirt_volume.rocky9, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .pool: planned value cty.StringVal("default") for a non-computed attribute
libvirt_cloudinit_disk.commoninit: Creating...
libvirt_volume.rocky9: Creating...
2023-06-14T09:46:58.646+1200 [INFO]  Starting apply for libvirt_cloudinit_disk.commoninit
2023-06-14T09:46:58.646+1200 [INFO]  Starting apply for libvirt_volume.rocky9
2023-06-14T09:46:58.646+1200 [DEBUG] libvirt_volume.rocky9: applying the planned Create change
2023-06-14T09:46:58.646+1200 [DEBUG] libvirt_cloudinit_disk.commoninit: applying the planned Create change
2023-06-14T09:46:58.646+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 [DEBUG] creating cloudinit: timestamp=2023-06-14T09:46:58.646+1200
2023-06-14T09:46:58.646+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 [INFO] cloudInit: {Name:commoninit.iso PoolName:default MetaData: UserData:#cloud-config
# vim: syntax=yaml

ssh_pwauth: False
users:
  - name: ansible
    homedir: /home/ansible
    shell: /bin/bash
    sudo: 'ALL=(ALL) NOPASSWD:ALL'
    ssh_authorized_keys:
      - 'ssh-ed25519 <redacted>'
NetworkConfig:version: 2
ethernets:
  ens3:
    dhcp4: true}: timestamp=2023-06-14T09:46:58.646+1200
2023-06-14T09:46:58.646+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 Creating new ISO: timestamp=2023-06-14T09:46:58.646+1200
2023-06-14T09:46:58.646+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 Creating ISO contents: timestamp=2023-06-14T09:46:58.646+1200
2023-06-14T09:46:58.646+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 ISO contents created: timestamp=2023-06-14T09:46:58.646+1200
2023-06-14T09:46:58.646+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 About to execute cmd: /usr/bin/mkisofs -output /tmp/cloudinit1030668553/commoninit.iso -volid cidata -joliet -rock /tmp/cloudinit1030668553/user-data /tmp/cloudinit1030668553/meta-data /tmp/cloudinit1030668553/network-config: timestamp=2023-06-14T09:46:58.646+1200
2023-06-14T09:46:58.646+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 [DEBUG] Locking "default": timestamp=2023-06-14T09:46:58.646+1200
2023-06-14T09:46:58.646+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 [DEBUG] Locked "default": timestamp=2023-06-14T09:46:58.646+1200
2023-06-14T09:46:58.649+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 ISO created at /tmp/cloudinit1030668553/commoninit.iso: timestamp=2023-06-14T09:46:58.649+1200
2023-06-14T09:46:58.649+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:58 [DEBUG] Locking "default": timestamp=2023-06-14T09:46:58.649+1200
2023-06-14T09:46:59.766+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:59 Image https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2 image is: 989265920 bytes: timestamp=2023-06-14T09:46:59.766+1200
2023-06-14T09:46:59.766+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:59 [DEBUG] Generated XML for libvirt volume:
  <volume>
      <name>rocky9</name>
      <capacity unit="B">989265920</capacity>
      <target>
          <format type="qcow2"></format>
          <permissions>
              <mode>644</mode>
          </permissions>
      </target>
  </volume>: timestamp=2023-06-14T09:46:59.766+1200
2023-06-14T09:46:59.767+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:59 [INFO] Volume about to be created was found and left as-is: rocky9: timestamp=2023-06-14T09:46:59.767+1200
2023-06-14T09:46:59.767+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:46:59 [INFO] Volume ID: /var/lib/libvirt/images/rocky9: timestamp=2023-06-14T09:46:59.767+1200
2023-06-14T09:47:00.034+1200 [INFO]  provider.terraform-provider-libvirt_v0.7.1: 2023/06/14 09:47:00 [DEBUG]: url resp status code 200 OK (retry #0): timestamp=2023-06-14T09:47:00.033+1200
libvirt_cloudinit_disk.commoninit: Still creating... [10s elapsed]
libvirt_volume.rocky9: Still creating... [10s elapsed]
libvirt_volume.rocky9: Still creating... [1m0s elapsed]
libvirt_cloudinit_disk.commoninit: Still creating... [1m0s elapsed]
libvirt_volume.rocky9: Still creating... [1m10s elapsed]
libvirt_cloudinit_disk.commoninit: Still creating... [1m10s elapsed]

AS this never completes I've truncated it at the 1 min mark

Additional information:

Do you have SELinux or Apparmor/Firewall enabled? Some special configuration? Nope, just stock standard arch

Have you tried to reproduce the issue without them enabled? N/A

NHAS commented 1 year ago

My bad it was hanging because a volume was created and never destroyed, but for some reason didnt error out.