output ip fail with bridge and I need it for my ansible inventory

itwars commented 4 years ago

System Information

Linux distribution

Ubuntu 18.04.4 LTS

Terraform version

+ provider.libvirt (unversioned)
+ provider.template v2.1.2

Provider and libvirt versions

terraform-provider-libvirt 0.6.2+git.1585292411.8cbe9ad0
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Running hypervisor: QEMU 2.11.1
Running against daemon: 4.0.0

Description of Issue/Question

Setup

provider "libvirt" {
  uri = "qemu:///system"
}
terraform {
  required_version = ">= 0.12"
}
resource "libvirt_pool" "debian" {
  name = "debian"
  type = "dir"
  path = "/srv/terraform-provider-libvirt-pool-debian"
}
resource "libvirt_volume" "debian-qcow2" {
  name   = "debian-qcow2"
  pool   = libvirt_pool.debian.name
  source = "https://cdimage.debian.org/cdimage/openstack/current/debian-10.3.2-20200406-openstack-amd64.qcow2"
  format = "qcow2"
}
resource "libvirt_volume" "disk_debian_resized" {
  name           = "disk"
  base_volume_id = libvirt_volume.debian-qcow2.id
  pool           = libvirt_pool.debian.name
  size           = 2 * 1024 * 1024 * 1024
}
data "template_file" "user_data" {
  template = file("${path.module}/cloud_init.cfg")
}
data "template_file" "network_config" {
  template = file("${path.module}/network_config.cfg")
}
resource "libvirt_cloudinit_disk" "commoninit" {
  name           = "commoninit.iso"
  user_data      = data.template_file.user_data.rendered
  network_config = data.template_file.network_config.rendered
  pool           = libvirt_pool.debian.name
}
resource "libvirt_domain" "domain-debian" {
  name   = "debian-terraform"
  memory = "2048"
  vcpu   = 2
  qemu_agent = true
  cloudinit = libvirt_cloudinit_disk.commoninit.id
  disk {
    volume_id = libvirt_volume.disk_debian_resized.id
  }
  network_interface {
    bridge = "br0"
    wait_for_lease = true
  }

provisioner "local-exec" {
    command = "virsh qemu-agent-command debian-terraform '{\"execute\":\"guest-network-get-interfaces\"}'|jq     '.return[1].\"ip-addresses\"[0].\"ip-address\"'| sed 's/\"//g'"
}

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }
  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }
  graphics {
    type        = "spice"
    listen_type = "address"
    autoport    = true
  }
}

output "ips" {
  # show IP, run 'terraform refresh' if not populated
  value = libvirt_domain.domain-debian.*.network_interface.0.addresses
}

Additional information:

terraform apply

libvirt_pool.debian: Creating...
libvirt_pool.debian: Creation complete after 5s [id=289d6454-ff34-4552-9c29-8b12042b0b35]
libvirt_cloudinit_disk.commoninit: Creating...
libvirt_volume.debian-qcow2: Creating...
libvirt_volume.debian-qcow2: Still creating... [10s elapsed]
libvirt_cloudinit_disk.commoninit: Still creating... [10s elapsed]
libvirt_volume.debian-qcow2: Creation complete after 11s [id=/srv/terraform-provider-libvirt-pool-debian/debian-qcow2]
libvirt_volume.disk_debian_resized: Creating...
libvirt_cloudinit_disk.commoninit: Creation complete after 11s [id=/srv/terraform-provider-libvirt-pool-debian/commoninit.iso;5e9f02e4-631b-5c20-ed09-64bed861e1a0]
libvirt_volume.disk_debian_resized: Creation complete after 0s [id=/srv/terraform-provider-libvirt-pool-debian/disk]
libvirt_domain.domain-debian: Creating...
libvirt_domain.domain-debian: Still creating... [10s elapsed]
libvirt_domain.domain-debian: Still creating... [20s elapsed]
...
libvirt_domain.domain-debian: Still creating... [5m0s elapsed]

Error: Error: couldn't retrieve IP address of domain.Please check following:
1) is the domain running proplerly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 timeout while waiting for state to become 'all-addresses-obtained' (last state: 'waiting-addresses', timeout: 5m0s)

  on debian10.tf line 36, in resource "libvirt_domain" "domain-debian":
  36: resource "libvirt_domain" "domain-debian" {

virsh qemu-agent-command debian-terraform '{"execute":"guest-network-get-interfaces"}'|jq '.return[1]."ip-addresses"[0]."ip-address"'| sed 's/\"//g'

Give me : 192.168.1.26

terraform refresh

Give me : Outputs:

ips = [
  [],
]

MalloZup commented 4 years ago

HI! if you are using bridge you need the qemu-agent installed on your vms otherwise wait_for_lease will not be able to detect the IP

itwars commented 4 years ago

qemu-agent is on my terraform file + I use it by requesting : virsh qemu-agent-command debian-terraform '{"execute":"guest-network-get-interfaces"}'

tips : when having my virsh command in a local-exec it give me:

libvirt_domain.domain-debian (local-exec): Executing: ["/bin/sh" "-c" "virsh qemu-agent-command debian-terraform '{\"execute\":\"guest-network-get-interfaces\"}'|jq '.return[1].\"ip-addresses\"[0].\"ip-address\"'| sed 's/\"//g'"]
libvirt_domain.domain-debian (local-exec): error: Guest agent is not responding: QEMU guest agent is not connected

It seems qemu-agent start too late?! It is not include in the https://cdimage.debian.org/cdimage/openstack/current/debian-10.3.2-20200406-openstack-amd64.qcow2 image, and install by cloud-init.

I just have an extra test using centos instead of debian (centos img include qemu-agent), virsh command return me the ip address as soon as the vm start, while still answer : libvirt_domain.domain-centos: Still creating...

itwars commented 4 years ago

Hum I just forgot to add:

qemu_agent = true

Anyway, I still don't have the ip:

libvirt_domain.domain-debian: Still creating... [20s elapsed]
libvirt_domain.domain-debian: Still creating... [30s elapsed]
libvirt_domain.domain-debian: Provisioning with 'local-exec'...
libvirt_domain.domain-debian (local-exec): Executing: ["/bin/sh" "-c" "virsh qemu-agent-command debian-terraform '{\"execute\":\"guest-network-get-interfaces\"}'|jq '.return[1].\"ip-addresses\"[0].\"ip-address\"'| sed 's/\"//g'"]
libvirt_domain.domain-debian (local-exec): error: Guest agent is not responding: QEMU guest agent is not connected
libvirt_domain.domain-debian: Creation complete after 31s [id=542fb731-b9c3-40fc-8b27-82ec0e725a85]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

ips = [
  [],
]

I wait more than 10 minutes, start again a terraform refresh and output is still:

Outputs:

ips = [
  [],
]

MalloZup commented 4 years ago

the provider itself doesn't add the qemu-agent pkg to your distro . You will need to download and install it before on the image itself if you wanna use it. hth

use guest agent: https://wiki.qemu.org/Features/GuestAgent

MalloZup commented 4 years ago

the provider itself doesn't add the qemu-agent pkg to your distro . You will need to download and install it before on the image itself if you wanna use it. hth

use guest agent: https://wiki.qemu.org/Features/GuestAgent

MalloZup commented 4 years ago

re-opening

MalloZup commented 4 years ago

sorry I clicked the wrong button and github was buggy not reachable.

Beeing that sad, there is not much we can do from the terraform libvirt side.

As you saw, either you can do a refresh later on to get the ip. But with the bridge mode and without the qemu-agent pkg installed , we can't get the IP adress of the machines.

MalloZup commented 4 years ago

sorry I clicked the wrong button and github was buggy not reachable.

Beeing that sad, there is not much we can do from the terraform libvirt side.

As you saw, either you can do a refresh later on to get the ip. But with the bridge mode and without the qemu-agent pkg installed , we can't get the IP adress of the machines.

MalloZup commented 4 years ago

re-opening

MalloZup commented 4 years ago

re-opening

itwars commented 4 years ago

As I told you I do install the qemu-agent during cloud-init phase :

#cloud-config
hostname: gitlab
fqdn: gitlab.local
manage_etc_hosts: True
growpart:
  mode: auto
  devices: ['/']
packages:
  - qemu-guest-agent

And in the Centos img it's already done by Centos people!

Even I can query qemu-agent when (and this commandline do work):

virsh qemu-agent-command debian-terraform '{"execute":"guest-network-get-interfaces"}'

Please read carefully what I write, pls.

So it's not a question, it's an issue, because it doesn't work. BR

MalloZup commented 4 years ago

@itwars ok sound un-usual issue. Sorry, I might have overseen your comment. Adapting thx

MalloZup commented 4 years ago

can you attach the output of the cmd

TF_LOG=DEBUG terraform apply? thx

itwars commented 4 years ago

I think in qemu_agent.go, you try to get the ip address too early, if you look in my .tf file above, event when I do at the very end a local-exec at the very end of the process, using virsh command I couldn't get ip. It's not a matter of seconds, but minutes:

MinTimeout: 4 * time.Second,
Delay:      4 * time.Second, // Wait this time before starting checks
Timeout:    30 * time.Second,

Anyway, here is it :

2020/04/21 19:42:49 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
  Use TF_LOG=TRACE to see Terraform's internal logs.
  ----
2020/04/21 19:42:49 [INFO] Terraform version: 0.12.24  
2020/04/21 19:42:49 [INFO] Go runtime version: go1.12.13
2020/04/21 19:42:49 [INFO] CLI args: []string{"/usr/local/bin/terraform", "apply"}
2020/04/21 19:42:49 [DEBUG] Attempting to open CLI config file: /home/vrh/.terraformrc
2020/04/21 19:42:49 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/04/21 19:42:49 [DEBUG] checking for credentials in "/home/vrh/.terraform.d/plugins"
2020/04/21 19:42:49 [INFO] CLI command args: []string{"apply"}
2020/04/21 19:42:49 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
  Use TF_LOG=TRACE to see Terraform's internal logs.
  ----
2020/04/21 19:42:49 [DEBUG] New state was assigned lineage "a64d111c-5f13-40ae-3896-8bcd51a58c63"
2020/04/21 19:42:49 [DEBUG] checking for provider in "."
2020/04/21 19:42:49 [DEBUG] checking for provider in "/usr/local/bin"
2020/04/21 19:42:49 [DEBUG] checking for provider in ".terraform/plugins/linux_amd64"
2020/04/21 19:42:49 [DEBUG] found provider "terraform-provider-template_v2.1.2_x4"
2020/04/21 19:42:49 [DEBUG] checking for provider in "/home/vrh/.terraform.d/plugins"
2020/04/21 19:42:49 [WARN] found legacy provider "terraform-provider-libvirt"
2020/04/21 19:42:49 [DEBUG] found valid plugin: "template", "2.1.2", "/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4"
2020/04/21 19:42:49 [DEBUG] found valid plugin: "libvirt", "0.0.0", "/home/vrh/.terraform.d/plugins/terraform-provider-libvirt"
2020/04/21 19:42:49 [DEBUG] checking for provisioner in "."
2020/04/21 19:42:49 [DEBUG] checking for provisioner in "/usr/local/bin"
2020/04/21 19:42:49 [DEBUG] checking for provisioner in ".terraform/plugins/linux_amd64"
2020/04/21 19:42:49 [DEBUG] checking for provisioner in "/home/vrh/.terraform.d/plugins"
2020/04/21 19:42:49 [INFO] backend/local: starting Apply operation
2020-04-21T19:42:49.717Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:42:49.783Z [DEBUG] plugin: starting plugin: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt args=[/home/vrh/.terraform.d/plugins/terraform-provider-libvirt]
2020-04-21T19:42:49.783Z [DEBUG] plugin: plugin started: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt pid=11120
2020-04-21T19:42:49.784Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt
2020-04-21T19:42:49.825Z [INFO]  plugin.terraform-provider-libvirt: configuring server automatic mTLS: timestamp=2020-04-21T19:42:49.825Z
2020-04-21T19:42:49.893Z [DEBUG] plugin.terraform-provider-libvirt: plugin address: address=/tmp/plugin312739390 network=unix timestamp=2020-04-21T19:42:49.892Z
2020-04-21T19:42:49.893Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:50.038Z [DEBUG] plugin: plugin process exited: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt pid=11120
2020-04-21T19:42:50.038Z [DEBUG] plugin: plugin exited
2020-04-21T19:42:50.038Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:42:50.108Z [DEBUG] plugin: starting plugin: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 args=[/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4]
2020-04-21T19:42:50.108Z [DEBUG] plugin: plugin started: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11134
2020-04-21T19:42:50.108Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
2020-04-21T19:42:50.119Z [INFO]  plugin.terraform-provider-template_v2.1.2_x4: configuring server automatic mTLS: timestamp=2020-04-21T19:42:50.119Z
2020-04-21T19:42:50.188Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:50.188Z [DEBUG] plugin.terraform-provider-template_v2.1.2_x4: plugin address: address=/tmp/plugin088692663 network=unix timestamp=2020-04-21T19:42:50.188Z
2020-04-21T19:42:50.330Z [DEBUG] plugin: plugin process exited: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11134
2020-04-21T19:42:50.330Z [DEBUG] plugin: plugin exited
2020-04-21T19:42:50.331Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/terraform args=[/usr/local/bin/terraform, internal-plugin, provisioner, local-exec]
2020-04-21T19:42:50.331Z [DEBUG] plugin: plugin started: path=/usr/local/bin/terraform pid=11144
2020-04-21T19:42:50.331Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/terraform
2020-04-21T19:42:50.353Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
2020-04-21T19:42:50.353Z [DEBUG] plugin.terraform:   Use TF_LOG=TRACE to see Terraform's internal logs.
2020-04-21T19:42:50.353Z [DEBUG] plugin.terraform:   ----
2020-04-21T19:42:50.376Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [INFO] Terraform version: 0.12.24  
2020-04-21T19:42:50.376Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [INFO] Go runtime version: go1.12.13
2020-04-21T19:42:50.376Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [INFO] CLI args: []string{"/usr/local/bin/terraform", "internal-plugin", "provisioner", "local-exec"}
2020-04-21T19:42:50.376Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [DEBUG] Attempting to open CLI config file: /home/vrh/.terraformrc
2020-04-21T19:42:50.376Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020-04-21T19:42:50.376Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [DEBUG] checking for credentials in "/home/vrh/.terraform.d/plugins"
2020-04-21T19:42:50.376Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [INFO] CLI command args: []string{"internal-plugin", "provisioner", "local-exec"}
2020-04-21T19:42:50.377Z [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2020/04/21 19:42:50 [INFO] Starting provisioner plugin local-exec
2020-04-21T19:42:50.378Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:50.378Z [DEBUG] plugin.terraform: plugin address: address=/tmp/plugin143109278 network=unix timestamp=2020-04-21T19:42:50.377Z
2020-04-21T19:42:50.386Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=11144
2020-04-21T19:42:50.386Z [DEBUG] plugin: plugin exited
2020/04/21 19:42:50 [INFO] terraform: building graph: GraphTypeValidate
2020/04/21 19:42:50 [DEBUG] adding implicit provider configuration provider.template, implied first by data.template_file.user_data
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "libvirt_domain.domain-debian" (*terraform.NodeValidatableResource) needs provider.libvirt
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "libvirt_pool.debian" (*terraform.NodeValidatableResource) needs provider.libvirt
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "libvirt_cloudinit_disk.commoninit" (*terraform.NodeValidatableResource) needs provider.libvirt
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "data.template_file.user_data" (*terraform.NodeValidatableResource) needs provider.template
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "data.template_file.network_config" (*terraform.NodeValidatableResource) needs provider.template
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "libvirt_volume.debian-qcow2" (*terraform.NodeValidatableResource) needs provider.libvirt
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "libvirt_volume.disk_debian_resized" (*terraform.NodeValidatableResource) needs provider.libvirt
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "libvirt_domain.domain-debian" references: [libvirt_cloudinit_disk.commoninit libvirt_volume.disk_debian_resized]
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "output.ips" references: [libvirt_domain.domain-debian]
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "provisioner.local-exec" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "provider.libvirt" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "provider.template" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "libvirt_volume.debian-qcow2" references: [libvirt_pool.debian]
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "libvirt_volume.disk_debian_resized" references: [libvirt_pool.debian libvirt_volume.debian-qcow2]
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "data.template_file.user_data" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "data.template_file.network_config" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "libvirt_pool.debian" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.commoninit" references: [data.template_file.user_data data.template_file.network_config libvirt_pool.debian]
2020/04/21 19:42:50 [DEBUG] Starting graph walk: walkValidate
2020-04-21T19:42:50.393Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/terraform args=[/usr/local/bin/terraform, internal-plugin, provisioner, local-exec]
2020-04-21T19:42:50.393Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:42:50.393Z [DEBUG] plugin: plugin started: path=/usr/local/bin/terraform pid=11167
2020-04-21T19:42:50.394Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/terraform
2020-04-21T19:42:50.417Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
2020-04-21T19:42:50.417Z [DEBUG] plugin.terraform:   Use TF_LOG=TRACE to see Terraform's internal logs.
2020-04-21T19:42:50.417Z [DEBUG] plugin.terraform:   ----
2020-04-21T19:42:50.441Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [INFO] Terraform version: 0.12.24  
2020-04-21T19:42:50.441Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [INFO] Go runtime version: go1.12.13
2020-04-21T19:42:50.441Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [INFO] CLI args: []string{"/usr/local/bin/terraform", "internal-plugin", "provisioner", "local-exec"}
2020-04-21T19:42:50.441Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [DEBUG] Attempting to open CLI config file: /home/vrh/.terraformrc
2020-04-21T19:42:50.441Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020-04-21T19:42:50.441Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [DEBUG] checking for credentials in "/home/vrh/.terraform.d/plugins"
2020-04-21T19:42:50.442Z [DEBUG] plugin.terraform: 2020/04/21 19:42:50 [INFO] CLI command args: []string{"internal-plugin", "provisioner", "local-exec"}
2020-04-21T19:42:50.442Z [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2020/04/21 19:42:50 [INFO] Starting provisioner plugin local-exec
2020-04-21T19:42:50.443Z [DEBUG] plugin.terraform: plugin address: address=/tmp/plugin447596451 network=unix timestamp=2020-04-21T19:42:50.442Z
2020-04-21T19:42:50.443Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:50.466Z [DEBUG] plugin: starting plugin: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 args=[/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4]
2020-04-21T19:42:50.466Z [DEBUG] plugin: plugin started: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11188
2020-04-21T19:42:50.466Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
2020-04-21T19:42:50.478Z [INFO]  plugin.terraform-provider-template_v2.1.2_x4: configuring server automatic mTLS: timestamp=2020-04-21T19:42:50.477Z
2020-04-21T19:42:50.550Z [DEBUG] plugin.terraform-provider-template_v2.1.2_x4: plugin address: address=/tmp/plugin704194620 network=unix timestamp=2020-04-21T19:42:50.550Z
2020-04-21T19:42:50.550Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:50.551Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:42:50.623Z [DEBUG] plugin: starting plugin: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt args=[/home/vrh/.terraform.d/plugins/terraform-provider-libvirt]
2020-04-21T19:42:50.624Z [DEBUG] plugin: plugin started: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt pid=11199
2020-04-21T19:42:50.624Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt
2020-04-21T19:42:50.666Z [INFO]  plugin.terraform-provider-libvirt: configuring server automatic mTLS: timestamp=2020-04-21T19:42:50.666Z
2020-04-21T19:42:50.733Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:50.733Z [DEBUG] plugin.terraform-provider-libvirt: plugin address: address=/tmp/plugin999691313 network=unix timestamp=2020-04-21T19:42:50.733Z
2020-04-21T19:42:50.740Z [DEBUG] plugin: plugin process exited: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11188
2020-04-21T19:42:50.740Z [DEBUG] plugin: plugin exited
2020-04-21T19:42:50.903Z [DEBUG] plugin: plugin process exited: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt pid=11199
2020-04-21T19:42:50.903Z [DEBUG] plugin: plugin exited
2020-04-21T19:42:50.903Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=11167
2020-04-21T19:42:50.903Z [DEBUG] plugin: plugin exited
2020/04/21 19:42:50 [INFO] backend/local: apply calling Refresh
2020/04/21 19:42:50 [INFO] terraform: building graph: GraphTypeRefresh
2020/04/21 19:42:50 [DEBUG] adding implicit provider configuration provider.template, implied first by data.template_file.user_data
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "data.template_file.network_config" (*terraform.NodeRefreshableDataResource) needs provider.template
2020/04/21 19:42:50 [DEBUG] ProviderTransformer: "data.template_file.user_data" (*terraform.NodeRefreshableDataResource) needs provider.template
2020/04/21 19:42:50 [DEBUG] pruning unused provider.libvirt
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "data.template_file.user_data" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "data.template_file.network_config" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "output.ips" references: []
2020/04/21 19:42:50 [DEBUG] ReferenceTransformer: "provider.template" references: []
2020/04/21 19:42:50 [DEBUG] Starting graph walk: walkRefresh
2020-04-21T19:42:50.906Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:42:50.972Z [DEBUG] plugin: starting plugin: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 args=[/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4]
2020-04-21T19:42:50.973Z [DEBUG] plugin: plugin started: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11211
2020-04-21T19:42:50.973Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
2020-04-21T19:42:50.983Z [INFO]  plugin.terraform-provider-template_v2.1.2_x4: configuring server automatic mTLS: timestamp=2020-04-21T19:42:50.983Z
2020-04-21T19:42:51.056Z [DEBUG] plugin.terraform-provider-template_v2.1.2_x4: plugin address: address=/tmp/plugin715908654 network=unix timestamp=2020-04-21T19:42:51.055Z
2020-04-21T19:42:51.056Z [DEBUG] plugin: using plugin: version=5
2020/04/21 19:42:51 [DEBUG] Resource state not found for node "data.template_file.user_data", instance data.template_file.user_data
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "data.template_file.user_data" references: []
2020/04/21 19:42:51 [DEBUG] Resource state not found for node "data.template_file.network_config", instance data.template_file.network_config
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "data.template_file.network_config" references: []
[0m[1mdata.template_file.network_config: Refreshing state...[0m
[0m[1mdata.template_file.user_data: Refreshing state...[0m
2020-04-21T19:42:51.207Z [DEBUG] plugin: plugin process exited: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11211
2020-04-21T19:42:51.207Z [DEBUG] plugin: plugin exited
2020/04/21 19:42:51 [INFO] backend/local: apply calling Plan
2020/04/21 19:42:51 [INFO] terraform: building graph: GraphTypePlan
2020/04/21 19:42:51 [DEBUG] adding implicit provider configuration provider.template, implied first by data.template_file.user_data
2020/04/21 19:42:51 [DEBUG] ProviderTransformer: "libvirt_cloudinit_disk.commoninit" (*terraform.NodePlannableResource) needs provider.libvirt
2020/04/21 19:42:51 [DEBUG] ProviderTransformer: "libvirt_domain.domain-debian" (*terraform.NodePlannableResource) needs provider.libvirt
2020/04/21 19:42:51 [DEBUG] ProviderTransformer: "libvirt_pool.debian" (*terraform.NodePlannableResource) needs provider.libvirt
2020/04/21 19:42:51 [DEBUG] ProviderTransformer: "data.template_file.user_data" (*terraform.NodePlannableResource) needs provider.template
2020/04/21 19:42:51 [DEBUG] ProviderTransformer: "data.template_file.network_config" (*terraform.NodePlannableResource) needs provider.template
2020/04/21 19:42:51 [DEBUG] ProviderTransformer: "libvirt_volume.debian-qcow2" (*terraform.NodePlannableResource) needs provider.libvirt
2020/04/21 19:42:51 [DEBUG] ProviderTransformer: "libvirt_volume.disk_debian_resized" (*terraform.NodePlannableResource) needs provider.libvirt
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_volume.debian-qcow2" references: [libvirt_pool.debian]
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_volume.disk_debian_resized" references: [libvirt_pool.debian libvirt_volume.debian-qcow2]
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "output.ips" references: [libvirt_domain.domain-debian]
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "provisioner.local-exec" references: []
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "provider.template" references: []
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "provider.libvirt" references: []
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.commoninit" references: [data.template_file.network_config libvirt_pool.debian data.template_file.user_data]
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_domain.domain-debian" references: [libvirt_cloudinit_disk.commoninit libvirt_volume.disk_debian_resized]
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_pool.debian" references: []
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "data.template_file.user_data" references: []
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "data.template_file.network_config" references: []
2020/04/21 19:42:51 [DEBUG] Starting graph walk: walkPlan
2020-04-21T19:42:51.213Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/terraform args=[/usr/local/bin/terraform, internal-plugin, provisioner, local-exec]
2020-04-21T19:42:51.213Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:42:51.228Z [DEBUG] plugin: plugin started: path=/usr/local/bin/terraform pid=11224
2020-04-21T19:42:51.228Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/terraform
2020-04-21T19:42:51.238Z [DEBUG] plugin.terraform: 2020/04/21 19:42:51 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
2020-04-21T19:42:51.238Z [DEBUG] plugin.terraform:   Use TF_LOG=TRACE to see Terraform's internal logs.
2020-04-21T19:42:51.238Z [DEBUG] plugin.terraform:   ----
2020-04-21T19:42:51.266Z [DEBUG] plugin.terraform: 2020/04/21 19:42:51 [INFO] Terraform version: 0.12.24  
2020-04-21T19:42:51.267Z [DEBUG] plugin.terraform: 2020/04/21 19:42:51 [INFO] Go runtime version: go1.12.13
2020-04-21T19:42:51.267Z [DEBUG] plugin.terraform: 2020/04/21 19:42:51 [INFO] CLI args: []string{"/usr/local/bin/terraform", "internal-plugin", "provisioner", "local-exec"}
2020-04-21T19:42:51.267Z [DEBUG] plugin.terraform: 2020/04/21 19:42:51 [DEBUG] Attempting to open CLI config file: /home/vrh/.terraformrc
2020-04-21T19:42:51.267Z [DEBUG] plugin.terraform: 2020/04/21 19:42:51 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020-04-21T19:42:51.267Z [DEBUG] plugin.terraform: 2020/04/21 19:42:51 [DEBUG] checking for credentials in "/home/vrh/.terraform.d/plugins"
2020-04-21T19:42:51.268Z [DEBUG] plugin.terraform: 2020/04/21 19:42:51 [INFO] CLI command args: []string{"internal-plugin", "provisioner", "local-exec"}
2020-04-21T19:42:51.268Z [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2020/04/21 19:42:51 [INFO] Starting provisioner plugin local-exec
2020-04-21T19:42:51.268Z [DEBUG] plugin.terraform: plugin address: address=/tmp/plugin082414671 network=unix timestamp=2020-04-21T19:42:51.268Z
2020-04-21T19:42:51.268Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:51.285Z [DEBUG] plugin: starting plugin: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt args=[/home/vrh/.terraform.d/plugins/terraform-provider-libvirt]
2020-04-21T19:42:51.286Z [DEBUG] plugin: plugin started: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt pid=11243
2020-04-21T19:42:51.286Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt
2020-04-21T19:42:51.326Z [INFO]  plugin.terraform-provider-libvirt: configuring server automatic mTLS: timestamp=2020-04-21T19:42:51.326Z
2020-04-21T19:42:51.393Z [DEBUG] plugin.terraform-provider-libvirt: plugin address: address=/tmp/plugin892775239 network=unix timestamp=2020-04-21T19:42:51.393Z
2020-04-21T19:42:51.394Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:51.394Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:42:51.468Z [DEBUG] plugin: starting plugin: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 args=[/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4]
2020-04-21T19:42:51.468Z [DEBUG] plugin: plugin started: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11256
2020-04-21T19:42:51.468Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
2020-04-21T19:42:51.480Z [INFO]  plugin.terraform-provider-template_v2.1.2_x4: configuring server automatic mTLS: timestamp=2020-04-21T19:42:51.480Z
2020-04-21T19:42:51.551Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:42:51.551Z [DEBUG] plugin.terraform-provider-template_v2.1.2_x4: plugin address: address=/tmp/plugin996603373 network=unix timestamp=2020-04-21T19:42:51.551Z
2020-04-21T19:42:51.553Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:42:51 [DEBUG] Configuring provider for 'qemu:///system': &{map[uri:0xc42057d040] <nil> <nil> 0xc42000cf20 map[] <nil> 0xc4204cc0c0 0xc4204cc000 0xc4204b86e0 false map[] {{0 0} 1} false false}
2020-04-21T19:42:51.555Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:42:51 [INFO] Created libvirt client
2020/04/21 19:42:51 [DEBUG] Resource instance state not found for node "libvirt_pool.debian", instance libvirt_pool.debian
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_pool.debian" references: []
2020/04/21 19:42:51 [DEBUG] Resource instance state not found for node "libvirt_volume.debian-qcow2", instance libvirt_volume.debian-qcow2
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_volume.debian-qcow2" references: []
2020/04/21 19:42:51 [DEBUG] Resource instance state not found for node "libvirt_volume.disk_debian_resized", instance libvirt_volume.disk_debian_resized
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_volume.disk_debian_resized" references: []
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "data.template_file.network_config" references: []
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "data.template_file.user_data" references: []
2020/04/21 19:42:51 [DEBUG] Resource instance state not found for node "libvirt_cloudinit_disk.commoninit", instance libvirt_cloudinit_disk.commoninit
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.commoninit" references: []
2020-04-21T19:42:51.698Z [DEBUG] plugin: plugin process exited: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11256
2020-04-21T19:42:51.698Z [DEBUG] plugin: plugin exited
2020/04/21 19:42:51 [DEBUG] Resource instance state not found for node "libvirt_domain.domain-debian", instance libvirt_domain.domain-debian
2020/04/21 19:42:51 [DEBUG] ReferenceTransformer: "libvirt_domain.domain-debian" references: []
2020/04/21 19:42:51 [WARN] Provider "registry.terraform.io/-/libvirt" produced an invalid plan for libvirt_domain.domain-debian, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .running: planned value cty.True does not match config value cty.NullVal(cty.Bool)
      - .fw_cfg_name: planned value cty.StringVal("opt/com.coreos/config") does not match config value cty.NullVal(cty.String)
      - .graphics[0].listen_address: planned value cty.StringVal("127.0.0.1") does not match config value cty.NullVal(cty.String)
      - .console[0].source_host: planned value cty.StringVal("127.0.0.1") does not match config value cty.NullVal(cty.String)
      - .console[0].source_service: planned value cty.StringVal("0") does not match config value cty.NullVal(cty.String)
      - .console[1].source_host: planned value cty.StringVal("127.0.0.1") does not match config value cty.NullVal(cty.String)
      - .console[1].source_service: planned value cty.StringVal("0") does not match config value cty.NullVal(cty.String)
2020-04-21T19:42:51.719Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:42:51 [DEBUG] cleaning up connection for URI: qemu:///system
2020-04-21T19:42:51.724Z [DEBUG] plugin: plugin process exited: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt pid=11243
2020-04-21T19:42:51.724Z [DEBUG] plugin: plugin exited
2020-04-21T19:42:51.726Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=11224
2020-04-21T19:42:51.726Z [DEBUG] plugin: plugin exited

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  [32m+[0m create
[0m
Terraform will perform the following actions:

[1m  # libvirt_cloudinit_disk.commoninit[0m will be created[0m[0m
[0m  [32m+[0m[0m resource "libvirt_cloudinit_disk" "commoninit" {
      [32m+[0m [0m[1m[0mid[0m[0m             = (known after apply)
      [32m+[0m [0m[1m[0mname[0m[0m           = "commoninit.iso"
      [32m+[0m [0m[1m[0mnetwork_config[0m[0m = <<~EOT
            #for virtio nic ethX
            #for en100X nic ensX
            #network-interfaces: |
            #  auto eth0
            #  iface eth0 inet static
            #  address 192.168.122.102
            #  network 192.168.122.0
            #  netmask 255.255.255.0
            #  broadcast 192.168.122.255
            #  gateway 192.168.122.1
            #  dns-nameservers 192.168.122.1 8.8.8.8
            #bootcmd:
            #  - ifdown eth0
            #  - ifup eth0

            version: 2
            ethernets:
              eth0:
                dhcp4: true
        EOT
      [32m+[0m [0m[1m[0mpool[0m[0m           = "debian"
      [32m+[0m [0m[1m[0muser_data[0m[0m      = <<~EOT
            #cloud-config
            hostname: gitlab
            fqdn: gitlab.local
            manage_etc_hosts: True
            growpart:
              mode: auto
              devices: ['/']
            packages:
              - qemu-guest-agent
            #power_state:
            # delay: "+3"
            # mode: reboot
            # message: Bye Bye
            # timeout: 30
            # condition: True
            users:
              - name: devops
                sudo: ALL=(ALL) NOPASSWD:ALL
                groups: users, admin
                home: /home/devops
                shell: /bin/bash
                ssh_authorized_keys:
                  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwRPdSosCYqP7ButhCIZS0aqF2lM5O/jsvvI/ddv1PWsP3Z6/3WxIvTCiBfNDmRLo1fta/lK3HZ5kaEkxarxxKjV02KtolB5t0MhUKxCKFnQM5b3lbp476gCxVolUHW9GDN/bKZ7c4mN4Wzfeig8h2lw+ttPNejgvm2HFpEwvln8yQ6P0Ledv7bX3TQuqgUjjp1FMdBA9b8Ug4cIl3OaBQTxVBgcVgWOC3UHShDAz/jT1uefyjVGRowTKOiFyyFcs+mQKUTD7peKk95IQ15fy8YBLUxxwsz9ilXgPfwk3AdlVpu22lHCweAwoosULc+/uDYWzk0fx5McLUC5gezPql vrh@node2
            final_message: "The system is finally up, after $UPTIME seconds"
        EOT
    }

[1m  # libvirt_domain.domain-debian[0m will be created[0m[0m
[0m  [32m+[0m[0m resource "libvirt_domain" "domain-debian" {
      [32m+[0m [0m[1m[0march[0m[0m        = (known after apply)
      [32m+[0m [0m[1m[0mcloudinit[0m[0m   = (known after apply)
      [32m+[0m [0m[1m[0memulator[0m[0m    = (known after apply)
      [32m+[0m [0m[1m[0mfw_cfg_name[0m[0m = "opt/com.coreos/config"
      [32m+[0m [0m[1m[0mid[0m[0m          = (known after apply)
      [32m+[0m [0m[1m[0mmachine[0m[0m     = (known after apply)
      [32m+[0m [0m[1m[0mmemory[0m[0m      = 2048
      [32m+[0m [0m[1m[0mname[0m[0m        = "debian-terraform"
      [32m+[0m [0m[1m[0mqemu_agent[0m[0m  = true
      [32m+[0m [0m[1m[0mrunning[0m[0m     = true
      [32m+[0m [0m[1m[0mvcpu[0m[0m        = 2

      [32m+[0m [0mconsole {
          [32m+[0m [0m[1m[0msource_host[0m[0m    = "127.0.0.1"
          [32m+[0m [0m[1m[0msource_service[0m[0m = "0"
          [32m+[0m [0m[1m[0mtarget_port[0m[0m    = "0"
          [32m+[0m [0m[1m[0mtarget_type[0m[0m    = "serial"
          [32m+[0m [0m[1m[0mtype[0m[0m           = "pty"
        }
      [32m+[0m [0mconsole {
          [32m+[0m [0m[1m[0msource_host[0m[0m    = "127.0.0.1"
          [32m+[0m [0m[1m[0msource_service[0m[0m = "0"
          [32m+[0m [0m[1m[0mtarget_port[0m[0m    = "1"
          [32m+[0m [0m[1m[0mtarget_type[0m[0m    = "virtio"
2020/04/21 19:42:51 [DEBUG] command: asking for input: "Do you want to perform these actions?"
          [32m+[0m [0m[1m[0mtype[0m[0m           = "pty"
        }

      [32m+[0m [0mdisk {
          [32m+[0m [0m[1m[0mvolume_id[0m[0m = (known after apply)
        }

      [32m+[0m [0mgraphics {
          [32m+[0m [0m[1m[0mautoport[0m[0m       = true
          [32m+[0m [0m[1m[0mlisten_address[0m[0m = "127.0.0.1"
          [32m+[0m [0m[1m[0mlisten_type[0m[0m    = "address"
          [32m+[0m [0m[1m[0mtype[0m[0m           = "spice"
        }

      [32m+[0m [0mnetwork_interface {
          [32m+[0m [0m[1m[0maddresses[0m[0m    = (known after apply)
          [32m+[0m [0m[1m[0mbridge[0m[0m       = "br0"
          [32m+[0m [0m[1m[0mhostname[0m[0m     = (known after apply)
          [32m+[0m [0m[1m[0mmac[0m[0m          = (known after apply)
          [32m+[0m [0m[1m[0mnetwork_id[0m[0m   = (known after apply)
          [32m+[0m [0m[1m[0mnetwork_name[0m[0m = (known after apply)
        }
    }

[1m  # libvirt_pool.debian[0m will be created[0m[0m
[0m  [32m+[0m[0m resource "libvirt_pool" "debian" {
      [32m+[0m [0m[1m[0mallocation[0m[0m = (known after apply)
      [32m+[0m [0m[1m[0mavailable[0m[0m  = (known after apply)
      [32m+[0m [0m[1m[0mcapacity[0m[0m   = (known after apply)
      [32m+[0m [0m[1m[0mid[0m[0m         = (known after apply)
      [32m+[0m [0m[1m[0mname[0m[0m       = "debian"
      [32m+[0m [0m[1m[0mpath[0m[0m       = "/srv/terraform-provider-libvirt-pool-debian"
      [32m+[0m [0m[1m[0mtype[0m[0m       = "dir"
    }

[1m  # libvirt_volume.debian-qcow2[0m will be created[0m[0m
[0m  [32m+[0m[0m resource "libvirt_volume" "debian-qcow2" {
      [32m+[0m [0m[1m[0mformat[0m[0m = "qcow2"
      [32m+[0m [0m[1m[0mid[0m[0m     = (known after apply)
      [32m+[0m [0m[1m[0mname[0m[0m   = "debian-qcow2"
      [32m+[0m [0m[1m[0mpool[0m[0m   = "debian"
      [32m+[0m [0m[1m[0msize[0m[0m   = (known after apply)
      [32m+[0m [0m[1m[0msource[0m[0m = "https://cdimage.debian.org/cdimage/openstack/current/debian-10.3.2-20200406-openstack-amd64.qcow2"
    }

[1m  # libvirt_volume.disk_debian_resized[0m will be created[0m[0m
[0m  [32m+[0m[0m resource "libvirt_volume" "disk_debian_resized" {
      [32m+[0m [0m[1m[0mbase_volume_id[0m[0m = (known after apply)
      [32m+[0m [0m[1m[0mformat[0m[0m         = (known after apply)
      [32m+[0m [0m[1m[0mid[0m[0m             = (known after apply)
      [32m+[0m [0m[1m[0mname[0m[0m           = "disk"
      [32m+[0m [0m[1m[0mpool[0m[0m           = "debian"
      [32m+[0m [0m[1m[0msize[0m[0m           = 2147483648
    }

[0m[1mPlan:[0m 5 to add, 0 to change, 0 to destroy.[0m

[0m[1mDo you want to perform these actions?[0m
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  [1mEnter a value:[0m [0m
2020/04/21 19:43:00 [INFO] backend/local: apply calling Apply
2020/04/21 19:43:00 [INFO] terraform: building graph: GraphTypeApply
2020/04/21 19:43:00 [DEBUG] Resource state not found for node "libvirt_cloudinit_disk.commoninit", instance libvirt_cloudinit_disk.commoninit
2020/04/21 19:43:00 [DEBUG] Resource state not found for node "libvirt_domain.domain-debian", instance libvirt_domain.domain-debian
2020/04/21 19:43:00 [DEBUG] Resource state not found for node "libvirt_pool.debian", instance libvirt_pool.debian
2020/04/21 19:43:00 [DEBUG] Resource state not found for node "libvirt_volume.debian-qcow2", instance libvirt_volume.debian-qcow2
2020/04/21 19:43:00 [DEBUG] Resource state not found for node "libvirt_volume.disk_debian_resized", instance libvirt_volume.disk_debian_resized
2020/04/21 19:43:00 [DEBUG] adding implicit provider configuration provider.template, implied first by data.template_file.user_data (prepare state)
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_pool.debian" (*terraform.NodeApplyableResourceInstance) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_cloudinit_disk.commoninit (prepare state)" (*terraform.NodeApplyableResource) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_pool.debian (prepare state)" (*terraform.NodeApplyableResource) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "data.template_file.user_data (prepare state)" (*terraform.NodeApplyableResource) needs provider.template
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_volume.disk_debian_resized (prepare state)" (*terraform.NodeApplyableResource) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_cloudinit_disk.commoninit" (*terraform.NodeApplyableResourceInstance) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_domain.domain-debian (prepare state)" (*terraform.NodeApplyableResource) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "data.template_file.network_config (prepare state)" (*terraform.NodeApplyableResource) needs provider.template
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_volume.debian-qcow2 (prepare state)" (*terraform.NodeApplyableResource) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_domain.domain-debian" (*terraform.NodeApplyableResourceInstance) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_volume.debian-qcow2" (*terraform.NodeApplyableResourceInstance) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ProviderTransformer: "libvirt_volume.disk_debian_resized" (*terraform.NodeApplyableResourceInstance) needs provider.libvirt
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "data.template_file.network_config (prepare state)" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_pool.debian" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "provisioner.local-exec" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_pool.debian (prepare state)" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_volume.debian-qcow2 (prepare state)" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.commoninit (prepare state)" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_domain.domain-debian" references: [libvirt_cloudinit_disk.commoninit (prepare state) libvirt_cloudinit_disk.commoninit libvirt_cloudinit_disk.commoninit libvirt_volume.disk_debian_resized (prepare state) libvirt_volume.disk_debian_resized libvirt_volume.disk_debian_resized]
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_volume.disk_debian_resized (prepare state)" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "data.template_file.user_data (prepare state)" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_volume.debian-qcow2" references: [libvirt_pool.debian libvirt_pool.debian libvirt_pool.debian (prepare state)]
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_volume.disk_debian_resized" references: [libvirt_volume.debian-qcow2 (prepare state) libvirt_volume.debian-qcow2 libvirt_volume.debian-qcow2 libvirt_pool.debian libvirt_pool.debian libvirt_pool.debian (prepare state)]
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "provider.libvirt" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "provider.template" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_domain.domain-debian (prepare state)" references: []
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "libvirt_cloudinit_disk.commoninit" references: [data.template_file.network_config (prepare state) libvirt_pool.debian libvirt_pool.debian libvirt_pool.debian (prepare state) data.template_file.user_data (prepare state)]
2020/04/21 19:43:00 [DEBUG] ReferenceTransformer: "output.ips" references: [libvirt_domain.domain-debian libvirt_domain.domain-debian libvirt_domain.domain-debian (prepare state)]
2020/04/21 19:43:00 [DEBUG] Starting graph walk: walkApply
2020-04-21T19:43:00.253Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:43:00.253Z [DEBUG] plugin: starting plugin: path=/usr/local/bin/terraform args=[/usr/local/bin/terraform, internal-plugin, provisioner, local-exec]
2020-04-21T19:43:00.254Z [DEBUG] plugin: plugin started: path=/usr/local/bin/terraform pid=11268
2020-04-21T19:43:00.254Z [DEBUG] plugin: waiting for RPC address: path=/usr/local/bin/terraform
2020-04-21T19:43:00.273Z [DEBUG] plugin.terraform: 2020/04/21 19:43:00 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
2020-04-21T19:43:00.273Z [DEBUG] plugin.terraform:   Use TF_LOG=TRACE to see Terraform's internal logs.
2020-04-21T19:43:00.273Z [DEBUG] plugin.terraform:   ----
2020-04-21T19:43:00.301Z [DEBUG] plugin.terraform: 2020/04/21 19:43:00 [INFO] Terraform version: 0.12.24  
2020-04-21T19:43:00.301Z [DEBUG] plugin.terraform: 2020/04/21 19:43:00 [INFO] Go runtime version: go1.12.13
2020-04-21T19:43:00.301Z [DEBUG] plugin.terraform: 2020/04/21 19:43:00 [INFO] CLI args: []string{"/usr/local/bin/terraform", "internal-plugin", "provisioner", "local-exec"}
2020-04-21T19:43:00.301Z [DEBUG] plugin.terraform: 2020/04/21 19:43:00 [DEBUG] Attempting to open CLI config file: /home/vrh/.terraformrc
2020-04-21T19:43:00.301Z [DEBUG] plugin.terraform: 2020/04/21 19:43:00 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020-04-21T19:43:00.302Z [DEBUG] plugin.terraform: 2020/04/21 19:43:00 [DEBUG] checking for credentials in "/home/vrh/.terraform.d/plugins"
2020-04-21T19:43:00.302Z [DEBUG] plugin.terraform: 2020/04/21 19:43:00 [INFO] CLI command args: []string{"internal-plugin", "provisioner", "local-exec"}
2020-04-21T19:43:00.303Z [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2020/04/21 19:43:00 [INFO] Starting provisioner plugin local-exec
2020-04-21T19:43:00.303Z [DEBUG] plugin.terraform: plugin address: address=/tmp/plugin419688592 network=unix timestamp=2020-04-21T19:43:00.303Z
2020-04-21T19:43:00.304Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:43:00.321Z [DEBUG] plugin: starting plugin: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 args=[/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4]
2020-04-21T19:43:00.322Z [DEBUG] plugin: plugin started: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11289
2020-04-21T19:43:00.322Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4
2020-04-21T19:43:00.333Z [INFO]  plugin.terraform-provider-template_v2.1.2_x4: configuring server automatic mTLS: timestamp=2020-04-21T19:43:00.333Z
2020-04-21T19:43:00.405Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:43:00.405Z [DEBUG] plugin.terraform-provider-template_v2.1.2_x4: plugin address: address=/tmp/plugin018212457 network=unix timestamp=2020-04-21T19:43:00.404Z
2020-04-21T19:43:00.405Z [INFO]  plugin: configuring client automatic mTLS
2020-04-21T19:43:00.477Z [DEBUG] plugin: starting plugin: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt args=[/home/vrh/.terraform.d/plugins/terraform-provider-libvirt]
2020-04-21T19:43:00.477Z [DEBUG] plugin: plugin started: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt pid=11300
2020-04-21T19:43:00.477Z [DEBUG] plugin: waiting for RPC address: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt
2020-04-21T19:43:00.518Z [INFO]  plugin.terraform-provider-libvirt: configuring server automatic mTLS: timestamp=2020-04-21T19:43:00.518Z
2020-04-21T19:43:00.587Z [DEBUG] plugin: using plugin: version=5
2020-04-21T19:43:00.587Z [DEBUG] plugin.terraform-provider-libvirt: plugin address: address=/tmp/plugin011232384 network=unix timestamp=2020-04-21T19:43:00.586Z
2020-04-21T19:43:00.592Z [DEBUG] plugin: plugin process exited: path=/home/vrh/kvm/devops-demo/terraform/gitlab/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=11289
2020-04-21T19:43:00.592Z [DEBUG] plugin: plugin exited
2020-04-21T19:43:00.734Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 [DEBUG] Configuring provider for 'qemu:///system': &{map[uri:0xc420143900] <nil> <nil> 0xc4201b7ba0 map[] <nil> 0xc4201b7d40 0xc4201b7c60 0xc4204e9400 false map[] {{0 0} 1} false false}
2020-04-21T19:43:00.737Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 [INFO] Created libvirt client
[0m[1mlibvirt_pool.debian: Creating...[0m[0m
2020/04/21 19:43:00 [DEBUG] libvirt_pool.debian: applying the planned Create change
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 [DEBUG] Locking "debian"
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 [DEBUG] Locked "debian"
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 [DEBUG] Pool with name 'debian' does not exist yet
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 [DEBUG] Generated XML for libvirt storage pool:
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt:   <pool type="dir">
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt:       <name>debian</name>
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt:       <target>
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt:           <path>/srv/terraform-provider-libvirt-pool-debian</path>
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt:       </target>
2020-04-21T19:43:00.743Z [DEBUG] plugin.terraform-provider-libvirt:   </pool>
2020-04-21T19:43:00.750Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 [INFO] Pool ID: 9db809ef-1488-4154-81d8-40a0de60a2e4
2020-04-21T19:43:00.750Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 Waiting for pool 9db809ef-1488-4154-81d8-40a0de60a2e4 to be active...
2020-04-21T19:43:00.750Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:00 [DEBUG] Waiting for state to become: [EXISTS]
2020-04-21T19:43:05.754Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 [DEBUG] Pool debian path: /srv/terraform-provider-libvirt-pool-debian
2020-04-21T19:43:05.755Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 [DEBUG] Unlocking "debian"
2020-04-21T19:43:05.755Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 [DEBUG] Unlocked "debian"
[0m[1mlibvirt_pool.debian: Creation complete after 5s [id=9db809ef-1488-4154-81d8-40a0de60a2e4][0m[0m
[0m[1mlibvirt_cloudinit_disk.commoninit: Creating...[0m[0m
[0m[1mlibvirt_volume.debian-qcow2: Creating...[0m[0m
2020/04/21 19:43:05 [DEBUG] libvirt_volume.debian-qcow2: applying the planned Create change
2020/04/21 19:43:05 [DEBUG] libvirt_cloudinit_disk.commoninit: applying the planned Create change
2020-04-21T19:43:05.773Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 [DEBUG] creating cloudinit
2020-04-21T19:43:05.773Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 [INFO] cloudInit: {Name:commoninit.iso PoolName:debian MetaData: UserData:#cloud-config
2020-04-21T19:43:05.773Z [DEBUG] plugin.terraform-provider-libvirt: hostname: gitlab
2020-04-21T19:43:05.773Z [DEBUG] plugin.terraform-provider-libvirt: fqdn: gitlab.local
2020-04-21T19:43:05.773Z [DEBUG] plugin.terraform-provider-libvirt: manage_etc_hosts: True
2020-04-21T19:43:05.773Z [DEBUG] plugin.terraform-provider-libvirt: growpart:
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt:   mode: auto
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt:   devices: ['/']
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt: packages:
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt:   - qemu-guest-agent
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt: #power_state:
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt: # delay: "+3"
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt: # mode: reboot
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt: # message: Bye Bye
2020-04-21T19:43:05.774Z [DEBUG] plugin.terraform-provider-libvirt: # timeout: 30
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt: # condition: True
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt: users:
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt:   - name: devops
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt:     sudo: ALL=(ALL) NOPASSWD:ALL
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt:     groups: users, admin
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt:     home: /home/devops
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt:     shell: /bin/bash
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt:     ssh_authorized_keys:
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt:       - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwRPdSosCYqP7ButhCIZS0aqF2lM5O/jsvvI/ddv1PWsP3Z6/3WxIvTCiBfNDmRLo1fta/lK3HZ5kaEkxarxxKjV02KtolB5t0MhUKxCKFnQM5b3lbp476gCxVolUHW9GDN/bKZ7c4mN4Wzfeig8h2lw+ttPNejgvm2HFpEwvln8yQ6P0Ledv7bX3TQuqgUjjp1FMdBA9b8Ug4cIl3OaBQTxVBgcVgWOC3UHShDAz/jT1uefyjVGRowTKOiFyyFcs+mQKUTD7peKk95IQ15fy8YBLUxxwsz9ilXgPfwk3AdlVpu22lHCweAwoosULc+/uDYWzk0fx5McLUC5gezPql vrh@node2
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt: final_message: "The system is finally up, after $UPTIME seconds"
2020-04-21T19:43:05.775Z [DEBUG] plugin.terraform-provider-libvirt:  NetworkConfig:#for virtio nic ethX
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #for en100X nic ensX
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #network-interfaces: |
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  auto eth0
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  iface eth0 inet static
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  address 192.168.122.102
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  network 192.168.122.0
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  netmask 255.255.255.0
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  broadcast 192.168.122.255
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  gateway 192.168.122.1
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  dns-nameservers 192.168.122.1 8.8.8.8
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #bootcmd:
2020-04-21T19:43:05.776Z [DEBUG] plugin.terraform-provider-libvirt: #  - ifdown eth0
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: #  - ifup eth0
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: 
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: version: 2
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: ethernets:
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt:   eth0:
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt:     dhcp4: true
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: }
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 Creating new ISO
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 Creating ISO contents
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 [DEBUG] Locking "debian"
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 [DEBUG] Locked "debian"
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 ISO contents created
2020-04-21T19:43:05.777Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 About to execute cmd: &{Path:/usr/bin/mkisofs Args:[mkisofs -output /tmp/cloudinit979276255/commoninit.iso -volid cidata -joliet -rock /tmp/cloudinit979276255/user-data /tmp/cloudinit979276255/meta-data /tmp/cloudinit979276255/network-config] Env:[] Dir: Stdin:<nil> Stdout:<nil> Stderr:<nil> ExtraFiles:[] SysProcAttr:<nil> Process:<nil> ProcessState:<nil> ctx:<nil> lookPathErr:<nil> finished:false childFiles:[] closeAfterStart:[] closeAfterWait:[] goroutine:[] errch:<nil> waitDone:<nil>}
2020-04-21T19:43:05.780Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 ISO created at /tmp/cloudinit979276255/commoninit.iso
2020-04-21T19:43:05.781Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:05 [DEBUG] Locking "debian"
2020-04-21T19:43:06.484Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:06 Image https://cdimage.debian.org/cdimage/openstack/current/debian-10.3.2-20200406-openstack-amd64.qcow2 image is: 571451904 bytes
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:06 [DEBUG] Generated XML for libvirt volume:
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:   <volume>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:       <name>debian-qcow2</name>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:       <capacity unit="B">571451904</capacity>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:       <target>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:           <format type="qcow2"></format>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:           <permissions>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:               <mode>644</mode>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:           </permissions>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:       </target>
2020-04-21T19:43:06.485Z [DEBUG] plugin.terraform-provider-libvirt:   </volume>
2020-04-21T19:43:06.526Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:06 [INFO] Volume ID: /srv/terraform-provider-libvirt-pool-debian/debian-qcow2
2020-04-21T19:43:06.634Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:06 [DEBUG]: url resp status code 200 OK (retry #0)
[0m[1mlibvirt_volume.debian-qcow2: Still creating... [10s elapsed][0m[0m
[0m[1mlibvirt_cloudinit_disk.commoninit: Still creating... [10s elapsed][0m[0m
2020-04-21T19:43:17.444Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 571451904 bytes uploaded
2020-04-21T19:43:17.445Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 Waiting for volume /srv/terraform-provider-libvirt-pool-debian/debian-qcow2 to be active...
2020-04-21T19:43:17.445Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Waiting for state to become: [EXISTS]
2020-04-21T19:43:17.447Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Volume debian-qcow2 format: qcow2
2020-04-21T19:43:17.447Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Unlocking "debian"
2020-04-21T19:43:17.447Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Unlocked "debian"
2020-04-21T19:43:17.447Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Locked "debian"
[0m[1mlibvirt_volume.debian-qcow2: Creation complete after 11s [id=/srv/terraform-provider-libvirt-pool-debian/debian-qcow2][0m[0m
[0m[1mlibvirt_volume.disk_debian_resized: Creating...[0m[0m
2020-04-21T19:43:17.459Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 374784 bytes uploaded
2020/04/21 19:43:17 [DEBUG] libvirt_volume.disk_debian_resized: applying the planned Create change
2020-04-21T19:43:17.460Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Unlocking "debian"
2020-04-21T19:43:17.460Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Unlocked "debian"
2020-04-21T19:43:17.461Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Locking "debian"
2020-04-21T19:43:17.461Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Locked "debian"
2020-04-21T19:43:17.465Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 374784 bytes downloaded
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 ISO reader: processing file /meta_dat.
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Generated XML for libvirt volume:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:   <volume>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:       <name>disk</name>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:       <capacity unit="bytes">2147483648</capacity>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:       <target>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:           <format type="qcow2"></format>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:           <permissions>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:               <mode>644</mode>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:           </permissions>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:       </target>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:       <backingStore>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:           <path>/srv/terraform-provider-libvirt-pool-debian/debian-qcow2</path>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:           <format type="qcow2"></format>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:       </backingStore>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:   </volume>
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 ISO reader: processing file /network_.
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 ISO reader: processing file /user_dat.
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG]: Read cloud-init from file: &{Name:commoninit.iso PoolName:debian MetaData: UserData:#cloud-config
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: hostname: gitlab
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: fqdn: gitlab.local
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: manage_etc_hosts: True
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: growpart:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:   mode: auto
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:   devices: ['/']
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: packages:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:   - qemu-guest-agent
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #power_state:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: # delay: "+3"
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: # mode: reboot
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: # message: Bye Bye
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: # timeout: 30
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: # condition: True
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: users:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:   - name: devops
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:     sudo: ALL=(ALL) NOPASSWD:ALL
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:     groups: users, admin
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:     home: /home/devops
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:     shell: /bin/bash
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:     ssh_authorized_keys:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:       - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwRPdSosCYqP7ButhCIZS0aqF2lM5O/jsvvI/ddv1PWsP3Z6/3WxIvTCiBfNDmRLo1fta/lK3HZ5kaEkxarxxKjV02KtolB5t0MhUKxCKFnQM5b3lbp476gCxVolUHW9GDN/bKZ7c4mN4Wzfeig8h2lw+ttPNejgvm2HFpEwvln8yQ6P0Ledv7bX3TQuqgUjjp1FMdBA9b8Ug4cIl3OaBQTxVBgcVgWOC3UHShDAz/jT1uefyjVGRowTKOiFyyFcs+mQKUTD7peKk95IQ15fy8YBLUxxwsz9ilXgPfwk3AdlVpu22lHCweAwoosULc+/uDYWzk0fx5McLUC5gezPql vrh@node2
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: final_message: "The system is finally up, after $UPTIME seconds"
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:  NetworkConfig:#for virtio nic ethX
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #for en100X nic ensX
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #network-interfaces: |
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  auto eth0
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  iface eth0 inet static
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  address 192.168.122.102
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  network 192.168.122.0
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  netmask 255.255.255.0
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  broadcast 192.168.122.255
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  gateway 192.168.122.1
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  dns-nameservers 192.168.122.1 8.8.8.8
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #bootcmd:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  - ifdown eth0
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: #  - ifup eth0
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: 
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: version: 2
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: ethernets:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:   eth0:
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt:     dhcp4: true
2020-04-21T19:43:17.466Z [DEBUG] plugin.terraform-provider-libvirt: }
2020/04/21 19:43:17 [WARN] Provider "registry.terraform.io/-/libvirt" produced an unexpected new value for libvirt_cloudinit_disk.commoninit, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .meta_data: was null, but now cty.StringVal("")
[0m[1mlibvirt_cloudinit_disk.commoninit: Creation complete after 11s [id=/srv/terraform-provider-libvirt-pool-debian/commoninit.iso;5e9f4cd5-46b8-8487-c050-ff6dc27525db][0m[0m
2020-04-21T19:43:17.501Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [INFO] Volume ID: /srv/terraform-provider-libvirt-pool-debian/disk
2020-04-21T19:43:17.501Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 Waiting for volume /srv/terraform-provider-libvirt-pool-debian/disk to be active...
2020-04-21T19:43:17.501Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Waiting for state to become: [EXISTS]
2020-04-21T19:43:17.503Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Volume disk format: qcow2
2020-04-21T19:43:17.503Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Unlocking "debian"
2020-04-21T19:43:17.503Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Unlocked "debian"
[0m[1mlibvirt_volume.disk_debian_resized: Creation complete after 1s [id=/srv/terraform-provider-libvirt-pool-debian/disk][0m[0m
2020/04/21 19:43:17 [WARN] Provider "registry.terraform.io/-/libvirt" produced an invalid plan for libvirt_domain.domain-debian, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .fw_cfg_name: planned value cty.StringVal("opt/com.coreos/config") does not match config value cty.NullVal(cty.String)
      - .running: planned value cty.True does not match config value cty.NullVal(cty.Bool)
      - .graphics[0].listen_address: planned value cty.StringVal("127.0.0.1") does not match config value cty.NullVal(cty.String)
      - .console[0].source_host: planned value cty.StringVal("127.0.0.1") does not match config value cty.NullVal(cty.String)
      - .console[0].source_service: planned value cty.StringVal("0") does not match config value cty.NullVal(cty.String)
      - .console[1].source_service: planned value cty.StringVal("0") does not match config value cty.NullVal(cty.String)
      - .console[1].source_host: planned value cty.StringVal("127.0.0.1") does not match config value cty.NullVal(cty.String)
[0m[1mlibvirt_domain.domain-debian: Creating...[0m[0m
2020/04/21 19:43:17 [DEBUG] libvirt_domain.domain-debian: applying the planned Create change
2020-04-21T19:43:17.527Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] setting computed for "network_interface.0.addresses" from ComputedKeys
2020-04-21T19:43:17.527Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] setting computed for "network_interface.0.addresses" from ComputedKeys
2020-04-21T19:43:17.528Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Create resource libvirt_domain
2020-04-21T19:43:17.611Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [TRACE] Capabilities of host 
anonical:} {Name:pc-1.0 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.9 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.7 MaxCPUs:255 Canonical:} {Name:xenfv MaxCPUs:128 Canonical:} {Name:pc-i440fx-wily MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.3 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.5 MaxCPUs:255 Canonical:} {Name:pc-i440fx-yakkety MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.1 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.2 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.0 MaxCPUs:255 Canonical:} {Name:pc-q35-yakkety MaxCPUs:255 Canonical:} {Name:pc-i440fx-bionic-hpb MaxCPUs:255 Canonical:} {Name:pc-q35-2.11 MaxCPUs:288 Canonical:} {Name:q35 MaxCPUs:288 Canonical:pc-q35-2.11} {Name:pc-i440fx-xenial MaxCPUs:255 Canonical:} {Name:xenpv MaxCPUs:1 Canonical:} {Name:pc-q35-2.10 MaxCPUs:288 Canonical:} {Name:pc-q35-bionic-hpb MaxCPUs:288 Canonical:} {Name:pc-q35-xenial MaxCPUs:255 Canonical:} {Name:pc-i440fx-artful MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.7 MaxCPUs:255 Canonical:} {Name:pc-q35-2.9 MaxCPUs:288 Canonical:} {Name:pc-0.15 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.5 MaxCPUs:255 Canonical:} {Name:pc-q35-2.7 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.11 MaxCPUs:255 Canonical:} {Name:pc MaxCPUs:255 Canonical:pc-i440fx-2.11} {Name:pc-q35-2.8 MaxCPUs:288 Canonical:} {Name:pc-q35-zesty MaxCPUs:288 Canonical:} {Name:pc-0.13 MaxCPUs:255 Canonical:} {Name:pc-q35-artful MaxCPUs:288 Canonical:} {Name:pc-0.14 MaxCPUs:255 Canonical:} {Name:pc-q35-2.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-trusty MaxCPUs:255 Canonical:} {Name:pc-q35-2.5 MaxCPUs:255 Canonical:} {Name:pc-q35-2.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.10 MaxCPUs:255 Canonical:} {Name:pc-0.11 MaxCPUs:255 Canonical:} {Name:pc-0.12 MaxCPUs:255 Canonical:} {Name:pc-q35-bionic MaxCPUs:288 Canonical:} {Name:pc-0.10 MaxCPUs:255 Canonical:}]}]} Features:0xc420473180}]}
2020-04-21T19:43:17.611Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [TRACE] Checking for x86_64/hvm against i686/hvm
2020-04-21T19:43:17.611Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [TRACE] Checking for x86_64/hvm against x86_64/hvm
2020-04-21T19:43:17.611Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Found 54 machines in guest for x86_64/hvm
2020-04-21T19:43:17.611Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [TRACE] Checking for x86_64/hvm against i686/hvm
2020-04-21T19:43:17.611Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [TRACE] Checking for x86_64/hvm against x86_64/hvm
2020-04-21T19:43:17.611Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Found 54 machines in guest for x86_64/hvm
2020-04-21T19:43:17.641Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Setting disk driver to 'qcow2' to match disk volume format
2020-04-21T19:43:17.641Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] scsiDisk: false
2020-04-21T19:43:17.641Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 filesystems: []
2020-04-21T19:43:17.642Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [INFO] Creating libvirt domain at qemu:///system
2020-04-21T19:43:17.643Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:17 [DEBUG] Generated XML for libvirt domain:
2020-04-21T19:43:17.643Z [DEBUG] plugin.terraform-provider-libvirt:   <domain type="kvm">
2020-04-21T19:43:17.643Z [DEBUG] plugin.terraform-provider-libvirt:       <name>debian-terraform</name>
2020-04-21T19:43:17.643Z [DEBUG] plugin.terraform-provider-libvirt:       <memory unit="MiB">2048</memory>
2020-04-21T19:43:17.643Z [DEBUG] plugin.terraform-provider-libvirt:       <vcpu>2</vcpu>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:       <os>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <type>hvm</type>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:       </os>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:       <features>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <pae></pae>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <acpi></acpi>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <apic></apic>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:       </features>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:       <cpu></cpu>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:       <devices>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <disk type="volume" device="disk">
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <driver name="qemu" type="qcow2"></driver>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <source pool="debian" volume="disk"></source>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <target dev="vda" bus="virtio"></target>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           </disk>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <disk type="file" device="cdrom">
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <driver name="qemu" type="raw"></driver>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <source file="/srv/terraform-provider-libvirt-pool-debian/commoninit.iso"></source>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <target dev="hdd" bus="ide"></target>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           </disk>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <interface type="bridge">
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <mac address="52:54:00:a2:7e:8a"></mac>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <source bridge="br0"></source>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <model type="virtio"></model>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           </interface>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <console>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <target type="serial" port="0"></target>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           </console>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <console>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <target type="virtio" port="1"></target>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           </console>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <channel>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <target type="virtio" name="org.qemu.guest_agent.0"></target>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           </channel>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <graphics type="spice" autoport="yes">
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <listen type="address" address="127.0.0.1"></listen>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           </graphics>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           <rng model="virtio">
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:               <backend model="random">/dev/urandom</backend>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:           </rng>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:       </devices>
2020-04-21T19:43:17.644Z [DEBUG] plugin.terraform-provider-libvirt:   </domain>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [INFO] Domain ID: 2521c3e0-0c38-46d8-aa6b-14464eacd4f4
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [DEBUG] Read resource libvirt_domain
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [DEBUG] read: obtained XML desc for domain:
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt: <domain type='kvm' id='44'>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <name>debian-terraform</name>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <uuid>2521c3e0-0c38-46d8-aa6b-14464eacd4f4</uuid>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <memory unit='KiB'>2097152</memory>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <currentMemory unit='KiB'>2097152</currentMemory>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <vcpu placement='static'>2</vcpu>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <resource>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:     <partition>/machine</partition>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   </resource>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <os>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:     <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:     <boot dev='hd'/>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   </os>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <features>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:     <acpi/>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:     <apic/>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:     <pae/>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   </features>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <clock offset='utc'/>
2020-04-21T19:43:18.041Z [DEBUG] plugin.terraform-provider-libvirt:   <on_poweroff>destroy</on_poweroff>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:   <on_reboot>restart</on_reboot>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:   <on_crash>destroy</on_crash>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:   <devices>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <emulator>/usr/bin/kvm-spice</emulator>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <disk type='volume' device='disk'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <driver name='qemu' type='qcow2'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <source pool='debian' volume='disk'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <backingStore type='file' index='1'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:         <format type='qcow2'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:         <source file='/srv/terraform-provider-libvirt-pool-debian/debian-qcow2'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:         <backingStore/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       </backingStore>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <target dev='vda' bus='virtio'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='virtio-disk0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </disk>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <disk type='file' device='cdrom'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <driver name='qemu' type='raw'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <source file='/srv/terraform-provider-libvirt-pool-debian/commoninit.iso'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <backingStore/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <target dev='hdd' bus='ide'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <readonly/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='ide0-1-1'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='drive' controller='0' bus='1' target='0' unit='1'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </disk>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <controller type='usb' index='0' model='piix3-uhci'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='usb'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </controller>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <controller type='pci' index='0' model='pci-root'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='pci.0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </controller>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <controller type='ide' index='0'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='ide'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </controller>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <controller type='virtio-serial' index='0'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='virtio-serial0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </controller>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <interface type='bridge'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <mac address='52:54:00:a2:7e:8a'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <source bridge='br0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <target dev='vnet0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <model type='virtio'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='net0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </interface>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <serial type='pty'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <source path='/dev/pts/2'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <target type='isa-serial' port='0'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:         <model name='isa-serial'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       </target>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='serial0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </serial>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <console type='pty' tty='/dev/pts/2'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <source path='/dev/pts/2'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <target type='serial' port='0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='serial0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </console>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <console type='pty'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <source path='/dev/pts/4'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <target type='virtio' port='1'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='console1'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </console>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <channel type='pty'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <source path='/dev/pts/3'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='channel0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='virtio-serial' controller='0' bus='0' port='1'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </channel>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <input type='mouse' bus='ps2'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='input0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </input>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <input type='keyboard' bus='ps2'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='input1'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </input>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <listen type='address' address='127.0.0.1'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </graphics>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <video>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <model type='cirrus' vram='16384' heads='1' primary='yes'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='video0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </video>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <memballoon model='virtio'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='balloon0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </memballoon>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <rng model='virtio'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <backend model='random'>/dev/urandom</backend>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <alias name='rng0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:       <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     </rng>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:   </devices>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:   <seclabel type='dynamic' model='dac' relabel='yes'>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <label>+64055:+113</label>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:     <imagelabel>+64055:+113</imagelabel>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt:   </seclabel>
2020-04-21T19:43:18.042Z [DEBUG] plugin.terraform-provider-libvirt: </domain>
2020-04-21T19:43:18.091Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [TRACE] Capabilities of host 
anonical:} {Name:pc-1.0 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.9 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.7 MaxCPUs:255 Canonical:} {Name:xenfv MaxCPUs:128 Canonical:} {Name:pc-i440fx-wily MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.3 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.5 MaxCPUs:255 Canonical:} {Name:pc-i440fx-yakkety MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.1 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.2 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.0 MaxCPUs:255 Canonical:} {Name:pc-q35-yakkety MaxCPUs:255 Canonical:} {Name:pc-i440fx-bionic-hpb MaxCPUs:255 Canonical:} {Name:pc-q35-2.11 MaxCPUs:288 Canonical:} {Name:q35 MaxCPUs:288 Canonical:pc-q35-2.11} {Name:pc-i440fx-xenial MaxCPUs:255 Canonical:} {Name:xenpv MaxCPUs:1 Canonical:} {Name:pc-q35-2.10 MaxCPUs:288 Canonical:} {Name:pc-q35-bionic-hpb MaxCPUs:288 Canonical:} {Name:pc-q35-xenial MaxCPUs:255 Canonical:} {Name:pc-i440fx-artful MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.7 MaxCPUs:255 Canonical:} {Name:pc-q35-2.9 MaxCPUs:288 Canonical:} {Name:pc-0.15 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.5 MaxCPUs:255 Canonical:} {Name:pc-q35-2.7 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.11 MaxCPUs:255 Canonical:} {Name:pc MaxCPUs:255 Canonical:pc-i440fx-2.11} {Name:pc-q35-2.8 MaxCPUs:288 Canonical:} {Name:pc-q35-zesty MaxCPUs:288 Canonical:} {Name:pc-0.13 MaxCPUs:255 Canonical:} {Name:pc-q35-artful MaxCPUs:288 Canonical:} {Name:pc-0.14 MaxCPUs:255 Canonical:} {Name:pc-q35-2.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-trusty MaxCPUs:255 Canonical:} {Name:pc-q35-2.5 MaxCPUs:255 Canonical:} {Name:pc-q35-2.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.10 MaxCPUs:255 Canonical:} {Name:pc-0.11 MaxCPUs:255 Canonical:} {Name:pc-0.12 MaxCPUs:255 Canonical:} {Name:pc-q35-bionic MaxCPUs:288 Canonical:} {Name:pc-0.10 MaxCPUs:255 Canonical:}]}]} Features:0xc420747740}]}
2020-04-21T19:43:18.092Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [TRACE] Checking for x86_64/hvm against i686/hvm
2020-04-21T19:43:18.092Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [TRACE] Checking for x86_64/hvm against x86_64/hvm
2020-04-21T19:43:18.092Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [DEBUG] Found 54 machines in guest for x86_64/hvm
2020-04-21T19:43:18.092Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [TRACE] Checking for x86_64/hvm against i686/hvm
2020-04-21T19:43:18.092Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [TRACE] Checking for x86_64/hvm against x86_64/hvm
2020-04-21T19:43:18.092Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [DEBUG] Found 54 machines in guest for x86_64/hvm
2020-04-21T19:43:18.119Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [TRACE] Capabilities of host 
anonical:} {Name:pc-1.0 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.9 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.7 MaxCPUs:255 Canonical:} {Name:xenfv MaxCPUs:128 Canonical:} {Name:pc-i440fx-wily MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.3 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.5 MaxCPUs:255 Canonical:} {Name:pc-i440fx-yakkety MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.1 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.2 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.0 MaxCPUs:255 Canonical:} {Name:pc-q35-yakkety MaxCPUs:255 Canonical:} {Name:pc-i440fx-bionic-hpb MaxCPUs:255 Canonical:} {Name:pc-q35-2.11 MaxCPUs:288 Canonical:} {Name:q35 MaxCPUs:288 Canonical:pc-q35-2.11} {Name:pc-i440fx-xenial MaxCPUs:255 Canonical:} {Name:xenpv MaxCPUs:1 Canonical:} {Name:pc-q35-2.10 MaxCPUs:288 Canonical:} {Name:pc-q35-bionic-hpb MaxCPUs:288 Canonical:} {Name:pc-q35-xenial MaxCPUs:255 Canonical:} {Name:pc-i440fx-artful MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.7 MaxCPUs:255 Canonical:} {Name:pc-q35-2.9 MaxCPUs:288 Canonical:} {Name:pc-0.15 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.5 MaxCPUs:255 Canonical:} {Name:pc-q35-2.7 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.11 MaxCPUs:255 Canonical:} {Name:pc MaxCPUs:255 Canonical:pc-i440fx-2.11} {Name:pc-q35-2.8 MaxCPUs:288 Canonical:} {Name:pc-q35-zesty MaxCPUs:288 Canonical:} {Name:pc-0.13 MaxCPUs:255 Canonical:} {Name:pc-q35-artful MaxCPUs:288 Canonical:} {Name:pc-0.14 MaxCPUs:255 Canonical:} {Name:pc-q35-2.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-trusty MaxCPUs:255 Canonical:} {Name:pc-q35-2.5 MaxCPUs:255 Canonical:} {Name:pc-q35-2.6 MaxCPUs:255 Canonical:} {Name:pc-i440fx-1.4 MaxCPUs:255 Canonical:} {Name:pc-i440fx-2.10 MaxCPUs:255 Canonical:} {Name:pc-0.11 MaxCPUs:255 Canonical:} {Name:pc-0.12 MaxCPUs:255 Canonical:} {Name:pc-q35-bionic MaxCPUs:288 Canonical:} {Name:pc-0.10 MaxCPUs:255 Canonical:}]}]} Features:0xc4206c6380}]}
2020-04-21T19:43:18.119Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [TRACE] Checking for x86_64/hvm against i686/hvm
2020-04-21T19:43:18.119Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [TRACE] Checking for x86_64/hvm against x86_64/hvm
2020-04-21T19:43:18.119Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [DEBUG] Found 54 machines in guest for x86_64/hvm
2020-04-21T19:43:18.120Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [DEBUG] fetching networking interfaces using qemu-agent
2020-04-21T19:43:18.120Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:18 [DEBUG] Waiting for state to become: [qemu-agent-done]
2020-04-21T19:43:22.121Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:22 [DEBUG] sending command to qemu-agent
2020-04-21T19:43:22.122Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:22 [DEBUG] command error: virError(Code=86, Domain=10, Message='Guest agent is not responding: QEMU guest agent is not connected')
2020-04-21T19:43:22.122Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:22 [TRACE] Waiting 4s before next try
2020-04-21T19:43:26.122Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:26 [DEBUG] sending command to qemu-agent
2020-04-21T19:43:26.123Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:26 [DEBUG] command error: virError(Code=86, Domain=10, Message='Guest agent is not responding: QEMU guest agent is not connected')
2020-04-21T19:43:26.123Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:26 [TRACE] Waiting 8s before next try
[0m[1mlibvirt_domain.domain-debian: Still creating... [10s elapsed][0m[0m
2020-04-21T19:43:34.124Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:34 [DEBUG] sending command to qemu-agent
2020-04-21T19:43:34.125Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:34 [DEBUG] command error: virError(Code=86, Domain=10, Message='Guest agent is not responding: QEMU guest agent is not connected')
2020-04-21T19:43:34.125Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:34 [TRACE] Waiting 10s before next try
[0m[1mlibvirt_domain.domain-debian: Still creating... [20s elapsed][0m[0m
2020-04-21T19:43:44.126Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:44 [DEBUG] sending command to qemu-agent
2020-04-21T19:43:44.127Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:44 [DEBUG] command error: virError(Code=86, Domain=10, Message='Guest agent is not responding: QEMU guest agent is not connected')
2020-04-21T19:43:44.127Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:44 [TRACE] Waiting 10s before next try
[0m[1mlibvirt_domain.domain-debian: Still creating... [30s elapsed][0m[0m
2020-04-21T19:43:48.121Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:48 [WARN] WaitForState timeout after 30s
2020-04-21T19:43:48.121Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:48 [WARN] WaitForState starting 30s refresh grace period
2020-04-21T19:43:48.121Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:48 [DEBUG] no interfaces could be obtained with qemu-agent: falling back to the libvirt API
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:48 [DEBUG] Interfaces info obtained with libvirt API:
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt: ([]libvirt.DomainInterface) {
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt: }
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt: 
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:48 [DEBUG] read: addresses for '52:54:00:A2:7E:8A': []
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:48 [DEBUG] read: ifaces for 'debian-terraform':
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt: ([]map[string]interface {}) (len=1 cap=1) {
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:     (map[string]interface {}) (len=10) {
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=11) "passthrough": (string) "",
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=14) "wait_for_lease": (bool) false,
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=12) "network_name": (string) "",
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=6) "bridge": (string) (len=3) "br0",
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=4) "vepa": (string) "",
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=7) "macvtap": (string) "",
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=3) "mac": (string) (len=17) "52:54:00:A2:7E:8A",
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=8) "hostname": (string) "",
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=9) "addresses": ([]string) <nil>,
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=10) "network_id": (string) ""
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt:     }
2020-04-21T19:43:48.123Z [DEBUG] plugin.terraform-provider-libvirt: }
2020/04/21 19:43:48 [WARN] Provider "registry.terraform.io/-/libvirt" produced an unexpected new value for libvirt_domain.domain-debian, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .kernel: was null, but now cty.StringVal("")
      - .firmware: was null, but now cty.StringVal("")
      - .initrd: was null, but now cty.StringVal("")
      - .autostart: was null, but now cty.False
      - .disk[0].block_device: was null, but now cty.StringVal("")
      - .disk[0].file: was null, but now cty.StringVal("")
      - .disk[0].scsi: was null, but now cty.False
      - .disk[0].url: was null, but now cty.StringVal("")
      - .disk[0].wwn: was null, but now cty.StringVal("")
      - .network_interface[0].macvtap: was null, but now cty.StringVal("")
      - .network_interface[0].passthrough: was null, but now cty.StringVal("")
      - .network_interface[0].vepa: was null, but now cty.StringVal("")
      - .network_interface[0].wait_for_lease: was null, but now cty.False
      - .console[0].source_path: was null, but now cty.StringVal("")
      - .console[1].source_path: was null, but now cty.StringVal("")
[0m[1mlibvirt_domain.domain-debian: Provisioning with 'local-exec'...[0m[0m
[0m[0mlibvirt_domain.domain-debian (local-exec): Executing: ["/bin/sh" "-c" "virsh qemu-agent-command debian-terraform '{\"execute\":\"guest-network-get-interfaces\"}'|jq '.return[1].\"ip-addresses\"[0].\"ip-address\"'| sed 's/\"//g'"]
[0m[0mlibvirt_domain.domain-debian (local-exec): error: Guest agent is not responding: QEMU guest agent is not connected
[0m[1mlibvirt_domain.domain-debian: Creation complete after 30s [id=2521c3e0-0c38-46d8-aa6b-14464eacd4f4][0m[0m
2020-04-21T19:43:48.183Z [DEBUG] plugin.terraform-provider-libvirt: 2020/04/21 19:43:48 [DEBUG] cleaning up connection for URI: qemu:///system
2020-04-21T19:43:48.189Z [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=11268
2020-04-21T19:43:48.190Z [DEBUG] plugin: plugin exited
2020-04-21T19:43:48.189Z [DEBUG] plugin: plugin process exited: path=/home/vrh/.terraform.d/plugins/terraform-provider-libvirt pid=11300
2020-04-21T19:43:48.190Z [DEBUG] plugin: plugin exited
[0m[1m[32m
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.[0m
[0m[1m[32m
Outputs:

ips = [
  [],
][0m

MalloZup commented 4 years ago

@itwars thx lot for issue.

Found this 'Guest agent is not responding: QEMU guest agent is not connected') I guess you are right. Actually with openSUSE that worked.

I think personally we have 2 options. Either we modify the timeout default increasing it to minutes

or we could give a variable to change the default timeout.

Imho, I guess we can increase the default timeout to max 5 minutes. How many minutes you need to wait?

nicolaballotta commented 4 years ago

I'm experiencing the same exact behaviour. Also adding a null_resource delay of 60 seconds unfortunately didn't help. If I do a terraform refresh immediately after the apply finished, the output is there.

MalloZup commented 4 years ago

hi @nicolaballotta thx for feedback. SO the question now is considering all the distro, how much maximal timeout we need to wait. From 3 Min to 5 Min I guess should do it, but Is better to have a good timeout instead of modifying it later.

nicolaballotta commented 4 years ago

@MalloZup I'm not sure (at least in my case) it's due to timeout. As I said I tried adding a null_resource with delay of 60 seconds and even more, but I don't get the output. While if I do a terraform refresh, the output is there immediately after the provisioning is finished and without any timeout set.

This is what I've in my debugging:

2020-04-22T10:15:28.422+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/04/22 10:15:28 [WARN] WaitForState timeout after 30s
2020-04-22T10:15:28.422+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/04/22 10:15:28 [WARN] WaitForState starting 30s refresh grace period
2020-04-22T10:15:28.422+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/04/22 10:15:28 [DEBUG] no interfaces could be obtained with qemu-agent: falling back to the libvirt API
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/04/22 10:15:28 [DEBUG] Interfaces info obtained with libvirt API:
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt: ([]libvirt.DomainInterface) {
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt: }
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/04/22 10:15:28 [DEBUG] read: addresses for '52:54:00:96:A1:67': []
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/04/22 10:15:28 [DEBUG] read: ifaces for 'master-01':
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt: ([]map[string]interface {}) (len=1 cap=1) {
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:     (map[string]interface {}) (len=10) {
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=10) "network_id": (string) "",
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=12) "network_name": (string) "",
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=6) "bridge": (string) (len=3) "br0",
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=11) "passthrough": (string) "",
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=4) "vepa": (string) "",
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=7) "macvtap": (string) "",
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=3) "mac": (string) (len=17) "52:54:00:96:A1:67",
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=8) "hostname": (string) "",
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=14) "wait_for_lease": (bool) false,
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:         (string) (len=9) "addresses": ([]string) <nil>
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt:     }
2020-04-22T10:15:28.423+0200 [DEBUG] plugin.terraform-provider-libvirt: }
2020/04/22 10:15:28 [WARN] Provider "registry.terraform.io/-/libvirt" produced an unexpected new value for libvirt_domain.ubuntu_master[0], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .firmware: was null, but now cty.StringVal("")
      - .kernel: was null, but now cty.StringVal("")
      - .autostart: was null, but now cty.False
      - .description: was null, but now cty.StringVal("")
      - .disk[0].block_device: was null, but now cty.StringVal("")
      - .disk[0].file: was null, but now cty.StringVal("")
      - .disk[0].scsi: was null, but now cty.False
      - .disk[0].url: was null, but now cty.StringVal("")
      - .disk[0].wwn: was null, but now cty.StringVal("")
      - .initrd: was null, but now cty.StringVal("")
      - .network_interface[0].network_name: was cty.StringVal("host-bridge"), but now cty.StringVal("")

And btw I tried both with Ubuntu 19.10 and Centos 7. Same exact behaviour.

itwars commented 4 years ago

I just have a quick look into vsphere provider :

https://github.com/terraform-providers/terraform-provider-vsphere/blob/d18d4f04c4535839d640ad391dd90179b0a6905d/vsphere/internal/helper/virtualmachine/virtual_machine_helper.go

// WaitForGuestIP waits for a virtual machine to have an IP address.
//
// The timeout is specified in minutes. If zero or a negative value is passed,
// the waiter returns without error immediately.
func WaitForGuestIP(client *govmomi.Client, vm *object.VirtualMachine, timeout int, ignoredGuestIPs []interface{}) error {
    if timeout < 1 {
        log.Printf("[DEBUG] Skipping IP waiter for VM %q", vm.InventoryPath)
        return nil
    }
    log.Printf(
        "[DEBUG] Waiting for an available IP address on VM %q (timeout = %dm)",
        vm.InventoryPath,
        timeout,

https://github.com/terraform-providers/terraform-provider-vsphere/blob/1c52bb0d1124d759df766edf47e4fe1442fc9a96/website/docs/r/virtual_machine.html.markdown

wait_for_guest_net_timeout - (Optional) The amount of time, in minutes, to wait for an available IP address on this virtual machine's NICs. Older versions of VMware Tools do not populate this property. In those cases, this waiter can be disabled and the wait_for_guest_ip_timeout waiter can be used instead. A value less than 1 disables the waiter. 
Default: 5 minutes.

Anyway, regarding the time line vm deployement we have the followinf steps :

Start VM
Cloud-Init :
1. change vm name
2. handle network stuff
3. ...
4. update repo + INSTALL QEMU-AGENT
5. start QUEMU-AGENT
Recieve provisionners actions

So regarding that timeline, timeout must be minutes depending of server power (on my LAB it's a Celeron 4 cores + 6Gb RM), network bandwidth, ...

nicolaballotta commented 4 years ago

@itwars as I said before, I'm not sure this will solve the issue. Or maybe we are experiencing two different issues. Look at this:

I tried putting a delay before the output. I tried with 60 seconds, 2mins, 5mins. No output at all. But if I do a terraform refresh, also without the timeout, immediately after the apply finished, I get the output.

itwars commented 4 years ago

huuumm, so Huston we have a problem with qemu-agent when it first request ip address?

nicolaballotta commented 4 years ago

huuumm, so Huston we have a problem with qemu-agent when it first request ip address?

yeah that's what seems to me

itwars commented 4 years ago

digging a little deeper : just after the deployment I catch ip on my dhcp server, and then ssh to the vm, got an interessing information :

sudo service qemu-guest-agent status
● qemu-guest-agent.service - QEMU Guest Agent
   Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static; vendor preset: enabled)
   Active: inactive (dead)

after rebooting the vm I got :

sudo service qemu-guest-agent status
● qemu-guest-agent.service - QEMU Guest Agent
   Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static; vendor preset: enabled)
   Active: active (running) since Wed 2020-04-22 11:29:49 UTC; 19min ago
 Main PID: 368 (qemu-ga)
    Tasks: 1 (limit: 2377)
   Memory: 2.0M
   CGroup: /system.slice/qemu-guest-agent.service
           └─368 /usr/sbin/qemu-ga

Apr 22 11:29:49 gitlab systemd[1]: Started QEMU Guest Agent.

So on Debian, it do install qemu-guest-agent, but seems not to start it! Could you please check on Ubuntu?

nicolaballotta commented 4 years ago

Yeah in Ubuntu it's the same and that's why I've this in my cloud_init.cfg:


packages:
  - qemu-guest-agent

runcmd:
  - [ systemctl, daemon-reload ]
  - [ systemctl, enable, qemu-guest-agent ]
  - [ systemctl, start, qemu-guest-agent ]```

itwars commented 4 years ago

On CentOS, agent automaticaly start as expected, without you runcmd. But output still failed, on work using terraform refresh

And your runcmd, start my agent as well on Debian, but still no ip, only with refresh!

Anyway, as local-exec is not able to get ip :

provisioner "local-exec" {
    command = <<EOT
    sleep 60
    virsh qemu-agent-command debian-terraform '{"execute":"guest-network-get-interfaces"}'|jq '.return[1]."ip-addresses"[0]."ip-address"'| sed 's/\"//g'
    EOT
  }

The query made by terraform-provider-libvirt against qemu-guest-agent came too early in the process, as local-exec become excuted later! So first increase timeout for qemu-guest-agent request up to 5 min (as in vsphere provider). And it could work?

I also notice the cloud-init phase is started is a "star and forget mode", meanning: if I add a sleep 120 in the runcmd, terraform process go on even if cloud-init isn't complete yet! So perhaps it request the agent while it is not yet installed?

packages:
  - qemu-guest-agent
runcmd:
  - [ systemctl, daemon-reload ]
  - [ systemctl, enable, qemu-guest-agent ]
  - [ systemctl, start, qemu-guest-agent ]
  - sleep 240
  - echo CLOUD-INIT ENDED

In the virsh console, cloudinit phase ended long time after terraform gave it empty ip output. So if cloudinit last too long (pkg to setup, user to create, network issue,...) it won't work.

itwars commented 4 years ago

Yes I did it changing timeout in code, I'll send my PR to close it!

MinTimeout:  1 * time.Minute,
Delay:        30 * time.Second, // Wait this time before starting checks
Timeout:     5 * time.Minute,

libvirt_domain.domain-debian: Creation complete after 2m30s [id=f0d1a0c8-322d-4585-a61f-7474a4ce4a0d]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

ips = [
  [
    "192.168.1.20",
    "fe80::5054:ff:fe56:2230",
  ],
]

itwars commented 4 years ago

Fix by PR : 5427432

itwars commented 4 years ago

Hi, I'm just back to business, and I do some update (ie:terraform 0.13), and I very disapointed: again it couldn't get ip address. I think it's again a timeout issue, because running this command few seconds after, I got my ips:

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

iprunner = [
  [],
]
ipsgitlab = [
  [],
]

for i in gitlab gitlab-runner ; do virsh qemu-agent-command $i '{"execute":"guest-network-get-interfaces"}'|jq '.ret
urn[1]."ip-addresses"[0]."ip-address"'| sed 's/\"//g'; done

192.168.1.34
192.168.1.22

Hum, I've just revert to terraform 0.12.24, and it works! So Huston we have a problem with v0.13?!

itwars commented 3 years ago

Hi guys,

As nobody answer, I've dig a little are here are the evidences :

output directive still don't work with last provider version and terraform v0.13.4
in debug mode (TF_LOG=DEBUG terraform apply) qemu and libvirt provider do return ip addresses

Here are some informations :

# Define KVM domain to create
resource "libvirt_domain" "virt-domain" {
  name = "srv-${random_id.randomId.hex}"
  memory = var.domain_memory
  vcpu   = 4

  cloudinit = libvirt_cloudinit_disk.commoninit.id
  qemu_agent = true
  network_interface {
     bridge = "br0"
     wait_for_lease = true
  }

  console {
    type = "pty"
    target_type = "serial"
    target_port = "0"
  }

    disk {
    volume_id = libvirt_volume.image-qcow2.id
  }

  graphics {
    type = "spice"
    listen_type = "address"
    autoport = true
  }
}
output "ip" {
  value = libvirt_domain.virt-domain.*.network_interface.0.addresses
}

2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt: 2020/10/13 10:35:31 [DEBUG] Interfaces obtained via qemu-agent: [{Name:eth0 Hwaddr:52:54:00:4a:14:a1 Addrs:[{Type:0 Addr:192.168.1.21 Prefix:24} {Type:1 Addr:fe80::5054:ff:fe4a:14a1 Prefix:64}]}]
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt: 2020/10/13 10:35:31 [DEBUG] read: addresses for '52:54:00:4A:14:A1': [192.168.1.21 fe80::5054:ff:fe4a:14a1]
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt: 2020/10/13 10:35:31 [DEBUG] read: ifaces for 'srv-92e99ab1':
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt: ([]map[string]interface {}) (len=1 cap=1) {
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:     (map[string]interface {}) (len=10) {
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=6) "bridge": (string) (len=3) "br0",
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=7) "macvtap": (string) "",
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=8) "hostname": (string) "",
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=10) "network_id": (string) "",
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=12) "network_name": (string) "",
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=3) "mac": (string) (len=17) "52:54:00:4A:14:A1",
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=14) "wait_for_lease": (bool) true,
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=9) "addresses": ([]string) (len=2 cap=2) {
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:                     (string) (len=12) "192.168.1.21",
2020-10-13T10:35:31.379Z [DEBUG] plugin.terraform-provider-libvirt:                     (string) (len=23) "fe80::5054:ff:fe4a:14a1"
2020-10-13T10:35:31.380Z [DEBUG] plugin.terraform-provider-libvirt:             },
2020-10-13T10:35:31.380Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=4) "vepa": (string) "",
2020-10-13T10:35:31.380Z [DEBUG] plugin.terraform-provider-libvirt:             (string) (len=11) "passthrough": (string) ""
2020-10-13T10:35:31.380Z [DEBUG] plugin.terraform-provider-libvirt:     }
2020-10-13T10:35:31.380Z [DEBUG] plugin.terraform-provider-libvirt: }
2020/10/13 10:35:31 [WARN] Provider "registry.terraform.io/dmacvicar/libvirt" produced an unexpected new value for module.ubuntu-server-1.libvirt_domain.virt-domain, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .autostart: was null, but now cty.False
      - .firmware: was null, but now cty.StringVal("")
      - .initrd: was null, but now cty.StringVal("")
      - .kernel: was null, but now cty.StringVal("")
      - .description: was null, but now cty.StringVal("")
      - .disk[0].wwn: was null, but now cty.StringVal("")
      - .disk[0].block_device: was null, but now cty.StringVal("")
      - .disk[0].file: was null, but now cty.StringVal("")
      - .disk[0].scsi: was null, but now cty.False
      - .disk[0].url: was null, but now cty.StringVal("")
      - .network_interface[0].passthrough: was null, but now cty.StringVal("")
      - .network_interface[0].vepa: was null, but now cty.StringVal("")
      - .network_interface[0].macvtap: was null, but now cty.StringVal("")
      - .console[0].source_path: was null, but now cty.StringVal("")
module.ubuntu-server-1.libvirt_domain.virt-domain: Creation complete after 26s [id=17911147-b30e-4028-afb5-9c571f0d912e]
2020-10-13T10:35:31.401Z [DEBUG] plugin.terraform-provider-libvirt: 2020/10/13 10:35:31 [DEBUG] cleaning up connection for URI: qemu:///system
2020-10-13T10:35:31.404Z [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/dmacvicar/libvirt/0.6.2/linux_amd64/terraform-provider-libvirt pid=44836
2020-10-13T10:35:31.404Z [DEBUG] plugin: plugin exited

In my previous report on this issue, I got output with empty values, but now it don't show any output!?

module.ubuntu-server-1.libvirt_volume.image-qcow2: Creating...
module.ubuntu-server-1.libvirt_cloudinit_disk.commoninit: Creation complete after 0s [id=/tmp/terraform_libvirt_provider_images_1/commoninit-53352038.iso;5f8590bb-d314-7f9a-a707-fb1beda522c6]
module.ubuntu-server-1.libvirt_volume.image-qcow2: Creation complete after 0s [id=/tmp/terraform_libvirt_provider_images_1/disk-53352038]
module.ubuntu-server-1.libvirt_domain.virt-domain: Creating...
module.ubuntu-server-1.libvirt_domain.virt-domain: Still creating... [10s elapsed]
module.ubuntu-server-1.libvirt_domain.virt-domain: Still creating... [20s elapsed]
module.ubuntu-server-1.libvirt_domain.virt-domain: Creation complete after 26s [id=606c7781-187c-445d-84ed-f936e464570b]

Apply complete! Resources: 6 added, 0 changed, 6 destroyed.

itwars commented 3 years ago

Sorry, my fault ... My tf file was shity!

DMW007 commented 3 years ago

I have a similar issue with a bridge: 2 VMs were build and should get a IP from the dedicated DHCP server in the network, where the bridge is connected to. But it works most of the time: On 9 from 10 test runs, it works well. The one missing run prints only 1/2 IPs like this:

Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Outputs:

ips = [
  [],
]
ips_db2 = [
  [
    "192.168.0.37",
    "2001:16b8:249c:8d00:5054:ff:fe42:d4aa",
    "fe80::5054:ff:fe42:d4aa",
  ],
]

In this situation, I can't fetch them. Also not after waiting some time, it still shows an empty array:

$ terraform output -json ips
[[]]

But after I ran terraform refresh, it prints the IP. I'm using Terraform v0.13.6 with the latest libvirt provider v0.6.3 and I'll run this some more time with the refresh call to see if this is a workaround for this issue.

Edit: Test results

I did 12 runs and noticed that one of them had no IP in the output after terraform apply, but then it appears after running terraform refresh:

Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Outputs:

ips = [
  [],
]
ips_db2 = [
  [
    "192.168.0.37",
    "2001:16b8:2435:5000:5054:ff:fe42:d4aa",
    "fe80::5054:ff:fe42:d4aa",
  ],
]
network_mode = bridge
data.template_file.cloudinit_data: Refreshing state... [id=ea66faeeeedfcf6281f98392a1234fa3a075ea52e171312822859fb61969c24b]
data.template_file.cloudinit_data_db2: Refreshing state... [id=e217fbcefb0ff64f45f96c2352302ab56bfbdc14a96f861b1c5f3d8bcf36eaae]
libvirt_pool.default: Refreshing state... [id=103b40c5-90c0-4cef-acb2-4b4043f025f4]
libvirt_network.cnx_network: Refreshing state... [id=841a9ecf-1fca-4844-9b16-5991c5321c90]
libvirt_volume.centos7-img: Refreshing state... [id=/tmp/kvm/cnx_centos7.qcow2]
libvirt_cloudinit_disk.cloudinit: Refreshing state... [id=/tmp/kvm/cloudinit.iso;60180f6f-b357-6cbb-7fe3-cc6a951f5709]
libvirt_cloudinit_disk.cloudinit_db2: Refreshing state... [id=/tmp/kvm/cloudinit_db2.iso;60180f6f-83df-36d0-531c-b81a0ec3d260]
libvirt_volume.centos7-img_db2: Refreshing state... [id=/tmp/kvm/cnx_centos7_db2.qcow2]
libvirt_domain.db2: Refreshing state... [id=1054172e-5c59-4b5b-bae6-f7d56581f3cc]
libvirt_domain.cnx: Refreshing state... [id=f6fa28c7-dc3c-4037-8e6e-87d86f3fe152]

Outputs:

ips = [
  [
    "192.168.0.39",
    "2001:16b8:2435:5000:5054:ff:fe4e:cc0f",
    "fe80::5054:ff:fe4e:cc0f",
  ],
]
ips_db2 = [
  [
    "192.168.0.37",
    "2001:16b8:2435:5000:5054:ff:fe42:d4aa",
    "fe80::5054:ff:fe42:d4aa",
  ],
]
network_mode = bridge

The first output is from apply, the second from refresh. For me it's a workaround to run terraform refresh, it took about 6 seconds.

dmacvicar / terraform-provider-libvirt