dmacvicar / terraform-provider-libvirt

Terraform provider to provision infrastructure with Linux's KVM using libvirt
Apache License 2.0
1.54k stars 457 forks source link

No interfaces obtained via qemu-agent #891

Closed matthijssjansen closed 2 years ago

matthijssjansen commented 2 years ago

System Information

Linux distribution

Ubuntu 20.04

Terraform version

1.0.6

Provider and libvirt versions

0.6.2


Checklist

Description of Issue/Question

Setup

Terraform: cloudedge.txt

Cloudinit: network_config_static.txt cloud_init.txt

Steps to Reproduce Issue

I want to create VMs and expose them to the local network via a network bridge br0. However, using Terraform the VMs are stuck on the following (from TF_LOG=DEBUG terraform apply)

2021-10-06T13:45:01.302Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] sending command to qemu-agent
2021-10-06T13:45:01.307Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] qemu-agent response: {"return":[{"name":"lo","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"127.0.0.1","prefix":8},{"ip-address-type":"ipv6","ip-address":"::1","prefix":128}],"statistics":{"tx-packets":6,"tx-errs":0,"rx-bytes":416,"rx-dropped":0,"rx-packets":6,"rx-errs":0,"tx-bytes":416,"tx-dropped":0},"hardware-address":"00:00:00:00:00:00"},{"name":"eth0","ip-addresses":[{"ip-address-type":"ipv6","ip-address":"fe80::5054:ff:fe7e:962e","prefix":64}],"statistics":{"tx-packets":29,"tx-errs":0,"rx-bytes":27610,"rx-dropped":5,"rx-packets":224,"rx-errs":0,"tx-bytes":7838,"tx-dropped":0},"hardware-address":"52:54:00:7e:96:2e"}]}
2021-10-06T13:45:01.307Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] Parsed response {Interfaces:[{Name:lo Hwaddr:00:00:00:00:00:00 IPAddresses:[{Type:ipv4 Address:127.0.0.1 Prefix:8} {Type:ipv6 Address:::1 Prefix:128}]} {Name:eth0 Hwaddr:52:54:00:7e:96:2e IPAddresses:[{Type:ipv6 Address:fe80::5054:ff:fe7e:962e Prefix:64}]}]}
2021-10-06T13:45:01.307Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] Interfaces obtained via qemu-agent: []
2021-10-06T13:45:01.307Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] no interfaces could be obtained with qemu-agent: falling back to the libvirt API
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] Interfaces info obtained with libvirt API:
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: ([]libvirt.DomainInterface) {
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: }
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] ifaces with addresses: []
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] 52:54:00:7E:96:2E doesn't have IP address(es) yet...
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] IP address not found for iface=52:54:00:7E:96:2E: will try in a while
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [TRACE] Waiting 10s before next try

Resulting in

Error: Error: couldn't retrieve IP address of domain.

The VMs have qemu-guest-agent running on them, which gives back this list of network interfaces available on the VM, but this list is missing something / incorrect according to Terraform. The VM ends up having no IP (using virsh console to login to the VM, followed by `hostname -i```.


Additional information:

I have tried many permutations of the given terraform and cloudinit file, including other terraform-provider-libvirt versions, other operating systems for the VM, using static IPs and / or dynamic IPs via DHCP. At this point I am out of ideas and would like some help in figuring out what goes wrong.

I am able to create VMs with static IPs and a network bridge using the Qemu CLI (without Terraform).

mxkle commented 2 years ago

Hi,

i think i have the same problem. I try to deploy a debian vm with this plugin but it runs into a timeout while trying to get the ip of the vm. On my dhcp server i can see that the vm successfully obtained an address but terraform isnt recognizing it.

my tf file:

terraform {
  required_version = ">= 0.13"
    required_providers {
      libvirt = {
        source  = "dmacvicar/libvirt"
        version = "0.6.11"
      }
    }
}

resource "libvirt_domain" "virt-machine" {
  count  = var.vm_count
  name   = format("${var.vm_hostname_prefix}%02d", count.index + var.index_start)
  memory = var.memory
  vcpu   = var.vcpu
  autostart  = var.autostart
  qemu_agent = true

  cloudinit = element(libvirt_cloudinit_disk.commoninit.*.id, count.index)

  network_interface {
    bridge         = var.bridge
    wait_for_lease = true
    hostname       = format("${var.vm_hostname_prefix}%02d", count.index + var.index_start)
  }

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }

  disk {
    volume_id = element(libvirt_volume.volume-qcow2.*.id, count.index)
  }

  graphics {
    type        = "spice"
    listen_type = "address"
    autoport    = true
  }
}

while terraform tries to obtain the ip of the vm, i can already find it via virsh:

virsh qemu-agent-command nomad01 '{"execute": "guest-network-get-interfaces"}'
{
  "return": [
    {
      "name": "lo",
      "ip-addresses": [
        {
          "ip-address-type": "ipv4",
          "ip-address": "127.0.0.1",
          "prefix": 8
        },
        {
          "ip-address-type": "ipv6",
          "ip-address": "::1",
          "prefix": 128
        }
      ],
      "statistics": {
        "tx-packets": 0,
        "tx-errs": 0,
        "rx-bytes": 0,
        "rx-dropped": 0,
        "rx-packets": 0,
        "rx-errs": 0,
        "tx-bytes": 0,
        "tx-dropped": 0
      },
      "hardware-address": "00:00:00:00:00:00"
    },
    {
      "name": "ens3",
      "ip-addresses": [
        {
          "ip-address-type": "ipv4",
          "ip-address": "192.168.2.117",
          "prefix": 24
        },
        {
          "ip-address-type": "ipv6",
          "ip-address": "fe80::5054:ff:fe62:d25f",
          "prefix": 64
        }
      ],
      "statistics": {
        "tx-packets": 56,
        "tx-errs": 0,
        "rx-bytes": 70690,
        "rx-dropped": 5,
        "rx-packets": 743,
        "rx-errs": 0,
        "tx-bytes": 8065,
        "tx-dropped": 0
      },
      "hardware-address": "52:54:00:62:d2:5f"
    },
    {
      "name": "docker0",
      "ip-addresses": [
        {
          "ip-address-type": "ipv4",
          "ip-address": "172.17.0.1",
          "prefix": 16
        }
      ],
      "statistics": {
        "tx-packets": 0,
        "tx-errs": 0,
        "rx-bytes": 0,
        "rx-dropped": 0,
        "rx-packets": 0,
        "rx-errs": 0,
        "tx-bytes": 0,
        "tx-dropped": 0
      },
      "hardware-address": "02:42:3e:b2:81:46"
    }
  ]
}

in my cloud init file i also install, start and enable the qemu-guest-agent.service. before terraform aborts i can connect to the vm via ssh and verify that its running:

$ sudo systemctl status qemu-guest-agent.service 
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static)
     Active: active (running) since Sun 2021-10-10 11:35:23 CEST; 23min ago
   Main PID: 455 (qemu-ga)
      Tasks: 2 (limit: 529)
     Memory: 2.3M
        CPU: 5ms
     CGroup: /system.slice/qemu-guest-agent.service
             └─455 /usr/sbin/qemu-ga

Oct 10 11:35:23 nomad01 systemd[1]: Started QEMU Guest Agent.

and here is the output from the terraform apply command:

2021-10-10T10:07:14.825Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] waiting for network address for iface=52:54:00:62:D2:5F
2021-10-10T10:07:14.826Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] Not implemented
2021-10-10T10:07:14.826Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] no interfaces could be obtained with qemu-agent: falling back to the libvirt API
2021-10-10T10:07:14.826Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] Interfaces info obtained with libvirt API:
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: ([]libvirt.DomainInterface) <nil>
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] ifaces with addresses: []
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] 52:54:00:62:D2:5F doesn't have IP address(es) yet...
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] IP address not found for iface=52:54:00:62:D2:5F: will try in a while
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [TRACE] Waiting 10s before next try

I found this issue here: https://github.com/dmacvicar/terraform-provider-libvirt/issues/710 its closed because there was already a fix for a problem with channel bindings.

remoe commented 2 years ago

qemu agent is not supported in latest version: #879

0.6.3 : works - https://github.com/dmacvicar/terraform-provider-libvirt/blob/v0.6.3/libvirt/domain.go#L157 v0.6.9-pre1 : not implemented! https://github.com/dmacvicar/terraform-provider-libvirt/blob/v0.6.9-pre1/libvirt/domain.go#L159

mxkle commented 2 years ago

Oh thanks for the info remoe, i somehow missed that. Sorry! I will switch to version 0.6.3 and try agian.

remoe commented 2 years ago

@mxkle , but this version would not work with terraform 1.x

everflux commented 2 years ago

Is there a version that supports the qemu-agent and works with terraform 1.0.x?

remoe commented 2 years ago

@everflux, no wait for https://github.com/dmacvicar/terraform-provider-libvirt/pull/ 873 is reviewed and merged in.

everflux commented 2 years ago

The PR is reviewed but it looks like it will not be merged, it is 1,5h years old and conflicts with the current main. I can understand the requested changes by @dmacvicar but I am afraid that it won't materialize.

dmacvicar commented 2 years ago

I have manually rebased and committed #873

I don't like the 120M image for the test, so the test is pending and I opened #913 to revisit it.