search

dmacvicar / terraform-provider-libvirt

Terraform provider to provision infrastructure with Linux's KVM using libvirt
Apache License 2.0
1.54k stars 457 forks source link

Unable to create simple qemu VM on aarch64 #976

Open voroskoi opened 1 year ago

voroskoi commented 1 year ago

System Information

Linux distribution

Alpine Linux Edge
Linux puruttya 5.15.69-0-rpi4 #1-Alpine SMP PREEMPT Thu Sep 22 09:04:24 UTC 2022 aarch64 GNU/Linux

Terraform version

Terraform v1.3.0
on linux_arm64
+ provider registry.terraform.io/dmacvicar/libvirt v0.6.14

Your version of Terraform is out of date! The latest version
is 1.3.1. You can update by downloading from https://www.terraform.io/downloads.html

Provider and libvirt versions

./terraform-provider-libvirt_v0.6.14 0.6.14

Description of Issue/Question

This is the first time I try using terraform, thank You very much for this provider! I would like to create a simple VM image, my main.tf is below.

I get the following error message:

voroskoi ~/terraform via 💠 default ❯ terraform apply
libvirt_volume.alpine: Refreshing state... [id=/home/voroskoi/libvirt/alpine]
libvirt_pool.default: Refreshing state... [id=f36e6d26-5694-401a-b5d6-036fea8a41c9]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # libvirt_domain.terraform_test will be created
  + resource "libvirt_domain" "terraform_test" {
      + arch        = (known after apply)
      + disk        = [
          + {
              + block_device = null
              + file         = null
              + scsi         = null
              + url          = null
              + volume_id    = "/home/voroskoi/libvirt/alpine"
              + wwn          = null
            },
        ]
      + emulator    = (known after apply)
      + firmware    = "/usr/share/AAVMF/AAVMF_CODE.fd"
      + fw_cfg_name = "opt/com.coreos/config"
      + id          = (known after apply)
      + machine     = (known after apply)
      + memory      = 512
      + name        = "terraform_test3"
      + qemu_agent  = false
      + running     = true
      + vcpu        = 1
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

libvirt_domain.terraform_test: Creating...
╷
│ Error: error defining libvirt domain: loader attribute 'readonly' cannot be specified when firmware autoselection is enabled
│
│   with libvirt_domain.terraform_test,
│   on main.tf line 24, in resource "libvirt_domain" "terraform_test":
│   24: resource "libvirt_domain" "terraform_test" {
│
╵

Without the firmware = "/usr/share/AAVMF/AAVMF_CODE.fd" line I get this error message: (I have to keep increasing the counter in name to avoid hitting domain already exists error.)

voroskoi ~/terraform via 💠 default ❯ terraform apply
libvirt_pool.default: Refreshing state... [id=f36e6d26-5694-401a-b5d6-036fea8a41c9]
libvirt_volume.alpine: Refreshing state... [id=/home/voroskoi/libvirt/alpine]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # libvirt_domain.terraform_test will be created
  + resource "libvirt_domain" "terraform_test" {
      + arch        = (known after apply)
      + disk        = [
          + {
              + block_device = null
              + file         = null
              + scsi         = null
              + url          = null
              + volume_id    = "/home/voroskoi/libvirt/alpine"
              + wwn          = null
            },
        ]
      + emulator    = (known after apply)
      + fw_cfg_name = "opt/com.coreos/config"
      + id          = (known after apply)
      + machine     = (known after apply)
      + memory      = 512
      + name        = "terraform_test4"
      + qemu_agent  = false
      + running     = true
      + vcpu        = 1
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

libvirt_domain.terraform_test: Creating...
╷
│ Error: error creating libvirt domain: operation failed: Unable to find any firmware to satisfy 'efi'
│
│   with libvirt_domain.terraform_test,
│   on main.tf line 24, in resource "libvirt_domain" "terraform_test":
│   24: resource "libvirt_domain" "terraform_test" {
│
╵

I have checked the examples here, maybe I should define the nvram, but I have no idea where those values come from.

Setup

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
  uri = "qemu:///system"
}

resource "libvirt_volume" "alpine" {
  name   = "alpine"
  source = "https://dl-cdn.alpinelinux.org/alpine/v3.16/releases/aarch64/alpine-virt-3.16.2-aarch64.iso"
}

resource "libvirt_pool" "default" {
  name = "default"
  type = "dir"
  path = "/home/voroskoi/libvirt"
}

resource "libvirt_domain" "terraform_test" {
  name = "terraform_test3"
  firmware = "/usr/share/AAVMF/AAVMF_CODE.fd"

  disk {
    volume_id = libvirt_volume.alpine.id
  }
}

Steps to Reproduce Issue

run terraform apply

Additional information:

I do not have neither AppArmor, nor SELinux installed. Iptables not running.

pfworks commented 5 months ago

I am stuck here as well. I have added the section below and still get the same error. I've even tried running from x86_64 and aarch64 terraform host.

nvram { file = "/usr/share/AAVMF/AAVMF_VARS.fd" }

pfworks commented 5 months ago

I've also tried using the AAVMF_VARS.{ms,snakeoil}.fd files

pfworks commented 5 months ago

OK, I think I know what's going on, but don't know how to fix it yet. Since we're passing the values, the generated file for qemu should not have "firmware = efi" in it. This needs to be removed. I'm looking in to how to do that.

pfworks commented 5 months ago

So it seems that for now, you have to specify nvram and firmware settings, and remove the "d.OS.Firmware = "efi"" from domain_def.go. This gets past this issue, and brings up another issue where devices are not properly getting

set, so virsh is throwing an error.

pfworks commented 5 months ago

I'm also investigating... setting "efi" should pick up the right values, so I'm looking into that as well.

pfworks commented 5 months ago

OK, so here's progress so far. Copy /usr/share/AAVMF/AAVMF_VARS.fd to /var/lib/libvirt/qemu/nvram/ Use a provider where I've taken out setting the firmware=efi in the qemu xml file Set firmware="/usr/share/AAVMF/AAVMF_CODE.fd" Set nvram { file="/var/lib/libvirt/qemu/nvram/"} Run my terraform plan and apply - can't connect to qemu agent virsh destroy and start, and machine comes up.

pfworks commented 5 months ago

So I think the thing to do is have nvram provide a template and a destination, generate the file and put it in the correct spot. I"m going to look through the code and see if I can figure out where that could/should happen. It seems some of this stuff is buried in go bindings, so I'll have to figure that out, too as I've written exactly 1 simple go program in my life.

something like

`firmware = "/usr/share/AAVMF/AAVMF_CODE.fd"

nvram { template="/usr/share/AAVMF/AAVMF_VARS.df" file="/var/lib/libvirt/qemu/nvram/hostname_VARS.fd" }`

file should default to the above and only need to be specified if it should be someplace else.

pfworks commented 5 months ago

For now I added the following in my terraform file:

resource "null_resource" "copy_nvram" { provisioner "local-exec" {command = "scp -p VARS.fd root@${var.kvmhost}:/var/lib/libvirt/qemu/nvram/${var.hostname}_VARS.fd"} }

and this does the trick. I now have an aarch64 machine installing from terraform.