CoreOS Ignition only works on first boot, so updates are no-ops

[FreekingDean]

System Information

Linux distribution

5.3.1-arch1-1-ARCH #1 SMP PREEMPT

Terraform version

Terraform v0.12.9

Provider and libvirt versions

terraform-provider-libvirt was not built correctly
Compiled against library: libvirt 5.5.0
Using library: libvirt 5.5.0
Running hypervisor: QEMU 4.1.0
Running against daemon: 5.5.0

Git Commit:

3a11558c00af441cfe41e06896922ff7e5041f7f

---

Checklist

• [x] Is it a bug or something that does not work as expected? Please make sure you fill the version information below:

Description of Issue

After updating an ignition config & running terraform apply I notice the changes reflected on disk, but not within the VM itself.

Setup

https://gist.github.com/FreekingDean/dd3735523433f49f7484613107555761

Steps to Reproduce Issue

Not sure if logs are relevant, but can add if needed!

---

Additional information:

Nothing I can think to note!

[MalloZup]

@zeenix cc

[MalloZup]

I think here we just miss the update operation on the CRUD of terraform. This might not be trivial to implement or at least need some research.

thx for issue

[zeenix]

terraform-provider-libvirt was not built correctly

What's that all about?

[MalloZup]

@zeenix nope that version is a cosmetic thing (https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/main.go#L17).

This come when users use RPM versions or other packages not build from source. I think the --version command might have other room of improvement there but the risk is that we over complicated things..

But for sure we could improve the errror msg :grin:

[dmacvicar]

After updating an ignition config & running terraform apply I notice the changes reflected on disk, but not within the VM itself.

I don't understand this part. The ignition resource is supposed to help automating the packaging of the resource as a volume, and then passing that to QEMU via firmware config.

What else are you expecting it to do, or what are you not seeing, when you say "not within the VM itself"?

[FreekingDean]

I think this is my misunderstanding of the coreos ignition setup: https://coreos.com/ignition/docs/latest/what-is-ignition.html#when-is-ignition-executed. It seems that changes to the coreos ignition system would never be reflected since it only happens on first boot. I wonder if its wise to add some documentation around this. For example heres how I am solving this: https://github.com/FreekingDean/tf-modules/blob/master/vm/main.tf#L67-L69

resource "libvirt_volume" "os" {
  name   = "os-qcow2_${var.orchistration_type}.${count.index}"
  pool   = libvirt_pool.pool.name
  source = local.os
  format = "qcow2"
  count = var.node_count
  depends_on = [
    libvirt_ignition.ignition
  ]
}

[FreekingDean]

FWIW, cloudinit seems like a better alternative as well, so I'll be migrating to that anyway. (Im going to re-title to be more accurate)

[dmacvicar]

I think this is my misunderstanding of the coreos ignition setup: https://coreos.com/ignition/docs/latest/what-is-ignition.html#when-is-ignition-executed. It seems that changes to the coreos ignition system would never be reflected since it only happens on first boot. I wonder if its wise to add some documentation around this.

Yes. Ignition is comparable to what AutoYaST or Kickstart do. It is not a configuration management system.

For example heres how I am solving this: https://github.com/FreekingDean/tf-modules/blob/master/vm/main.tf#L67-L69

resource "libvirt_volume" "os" {
  name   = "os-qcow2_${var.orchistration_type}.${count.index}"
  pool   = libvirt_pool.pool.name
  source = local.os
  format = "qcow2"
  count = var.node_count
  depends_on = [
    libvirt_ignition.ignition
  ]
}

What is the advantage of making the os depend on the ignition data explicitly?

If I understand correctly, what you are doing is making the volume to be recreated if ignition data changes, which results in the profile to be reapplied, while in the default case, it is the VM which depends on the profile, which means the VM gets re-created on change, but not the volume (which is the only mutable and therefore changed resource by the ignition profile).

Do I understand correctly?

[FreekingDean]

The way coreos ignition decides to run is based on a file in the coreos/first_boot directory. After it is run once it will not run again. It is also not an immutable series of functions, so I need a fresh "install" of coreos if I want ignition to successfully run again.

[dmacvicar]

Makes sense.

Unfortunately, the ignition resource is attached to the VM, as it is the firmware configuration and the volume attachment that needs to be modified. That seems to skip the dependency between the ignition profile and the volume, which you very well do manually with a _dependson.

Do you see a better way of fixing this? Otherwise we could just add a note to the documentation.