Allow executing commands following creation in `incus_instance`

[tregubovav-dev]

This is follow up to the closed issue https://github.com/lxc/terraform-provider-incus/issues/27.

Preamble

Incus provider for terraform/open-tofu does not allow to deploy and provision any packages in the deployed container without 3'rd party tools. This impacts on deployment stability and possible compatibility issues with these 3rd party tools.

Problem

Infrastructure requires to deploy Linux container(s), and install and configure specific services on them. As incus provider does not provide functionality to install packages and/or run shell command in the container, we should use other solutions provided by incus and or terraform/open-tofu.

• cloud-init

  • Pros
  • Well documented and widely used solution.
  • Configuration abstracted from Linux distribution and allows to deploy most of packages and configure them
  • Allow to run any commands
  • Cons
  • Complex configurations require understanding of execution order of cloud-init modules
  • Cloud-init requires 90-120MB RAM provisioned per container, which may impact on deployment on IoT devices
  • Terraform/open-tofu can't identify failures in the cloud-init provisioning directly.

• terraform/open-tofu local provisioner

  • Pros
  • Local provisioner uses incus command-line client to execute commands which are well documented
  • Can be added to any resource and order of command execution can be controlled by regular terraform/open-tofu dependencies
  • Allows to run any commands
  • Cons
  • Requires installing and configuring incus client
  • There is not way to use provider configuration in the local provisioner (terraform/open-tofu does not allow to get provider configuration in the resources)

• using ansible provider

  • Pros
  • Well documented and widely used solution.
  • Configuration abstracted from Linux distribution and allows to deploy most of packages and configure them
  • Allow to run any commands
  • Cons
  • Configuration in Ansible playbook and in terraform/open-tofu provider are not synchronized. This may require additional maintenance.

    Proposed solution

    To implement provisioner incus-exec which can run any command directly in the incus container or VM. Provisioner incus-exec section can be added to most of resources, for example to incus_instance, incus_file, incus_storage_volume, terraform_data, etc. This provisioner can use the same schema as local-exec provisioner.

[stgraber]

For reference, here is the Terraform documentation on provisioners: https://developer.hashicorp.com/terraform/language/resources/provisioners/syntax

Note the emphasis on this being a last resort type thing and cloud-init or similar cloud meta-data being recommended over the use of provisioners.

I'm also struggling to find any good documentation on having a terraform provider expose additional provisioners. Do you have any link to another Terraform provider offering this?

[tregubovav-dev]

Hello Stéphane!

Terraform and Open-tofu have 3 provisioners and they are well documented:

• file provisioner. lxd and incus providers have better replacement for this provisioner with lxd_file and incus_file resource respectively.
• local-exec provisioner. This provisioner allows to execute any commands on the local host system where terraform/open-tofu is executed; including incus or lxd client commands.
• remote-exec provisioner which can execute command at remote host via ssh or winrm. I clearly understand why Terraform declares using these provisioners as a last resort type. It's because they do not have any control on transport used for file and remote-exec provisioners.

However, lxd and incus have built-in mechanism to run any command in container directly without using and/or configuring any external tools and this mechanism can be used for incus provisioner

I've prepared working example how to use local-exec provisioners with incus provider.

# provider
terraform {
  required_providers {
    incus = {
      source = "lxc/incus"
      version = "0.1.2"
    }
  }
}

# provider configuration
provider "incus" {
  remote {
    name = "<incus cluster node FQDN or IP>"
    scheme = "https"
    default = true
  }
}

# variable declarations
variable "remote" {
    type = string
    description = "Cluster host for accepting configuration"
}

variable "project" {
    type = string
    description = "LXC or Incus Project name"
}

variable "profiles" {
    type = list(string)
    description = "Profiles will be attached to the instance(s)"
    default = []
}

variable "system_dns" {
    type = list(string)
    description = "List of system DNS servers for /etc/resolv.conf"
}

variable "instances" {
    type = object({
        image = string
        name_template = string
        inst = map(
            object({
                target = string
                ip = string
                gw = string
            })
        )
    })
    description = "Instances' configuration"
}

# infrastructure configuration code
resource "incus_profile" "app_dns" {
    name = "app.test.dns"
    project = var.project
    description = "Profile to deploy System DNS Servers"

    config = {
        "boot.autostart" = true
        "limits.cpu" = 2
        "limits.memory" = "256MB"
    }

    device {
        type="nic"
        name = "eth0"
        properties = {
            name = "eth0"
            nictype = "bridged"
            parent = "br90"
        }
    }

    device {
        type = "disk"
        name = "root"
        properties = {
            path = "/"
            pool = "remote"
        }
    }
}

resource "incus_instance" "app_dns_instance" {
    for_each = var.instances.inst

    project = var.project
    image = var.instances.image
    target = each.value.target
    name = format(var.instances.name_template, each.key)
    profiles = concat([incus_profile.app_dns.name], var.profiles)
    wait_for_network = false
}

resource "incus_instance_file" "resolv_conf" {
    for_each = var.instances.inst

    project = var.project
    instance = incus_instance.app_dns_instance[each.key].name
    target_path = "/etc/resolv.conf"
    mode = "0644"

    content = <<EOF
%{ for value in var.system_dns ~}
nameserver ${value}
%{ endfor ~}
search test.tld
EOF
}

resource "incus_instance_file" "interfaces" {
    for_each = var.instances.inst

    project = var.project
    instance = incus_instance.app_dns_instance[each.key].name
    target_path = "/etc/network/interfaces"
    mode = "0644"

    content = <<-EOF
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address ${each.value.ip}
    gateway ${each.value.gw}
EOF

}

resource "incus_instance_file" "dnsmasq_d_conf" {
    for_each = var.instances.inst

    depends_on = [
        terraform_data.dnsmasq_d_install
    ]
    project = var.project
    instance = incus_instance.app_dns_instance[each.key].name
    target_path = "/etc/dnsmasq.conf"
    mode = "0644"
    content = "conf-dir=/etc/dnsmasq.d,*.conf"
}

resource "incus_instance_file" "dnsmasq_d_base" {
    for_each = var.instances.inst

    depends_on = [
        terraform_data.dnsmasq_d_install
    ]
    project = var.project
    instance = incus_instance.app_dns_instance[each.key].name
    target_path = "/etc/dnsmasq.d/base.conf"
    mode = "0644"
    content = <<EOF
server=1.1.1.3
server=1.0.0.3
interface=eth0
no-dhcp-interface=eth0
no-resolv
#expand-hosts
domain=home.my.somewhere
cache-size=1500
neg-ttl=10
dns-forward-max=4096
EOF
}

resource "incus_instance_file" "dnsmasq_d_local" {
    for_each = var.instances.inst

    depends_on = [
        terraform_data.dnsmasq_d_install
    ]
    project = var.project
    instance = incus_instance.app_dns_instance[each.key].name
    target_path = "/etc/dnsmasq.d/local.conf"
    mode = "0644"
    content = <<EOF
# Add local network DNS servers here, with domain specs 
# 
#server=/localnet/192.168.0.1
EOF
}

resource "terraform_data" "ip_update" {
    for_each = var.instances.inst

    triggers_replace = [
        each.value.ip
#        ,incus_instance.app_dns_instance[each.key],
#        ,incus_instance_file.dnsmasq_d_conf[each.key]
    ]

    provisioner "local-exec" {
        command = format("incus exec ${var.remote}:${incus_instance.app_dns_instance[each.key].name} --project ${var.project} -- service networking restart")
    }
}

resource "terraform_data" "dnsmasq_d_install" {
for_each = var.instances.inst

    depends_on = [
        incus_instance_file.interfaces,
        incus_instance_file.resolv_conf
    ]        
    provisioner "local-exec" {
        command = format("incus exec ${var.remote}:${incus_instance.app_dns_instance[each.key].name} --project ${var.project} -- apk add dnsmasq-dnssec")
    }
}

resource "terraform_data" "dnsmasq_d_restart" {
    for_each = var.instances.inst

    triggers_replace = [
        incus_instance_file.dnsmasq_d_base[each.key],
        incus_instance.app_dns_instance[each.key]
    ]

    provisioner "local-exec" {
        command = format("incus exec ${var.remote}:${incus_instance.app_dns_instance[each.key].name} --project ${var.project} -- sh -c 'service dnsmasq stop; service dnsmasq start'")
    }
}

Code will be more clean and stable If local-exec provisioner can be replace with incus-exec provisioner.

[stgraber]

@tregubovav-dev do you have any pointers on how to implement a custom provisioner? All I can find from hashicorp is documentation why this is a bad idea and telling you that there's a fixed set of built-in provisioners.

If it's possible to define an actual additional provisioner, then I'm not too opposed to it.

What I'm opposed to is having Resources abused for this kind of thing, so I don't want to see an incus_instance_exec resource be added for this as it's not an actual resource which can be compared to its plan.

For that matter, I think we should remove:

• incus_instance_file
• incus_storage_volume_copy (should be done through incus_storage_volume specifying an existing volume/snapshot as source)
• incus_image_publish (should be done through incus_image specifying an instance/snapshot as source)

@maveonair @adamcstephens @mdavidsen what do you all think?

[tregubovav-dev]

Here are links to:

• Terraform:
  • Generic provisioner interface and schema: https://github.com/hashicorp/terraform/tree/main/internal/provisioners
  • built-in provisioners implementation: https://github.com/hashicorp/terraform/tree/main/internal/builtin/provisioners
• Open-tofu:
  • Generic provisioner interface and schema: https://github.com/opentofu/opentofu/tree/main/internal/builtin/provisioners
  • built-in provisioners implementation: https://github.com/opentofu/opentofu/tree/main/internal/provisioners

Here is one of the discussions about 'docker-exec' provisioner: https://github.com/hashicorp/terraform/issues/4686

If you decide removing incus_instance_file resource, you should consider to add incus_file provisioner to keep plugin functionality.

P.S. Looking deeply to several terraform discussions I'm thinking whether it's possible to extend built-in remote-exec and file provisioner with custom connection like incus-https and incus-unix?

[tregubovav-dev]

I had short discussion with Terraform team in the thread and get very clear clarification about provisioners support by Terraform. Based on that I have to change my opinion and offer you different direction in supporting instances bootstrapping by lxd/incus provider and adding exec block in addition to file block to incus_instance resource. file and exec blocks should be executed in order defined in the resource.

resource "incus_instance" "app_dns_instance" {
    project = var.project
    image = var..image
    target = var.target
    name = var.name
    profiles = var.profiles

    file {
      target_path = "/etc/resolv.conf"
      mode = "0644"
      uid = 0
      gid = 0
      content = <<EOF
%{ for value in var.system_dns ~}
nameserver ${value}
%{ endfor ~}
search test.tld
EOF
    }

    file {
    target_path = "/etc/network/interfaces"
    mode = "0644"

    content = <<-EOF
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address ${var.ip}
    gateway ${var.gw}
EOF
    }

    exec {
      inline = [
        "service networking restart",
         "apk update & apk add dnsmasq-sec"
      ]
    }

    file {
      # copy directory with subdirectories
      target_path = "/etc/dnsmasq.d"
      source_path = var.dnsmasq_conf
      recursive = true 
    }

    exec {
      script = "/etc/dnsmasq.d/init.sh"
      on_fail = continue
    }
}

[stgraber]

Yeah, a post-create exec thing within the instance resource should be more reasonable and similar to the file option that's already supported there.

[tregubovav-dev]

Just for sure terraform-provider-lxd provides execs block in the lxd_instance resource since v2.0.0. Here is a related thread and provider's code. I'm not sure whether it's legally to use the same execs block schema from the terraform-provider-lxd provider or we must define our own schema?

[stgraber]

The LXD provider is still under the MPL so we can import code from it just fine, we'll just want to make sure we're happy with the way it's done.

[tregubovav-dev]

I have migrated my cluster from LXD to Incus recently. However, I will try to deploy single VM for LXD and will try whether lxd provider's execs works for me.

---

I did small experiment and I can say: execs block in lxd_instance' (terraform-provider-lxd` provider) works well for me with couple drawbacks:

1. Order of commands execution defined in execs block is unclear. I asked this question here
2. There is no way to define order of specific files uploads and execution commands. All files always uploaded before commands in execs block start executing. This is minor issue and does not prevent using execs block functionality.