How to provision an custom image and make use of `user_data`

[jlarfors]

Hello, I am building a custom image using Packer that I later want to provision with Terraform and make use of user_data when provisioning with Terraform.

However, seems the user_data has no effect when provisioning with Terraform.

Any way to get this to work?

Packer

We have something liek this for packer:

packer {
    required_plugins {
        upcloud = {
            version = "v1.2.0"
            source = "github.com/UpCloudLtd/upcloud"
        }
    }
}

source "upcloud" "vault" {
  username = "${var.username}"
  password = "${var.password}"
  zone = "fi-hel1"
  storage_name = "Debian GNU/Linux 11"
  template_prefix = "deb-hashi-vault"
}

build {
  sources = ["source.upcloud.vault"]
  // ... more provisioning stuff here
}

Terraform

And something liek this with Terraform:

resource "upcloud_network" "private_network" {
  name = "example_private_net"
  zone = "fi-hel1"

  # router = upcloud_router.example_router.id

  ip_network {
    address            = "10.0.0.0/24"
    dhcp               = true
    dhcp_default_route = false
    family             = "IPv4"
    gateway            = "10.0.0.1"
  }
}

resource "upcloud_server" "vault" {
  hostname = "vault-server"
  zone     = "fi-hel1"
  plan     = "1xCPU-1GB"

  template {
    # uuid of packer built image
    storage = var.custom_image
    size = 25
  }

  network_interface {
    type = "public"
  }

  // Simple user_data test
  user_data = "#!/bin/bash\ntouch /tmp/test.txt"
}

[Darep]

user_data does not work in a custom image unless you have specifically implemented it into your custom image. E.g. if you use our public template for Debian, user_data should work normally.

Are you handling the user_data field somehow in your image, or is your image based on one of our public templates by any chance? If you need to implement this into your custom image, you could for example try with cloud-init & our metadata API. UpCloud has a small tutorial for the metadata API: https://upcloud.com/community/tutorials/upcloud-metadata-service/ . The metadata API needs to be enabled on server creation.

Let me know if this helps :blush:

[jlarfors]

Thanks for the reply.

The base image for packer is one of your public templates Debian GNU/Linux 11, so this should work, right? Do you need anything to reproduce this? I think my example should be pretty complete to reproduce it.

I also tested cloud-init with one of your public Debian images but wasn't sure what would need to be configured to make that work... It would be preferable if cloud-init could be "enabled" when using packer but I don't know the internals of all this, only have experience using cloud-init with other clouds. So if you have some pointers that'd be awesome!

[Darep]

Ahh, yeah, at least I was under the impression that it should work 😲 Weird! 😄 I will ask some UpCloudians if they know what's up & if someone could investigate this more deeply.

[ka-myl]

Hi @jlarfors

user_data is not exactly supported for custom images. However if the custom image is based on UpCloud official template, you could get this to work by:

1. Installing (or updating) cloud-init v21.1+
2. Removing /var/lib/cloud directory (to make sure cloud-init runs on the next boot)

The caveat with Debian 11 template is that there isn't a recent-enough cloud-init version available in main repository, so to get it working you might need to built cloud-init from source, or install from testing (which might have some side effects, so it's not really optimal production setup). The following packer config worked for me:

packer {
    required_plugins {
        upcloud = {
            version = "v1.2.0"
            source = "github.com/UpCloudLtd/upcloud"
        }
    }
}

source "upcloud" "vault" {
  username = "${var.username}"
  password = "${var.password}"
  zone = "fi-hel1"
  storage_name = "Debian GNU/Linux 11"
  template_prefix = "vault-image"
}

build {
  sources = ["source.upcloud.vault"]

  provisioner "shell" {
    inline = [
      "cp /etc/apt/sources.list /etc/apt/sources.list.bak",
      "echo 'deb http://deb.debian.org/debian/ testing main' >> /etc/apt/sources.list",
      "apt-get update",
      "DEBIAN_FRONTEND=noninteractive apt -y install cloud-init=21.4-1",
      "rm -rf /var/lib/cloud",
      "cp -f /etc/apt/sources.list.bak /etc/apt/sources.list",
      # ... more provisioning stuff here
    ]
  }
}

And then with Terraform:

resource "upcloud_server" "vault" {
  hostname = "vault-server"
  zone     = "fi-hel1"
  plan     = "1xCPU-1GB"
  metadata = true

  template {
    # Custom image based on Debian 11
    storage = var.custom_image
    size = 25
  }

  network_interface {
    type = "public"
  }

  login {
    keys = [
      "my-ssh-pub-key"
    ]
  }

  // Simple user_data test
  user_data = "#!/bin/bash\ntouch /tmp/test.txt"
}

Note that metadata has to be allowed for the server for this to work.

Please also note that default cloud-init config disables root login and creates a debian user instead. So to be able to log into the server properly, you might need to tweak the /etc/cloud/cloud.cfg file for your custom image. I took a shortcut and set the SSH key via Terraform login block (which just works for debian user, but is not exactly optimal).

I hope this clears things out a bit, let us know if you have any other questions :)

[giacomo-alzetta-aiven]

Has there been any work on supporting user_data with custom images without doing strange hacks on the image size?

In our case we are just using the API. The API documentation describes metadata and user_data and it seems to imply that you just need to specify "metadata": "yes", "user_data": "<script>" to have the script run by root at initialization time, but this ticket seem to imply you actually have to SSH into the node, do some hacks and reboot to have cloud-init run the script?

We want to get rid of all SSH connections as root to our VMs for security reasons so that's not good enough for us.

[Darep]

I also haven't tried with Packer, but I can confirm user_data works with custom images in UpCloud API, if the custom image is built from one of UpCloud's more recent templates with cloud-init built-in! :) Need to use the cloud-init config (more examples: https://github.com/canonical/cloud-init/tree/main/doc/examples) with these.

I've personally used UpCloud's Debian 12, Ubuntu 22 and Ubuntu 24 templates to create the custom image, and then supplied user_data with a custom cloud-init config when creating a server from the custom image. That has worked for me nicely

For example, here's what I've supplied to the UpCloud API to create an app user with access to docke on server creation with my custom image:

{ "server": {
  ...
  "user_data": "#cloud-config
users:
- name: app
  groups: docker
  shell: /bin/bash
  ssh_authorized_keys:
    - ${SSH_KEY}"
}

[giacomo-alzetta-aiven]

Sorry, we are not actually using packer, I just thought that the underlying problem is the same we are having by pushing images directly via the UpCloud API.

We are building from scratch our images, we use the http_import functionality to copy the raw images to a storage from some bucket and then templatize it. So is this already supposed to work as you say? The templatize operation does not allow to specify a template_type: cloud-init AFAIK.

[Darep]

Yeah, I haven't used Packer lately either 😊

If you are using http_import then I'd suggest to try to add a cloud-init setup to the custom image. The cloud-init setup needs to match exactly as they are in UpCloud's own templates. Might be difficult to get it right. Easiest way to do this is to create a new custom image from UpCloud's Debian 12, Ubuntu 22 or Ubuntu 24 -template. Or try deploying one of those templates and copy & modify relevant bits from there into your custom image 😅

Note also that you need to have metadata enabled when you create the server.