Figure out Tinkerbell workflow for provisioning Ubuntu instance with cloud-init userdata

[invidian]

To be able to use existing cloud-init configs generated by Cluster API, we should figure out a way how Tinkerbell can provision Ubuntu on worker nodes with specific cloud-init userdata specified.

This should allow easy boostrapping of clusters.

Regarding the cloud-init userdata delivery to the provisioning process, I think for MVP we could store userdata base64 encoded in workflow template directly, the same as we do for installing Flatcar on Tinkerbell: https://github.com/kinvolk/tinkerbell.org/blob/invidian/flatcar/content/examples/flatcar-container-linux/_index.md#preparing-template.

While a bit hacky, this will allow delivering userdata without any middleware like Hegel, which should make things simple for the start.

#

I figured out how to install Ubuntu Cloud ISO image on machine and make it use cloud-init. Here are the steps:

• Download some live Ubuntu image. In my case I used https://releases.ubuntu.com/20.04/ubuntu-20.04.1-live-server-amd64.iso
• Boot this image on machine which needs to be provsioned. In my case I used libvirt.

• Run the following commands to install Ubuntu Cloud on virtual machine disk:

  cat <<EOF > 90_dpkg.cfg
  datasource_list: [ None ]
  datasource:
  None:
    metadata:
      local-hostname: bar
    userdata_raw: |
      #cloud-config
  
      users:
        - name: root
          ssh_authorized_keys:
            - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC96Ge1knq4wq3EUSCzYUPsumdLIi1B7Z/lv478G8kCgrxH5OAncqrWIRP+hG1WThe+yi60vTd1jeBc3YzOtTqSFRDnpSNNx++Lvh0/KRN6d1advX8XnmPHMvCPBtJXoRjuQLh7CS8VrP+VA3QY1y8mRoX2710pShyf2MiVI5G0Vi32GBKWJ1MRvP9kGAeay50HJsvF12AGz0VJImVn+eqtVlgPvmF6K+NqgEMdz8660avi1RKrWhLPAym82Am6WEoWQdSx01OrVlGW6nBMys3/fMAVAvYlUwolfPIPB7tl7Vh/xMeyzX6QW7BEVeGRTdzaGACl8vgbZ4BPeQiMihOWmnKHw//BDmGLYWh+2oDdvfNNL+kgwTg4YgN4NL1Crk5rPCB2WHdiwTj5I8N+NLY7K1u0vq8Loe+cZciuRIN3495sL0m9eiXkKcknM9j4ApqDe+WMiv1gojB6o8yE0RV4CmYgJa9TFwDv9+ib+G1tDCsD+ULUn9g9ARtLaHtQX5XW+adAVkXtk+DEqWHdvfc8uv298aQ6UxexKcsF4uqj88c2gDiFjjrddqt3YScPV4d3KQzBuVIO6whbCzHnEUBqfkDhogrEronEC5+wRav5NsvntucDcKDIMpQCgHTIqSOKk8wog0a1iPrEFaHs21W2CsxGfDYuKiFfSF1JvpMwnw== mateusz@kinvolk.io
  EOF
  
  cat <<EOF > run.sh
  #!/bin/bash
  
  set -euxo pipefail
  
  # Having kernel in /boot is required by virt-customize, so we need to install some kernel, as
  # Ubuntu server live image has no kernel in /boot.
  #
  # However, installing kernel requires writable /lib/modules and Ubuntu server live image has it
  # read-only, so we workaround it by creating tmpfs on top of it.
  mount -t tmpfs tmpfs /lib/modules
  
  apt update && apt install -y libguestfs-tools linux-image-generic
  
  wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
  qemu-img convert -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw
  rm focal-server-cloudimg-amd64.img
  dd bs=4M if=focal-server-cloudimg-amd64.raw of=/dev/vda conv=fdatasync  status=progress
  virt-customize -a /dev/vda --copy-in ./90_dpkg.cfg:/etc/cloud/cloud.cfg.d/ --root-password password:somepassword
  reboot
  EOF
  
  chmod +x run.sh

• If needed, modify disk path and public SSH key part in files created above. In my case, my VM has single disk /dev/vda.
• Run the script:

  ./run.sh

• If everything went well, reboot the machine:

  reboot

Now, your VM should boot Ubuntu Cloud with provided user-data applied.

[invidian]

Next step is convert the script into Tinkerbell workflow.

Some things to consider:

• virt-customize may not work on OSIE inside Docker container. If it's not, we can copy files over by mounting the new partition I guess.

[displague]

https://github.com/canonical/cloud-init/pull/680 is relevant since the metadata formats are related.

[displague]

I was discussing this with @dustinmiller1337 @pereztr5 and @mmlb very recently. In the Equinix Metal world, the DMI fields do not uniformly identify that the devices are running in EM. It is advantageous for us to do this so that users and tools like cloud-init can identify their environment.

In Tinkerbell, I don't know if we can make the same assumptions about manipulating the device DMI. In some cases, the DMI can only be updated with hardware-specific software.

This problem is top on my mind now, especially as it relates to this conversation: https://github.com/canonical/cloud-init/pull/680#discussion_r536386252. It is not preferable for cloud-init to base its data source detection on the existence of a known metadata service.

[invidian]

Okay, I figured out how to run the code above as a Workflow. Findings:

• virt-customize works on OSIE, but pulling it into Docker image makes installation image quite big (1GB+), which causes troubles on OSIE, as on the worker with 4GB of RAM, the tmpfs is 50% RAM size, so 2GB and this image can barely unpack on it. I switched to use simply partprobe + mount which seems simpler and also does the job.

Dockerfile content for for ubuntu-install image:

FROM alpine:3.12

RUN apk add -U qemu-img

And workflow template:

version: "0.1"
name: ubuntu-install
global_timeout: 1800
tasks:
  - name: "ubuntu-install"
    worker: "{{.machine1}}"
    volumes:
      - /dev:/dev
      - /statedir:/statedir
    actions:
      - name: "dump-cloud-init"
        image: ubuntu-install
        command:
          - sh
          - -c
          - |
            echo '{{.cloudInit}}' | base64 -d > /statedir/90_dpkg.cfg
      - name: "download-image"
        image: ubuntu-install
        command:
          - sh
          - -c
          - |
            # TODO: Pull image from Tinkerbell nginx and convert it there, so we can pipe
            # wget directly into dd.
            /usr/bin/wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img -O /statedir/focal-server-cloudimg-amd64.img
      - name: "write-image-to-disk"
        image: ubuntu-install
        command:
          - sh
          - -c
          - |
            /usr/bin/qemu-img convert -f qcow2 -O raw /statedir/focal-server-cloudimg-amd64.img /dev/vda
      - name: "write-cloud-init-config"
        image: ubuntu-install
        command:
          - sh
          - -c
          - |
            set -eux
            partprobe /dev/vda
            mkdir -p /mnt/target
            mount -t ext4 /dev/vda1 /mnt/target
            cp /statedir/90_dpkg.cfg /mnt/target/etc/cloud/cloud.cfg.d/
            umount /mnt/target
      - name: "reboot" # This task shouldn't really be there, but there is no other way to reboot the Tinkerbell Worker into target OS in Tinkerbell for now.
        image: ubuntu-install
        command:
          - sh
          - -c
          - |
            echo 1 > /proc/sys/kernel/sysrq; echo b > /proc/sysrq-trigger

Then, create a workflow like:

tink workflow create -t <template ID> -r '{"machine1": "<mac>", "cloudInit": "<base64 encoded cloud-init data>"}'

[invidian]

The MVP part is figured out, so let's close this as done.