vSphere Guestinfo Datasource

[mitchellmaler]

Is your feature request related to a problem? Please describe. Today you have to mount a drive for each node that is created with the correct guestinfo. This add extra overhead.

Describe the solution you'd like Add vsphere guestinfo as a datasource for the user data. This can then be provided as metadata to a vm instead of mounted as a disk.

Describe alternatives you've considered None

Additional context Add any other context or screenshots about the feature request here.

[brlbil]

K3OS uses linuxkit's metadata package to for cloud-init data source. There is already an open issue in that project for the vSphere guestinfo data source.

[nhi-vanye]

Need this to migrate from RancherOS.

[johnlaur]

It's really crummy that I cannot seem to discover any way to pass install parameters to a k3os template in vsphere via the vapp properties. I would like to orchestrate k3os cluster deployment using terraform. I am curious what anyone else may be experimenting with as a workaround?

Until linuxkit adds something, it looks like the best approach in the interim might be to script up a small ova builder that retrieves the vapp properties either via guestinfo (k3os ships with vm tools, but i dont know if the installer has them) or a second cdrom device, and if present executes the install.

Has anyone else implemented any similar method?

[voima-eetu]

@johnlaur I had this issue with vmware vcloud director.

I solved the issue by creating iso image with script(ran local_exec in terraform) uploaded that iso to vCloud, and straight after creating vcd_vm_app I attached disk with vcd_inserted_media.

create_iso.sh

if [ $# -eq 1 ]; then
        FILENAME=$1
        echo "Building image to $FILENAME ..."
elif [ $# -gt 1 ]; then
        echo 'Usage: ./build.sh [filename]'
        exit 1
else
        echo "Building image to $FILENAME ..."
fi

genisoimage -output $FILENAME -volid cidata -joliet $USERDATA 2> /tmp/build.log

FILESIZE=$(stat -c %s $FILENAME 2>/dev/null)
COLUMNS=$(tput cols)
if [[ $FILESIZE > 0 ]]; then
        printf '%s (%d bytes) ... done!\n' $FILENAME $FILESIZE
else
        printf 'Something went wrong while trying to make %s\n' $FILENAME
fi

Notice that using -rock flag is not possible(because of https://github.com/linuxkit/linuxkit/issues/3562) and that is why maximum user.data size is 2048kb.(At least with my testing)

---

main.tf

resource "null_resource" "create_cdrom" {
  depends_on = [local_file.user-data]

  triggers = { 
    user_data = local.user_data
  }

  provisioner "local-exec" {
    command = "${path.module}/cloud_init/create_iso.sh"
    environment = { 
      FILENAME = "/tmp/${var.vm_name}.iso"
      USERDATA = local_file.user-data.filename
    }   
  }
}

[johnlaur]

@voima-eetu thanks very much for documenting your solution.

I am currently testing with the patch from this PR. It adds support for linuxkit pulling from the guestinfo.userdata (as base64 or gzip+base64 specified in guestinfo.userdata.encoding) in a similar way to vmware's out of tree cloud-init plugin

Although I'd prefer to pass the config template in guestinfo.userdata, I'm not a big fan of having to do it by patching linuxkit and rebuilding k3os entirely; I'd rather a solution that can be performed by modifying the release iso before attaching it to the vmware template.

Assuming the guestinfo.userdata key contains a gzip+base64 encoded file, retrieving it in the VM is as simple as:

vmware-rpctool info-get guestinfo.userdata | base64 -d | gunzip -c > k3os.yaml

My assumption is that I could modify the iso by adding a script that retrieves this data and starts the installer without needing linuxkit to support this mechanism for cloud-init data. I don't really see a use case in my environment for modifying userdata after a node is installed, so to me it doesnt matter whether or not linuxkit can pick it up or whether or not it would survive a k3os upgrade

[schnerring]

I am currently testing with the patch from this PR. [...] Although I'd prefer to pass the config template in guestinfo.userdata, I'm not a big fan of having to do it by patching linuxkit and rebuilding k3os entirely; I'd rather a solution that can be performed by modifying the release iso before attaching it to the vmware template.

Unfortunately the linuxkit maintainers seem to ignore the PR for no reason I can see. The maintainers also stopped giving feedback to my PR adding Azure support to linuxkit despite other PRs still being merged. I specifically added this provider to use k3os on Azure (#101) (and it works great) but to-date you'd have to use your own linuxkit fork and build k3os yourself to do that.

[thehedgefrog]

Unfortunately the linuxkit maintainers seem to ignore the PR for no reason I can see. The maintainers also stopped giving feedback to my PR adding Azure support to linuxkit despite other PRs still being merged. I specifically added this provider to use k3os on Azure (#101) (and it works great) but to-date you'd have to use your own linuxkit fork and build k3os yourself to do that.

This is infuriating, the same is happening with vSphere. I was able to bypass this in my homelab by leveraging several Terraform provisioners to dynamically write a config.yaml file and then copy it via SSH to the VM before rebooting it, which will give the same end result but in a much, much more convoluted way. I think the k3os team should consider dropping linuxkit in favour of a solution with a more responsive dev team, as the linuxkit maintainers have demonstrated they have no interest in implementing ideas by the wider open-source community, which in return is pretty much against the open-source philosophy itself.

[brlbil]

I am the author for vSphere PR to linuxkit and I gave up on that PR due to lack of interest. We use vShpere provider in production and CDROM provider in testing with K3OS by maintaining our own metadata package with just these two providers. Also, the CDROM provider has issues and it has been waiting for go-diskfs library to be updated in linuxkit. All this shows that linuxkit metadata package is just not maintained well enough. So I propose either creating a separate metadata package so it might be used by other opensource projects as well or pulling the metadata package in K3OS and further develop in this repo.

[chris93111]

@brlbil do you have custom solution for this ? except the solution mentioned by @johnlaur (vmware-rpctool) I couldn't find anything else