Cannot get k3os build using packer and vmware

[aram535]

I'm trying to deploy to a vSphere 7.0.1.

$ k3os --version k3os version v0.19.4-dev.5 $ uname --kernel-release --kernel-version 5.4.0-54-generic #60 SMP Fri Nov 20 17:09:10 UTC 2020

Describe the bug $ cd k3os/package/packer/vsphere/server $ packer build -var-file k3os-server-variables.json k3os-server.json
vsphere-iso: output will be in this color.

==> vsphere-iso: Creating VM... ==> vsphere-iso: Customizing hardware... ==> vsphere-iso: Mounting ISO images... ==> vsphere-iso: Adding configuration parameters... ==> vsphere-iso: Set boot order... ==> vsphere-iso: Power on VM... ==> vsphere-iso: Waiting 40s for boot... ==> vsphere-iso: Typing boot command... [ ... the VM boots, I see the rancher configuration happening ... ] ==> vsphere-iso: Waiting for IP...

The VM is created, started, boots, is k3 is enabled and configured... then reboots(?) and that's where the packer install sits until it times out and powers down the instance.

If I suspend the packer install at this point I have a working k3os VM.

To Reproduce $ packer build -var-file k3os-server-variables.json k3os-server.json

Expected behavior Finish the VM, disconnect, and provide the URL to connect to Rancher.

Actual behavior If left, the install times out and removes the VM.

[pysen]

It's hard to answer your question without the .json config file contents. Anyway this is what works for me to setup a k3os node using packer. Set headless to false if build output is neccessary and I have the k3os-server.yaml config file in the http directory but this can be changed to any directory in the project structure.

k3os-server.json contents:

{
  "_comment": "Build with `packer build k3os-server.json`",
  "builders": [
    {
      "type": "qemu",
      "communicator": "none",
      "disk_size": "{{ user `disk_size` }}",
      "headless": "{{ user `headless` }}",
      "http_directory": "http",
      "iso_checksum": "{{ user `iso_checksum` }}",
      "iso_urls": [
        "{{ user `iso_url` }}"
      ],
      "output_directory": "output-{{ user `vm_name` }}",
      "boot_wait": "1s",
      "vm_name": "{{ user `vm_name` }}",
      "qemuargs": [
        [ "-m", "{{ user `memory` }}" ],
        [ "-smp", "cpus={{ user `cpus`}}"]
      ],
      "boot_command": [
        "<wait>",
        "<tab>",
        "<down>",
        "<wait>",
        "e",
        "<down>",
        "<down>",
        "<down>",
        "<down>",
        "<down>",
        "<down>",
        "<end>",
        " ",
        "k3os.install.device=/dev/vda",
        " ",
        "k3os.install.power_off=true",
        " ",
        "k3os.install.silent=true",
        " ",
        "k3os.install.config_url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/k3os-server.yaml",
        "<F10>"
      ],
      "shutdown_command": "poweroff"
    }
  ],
  "provisioners": [
  ],
  "variables": {
    "disk_size": "30G",
    "cpus" : "4",
    "memory" : "8192",
    "headless": "true",
    "iso_checksum": "ab15fcd4c4f5ed8805932523f85594f6844fe611fc50f8bc9a82ddd9bd326db0",
    "iso_url": "https://github.com/rancher/k3os/releases/download/v0.11.1/k3os-amd64.iso",
    "version" : "1",
    "vm_name": "k3os"
  }
}

[Mohitsharma44]

Seeing same behavior on release v0.19.5-rc.1 running on Vsphere 7.0.1

Here is my json file:

{
  "builders": [
    {
      "CPUs": 2,
      "RAM": 4096,
      "RAM_reserve_all": true,
      "boot_command": [
        "<enter>",
        "<wait40>",
        "rancher",
        "<enter>",
        "sudo k3os install",
        "<enter>",
        "1",
        "<enter>",
        "2",
        "<enter>",
        "Y",
        "<enter>",
        "http://{{ .HTTPIP }}:{{ .HTTPPort }}/cloud-config.yml",
        "<enter>",
        "y",
        "<enter>"
      ],
      "http_directory": ".",
      "boot_wait": "30s",
      "convert_to_template": "true",
      "datastore": "Internal_ssd_0_and_1",
      "disk_controller_type": "pvscsi",
      "folder": "VMTemplates",
      "guest_os_type": "other4xLinux64Guest",
      "host": "esxi01.sharmamohit.com",
      "insecure_connection": "true",
      "iso_paths": [
        "[Silo] ISOs/k3os-amd64.iso"
      ],
      "network_adapters": [
        {
          "network": "DInternetPortGroup0",
          "network_card": "vmxnet3"
        }
      ],
      "password": "{{ user `vcenter_password` }}",
      "shutdown_command": "sudo poweroff",
      "ssh_password": "{{user `ssh_password`}}",
      "ssh_username": "{{user `ssh_username`}}",
      "storage": [
        {
          "disk_size": 10240,
          "disk_thin_provisioned": true
        }
      ],
      "type": "vsphere-iso",
      "username": "{{ user `vcenter_username` }}",
      "vcenter_server": "{{ user `vcenter_host` }}",
      "vm_name": "k3os-0.19.5rc1-template"
    }
  ],
  "provisioners": [
    {
      "execute_command": "{{ .Vars }} sudo -E sh '{{ .Path }}'",
      "inline": [
        "echo hello world"
      ],
      "type": "shell"
    }
  ]
}

[Mohitsharma44]

fwtw, I've even tried --takeover with install script (for v0.19.5-rc.1 as well asv0.11.1) on a fresh ubuntu 20.04 but thats causing kernel panic (same as described here: https://github.com/rancher/k3os/issues/622). provisioning block that I tried with ubuntu 20.04 template:

    {
      "inline": [
        "sudo apt-get update -y",
        "sudo apt-get install -y dosfstools grub2 parted curl",
        "sudo bash -x /tmp/k3os-install.sh --takeover --poweroff --debug --tty ttyS0 --config /tmp/k3os-config.yml --no-format $(findmnt / -o SOURCE -n) \"{{user `k3os_iso_url`}}\""
      ],
      "type": "shell",
      "expect_disconnect": true
    },
    {
      "inline": [
        "set -x; sudo systemd-run --on-active=3 --timer-property=AccuracySec=100ms sudo systemctl reboot --force --force; sync; echo Rebooting"
      ],
      "expect_disconnect": true,
      "type": "shell"
    }

I've tried running install script manually on an ubuntu vm as well but it ends up with kernel panic too.

I guess Im going to stick with k3s on an ubuntu base for now.

[pkeech]

I have a Packer script working great with v0.11.1. As soon as I switch to v0.19.5-rc.1 the script eventually times out waiting for an IP address.

Not knowing too much about how virtual machines report their IP addresses to vSphere, it looks like the open-vm-tools is missing files in v0.19.5-rc.1.

[joace]

@pkeech I didn't build k3os with packer on VMware, but on Proxmox, and hit exact same issue as you, my solution may be helpful. I finally figured out that with v0.19.5-rc.1 image, the qemu-agent couldn't start. to report back IP, so packer couldn't connect to ssh for the next provisioning. The reason GA not start may be:

1. The GA_PATH (in qemu-guest-agent under /etc/conf.d) set to a folder not exist in the system after boot, e.g. /dev/virtio-ports in below picture
2. Sometimes, even that folder exists, but the org.qemu.guest_agent.0 (this is a symlink) porting to a different device, in my case, it's linked to vport1p1, but vport0p1 is actual GA device.

And I found by default, memory balloon virtio device also enabled, so there are 2 virtio device exist in the system, but the number for GA is not fixed, sometime is vport0p1, sometime vport1p1, not clear what's the magic here, maybe related to different host. But if disabled balloon, only vport0p1 present now. If VMware packer builder can disable memory balloon, you just need to create a custom config with write_files as below to workaround it.

write_files:

• path: /etc/conf.d/qemu-guest-agent content: |- GA_METHOD="virtio-serial" GA_PATH="/dev/vport0p1" owner: root:root permissions: '0644'

Unfortunately, Proxmox builder can't disable, I have to manually install a VM with iso, and edit VM config to remove balloon device, then convert it to a template, which along with above qemu-guest-agent config file, i haven't seen failed deployment.

[drbothen]

After researching this, I think this has to due with this 05-base Dockerfile

Per this link in version 3.12 of Alpine the open-vm-tools package was split into several different packages. Currently we are just installing open-vm-tools. we need to add the additional packages open-vm-tools-deploypkg, open-vm-tools-guestinfo, open-vm-tools-static, open-vm-tools-vmbackup to replicate previous behavior