Packer boot_command remains as default kernel boot command in /etc/default/grub causing problems with cloud init

[erSitzt]

Overview of the Issue

When using Packer to create a vmware template for Ubuntu 22.04 i ran into issues with cloud-init when deploying from that template.

This is my boot_command for packer

boot_command = [
    "c<wait10s>",
    "linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;seedfrom=http://{{ .HTTPIP }}:{{ .HTTPPort }}/\"",
    "<enter><wait5s>",
    "initrd /casper/initrd",
    "<enter><wait5s>",
    "boot",
    "<wait5s><enter><wait300>"
    ]

When deploying from the created template, cloud-init reads the kernel parameters and sets the datasource to nocloud-net ignoring VMware guest extra_config userdata/metadata.

This seems to happen with 22.04.02 and later... somehow 22.04.1 even when updated still works with this present.

As these parameters are not really useful after creating the template with packer, i was wondering how and why these end up as the default in the resulting template ? And most examples i could find, do not address this... only a few use sed to correct the grub config in a cleanup script at the end of the packer build.

Reproduction Steps

Create a packer vmware template using the ubuntu 22.04.x isos

Packer version

From packer version 1.10

Simplified Packer Template

source "vsphere-iso" "ubuntu" {

  vcenter_server        = var.vsphere_server
  cluster               = var.vsphere_cluster
  username              = var.vsphere_username
  password              = var.vsphere_password
  insecure_connection   = "true"
  datacenter            = var.vsphere_datacenter
  datastore             = var.vsphere_datastore
  folder                = var.vsphere_folder

  CPUs                  = var.cpu_num
  RAM                   = var.mem_size
  RAM_reserve_all       = true
  disk_controller_type  = ["pvscsi"]
  guest_os_type         = var.vsphere_guest_os_type
  iso_checksum          = var.os_iso_checksum
  iso_url               = var.os_iso_url
  http_directory        = "http"

  network_adapters {
    network             = var.vsphere_network
    network_card        = "vmxnet3"
  }

  storage {
    disk_size             = var.disk_size
    disk_thin_provisioned = true
  }

  vm_name               = var.vsphere_vm_name
  convert_to_template   = "true"
  communicator          = "ssh"
  ssh_username          = var.ssh_username
  ssh_password          = var.ssh_password
  ssh_timeout           = "30m"
  ssh_handshake_attempts = "100000"

  boot_order            = "disk,cdrom,floppy"
  boot_wait             = "3s"
  boot_command          = var.boot_command
  shutdown_command      = "echo '${var.ssh_password}' | sudo -S -E shutdown -P now"
  shutdown_timeout      = "15m"

  configuration_parameters = {
    "disk.EnableUUID" = "true"
  }
}

build {
  sources = ["source.vsphere-iso.ubuntu"]

  provisioner "shell" {
    execute_command = "echo '${var.ssh_password}' | {{.Vars}} sudo -S -E bash '{{.Path}}'"
    environment_vars = [
      "BUILD_USERNAME=${var.ssh_username}",
    ]
    scripts = var.shell_scripts
    expect_disconnect = true
  }

}

Operating system and Environment details

Ubuntu 22.04 WSL

Log Fragments and crash.log files

Include appropriate log fragments. If the log is longer than a few dozen lines, please include the URL to the gist of the log or use the Github detailed format instead of posting it directly in the issue.

Set the env var PACKER_LOG=1 for maximum log detail.

[erSitzt]

Just as an example

ubuntu@rke2-zentrale-infra-agent-1:~$ cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="autoinstall ds=nocloud-net;seedfrom=http://172.16.x.x:8012/"

This is from a VM that was deployed with 22.04.1 iso and updated during autoinstall to 22.04.2... cloud-init still working afterwards The same is present in VMs deployed with 22.04.2 iso, but those fail with cloud-init, because only nocloud-net datasource is used

[tenthirtyam]

I'd recommend opening this on the project for the Packer Plugin for vSphere and closing this one.

[tenthirtyam]

By the way, why not use the VMware datasource that was upstreamed to cloud-init?

[erSitzt]

@tenthirtyam what do you mean ?

My case looks like almost every example for packer / ubuntu with vmware... autoinstall / subiquity to install ubuntu and create the template cloud-init with VMware datasource when deploying from the template.. <== this is where the problem occurs, so isnt this the upstreamed VMware datasource ???

[tenthirtyam]

Try removing nocloud in the boot and then set the cloud-init configuration and the VMware data source during your provisioning phase.

You can find examples in my other project: https://github.com/vmware-samples/packer-examples-for-vsphere/blob/develop/builds/linux/ubuntu/22-04-lts/linux-ubuntu.pkr.hcl

[erSitzt]

I dont understand, you set the same boot command in your example.

[tenthirtyam]

Are you configuring the VMware data source later with a provisioner?

[erSitzt]

I'm using terraform to deploy the template afterwards. The problem has nothing to do with any datasource, only with the boot_command remaining as GRUB_CMDLINE_LINUX_DEFAULT.

There are two bugs at play here

https://bugs.launchpad.net/subiquity/+bug/1956452 and https://github.com/canonical/cloud-init/issues/4977

@tenthirtyam and even your example would cause this issue, as your are using --- in the beginning of the kernel line

This is my version to avoid the problems

boot_command = [
    "c<wait10s>",
    "linux /casper/vmlinuz autoinstall ds=\"nocloud-net;seedfrom=http://{{ .HTTPIP }}:{{ .HTTPPort }}/\" ---",
    "<enter><wait5s>",
    "initrd /casper/initrd",
    "<enter><wait5s>",
    "boot",
    "<wait5s><enter><wait300>"
    ]

Putting it at the end will result in a clean template, without any kernel parameters not needed in the final template. Maybe there should be a hint in the examples on what --- does...

[erSitzt]

Not a bug here