feat: local and remote cache options

Summary

Adds ability to perform cleanup of items added to the remote cache upon build completion in addition to the existing cancelled or halted.

To enable, set remote_cache_cleanup = true. Defaults to false.
Adds the ability to specify the datastore for the placement of the remote cache.

To enable, set remote_cache_datastore = "myCacheDatastoreName". If not set, the datastore of the virtual machine is used.
Adds the ability to specify the path on the datastore for the placement of the remote cache.

To enable, set remote_cache_path = "foo/bar"

If not set, the default path is packer_cache at the root of the datastore.
Adds the ability to overwrite items in the remote cache.

To enable, set remote_cache_overwrite = true.
Adds the ability to overwrite items in the local cache.

To enable, set local_cache_overwrite = true.

If local_cache_overwrite = true, remote_cache_overwrite is implicitly be set to true. This is to ensure consistency between the local and remote cache.

Testing

Example configuration:

source "vsphere-iso" "linux-photon" {

  // vCenter Server Endpoint Settings and Credentials
  vcenter_server      = var.vsphere_endpoint
  username            = var.vsphere_username
  password            = var.vsphere_password
  insecure_connection = var.vsphere_insecure_connection

  // vSphere Settings
  datacenter                     = var.vsphere_datacenter
  cluster                        = var.vsphere_cluster
  host                           = var.vsphere_host
  datastore                      = var.vsphere_datastore
  folder                         = var.vsphere_folder
  resource_pool                  = var.vsphere_resource_pool
  set_host_for_datastore_uploads = var.vsphere_set_host_for_datastore_uploads

  // Cache Settings
  local_cache_overwrite          = var.common_local_cache_overwrite 
  remote_cache_cleanup           = var.common_remote_cache_cleanup  
  remote_cache_overwrite         = var.common_remote_cache_overwrite 
  remote_cache_datastore         = var.common_remote_cache_datastore 
  remote_cache_path              = var.common_remote_cache_path

  // Virtual Machine Settings
  vm_name              = local.vm_name
  guest_os_type        = var.vm_guest_os_type
  firmware             = var.vm_firmware
  CPUs                 = var.vm_cpu_count
  cpu_cores            = var.vm_cpu_cores
  CPU_hot_plug         = var.vm_cpu_hot_add
  RAM                  = var.vm_mem_size
  RAM_hot_plug         = var.vm_mem_hot_add
  cdrom_type           = var.vm_cdrom_type
  disk_controller_type = var.vm_disk_controller_type
  storage {
    disk_size             = var.vm_disk_size
    disk_thin_provisioned = var.vm_disk_thin_provisioned
  }
  network_adapters {
    network      = var.vsphere_network
    network_card = var.vm_network_card
  }
  vm_version           = var.common_vm_version
  reattach_cdroms      = var.vm_cdrom_count
  tools_upgrade_policy = var.common_tools_upgrade_policy
  notes                = local.build_description

  // Removable Media Settings
  iso_url      = "https://packages.vmware.com/photon/5.0/GA/iso/photon-5.0-dde71ec57.x86_64.iso"
  iso_checksum = "6a7a258399a258da742032987c043ab25503698d35edafaf1ae000f12127da1a161d8b84caa17fd8f23d129e81e1faa7ab087c20ab9229772a643f8f9475305f"

  http_content = var.common_data_source == "http" ? local.data_source_content : null
  cd_content   = var.common_data_source == "disk" ? local.data_source_content : null

  // Boot and Provisioning Settings
  http_ip       = var.common_http_ip
  http_port_min = var.common_http_port_min
  http_port_max = var.common_http_port_max
  boot_order    = var.vm_boot_order
  boot_wait     = var.vm_boot_wait
  boot_command = [
    "<esc><wait>c",
    "linux /isolinux/vmlinuz root=/dev/ram0 loglevel=3 insecure_installation=1 ${local.data_source_command} photon.media=cdrom",
    "<enter>",
    "initrd /isolinux/initrd.img",
    "<enter>",
    "boot",
    "<enter>"
  ]
  ip_wait_timeout   = var.common_ip_wait_timeout
  ip_settle_timeout = var.common_ip_settle_timeout
  shutdown_command  = "echo '${var.build_password}' | sudo -S -E shutdown -P now"
  shutdown_timeout  = var.common_shutdown_timeout

  // Communicator Settings and Credentials
  communicator       = "ssh"
  ssh_proxy_host     = var.communicator_proxy_host
  ssh_proxy_port     = var.communicator_proxy_port
  ssh_proxy_username = var.communicator_proxy_username
  ssh_proxy_password = var.communicator_proxy_password
  ssh_username       = var.build_username
  ssh_password       = var.build_password
  ssh_port           = var.communicator_port
  ssh_timeout        = var.communicator_timeout
}

When remote_cache_cleanup = true:

CONFIRM: Build a VMware Photon OS 5 Template for VMware vSphere?

Continue? (y/n)
y
Building a VMware Photon OS 5 Template for VMware vSphere...
Initializing HashiCorp Packer and required plugins...
Starting the build....
packer build -force -on-error=ask 
vsphere-iso.linux-photon: output will be in this color.

==> vsphere-iso.linux-photon: Creating CD disk...
    vsphere-iso.linux-photon: Creating hybrid image...
    vsphere-iso.linux-photon: Done copying paths from CD_dirs
==> vsphere-iso.linux-photon: Uploading packer852352789.iso to [local-hdd-01] packer_cache...
==> vsphere-iso.linux-photon: Creating virtual machine...
==> vsphere-iso.linux-photon: Customizing hardware...
==> vsphere-iso.linux-photon: Adding SATA controller...
==> vsphere-iso.linux-photon: Mounting ISO images...
==> vsphere-iso.linux-photon: Adding configuration parameters...
==> vsphere-iso.linux-photon: Set boot order...
==> vsphere-iso.linux-photon: Power on VM...
==> vsphere-iso.linux-photon: Waiting 2s for boot...
==> vsphere-iso.linux-photon: Typing boot command...
==> vsphere-iso.linux-photon: Waiting for IP...
==> vsphere-iso.linux-photon: IP address: 172.16.14.158
==> vsphere-iso.linux-photon: Using SSH communicator to connect: 172.16.14.158
==> vsphere-iso.linux-photon: Waiting for SSH to become available...
==> vsphere-iso.linux-photon: Connected to SSH!
==> vsphere-iso.linux-photon: Provisioning with Ansible...
    vsphere-iso.linux-photon: Setting up proxy adapter for Ansible....
    vsphere-iso.linux-photon: Executing Ansible Galaxy
    vsphere-iso.linux-photon: Starting galaxy collection install process
    vsphere-iso.linux-photon: Process install dependency map
    vsphere-iso.linux-photon: Starting collection install process
    vsphere-iso.linux-photon: Downloading https://galaxy.
    ...........
==> vsphere-iso.linux-photon: Executing shutdown command...
==> vsphere-iso.linux-photon: Deleting Floppy drives...
==> vsphere-iso.linux-photon: Ejecting CD-ROM media...
==> vsphere-iso.linux-photon: Removing CD-ROM devices...
==> vsphere-iso.linux-photon: Convert VM into template...
==> vsphere-iso.linux-photon: Removing [local-hdd-01] packer_cache/packer852352789.iso...
    vsphere-iso.linux-photon: Closing sessions ....
==> vsphere-iso.linux-photon: Running post-processor:  (type manifest)
Build 'vsphere-iso.linux-photon' finished after 4 minutes 4 seconds.

==> Wait completed after 4 minutes 4 seconds

==> Builds finished. The artifacts of successful builds are:
--> vsphere-iso.linux-photon: linux-photon-5.0-develop
--> vsphere-iso.linux-photon: linux-photon-5.0-develop
Build Complete.
Press Enter to continue.

When remote_cache_cleanup = false or not included:

CONFIRM: Build a VMware Photon OS 5 Template for VMware vSphere?

Continue? (y/n)
y
Building a VMware Photon OS 5 Template for VMware vSphere...
Initializing HashiCorp Packer and required plugins...
Starting the build....
packer build -force -on-error=ask 
vsphere-iso.linux-photon: output will be in this color.

==> vsphere-iso.linux-photon: Creating CD disk...
    vsphere-iso.linux-photon: Creating hybrid image...
    vsphere-iso.linux-photon: Done copying paths from CD_dirs
==> vsphere-iso.linux-photon: Uploading packer492584107.iso to [local-hdd-01] packer_cache...
==> vsphere-iso.linux-photon: Creating virtual machine...
==> vsphere-iso.linux-photon: Customizing hardware...
==> vsphere-iso.linux-photon: Adding SATA controller...
==> vsphere-iso.linux-photon: Mounting ISO images...
==> vsphere-iso.linux-photon: Adding configuration parameters...
==> vsphere-iso.linux-photon: Set boot order...
==> vsphere-iso.linux-photon: Power on VM...
==> vsphere-iso.linux-photon: Waiting 2s for boot...
==> vsphere-iso.linux-photon: Typing boot command...
==> vsphere-iso.linux-photon: Waiting for IP...
==> vsphere-iso.linux-photon: IP address: 172.16.14.164
==> vsphere-iso.linux-photon: Using SSH communicator to connect: 172.16.14.164
==> vsphere-iso.linux-photon: Waiting for SSH to become available...
==> vsphere-iso.linux-photon: Connected to SSH!
==> vsphere-iso.linux-photon: Provisioning with Ansible...
    vsphere-iso.linux-photon: Setting up proxy adapter for Ansible....
    vsphere-iso.linux-photon: Executing Ansible Galaxy
    vsphere-iso.linux-photon: Starting galaxy collection install process
    vsphere-iso.linux-photon: Process install dependency map
    vsphere-iso.linux-photon: Starting collection install process
    vsphere-iso.linux-photon: Downloading https://galaxy.
    ...........
==> vsphere-iso.linux-photon: Executing shutdown command...
==> vsphere-iso.linux-photon: Deleting Floppy drives...
==> vsphere-iso.linux-photon: Ejecting CD-ROM media...
==> vsphere-iso.linux-photon: Removing CD-ROM devices...
==> vsphere-iso.linux-photon: Convert VM into template...
    vsphere-iso.linux-photon: Closing sessions ....
==> vsphere-iso.linux-photon: Running post-processor:  (type manifest)
Build 'vsphere-iso.linux-photon' finished after 3 minutes 49 seconds.

==> Wait completed after 3 minutes 49 seconds

==> Builds finished. The artifacts of successful builds are:
--> vsphere-iso.linux-photon: linux-photon-5.0-develop
--> vsphere-iso.linux-photon: linux-photon-5.0-develop
Build Complete.
Press Enter to continue.

And with:

remote_cache_datastore = "local-ssd-02"
remote_cache_path = "bar/packer_cache"

CONFIRM: Build a VMware Photon OS 5 Template for VMware vSphere?

Continue? (y/n)
y
Building a VMware Photon OS 5 Template for VMware vSphere...
Initializing HashiCorp Packer and required plugins...
Starting the build....
packer build -force -on-error=ask 
vsphere-iso.linux-photon: output will be in this color.

==> vsphere-iso.linux-photon: Retrieving ISO
==> vsphere-iso.linux-photon: Trying https://packages.vmware.com/photon/5.0/GA/iso/photon-5.0-dde71ec57.x86_64.iso
==> vsphere-iso.linux-photon: Trying https://packages.vmware.com/photon/5.0/GA/iso/photon-5.0-dde71ec57.x86_64.iso?checksum=sha512%3A6a7a258399a258da742032987c043ab25503698d35edafaf1ae000f12127da1a161d8b84caa17fd8f23d129e81e1faa7ab087c20ab9229772a643f8f9475305f
    vsphere-iso.linux-photon: photon-5.0-dde71ec57.x86_64.iso 4.31 GiB / 4.31 GiB [========================] 100.00% 14m42s
==> vsphere-iso.linux-photon: https://packages.vmware.com/photon/5.0/GA/iso/photon-5.0-dde71ec57.x86_64.iso?checksum=sha512%3A6a7a258399a258da742032987c043ab25503698d35edafaf1ae000f12127da1a161d8b84caa17fd8f23d129e81e1faa7ab087c20ab9229772a643f8f9475305f => /Users/johnsonryan/.cache/packer/f9a68f4fbeb2d077f82973894ae54e6af54eea7d.iso
==> vsphere-iso.linux-photon: Creating CD disk...
    vsphere-iso.linux-photon: Creating hybrid image...
    vsphere-iso.linux-photon: Done copying paths from CD_dirs
==> vsphere-iso.linux-photon: Uploading f9a68f4fbeb2d077f82973894ae54e6af54eea7d.iso to [local-ssd-02] bar/packer_cache...

Results: make test

johnsonryan@johnsonryaMMD6R packer-plugin-vsphere % make build
johnsonryan@johnsonryaMMD6R packer-plugin-vsphere % make test
?       github.com/hashicorp/packer-plugin-vsphere      [no test files]
?       github.com/hashicorp/packer-plugin-vsphere/builder/vsphere/common/testing       [no test files]
?       github.com/hashicorp/packer-plugin-vsphere/builder/vsphere/examples/driver      [no test files]
?       github.com/hashicorp/packer-plugin-vsphere/version      [no test files]
ok      github.com/hashicorp/packer-plugin-vsphere/builder/vsphere/clone        2.071s
ok      github.com/hashicorp/packer-plugin-vsphere/builder/vsphere/common       5.065s
ok      github.com/hashicorp/packer-plugin-vsphere/builder/vsphere/driver       7.056s
ok      github.com/hashicorp/packer-plugin-vsphere/builder/vsphere/iso  2.937s
ok      github.com/hashicorp/packer-plugin-vsphere/builder/vsphere/supervisor   6.441s
ok      github.com/hashicorp/packer-plugin-vsphere/post-processor/vsphere       4.994s
ok      github.com/hashicorp/packer-plugin-vsphere/post-processor/vsphere-template      5.825s

Reference

The following is in the top 5 of most requested (open) enhancements.

Closes #14 Closes #55 Closes #88

hashicorp / packer-plugin-vsphere

feat: local and remote cache options #399

Summary

Testing

Reference