`post-processor vsphere`: Handle prompts from ovftool when using custom CA

[goldstar611]

Overview of the Issue

https://github.com/hashicorp/packer/pull/7314 closed https://github.com/hashicorp/packer/issues/7234, for error message: Write 'yes' or 'no'.

I recently upgraded my packer version and started using the plugins and this "write yes or no" message appears endlessly in the packer log.

Adding --noSSLVerify to ovftool parameters resolves the endless, interactive prompts.

Reproduction Steps

1. Set up vcenter with an ESXi host OR set up nginx/apache server with a self signed SSL Cert (reproduction only) the real issue is when a custom CA bundle is being used.
2. Create a packer template that uses the vshpere post-processor and targets the above server
3. Wait for packer to finish building the VM and upload to the vcenter endpoint
4. Endless interactive prompt Write 'yes' or 'no' will be displayed until the packer process is killed (Ctrl+C)

Packer Version

1.9.2

Plugin Version and Builders

Please provide the plugin version.

vsphere 1.0 vmware 1.0

Please select the builder.

• [x] vsphere-iso

VMware vSphere Version

7.0

Guest Operating System

Windows 10

Simplified Packer Buildfile

Packer Buildfile

packer { required_plugins { vmware = { source = "github.com/hashicorp/vmware" version = "~> 1" } vsphere = { source = "github.com/hashicorp/vsphere" version = "~> 1" } } } variable "iso_checksum" { type = string default = "send_via_command_line" } variable "iso_url" { type = string default = "send_via_command_line" } source "vmware-iso" "autogenerated_1" { boot_command = [] boot_wait = "15m" communicator = "winrm" cpus = "2" disk_adapter_type = "lsisas1068" disk_size = "61440" disk_type_id = "1" floppy_files = ["./windows/10/answer_files/", "./windows/10/scripts/"] guest_os_type = "windows9-64" headless = "false" http_directory = "./windows/10/tools/" iso_checksum = "${var.iso_checksum}" iso_url = "${var.iso_url}" memory = "4096" network = "nat" network_adapter_type = "E1000E" shutdown_command = "shutdown /s /t 10 /f /d p:4:1 /c \"Packer Shutdown\"" shutdown_timeout = "45m" version = 14 vm_name = "temp_windows10x64_packer" vmx_data = { "RemoteDisplay.vnc.enabled" = "false" "RemoteDisplay.vnc.port" = "5900" "isolation.tools.hgfs.disable" = "true" "mks.enable3d" = "true" "tools.upgrade.policy" = "manual" } vmx_remove_ethernet_interfaces = false vnc_port_max = 5980 vnc_port_min = 5900 winrm_password = "winrm_password" winrm_timeout = "60m" winrm_username = "winrm_username" } build { sources = ["source.vmware-iso.autogenerated_1"] post-processor "vsphere" { cluster = "blah" datacenter = "Network" datastore = "datastore" host = "vcenter" overwrite = true password = "PASSWORD" username = "user" vm_folder = "vmFolder" vm_name = "temp_windows10x64_packer" vm_network = "network" } }

Operating System and Environment Details

Debian 12

Log Fragments and crash.log Files

ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Calling OVFtool to upload vmESC[0m

2023/08/01 15:29:58 packer-plugin-vsphere_v1.2.1_x5.0_linux_amd64 plugin: 2023/08/01 15:29:58 [INFO] (vsphere): starting ovftool command: ovftool --acceptAllEulas --name=windows10x64_packer --datastore=datastore --diskMode=thick --vmFolder=vmFolder --network=network --overwrite output-autogenerated_1/temp_windows10x64_packer.vmx vi://user%40vcenter:PASSWORD@vcenter/Network/host/blah

2023/08/01 15:29:58 packer-plugin-vsphere_v1.2.1_x5.0_linux_amd64 plugin: 2023/08/01 15:29:58 [INFO] (shell-local communicator): Executing local shell command [ovftool --acceptAllEulas --name=windows10x64_packer --datastore=datastore --diskMode=thick --vmFolder=vmFolder --network=network --overwrite output-autogenerated_1/temp_windows10x64_packer.vmx vi://user%40vcenter:PASSWORD@vcenter/Network/host/blah]
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Opening VMX source: output-autogenerated_1/temp_windows10x64_packer.vmxESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Accept SSL fingerprint (AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB) for host vcenter as target type.ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Fingerprint will be added to the known host fileESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m

...forever

[goldstar611]

in lieu of disabling SSL checking by default, I think a viable option is to handle the interactive prompt gracefully

[goldstar611]

I should mention that it's probably not necessary to set up a complete vcenter/vshpere instance. I'm sure that targeting a nginx or apache server with a self signed cert would be sufficient to hit the issue described.

[tenthirtyam]

Appears that the issue is in the post-processor:

func (p *PostProcessor) ValidateOvfTool(args []string, ofvtool string) error {
    args = append([]string{"--verifyOnly"}, args...)
    var out bytes.Buffer
    cmdCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    cmd := exec.CommandContext(cmdCtx, ovftool, args...)
    cmd.Stdout = &out

Ref: https://github.com/hashicorp/packer-plugin-vsphere/blob/56a37102c278fae91807dfd343d54eede06d313c/post-processor/vsphere/post-processor.go#L220C1-L226

May be fixed with:

func (p *PostProcessor) ValidateOvfTool(args []string, ofvtool string) error {
    args = append([]string{"--noSSLVerify", "--verifyOnly"}, args...)
    var out bytes.Buffer
    cmdCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    cmd := exec.CommandContext(cmdCtx, ovftool, args...)
    cmd.Stdout = &out

Additional reference from the packer-plugin-vmware change. https://github.com/hashicorp/packer/pull/7314/files

cc @nywilken

[rgl]

Please do not unconditionally disable TLS verification, as that defeats its purpose; instead, somehow, use the value from https://developer.hashicorp.com/packer/plugins/builders/vsphere/vsphere-iso#insecure_connection to decide.

[goldstar611]

Please do not unconditionally disable TLS verification, as that defeats its purpose; instead, somehow, use the value from https://developer.hashicorp.com/packer/plugins/builders/vsphere/vsphere-iso#insecure_connection to decide.

I agree, but you should know it's already hardcoded in at least 2 places.

https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/step_export.go#L42

https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/driver_esx5.go#L373

[goldstar611]

I've updated the title to propose better handling of the prompts since disabling ovftool's SSL check can simply be added to the options parameter currently:

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    options    = ["--noSSLVerify"]
  }
}

[goldstar611]

Any idea when using own certificate authority will be working again?

[tenthirtyam]

I agree, but you should know it's already hardcoded in at least 2 places.

https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/step_export.go#L42

https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/driver_esx5.go#L373

These references are from a seperate plugin.

[goldstar611]

I agree, but you should know it's already hardcoded in at least 2 places. https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/step_export.go#L42 https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/driver_esx5.go#L373

These references are from a seperate plugin.

correct, that is just auxiliary information for rgl who doesn't want ssl disabled by default. He should know it already is disabled by default in other vmware-owned plugins

[tenthirtyam]

in other vmware-owned plugins

HashiCorp maintains these plugins, but we (VMware) do collaborate on them. 😄

[tenthirtyam]

Taking a look at this the following should be possible currently:

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    options = ["--TargetSSLThumbprint AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB"]
  }
}

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    options = ["--TargetPEM /usr/lib/vmware-ovftool/certs/cacert.pem"]
  }
}

As an alternative, the following could be explicitly added:

import (
  ...
  "os"
  ...
)

type Config struct {
  common.PackerConfig `mapstructure:",squash"`
  ..
  TargetSSLThumbprint string `mapstructure:"target_ssl_thumbprint"`
  TargetPEM string `mapstructure:"target_pem"`
  ...
}

...

func (p *PostProcessor) BuildArgs(source, ovftool_uri string) ([]string, error) {
  args := []string{
    "--acceptAllEulas",
    fmt.Sprintf(`--name=%s`, p.config.VMName),
    fmt.Sprintf(`--datastore=%s`, p.config.Datastore),
  }

  if p.config.Insecure {
    args = append(args, fmt.Sprintf(`--noSSLVerify=%t`, p.config.Insecure))
  }

  if p.config.TargetSSLThumbprint != "" {
    args = append(args, fmt.Sprintf("--targetSSLThumbprint=%s", p.config.TargetSSLThumbprint))
  }

  if p.config.TargetPEM != "" {
    pemBytes, err := os.ReadFile(p.config.TargetPEM)
    if err != nil {
      return nil, fmt.Errorf("failed to read PEM file: %w", err)
    }
    args = append(args, fmt.Sprintf("--targetPEM=%s", string(pemBytes)))
  }

  ...

  if len(p.config.Options) > 0 {
    args = append(args, p.config.Options...)
  }

  args = append(args, source)
  args = append(args, ovftool_uri)

  return args, nil
}

Then you could use:

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    target_ssl_thumbprint = "AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB"
  }
}

or

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    target_pem = "/usr/lib/vmware-ovftool/certs/cacert.pem"
  }
}

Let me know your thoughts.

cc @nywilken

Ryan Johnson Senior Staff Solutions Architect | Product Engineering @ VMware, Inc.

[goldstar611]

Hi Ryan, Thanks for spending some time on this again.

Let me know your thoughts.

CA/Browser forum requires that TLS certificates have a max validity of about 1 year (398 days) so if we used a finger/thumb print, then that would need to be updated once a year. It's better than disabling SSL/TLS altogether security wise but not as good as verifying to a trusted certificate authority.

If --TargetPEM is available today under options, I'm all in and this would resolve my issue. I will try this out and verify it works as intended.

[tenthirtyam]

Great - let me know. May be ideal to add an example to the docs if that's good for you.

[tenthirtyam]

Any update @goldstar611?

[goldstar611]

Hey Ryan,

Thanks for the ping. I made the corresponding changes to my .pkr.hcl file to remove --noSSLVerify and add --TargetPEM /usr/lib/vmware-ovftool/certs/cacert.pem but ovftool failed twice. Once for unknown option, then once for unexpected parameter. The correct case and an equal sign is needed, like --targetPEM=/usr/lib/vmware-ovftool/certs/cacert.pem

With the params sent correctly, it's now a fight with ovftool to accept my certificate file. I've tried several base64 encoded certificates (the standard PEM format) but each invocation returns:

Error: Invalid PEM file: <file location>

Not even the file at /usr/lib/vmware-ovftool/certs/cacert.pem works.

[Edit] This targetPEM option in ovftool seems to be broken in my limited experience. [Edit2] removed [Edit3] ovftool seems to only accept a leaf certificates (not root CA certificates) so this option has no advantage over the SSL thumbprint option since it will be changing each year.

[goldstar611]

My last message was a bit of a mess so let me summarize here.

• vsphere post-processor does not handle ovftool prompts (stemming from use of a custom CA)
• ovftool doesn't accept CA certs in the --targetPEM parameter

I've changed the title back to Handle prompts from ovftool when using custom CA because, as I see it, that would resolve my biggest concern of an automated process never returning. The 2nd bullet point is a limitation in ovftool, so outside the scope of this issue I think.

[tenthirtyam]

I'm working on this one.

[tenthirtyam]

Hi @goldstar611 - I've tested the following changes that does allow an option and mre graceful failure.

func (p *PostProcessor) PostProcess(ctx context.Context, ui packersdk.Ui, artifact packersdk.Artifact) (packersdk.Artifact, bool, bool, error) {
        ... 

        ... Existing
    ui.Message("Validating username and password...")
    err = p.ValidateOvfTool(args, ovftool, ui)
    if err != nil {
        return nil, false, false, err
    }
    ... Existing

        ...
}

func (p *PostProcessor) ValidateOvfTool(args []string, ofvtool string, ui packersdk.Ui) error {
    args = append([]string{"--verifyOnly"}, args...)
    if p.config.Insecure {
        args = append(args, "--noSSLVerify")
        ui.Message("Skipping SSL thumbprint verification; insecure flag set to true...")
    }
    var out bytes.Buffer
    cmdCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    cmd := exec.CommandContext(cmdCtx, ovftool, args...)
    cmd.Stdout = &out

    // Need to manually close stdin or else the ofvtool call will hang
    // forever in a situation where the user has provided an invalid
    // password or username
    stdin, err := cmd.StdinPipe()
    if err != nil {
        return err
    }
    defer stdin.Close()

    if err := cmd.Run(); err != nil {
        outString := out.String()
        if strings.Contains(outString, "Enter login information for source") {
            err = fmt.Errorf("error running ovftool with --verifyOnly; the username " +
                "or password you provided may be incorrect")
            return err
        } else if strings.Contains(outString, "Accept SSL fingerprint") {
            err = fmt.Errorf("error running ovftool with --verifyOnly; the ssl thumbprint " +
                "returned by the server is not trusted. manually accept the thumbprint " +
                "with ovftool, or set the insecure flag to true for this post-processor.")
            return err
        }
    }
    return nil
}

TL;DR -

1. If insecure is not provided or is set to false explicitly for the post-processor and the certificate is not trusted, it will fail with an error by capturing "Accept SSL fingerprint" in the stdout. Unfortunately, I'm not able to get it to pause and allow for user input with a fmt.Scanln(&response). (I've tried endlessly to get that to work. ¯\_(ツ)_/¯ )

2. If insecure is set to true, it will proceed and present a message.

=> vsphere-iso.linux-photon: Running post-processor:  (type vsphere)
    vsphere-iso.linux-photon (vsphere): Uploading /Users/ryan/Library/Mobile Documents/com~apple~CloudDocs/Code/Personal/<sensitive>-examples-for-vsphere12/artifacts/linux-photon-5.0-develop/linux-photon-5.0-develop.ovf to m01-vc01.rainpole.io
    vsphere-iso.linux-photon (vsphere): Validating username and password...
    vsphere-iso.linux-photon (vsphere): Skipping SSL thumbprint verification; insecure flag set to true...
    vsphere-iso.linux-photon (vsphere): Uploading virtual machine...
    vsphere-iso.linux-photon (vsphere): Opening OVF source: /Users/ryan/Library/Mobile Documents/com~apple~CloudDocs/Code/Personal/<sensitive>-examples-for-vsphere12/artifacts/linux-photon-5.0-develop/linux-photon-5.0-develop.ovf
    vsphere-iso.linux-photon (vsphere): The manifest validates

I'll put in a pull request tomorrow for these changes to help mitigate the issue of interacting with ovftool's options. I hope this will help.

cc: @nywilken @lbajolet-hashicorp

Ryan

[tenthirtyam]

Please note that I'm also trying to track down the issue seen in https://github.com/hashicorp/packer-plugin-vsphere/issues/279 after the completion, but I'm mostly convinced this is a packersdk issue.

vsphere-iso.linux-photon (vsphere): Deploying to VI: vi://administrator%40vsphere.local@m01-vc01.rainpole.io:443/m01-dc01/host/m01-cl01
vsphere-iso.linux-photon (vsphere): Transfer Completed
vsphere-iso.linux-photon (vsphere): Completed successfully
Build 'vsphere-iso.linux-photon' errored after 5 minutes 38 seconds: 1 error(s) occurred:

* Error destroying builder artifact: reading body msgpack decode error [pos 1556]: reflect.Set: value of type map[interface {}]interface {} is not assignable to type error; bad artifact: []string(nil)

==> Wait completed after 5 minutes 38 seconds

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso.linux-photon: 1 error(s) occurred:

* Error destroying builder artifact: reading body msgpack decode error [pos 1556]: reflect.Set: value of type map[interface {}]interface {} is not assignable to type error; bad artifact: []string(nil)

==> Builds finished but no artifacts were created.