Can't use Ansible provisioner with just SSH username and password

Overview of the Issue

I am trying to run an Ansible script for my vsphere image build but it looks like it keeps trying to authenticate using a key when all it should use is a username/password. This username and password works fine during the VM build and I can also SSH into the machine during this time but it keeps failing for the ansible provisioner.

Reproduction Steps

Try running the Ansible provisioner on a vsphere-iso image build using only username and password to run the ansible file. No key exists.

Plugin and Packer version

1.8.6

Simplified Packer Buildfile

    build {
      sources = ["source.vsphere-iso.ubuntu"]

      provisioner "ansible" {
        playbook_file = "./ansible/main.yml"
        user = local.admin_user
        extra_arguments = [
          "--extra-vars", "ansible_ssh_pass=${local.admin_pass}",
        ]
      }
    }

Operating system and Environment details

Using packer from MacOS and building an Ubuntu 22.04 image on vsphere

Log Fragments and crash.log files

vsphere-iso.ubuntu: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '[127.0.0.1]:55497' (RSA) to the list of known hosts.\r\nsign_and_send_pubkey: no mutual signature supported\r\test@127.0.0.1: Permission denied (publickey).", "unreachable": true}

hashicorp / packer-plugin-ansible